An evolutionary reasoning engine with author-recused, context-isolated LLM judging.
ReasonLineage improves responses through repeated generation, critique, selection, mutation, and crossover. It preserves the ancestry of each response and separates structural checks from the LLM reviews used to calculate fitness. The same process can be applied to answers, analyses, designs, and scientific hypotheses.
- Overview — what ReasonLineage does
- Evolution — non-author review and selection
- jev — check, optionally revise, recheck
- Architecture
- Quick Start
- Usage in Detail
- Environment variables and credentials
- Testing
- License
| Capability | What it does |
|---|---|
| Response evolution | A panel of models writes responses. Survivors are kept, and new responses come from mutation and crossover. Every response keeps its parent links. |
| Non-author fitness | Models other than the author review each response. Reviews are aggregated by the median for each criterion and combined using the task's weights. |
| jev structural checks | Noul, Score, and Choice assessments check each new response before peer scoring. A confirmed gap can trigger one revision by the original author. jev values do not enter fitness. |
| Recorded runs | Model calls, events, and checks are written to a run folder, so a stopped run can be resumed and a finished run replayed, diagnosed, or scored against a gold file. |
Entry points:
- CLI:
reasonlineage run | resume | diagnose | gold(alsopython -m reasonlineage) - Scripts: judge calibration and backend checks in
scripts/ - Workflow: the detailed workflow, written as an agent skill
The figure illustrates a four-model configuration: Opus 5.5 (Anthropic), GPT-6.1 sol (OpenAI), Grok 4.7 (xAI), and GLM-5.3-flash (Z.ai). Model availability and the active reviewer panel are set by the setup file. All displayed fitness values are illustrative.
The engine was built to test one question. When a model scores its own answers, or judges share the scoring record, does fitness keep rising across generations while correctness stays flat? And does removing the author from scoring, and judging each response in isolation, prevent it? To compare those conditions fairly, the code fixes who writes, who judges, and what is checked, so every condition runs the same procedure.
- With N active models, each writes ⌈4 / N⌉ initial responses, so the population holds P = N × ⌈4 / N⌉ responses (two each with three models, one each with four). Every active model except the author reviews each response; with a single model, it reviews its own responses in separate, fresh calls.
- Reviews are aggregated by the median for each criterion and combined using the task's weights to obtain fitness.
- P − 2 responses survive selection. Retained responses reuse their scores.
- Each later generation adds one mutation, written by the parent response's author, and one crossover, written by a model that authored neither parent. If every active model wrote one of the parents, one of them writes the crossover and the run records a
synthesizer_fallbackanomaly. - New responses are checked and reviewed before the next selection. The best evaluated response is returned with its lineage.
Survivors are responses; keeping P − 2 of them does not shrink the roster, and every active model keeps writing and reviewing.
| The engine (code) decides | The models do |
|---|---|
| who judges each response, and what the judge sees | write responses: initial, mutation, crossover |
| fitness: the per-criterion median, then the weighted sum | review and critique |
| selection and the generation count | choose which parts of each parent a crossover drops |
| records, protocol version, and mode checks |
Two run options vary the judging, for experiments on judging bias. The defaults are recuse and isolated.
--judges |
Who scores a response |
|---|---|
recuse |
every panel model except the author (a one-model panel judges its own responses, each in a fresh session) |
self |
the author only |
same_family |
other panel models from the author's company |
other_vendor |
panel models from other companies |
all |
every panel model, the author included |
--context |
What the judge sees besides the question and the response |
|---|---|
isolated |
nothing else |
author_label |
the name of the author model |
shared_trace |
the current population, with scores from earlier selections |
Peer fitness and structural completeness measure different things. A response can be complete in form and still cite papers that do not support it. An interesting idea can lack the validation strategy that would make it a comparable research proposal. The J1 gate targets the second kind of gap: it decides when to send a response back and what to fix.
The combined jev gate uses three parallel typed assessments:
| Type | Short label | Role |
|---|---|---|
| Noul | Present? | Check whether the required response or proposal is present. |
| Score | Adequate? | Check whether required components are sufficiently developed. |
| Choice | Priority gap? | Identify the component that most needs attention. |
Confirmed gaps can trigger one revision by the original author, followed by reinspection. Responses without confirmed gaps proceed directly to peer scoring. Unclear judgments can be rechecked once, and unresolved issues are recorded.
In the combined profile, Score's adequacy threshold is applied to P(level 2) + P(level 3) < 0.5. Choice selects the highest-probability priority gap, and a targeted revision requires agreement with the insufficient component scores.
The structural gate does not directly add jev values to fitness; fitness remains the responsibility of the LLM reviewers. J2 diagnostics are recorded separately and do not drive survival or revision. In the code, checker results go to the run's checks.jsonl, never to the engine's event log, and J1's revision decision is the only way a check affects the run.
The rules above belong to the combined profile, src/reasonlineage/protocol/profiles/jev-combined, which sets j1.enforce to true; select it with reasonlineage run TASK --protocol jev-combined. The default protocol that reasonlineage run loads, src/reasonlineage/protocol/protocol.json, has its own question set and sets j1.enforce to false: the gate records what it would revise without requesting the revision.
Interactive animation · MP4 · SVG · High-resolution PNG · Figure caption and production notes
Download the HTML animation and open it locally to use its playback controls; GitHub displays its source. The animation is 28 seconds long and demonstrates both the optional revision path and the direct path to peer scoring.
task (*.json) + setup (configs/main.json) + protocol (default or --protocol profile)
│
▼
cli.py checks pinned versions and the judging condition, writes header.json
│
▼
engine/runner.py generation 0 → J1 gate → non-author review → selection
→ mutation + crossover → … → result.json
├── backends/ claude · codex · grok CLIs, OpenAI-compatible HTTP
├── checks/gate.py J1: jev or a local LLM checker → checks.jsonl
└── core/ fitness, selection, roles
│
▼
run folder header.json · events.jsonl · calls/ · checks.jsonl · result.json
├── resume continues from the recorded calls
├── diagnose J2 and PMID check → diagnostics/
└── gold gold scoring → diagnostics/gold.jsonl, summary
| Component | Location |
|---|---|
Command line (run, resume, diagnose, gold) |
src/reasonlineage/cli.py |
| Run loop, policies, call recording and replay, event log, run header, judging conditions, setup and task loading | src/reasonlineage/engine/ |
| Fitness, selection, roles, category weights, population state | src/reasonlineage/core/ |
Model backends: claude, codex, grok CLIs and OpenAI-compatible http |
src/reasonlineage/backends/ |
| J1 gate, jev client, local LLM checker, combined jev rules | src/reasonlineage/checks/ |
| J2 diagnostics, PMID check, gold scoring | src/reasonlineage/diagnose/ |
Packaged protocol (generations, timeouts, token caps, J1/J2 questions), prompts, and selectable profiles (profiles/) |
src/reasonlineage/protocol/ |
| Default setup and example task | configs/main.json, examples/example-ipf.json |
- Python 3.11 or later. The core package has no runtime Python dependencies.
- For live runs, only the backends your setup uses:
claude: the Claude Code CLI, logged in with a subscriptioncodex: the Codex CLI, logged in with ChatGPTgrok: the grok CLI, logged in (grok login)http: an OpenAI-compatible endpoint, and a key file if it needs a key
- For the jev checker: a jev API key (§7).
python -m pip install -e '.[dev]'
reasonlineage --help
python -m pytest -qTests exclude live model/service calls by default.
A setup file names the panel and the checkers. The default is configs/main.json:
| File | Panel | Checker for public / private tasks |
|---|---|---|
configs/main.json (default) |
opus, gpt, grok, glm (glm-5.3) |
jev / a local DeepSeek endpoint |
A three-model setup with jev for public tasks looks like this:
{
"panel": [
{"name": "gpt", "backend": "codex", "model": "gpt-6.1-sol", "effort": "xhigh"},
{"name": "grok", "backend": "grok", "model": "grok-4.7", "effort": "xhigh"},
{"name": "glm", "backend": "http", "model": "glm-5.3-flash", "effort": "max",
"base_url": "https://ollama.com/v1", "family": "zai", "api_key_file": "~/.config/reasonlineage/ollama.env"}
],
"checkers": {
"jev": {"version": "jev-1.13.0", "api_key_env": "JEV_API_KEY", "api_key_file": "~/.config/reasonlineage/jev.env"}
},
"checker_for": {"public": "jev"},
"concurrency_per_backend": 2
}backendisclaude,codex,grok, orhttp. Anhttpmember also needsbase_urlandfamily(the company that made the model).checker_formapspublicandprivatetasks tojevor to a local checker listed undercheckers. jev is an outside service, and the loader rejects a setup that sends private tasks to it.- Experiment mode refuses model names containing
latestorpreview. Pin versions.
{
"id": "ipf-example",
"question": "Which approved drug is most promising to repurpose for idiopathic pulmonary fibrosis (IPF)? Rank the candidates, and give the mechanism, the key evidence, and a validation experiment for the top pick.",
"candidates": ["metformin", "imatinib", "sildenafil"],
"category": "biomedical",
"data_sensitivity": "public"
}| Field | Required | Meaning |
|---|---|---|
id, question |
yes | non-empty strings |
candidates |
no | at least two distinct items to rank |
category |
no (default biomedical) |
picks the fitness weights: general, math, science, creative, ethical, coding, analytical, biomedical |
data_sensitivity |
yes | public or private; picks the checker through checker_for |
Leave out candidates for an open question, the kind of task the jev-combined profile checks.
Then start a run as shown in §6.1.
# Defaults: configs/main.json, experiment mode, recuse + isolated, generations from protocol.json (3)
# Calls the models in the setup; ends with a "Done:" line naming the run folder under runs/ (ignored by Git)
reasonlineage run examples/example-ipf.json
# A fixed seed and another output folder
reasonlineage run examples/example-ipf.json --seed 1 --out runs/demo
# Generation 0 only
reasonlineage run examples/example-ipf.json --generations 0 --seed 1
# The combined jev profile (§3) for an open question with an embedded evidence packet; needs a jev checker
# (see src/reasonlineage/protocol/profiles/jev-combined/README.md)
reasonlineage run my-question.json --protocol jev-combined
# Judge the generation 0 of an earlier run under another condition
reasonlineage run TASK --gen0-from runs/RUN_FOLDER --judges self --context isolated
# Reuse every answer an earlier run recorded, with its seed and conditions
reasonlineage run TASK --replay runs/RUN_FOLDER --generations 3| Option | Default | Meaning |
|---|---|---|
--config |
configs/main.json |
setup file with the panel and checkers |
--protocol |
default |
default, a packaged profile (jev-combined), or a folder with protocol.json and prompts/; recorded in header.json and reused by resume and diagnose |
--mode |
experiment |
experiment will not start unless model and jev versions are pinned, the generation count is fixed, the panel can serve the judging condition, and a checker exists when the protocol turns the J1 gate on. exploration checks only the judging condition. |
--judges, --context |
recuse, isolated |
the judging condition (§2.3) |
--generations |
protocol.json (3) |
with --replay, the replayed run's count |
--seed |
random | recorded in header.json |
--replay RUN_DIR |
reuse every recorded answer, with that run's seed, conditions and protocol; refused if the protocol files differ | |
--gen0-from RUN_DIR |
reuse generation 0 of a run of the same task and protocol | |
--out |
runs |
folder for run folders |
A run folder holds:
| Path | Contents |
|---|---|
header.json |
task, setup, options, protocol version, hash and source, CLI versions, engine commit |
events.jsonl |
the engine's event log |
calls/ |
one record per model call, reused by resume and --replay |
checks.jsonl |
J1 checker results, when a checker runs |
unusable/ |
outputs that could not be used, per attempt |
result.json |
the final summary, written when the run finishes |
diagnostics/ |
j2.jsonl and pmid.jsonl from diagnose, gold.jsonl from gold |
Exit codes for run and resume: 0 done, 1 cannot start or continue, 2 stopped (fix the cause, then resume), 143 interrupted by SIGTERM or SIGHUP (resume).
reasonlineage resume runs/RUN_FOLDERresume rebuilds the run from header.json and the recorded calls. It reloads the protocol recorded in header.json and stops if those files changed since the run started, and it warns if a CLI version changed.
# J2 diagnostics and the PMID check (both call outside services)
reasonlineage diagnose runs/RUN_FOLDER
# Also ask J2 whether each answer reports a clinical-trial outcome for the listed items
reasonlineage diagnose runs/RUN_FOLDER --gold-items items.json # {"items": [{"item": ..., "outcome_date": "YYYY-MM-DD"}]}- J2 runs the task's checker over the run and writes
diagnostics/j2.jsonl. It is skipped when no checker is available. - PMID check looks up each cited PMID in PubMed and checks that the record names the item it supports. It writes
diagnostics/pmid.jsonl.
reasonlineage gold --gold my-gold.json runs/RUN_A runs/RUN_B \
--out report.json --analysis analysis.mdgold makes no model calls. It prints a summary, writes diagnostics/gold.jsonl in each run folder, and, with --out and --analysis, saves the summary as JSON and the level-1 analysis as Markdown. The gold file has the form {"tasks": {task id: {"approved": [...], "failed": [...]}}}. A folder that is not a run, whose task is not in the gold file, whose candidates differ from the gold drugs, or whose run did not finish is left out with a warning.
| Script | What it does | Calls models |
|---|---|---|
scripts/make_candidates.py |
every roster model writes one strong and one deliberately flawed answer to the same question, for judge calibration | yes |
scripts/calibrate_judges.py |
every judge scores every candidate --repeats times in isolated calls; reports leniency, noise, and recusal bias |
yes |
scripts/calibrate_panel.py |
every roster judge scores every candidate, its own included; reports leniency and self- and family preference | yes |
scripts/check_backends.py |
one isolated scoring call per backend/model; reports context size and latency | yes |
scripts/smoke_claude_cli.py |
repeats a Strict Evaluator request concurrently through the Claude CLI; reports latency and score noise | yes |
The first line of each file says what it does.
| Variable | Read by | Purpose |
|---|---|---|
JEV_API_KEY |
reasonlineage run, resume, diagnose, when the task's checker is jev |
The jev API key. The name is the default of checkers.jev.api_key_env in a setup. When the variable is unset, the CLI reads the file in checkers.jev.api_key_file. |
No other environment variable is read. Other credentials live in files:
api_key_fileon anhttppanel member or local checker: a file outside the repo that holds only the key. The program reads it and never prints it. Anhttpmember without one sends a placeholder key.- CLI logins: the
claude,codex, andgrokbackends use each CLI's own login. The grok backend runs with a temporary home that holds only the grok login, so no user configuration loads.
# Offline suite; tests marked `live` are deselected by default
python -m pytest -q
# A single file
python -m pytest -q tests/test_cli.py
# Live tests only: these call real models or services
python -m pytest -q -m livepyproject.toml registers the live marker and sets addopts = "-m 'not live'".
MIT — see LICENSE.

