Repository navigation
Replies: 2 comments
LLM metadata curation of public sequencing archives: prior artResearch, not a decision. Written 2026-10-13 to inform an LLM-heavier metadata front end (the The systemsPeer-reviewed
Preprints
Not found. No NCBI paper on GPT-4 harmonisation of BioSample attributes turned up; NCBI's 960 CELLxGENE schema as the target vocabularychanzuckerberg/single-cell-curation,
Organism left Techniques that make output checkable
Self-consistency voting (Wang et al., arXiv:2203.11171) and OAK What carries over
|
|
Moved to liulab-data: Bulk metadata for all of GEO and SRA: sources, sizes, change feeds. Bulk data management belongs to liulab-data, and seqforge consumes it. Discuss the sources there. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Research, not a decision. Written 2026-10-13 for review before any work starts. The bulk
sources (where the metadata lives, how big it is, how it updates) moved to
liulab-data#17, since bulk metadata belongs to liulab-data. This note covers what the
code does today, the questions a corpus front end has to answer, and two or three options for each,
with a leaning where there is one. Every claim about the code was read from the tree at
744ba10.What the code does today
recordsmakes per-accession NCBI calls: one Entrezelinkwalk, thenefetchfor SRA, BioSample and BioProject in batches of 100io/archive.pyfetch_recordsArchiveRecordio/archive.py,io/remote.pygeo_softArchiveRecordSetis source-tagged (source: str) and holds four levels: project, sample, experiment, runmodels/records.pyassertedvalue at rung 0, marked declared. Only the 960 NCBI-harmonised BioSample attribute names qualifyresolve/records.pylibrary_strategyis never mapped to a chemistry.library_sourcecan only veto a bulk hypothesisresolve/engine.py,resolve/escalate.pyharvest/meter.pyharvest/plan.pyworkspace.py,evals/records/<ACC>.jsoncache is written and never read back automaticallycli/io.py,cli/run.pyA. Where the metadata half ends
The request said "per-study manifests ready to compile". That cannot come from metadata alone. The
manifest's library section (which chemistry, which file plays which role) is decided from bytes, and
paperwork is only a hypothesis (ADR-0043, R4). So the metadata half has to stop short of a manifest.
probe-remote,resolve)--max-readsand--max-bytes(R3), so it is megabytes per run, not gigabytesB. The corpus ladder needs its own name, and probably a different order
The name. "Rung" is taken. It means which step settled one field (0 = metadata, 2–3 = bytes,
6 = alignment, 7 = a human), and it is stored on every
Evidencedvalue. The proposed 0–5 laddermeasures something else: how much was fetched and spent on one study. A study can be at
corpus step 1 while its chemistry field reads rung 3. These are two axes, so they need two words.
Candidate names for the corpus axis: reach, depth, stage. "Level" and "tier" are on
Rung's avoid list.
The order. The proposed order is: structured, LLM on GEO text, manuscript, decision model, snippet
probe, full download. Two problems:
has already flagged. Most flagged ambiguities are processing-divergent chemistry ties, and those come
out of scoring bytes. On metadata alone, the metadata resolver already decides: the stronger basis
wins, and equal authorities that disagree leave the field null with a warning (ADR-0010). So that
side has nothing to escalate.
costs no tokens and decides the chemistry outright. A manuscript costs a PDF fetch plus tokens, and
only adds sample facts.
Which is cheaper per study, a snippet probe or a manuscript pass, is a number to measure on the pilot,
not to assume.
C. What the "rung-0 table" is in seqforge's terms
The structured pre-parse is a projection of archive records: typed columns that a submitter filled
in. In seqforge's terms, these are declared positions.
ArchiveRecordSetper deposit, the same typefetch_recordsreturns today but with a newsourcetag. A columnar table (Parquet, queried with DuckDB) is built from those filesEither way, the GEO text has to enter the record set. That needs a decision: does a GSE/GSM become
project and sample records carrying
free_text(the existing levels), or a new level? The existinglevels fit, since a deposit is defined as a BioProject or a GEO series (ADR-0021's "Deposit").
The hypothesis line holds: a rung-0 assay class can choose which studies to process and can seed
the chemistry hypothesis. It never enters evidence.
D. The corpus database: master record, or index?
R5 says disk is the state and only
seqforge/cache/is safe to delete.attempts.jsonl: the step reached, exit code, refusals, tokens and money spent, and the manifest hash. The DB is rebuilt from these files and can be deletedA new dependency follows from either option. Today the repo has pandas and no SQLite, DuckDB or Parquet.
E. The unit of work
GSM can appear in several series. Processing per GSE would pay for shared samples twice and give
them two manifests. ADR-0021 already defines the deposit as the unit.
HARVEST_VERSION, the model or the KBchanges. The record-set hash covers the bulk dump's own update dates, so an incremental dump update
marks exactly the studies it touched.
F. Model cost at scale
corpus re-run without it pays again for everything.
The provider seam (
LLMProvider.complete_json) is synchronous today.liulab-data#17), and much of the sequencing part is assays the KB has no spec for.
G. The source seam
Today there are two ad hoc paths:
io resolve).A source adapter would bundle three jobs:
ArchiveRecordSet.GEO bulk, SRA bulk, ENA and ArrayExpress each implement those three. The existing per-accession
fetch_recordsthen becomes the fallback for any study a dump does not cover.H. The big switch: let the model do much more in the metadata half
The proposal: keep the bytes-to-pipeline half deterministic, and let a language model do far more of
the metadata half. GEO studies are too varied and too messy for hand-written parsers. What comes out
should be structured, in a controlled vocabulary, and easy for code to join on. Prior art is
collected in
the prior-art note (posted as a comment below when finished).
What this changes.
AGENTS.mdtoday gives the model exactly two jobs: read prose intoquote-checked assertions, and arbitrate an ambiguity code has already flagged. It calls any further
job a warning sign. The switch is a deliberate change to that paragraph, scoped to the metadata half.
It would need a decision record that says so, rather than a slow drift.
What makes randomness acceptable. The determinism that matters is in the compile, not in the
model. If every model output is written to disk as a content-addressed artifact, keyed by input hash,
prompt version and model, then the model runs once and everything after it reads a frozen file. That
is how a probe
Observationalready works. Re-running the compile gives the same answer. Re-runningthe model is a deliberate, versioned event, never a side effect.
Possible model jobs, and the check each one gets from code:
UBERON:0002421ctrl_rep2_R1.fq→ GSM1234567What stays with code: the vocabulary and schema, every check above, precedence between sources,
and everything compose reads. Chemistry stays decided by bytes.
Three choices to think about:
inferredwith a confidence, and code may use it; (b) used for search and filtering only, never in a manifest; (c) used in a manifest only after it agrees with a second, independent model passOpen questions only you can answer
manifest field (choice (a)), or only the searchable catalogue (choice (b))?
ontologies), or seqforge's own?
All reactions