Repository navigation
Conversation
…llapsing across wells Pools reads by droplet barcode across all wells of a reaction (on-disk barcode shards, one shard in memory at a time), collapses UMIs per barcode across wells (directional adjacency), clusters each barcode once with the legacy greedy clustering + POA, assigns each molecule to its majority well, counts molecules per contig per well, annotates per shard (mmseqs_threads=1) and pairs by shared home well with pooled_pairing.pair_pooled (leakage estimated over the reaction). Writes pairs/pairs.parquet, contigs/contigs.parquet, contigs/contig_wells.parquet. Default stays pooling="per_well"; legacy=True forces per-well v2-base behaviour. estimate_leakage is split so totals can be aggregated per shard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- one legacy flag (forces tso/orientation fixes off and pooling='per_well') - cross-well mode now honours tso_g_max, barcode_prefer_exact and mmseqs_threads (parse_barcodes and abstar were called with hard-coded/default values) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
legacy also forces pooling='per_well'; the per-well-only options (umi_collapse, pairing, min_dominance, collapse_clonal, support_columns, qc) are documented as not applying in cross-well mode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A new cross-well processing mode,
pairplex.run(..., pooling="across_wells")(CLI--pooling across_wells). The default staysper_well, so the two architectures can be compared.legacy=True(--legacy) forces per-well v2-base behaviour.Why
Barcoded cDNA leaks out of cells before the well split. A released molecule keeps its barcode and UMI and lands in a random well, but PCR happens after the split, so each molecule's reads still sit in one well. Per-well processing therefore sees a cell's chains scattered over wells. In SimPlex, 69–73% of cells have one chain missing from their home well.
pair-pooledpools only finished per-well contigs. This mode pools at read level.Pipeline (
pairplex/crosswell.py)parse_barcodes. The well number is the last integer in the file stem (VDJ_123,well042). Reads stream into on-disk barcode-hash shards through one append-only parquet writer per shard. The default is about 1 shard per 160 MB of gzipped input (~1M reads); override withcrosswell_shards. Memory is bounded by one shard plus a 1M-read parse buffer, never the whole reaction.greedy_clusteringand threshold, then make one POA consensus per cluster as legacy does. Contigs are named<barcode>_contig-<i>. The legacy filters apply to pooled reads and corrected UMIs.n_wis the number of molecules assigned to well w.abstar.run(mmseqs_threads=1).estimate_leakagewas split intoleakage_from_totalsso species totals can be aggregated per shard. Each shard is then paired with the unchangedpooled_pairing.pair_pooled, which gives the home-well posterior, reciprocal best pairs, confidence and margin. A barcode always lives in one shard.Outputs:
pairs/pairs.parquet: contract schema, plusumis:0/1(molecules at the home well),umis_home,umis_pooled,n_wells,p_homeper chain, andmargin.contigs/contigs.parquet: one row per contig, with annotation.contigs/contig_wells.parquet: contig, well, umis, reads, umis_present.This mode does not write per-well
parsed/,metadata/,consensus/orannotated/files. Stats go to the provenance run log: molecules, multi-well molecule fraction, hopped-read fraction, leakage, timings, peak RSS.Tests
pairplex/tests/test_crosswell.py). They cover well parsing, sharding (determinism, every barcode in one shard, streamed parsing), directional collapsing (count rule, transitive absorption, cross-well counts), molecule well assignment and hop accounting, per-barcode clustering and counts, leakage refactor equivalence, routing (default andlegacy=Truetake the per-well path), and an end-to-end run on 2 SimPlex wells that checks thepairs.parquetcontract schema.legacy=True, pooling="across_wells"give identicalmetadata/,consensus/andannotated/contents. Row order differs, because v2-base itself collects results withas_completed.Benchmark
Verdict: FAIL (
A1xw-crosswell-v1, kwargs{"pooling": "across_wells"}, all 4 datasets). Branchpairsis compared against the baselinepooledset. This PR stays a draft. The architecture is here for the planned comparison; it does not meet the acceptance rule as-is.Sims (full):
Confidence bins, sim_cal (n / precision), baseline pooled -> cross-well: <0.1 830/0.62 -> 1228/0.55; 0.1-0.3 417/0.59 -> 387/0.83; 0.3-0.6 283/0.66 -> 7/1.00; 0.6-0.9 14/0.86 -> 25/0.96; >=0.9 41/0.88 -> 6/1.00.
Real:
Runtime: 17386 s -> 15508 s (PASS). Per dataset: sim_cal 164 s, sim_imp 199 s, real_d07 6149 s (34 shards), real_d02 9011 s (48 shards).
Peak RSS (from the provenance log): sims 1.7–1.9 GB; real_d07 3.2 GB main / 2.7 GB largest worker; real_d02 2.5 GB.
Cross-well stats (sim_cal): reads off the molecule's main well 1.7%, the same as the measured index-hopping rate. 12.6% of molecules have a stray read in another well. Estimated leakage 0.75 (pooled baseline: 0.67); real data about 0.88.
Why it does not win (as measured)
pooledbaseline. The sims contain no such cells, so they can't measure it. On real data no lineage of >=5 cells is lost.🤖 Generated with Claude Code