Skip to content

Cross-well processing mode (pooling=across_wells) with UMI collapsing across wells - #14

Draft
bnemoz wants to merge 3 commits into
mainfrom
v2/crosswell-mode
Draft

bnemoz wants to merge 3 commits into
mainfrom
v2/crosswell-mode

Conversation

@bnemoz

@bnemoz bnemoz commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

What

A new cross-well processing mode, pairplex.run(..., pooling="across_wells") (CLI --pooling across_wells). The default stays per_well, so the two architectures can be compared. legacy=True (--legacy) forces per-well v2-base behaviour.

Why

Barcoded cDNA leaks out of cells before the well split. A released molecule keeps its barcode and UMI and lands in a random well, but PCR happens after the split, so each molecule's reads still sit in one well. Per-well processing therefore sees a cell's chains scattered over wells. In SimPlex, 69–73% of cells have one chain missing from their home well. pair-pooled pools only finished per-well contigs. This mode pools at read level.

Pipeline (pairplex/crosswell.py)

  1. Parse. Every well file goes through the existing parse_barcodes. The well number is the last integer in the file stem (VDJ_123, well042). Reads stream into on-disk barcode-hash shards through one append-only parquet writer per shard. The default is about 1 shard per 160 MB of gzipped input (~1M reads); override with crosswell_shards. Memory is bounded by one shard plus a 1M-read parse buffer, never the whole reaction.
  2. Collapse UMIs per barcode across all wells with UMI-tools directional adjacency: b is absorbed by a if Hamming(a,b)==1 and count(a) ≥ 2·count(b)−1, where counts are reads per raw UMI pooled over wells.
  3. Cluster each barcode's pooled reads once, with the legacy greedy_clustering and threshold, then make one POA consensus per cluster as legacy does. Contigs are named <barcode>_contig-<i>. The legacy filters apply to pooled reads and corrected UMIs.
  4. Molecule wells. Within each contig, each molecule goes to the well holding most of its reads (ties go to the lowest well). Reads in other wells count as hopped. n_w is the number of molecules assigned to well w.
  5. Annotate passing contigs per shard with abstar.run(mmseqs_threads=1).
  6. Pair. Leakage f is estimated over the whole reaction; estimate_leakage was split into leakage_from_totals so species totals can be aggregated per shard. Each shard is then paired with the unchanged pooled_pairing.pair_pooled, which gives the home-well posterior, reciprocal best pairs, confidence and margin. A barcode always lives in one shard.

Outputs:

  • pairs/pairs.parquet: contract schema, plus umis:0/1 (molecules at the home well), umis_home, umis_pooled, n_wells, p_home per chain, and margin.
  • contigs/contigs.parquet: one row per contig, with annotation.
  • contigs/contig_wells.parquet: contig, well, umis, reads, umis_present.

This mode does not write per-well parsed/, metadata/, consensus/ or annotated/ files. Stats go to the provenance run log: molecules, multi-well molecule fraction, hopped-read fraction, leakage, timings, peak RSS.

Tests

  • 39 pass (pairplex/tests/test_crosswell.py). They cover well parsing, sharding (determinism, every barcode in one shard, streamed parsing), directional collapsing (count rule, transitive absorption, cross-well counts), molecule well assignment and hop accounting, per-barcode clustering and counts, leakage refactor equivalence, routing (default and legacy=True take the per-well path), and an end-to-end run on 2 SimPlex wells that checks the pairs.parquet contract schema.
  • Per-well equivalence on 2 SimPlex wells: v2-base, this branch's default and legacy=True, pooling="across_wells" give identical metadata/, consensus/ and annotated/ contents. Row order differs, because v2-base itself collects results with as_completed.

Benchmark

Verdict: FAIL (A1xw-crosswell-v1, kwargs {"pooling": "across_wells"}, all 4 datasets). Branch pairs is compared against the baseline pooled set. This PR stays a draft. The architecture is here for the planned comparison; it does not meet the acceptance rule as-is.

Sims (full):

[PASS] sim_cal / pairs (vs baseline pooled)
   precision            0.627 ->      0.627
   cell_recall          0.152 ->      0.158
   emitted               1585 ->       1653
   n_conf0.9               41 ->          6
   prec_conf0.9         0.878 ->      1.000

[FAIL] sim_imp / pairs (vs baseline pooled)
   precision            0.967 ->      0.963
   cell_recall          0.871 ->      0.867
   emitted               6368 ->       6364
   n_conf0.9             4387 ->       4198
   prec_conf0.9         0.987 ->      0.986

Confidence bins, sim_cal (n / precision), baseline pooled -> cross-well: <0.1 830/0.62 -> 1228/0.55; 0.1-0.3 417/0.59 -> 387/0.83; 0.3-0.6 283/0.66 -> 7/1.00; 0.6-0.9 14/0.86 -> 25/0.96; >=0.9 41/0.88 -> 6/1.00.

Real:

  • real_d02 FAIL, clonal purity −2.4 pts. Confident (>=0.9) pairs almost disappear.
  • real_d07 FAIL, clonal purity −3.1 pts. Confident (>=0.9) pairs almost disappear.
  • Lineage retention: PASS on both. No baseline lineage of >=5 cells is lost; 6 (d02) and 4 (d07) 3-cell clones are lost. Light-chain coherence is lower on both.

Runtime: 17386 s -> 15508 s (PASS). Per dataset: sim_cal 164 s, sim_imp 199 s, real_d07 6149 s (34 shards), real_d02 9011 s (48 shards).
Peak RSS (from the provenance log): sims 1.7–1.9 GB; real_d07 3.2 GB main / 2.7 GB largest worker; real_d02 2.5 GB.

Cross-well stats (sim_cal): reads off the molecule's main well 1.7%, the same as the measured index-hopping rate. 12.6% of molecules have a stray read in another well. Estimated leakage 0.75 (pooled baseline: 0.67); real data about 0.88.

Why it does not win (as measured)

  • Molecules really do live in one well. 98.3% of reads sit in their molecule's main well, so the premise holds.
  • Pooling clusters does not add recall. Recall is within ±0.6 pt of pooled pairing on both sims. The shared-home-well pairing model, not per-well contig loss, limits what can be paired.
  • Lower confidence. Corrected-molecule counts are lower than per-well raw-UMI counts, which UMI errors inflate, and the estimated f is higher. Both flatten the home-well posterior, so far fewer pairs reach 0.9. Confidence looks better calibrated in sim_cal.
  • Lower real purity. On real data the mode emits 26–66% more pairs, mostly at low confidence, and purity drops.
  • Molecules split across clusters. 18.5% of molecules (corrected UMIs) span more than one cluster of their barcode. Greedy clustering splits molecules (fragments), which is the fix-2 lane.
  • Same-clone cells sharing a barcode in different wells collapse into one chain species and give at most one pair, exactly as in the pooled baseline. The sims contain no such cells, so they can't measure it. On real data no lineage of >=5 cells is lost.

🤖 Generated with Claude Code

bnemoz and others added 3 commits November 17, 2026 16:24
…llapsing across wells

Pools reads by droplet barcode across all wells of a reaction (on-disk barcode
shards, one shard in memory at a time), collapses UMIs per barcode across wells
(directional adjacency), clusters each barcode once with the legacy greedy
clustering + POA, assigns each molecule to its majority well, counts molecules
per contig per well, annotates per shard (mmseqs_threads=1) and pairs by shared
home well with pooled_pairing.pair_pooled (leakage estimated over the reaction).

Writes pairs/pairs.parquet, contigs/contigs.parquet, contigs/contig_wells.parquet.
Default stays pooling="per_well"; legacy=True forces per-well v2-base behaviour.
estimate_leakage is split so totals can be aggregated per shard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- one legacy flag (forces tso/orientation fixes off and pooling='per_well')
- cross-well mode now honours tso_g_max, barcode_prefer_exact and mmseqs_threads
  (parse_barcodes and abstar were called with hard-coded/default values)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
legacy also forces pooling='per_well'; the per-well-only options (umi_collapse, pairing,
min_dominance, collapse_clonal, support_columns, qc) are documented as not applying in
cross-well mode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant