docs.plus: our measured DOCX round-trip result, and an invitation to correct it in a pull request #662

We measured docs.plus together with eleven other open-source DOCX tools and published the whole thing — the corpus, the harnesses, the scorer, the logs and the reports — so that every number below can be checked and corrected.

Everything is here: https://github.com/arthrod/editors_trial

What the trial is

One input: fixture/complex.docx, a 35-part hand-built OOXML document that deliberately contains tracked changes, complex and simple fields, five content controls, tables with merges and nested tables, three sections including a landscape two-column one, a bar chart, a SmartArt diagram, an OLE object with an embedded workbook, an image, footnotes, endnotes, comments, headers and footers, and a set of namespace extensions Word writes.

Every editor was driven through its own documented import and export path — docs.plus's own entry point, not a re-implementation of it — and the result was scored by a 99-check diff at package-part, element, attribute and text level. Every output was re-opened to confirm it is still a valid document. The scorer, the harnesses, the fixture and the raw logs are in the repository.

What we measured for docs.plus

21.1% weighted, 24.1% critical (7/29), 20/99 checks, 91.7% text retention, 11/35 parts kept — 79 failed checks.

docs.plus keeps 91.7% of the text — among the highest in the trial — and the reason it loses the revision model is a single, identifiable seam rather than a design flaw.

The revision loss is mammoth, and we can point at the lines. mammoth/lib/docx/body-reader.js:447 unwraps w:ins and :450 unwraps w:moveTo, so insertions become ordinary text; :729 sets ignoreElements["w:del"] = true and :732 sets ignoreElements["w:moveFrom"] = true, so deletions and move-outs are dropped outright; :727 drops w:sectPr entirely. Counted across the whole chain — fixture → mammoth HTML → rendered HTML → Tiptap JSON → output — w:ins goes 3→0→0→0→0, w:del 3→0→0→0→0 and w:moveFrom 1→0→0→0→0. No author, no date and no w16du:dateUtc survives anywhere downstream. That is the import step, not the editor.

Two things worth knowing that are not in the score:

• On the hostile-input axis, docs.plus cleaned six of the ten hostile packages — the decompression bomb, the entry flood, the attribute bomb, the zip-slip, the external-relationship bomb and the active-content package — and refused the entity bomb and the corrupt-parts package (Report D).
• In the rendering report, docs.plus moves the footnote and endnote bodies to the end of the document as a numbered list, instead of placing them at the foot of their page (Report C §2.10). Your round trip keeps the text; the page is where it lands that differs.

The product itself — every heading with its own chatroom, the table of contents showing who is in the room, an MCP path so Claude or ChatGPT can read the documents an account can open — is not something a DOCX fidelity benchmark can score, and we would rather say that than pretend the number describes it.

The reports, and what each one is

• Report A — Serialization by Namespace — what each of the corpus's 27 namespace vocabularies does, and what it costs when it is lost. One section per namespace: what the vocabulary is in WordprocessingML, what each editor's own source does with it, and what each editor actually did to it, measured.

• Report A2 — Pass-Through — the complement of fidelity: of the bytes that came out, how many did the editor never even look at. Pass-through is the strongest possible fidelity and the weakest possible editing, and it is the same fact.

• Report B — Malformed Parts, Causes, Detection and Repair — twelve defect classes that Microsoft Word refuses to open, found by bisecting the corpus against Word itself; what malformed parts look like, what causes them, and whether any editor repairs them.

• Report C — Rendering Versus Word — what each editor's own running application puts on the page, screenshotted and compared with Word's own PDF export of the same corpus. Fidelity in the file and fidelity on the page are two different results, and this report keeps them apart.

• Report D — Safety, Resource Bombs and Hostile Input — ten hostile packages — decompression, entry flood, attribute bomb, deep nesting, entities, zip-slip, external relationships, active content, layout bomb, corrupt parts — graded on a separate axis, because preserving a hostile payload is not the same result as a crash.

• The per-tool note for docs.plus: https://github.com/arthrod/editors_trial/blob/main/reports/extras/docsplus.md

• What each measured tool is: https://github.com/arthrod/editors_trial/blob/main/reports/vendors.md

If anything here is wrong, please open a pull request

We would rather have a correction from you than a quiet disagreement. Three things are worth correcting, in this order of usefulness:

1. A number. Every figure in the reports is produced by python3 build/score_all.py over the committed outputs in out/, and it is checked against the scorer's own output by build/check_report_consistency.py, so a figure that cannot be reproduced fails the gate rather than slipping through. If one of ours is wrong, open a PR against the file that carries it — https://github.com/arthrod/editors_trial/blob/main/reports/extras/docsplus.md for the per-tool note, or the report that quotes it — and if the figure comes from the measurement rather than from prose, include the corrected out/docsplus/score.json too. build/test_report_claims.py enforces that the per-tool note still agrees with the shipped scorer, so it will tell you whether a correction is self-consistent.
2. A reading of your code. The [CODE] citations name a file and a line. If we read your implementation wrong — or if it has moved on since the commit we pinned — a PR that fixes the citation is as welcome as one that fixes a number.
3. A judgement. Some of the report's claims are readings, not measurements: what a design decision "means", which of two behaviours is the better one. If you disagree with a judgement, the place to say so is a pull request against the report, not a new issue.

We are not going to argue about the framing of a published number. If our reading is wrong, the fix belongs in the repository, in public, where the next person can check it.

What the trial does not claim

• One document. The corpus is deliberately hostile, but it is one document, not a corpus survey.
• A round trip, not an editing session. Every project was exercised on import → export with no edits. Selective-save behaviour that only triggers on a dirty block is therefore not distinguished from unconditional regeneration, except where the project documents otherwise.
• Word as a validator, not as an oracle. The corpus is checked against the ECMA-376 schemas and opened in Microsoft Word 16.115, but the outputs were judged by the scorer and by LibreOffice. A document that Word opens but that the scorer flags is still a loss.
• A moving instrument. The checker gained checks over the life of the study; the published figures come from the current 99-check scorer, and the repository is the record of that.

Reproducing it

git clone https://github.com/arthrod/editors_trial.git && cd editors_trial
python3 build/score_all.py                       # the scorecard table
python3 build/check_roundtrip.py fixture/complex.docx out/docsplus/complex.docx
bash build/run_second_pass.sh                    # the idempotence table