Skip to content

fix: drop stale split offsets from cleaned chunks in DocumentCleaner - #13846

Closed
sunhouy wants to merge 1 commit into
deepset-ai:mainfrom
sunhouy:fix/document-cleaner-stale-split-offsets
Closed

sunhouy wants to merge 1 commit into
deepset-ai:mainfrom
sunhouy:fix/document-cleaner-stale-split-offsets

Conversation

@sunhouy

@sunhouy sunhouy commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Related Issues

No issue for this one, reporting it here directly. It's not the same as the split_idx_start problems in #13806 and #13480: those are about the offsets a splitter computes, this one is about the offsets DocumentCleaner keeps after it rewrites the text of a chunk.

Proposed Changes:

DocumentCleaner rewrites the content of every chunk it gets, but it copied the metadata of the input document unchanged. Two of those fields, split_idx_start and _split_overlap, are computed by the splitters over the text of the whole source document, and cleaning a chunk changes its length. After that the offsets no longer describe the chunk they are attached to.

SentenceWindowRetriever.merge_documents_text uses split_idx_start to cut away the part of a chunk that was already emitted, so it merges the neighbours at the wrong positions and returns windows with the text glued together. DocumentPreprocessor splits first and then cleans, so this happened with default settings:

text = "Alpha beta.  Gamma   delta.  Epsilon zeta."
chunks = DocumentSplitter(split_by="sentence", split_length=1, split_overlap=0, split_threshold=0).run(
    documents=[Document(content=text)]
)["documents"]
cleaned = DocumentCleaner().run(documents=chunks)["documents"]
# the offsets are 0, 13, 29 but the chunks are now 11, 11 and 13 characters long
# window: 'Alpha beta.Gamma delta.Epsilon zeta.'

What the change does:

  • DocumentCleaner drops split_idx_start and _split_overlap from every chunk of a source that had at least one of its chunks rewritten. The decision is per source rather than per chunk, because the offsets of one chunk only make sense next to the offsets of its siblings: a chunk that came out unchanged can still sit next to a rewritten one and be merged at the wrong place.
  • Chunks whose source was not touched keep their offsets, and metadata that is not an offset (split_id, source_id, page_number) is always kept. Documents without a split_id are left alone entirely, since nothing tells us the offsets belong to a split.
  • The fallback in merge_documents_text, the one taken when the offsets are missing, joined the contents with "". With the offsets gone that turned 'Sentence 1.' and 'Sentence 2.' into 'Sentence 1.Sentence 2.', so it now joins with a space.

The window for the example above is 'Alpha beta. Gamma delta. Epsilon zeta.', which is the cleaned text. The cleaned chunk text and the chunk IDs are unchanged, so nothing changes for anyone who does not read split_idx_start. I did not try the reverse (clean before splitting) that #13481 proposed.

How did you test it?

Added tests in test/components/preprocessors/test_document_cleaner.py:

  • test_drops_split_offsets_of_rewritten_chunks: the offsets go away, split_id and source_id stay.
  • test_drops_split_offsets_for_all_chunks_of_a_rewritten_source: the unchanged sibling loses its offset too.
  • test_keeps_split_offsets_of_untouched_chunks: no rewrite, no removal.
  • test_split_offsets_survive_on_documents_without_split_id: metadata without a split_id is not touched.

And one end to end in test/components/retrievers/test_sentence_window_retriever.py, test_run_after_cleaning_does_not_glue_chunks_together, which builds the window from cleaned chunks and asserts it matches the cleaned text. Three existing tests in that file and the three matching ones in test_sentence_window_retriever_async.py asserted the glued output ('Sentence 1.Sentence 2....', 'a1.a2.a3.', '1.2.3.') as expected: they build documents without split_idx_start on purpose, and now expect the space separated text.

Before the change: 6 failed, 61 passed on test/components/preprocessors/test_document_cleaner.py and test/components/retrievers/test_sentence_window_retriever.py (the 4 new tests plus the 3 updated ones, which fail on their new expectations).
After: pytest test/components/preprocessors test/components/retrievers -p no:recording gives 886 passed, 21 skipped. ruff check and ruff format --check are clean on the touched files.

Notes for the reviewer

Checklist

  • I have read the contributors guidelines and the code of conduct.
  • I have updated the related issue with new insights and changes.
  • I have added unit tests and updated the docstrings.
  • I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
  • I have documented my code.
  • I have added a release note file, following the contributors guidelines.
  • I have run pre-commit hooks and fixed any issue.

DocumentCleaner rewrites the content of each chunk but kept the metadata the splitter
had attached to it. split_idx_start and _split_overlap are computed over the text of
the whole source document, so as soon as cleaning changes the length of a chunk they
point at the wrong place. SentenceWindowRetriever then merges the neighbouring chunks
at those offsets and returns windows with the text glued together, which is what
DocumentPreprocessor produced with default settings.

The cleaner now drops both fields for every chunk of a source that had any of its
chunks rewritten, and keeps the rest of the split metadata. When the offsets are gone
the retriever falls back to joining the contents, so that path now uses a space
separator instead of concatenating the chunks without one.
@sunhouy
sunhouy requested a review from a team as a code owner November 20, 2026 18:24
@sunhouy
sunhouy requested review from anakin87 and removed request for a team November 20, 2026 18:24
@vercel

vercel Bot commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

@sunhouy is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

@github-actions

Copy link
Copy Markdown
Contributor

Hi @sunhouy, thanks for your interest in contributing to Haystack! 🙏

We're closing this pull request because the issue it links already has an open PR:

To keep the review queue manageable, we review one PR per issue. If you think your approach has something to add, please leave a comment on the existing PR so we can discuss it there. Before picking up an issue, please check whether a PR is already linked to it.

Thanks for understanding!

This is an automated message to help us keep the review queue healthy.

@github-actions github-actions Bot closed this Nov 20, 2026
@github-actions github-actions Bot added P2 Medium priority, add to the next sprint if no P1 available topic:tests type:documentation Improvements on the docs and removed P2 Medium priority, add to the next sprint if no P1 available labels Nov 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  haystack/components/preprocessors
  document_cleaner.py
  haystack/components/retrievers
  sentence_window_retriever.py
  haystack/core/pipeline
  pipeline.py
Project Total  

This report was generated by python-coverage-comment-action

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

topic:tests type:documentation Improvements on the docs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant