Repository navigation
Conversation
DocumentCleaner rewrites the content of each chunk but kept the metadata the splitter had attached to it. split_idx_start and _split_overlap are computed over the text of the whole source document, so as soon as cleaning changes the length of a chunk they point at the wrong place. SentenceWindowRetriever then merges the neighbouring chunks at those offsets and returns windows with the text glued together, which is what DocumentPreprocessor produced with default settings. The cleaner now drops both fields for every chunk of a source that had any of its chunks rewritten, and keeps the rest of the split metadata. When the offsets are gone the retriever falls back to joining the contents, so that path now uses a space separator instead of concatenating the chunks without one.
Contributor
|
@sunhouy is attempting to deploy a commit to the deepset Team on Vercel. A member of the Team first needs to authorize it. |
Contributor
|
Hi @sunhouy, thanks for your interest in contributing to Haystack! 🙏 We're closing this pull request because the issue it links already has an open PR:
To keep the review queue manageable, we review one PR per issue. If you think your approach has something to add, please leave a comment on the existing PR so we can discuss it there. Before picking up an issue, please check whether a PR is already linked to it. Thanks for understanding! This is an automated message to help us keep the review queue healthy. |
Contributor
Coverage reportClick to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||||||||||||||||||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issues
No issue for this one, reporting it here directly. It's not the same as the
split_idx_startproblems in #13806 and #13480: those are about the offsets a splitter computes, this one is about the offsetsDocumentCleanerkeeps after it rewrites the text of a chunk.Proposed Changes:
DocumentCleanerrewrites the content of every chunk it gets, but it copied the metadata of the input document unchanged. Two of those fields,split_idx_startand_split_overlap, are computed by the splitters over the text of the whole source document, and cleaning a chunk changes its length. After that the offsets no longer describe the chunk they are attached to.SentenceWindowRetriever.merge_documents_textusessplit_idx_startto cut away the part of a chunk that was already emitted, so it merges the neighbours at the wrong positions and returns windows with the text glued together.DocumentPreprocessorsplits first and then cleans, so this happened with default settings:What the change does:
DocumentCleanerdropssplit_idx_startand_split_overlapfrom every chunk of a source that had at least one of its chunks rewritten. The decision is per source rather than per chunk, because the offsets of one chunk only make sense next to the offsets of its siblings: a chunk that came out unchanged can still sit next to a rewritten one and be merged at the wrong place.split_id,source_id,page_number) is always kept. Documents without asplit_idare left alone entirely, since nothing tells us the offsets belong to a split.merge_documents_text, the one taken when the offsets are missing, joined the contents with"". With the offsets gone that turned'Sentence 1.'and'Sentence 2.'into'Sentence 1.Sentence 2.', so it now joins with a space.The window for the example above is
'Alpha beta. Gamma delta. Epsilon zeta.', which is the cleaned text. The cleaned chunk text and the chunk IDs are unchanged, so nothing changes for anyone who does not readsplit_idx_start. I did not try the reverse (clean before splitting) that #13481 proposed.How did you test it?
Added tests in
test/components/preprocessors/test_document_cleaner.py:test_drops_split_offsets_of_rewritten_chunks: the offsets go away,split_idandsource_idstay.test_drops_split_offsets_for_all_chunks_of_a_rewritten_source: the unchanged sibling loses its offset too.test_keeps_split_offsets_of_untouched_chunks: no rewrite, no removal.test_split_offsets_survive_on_documents_without_split_id: metadata without asplit_idis not touched.And one end to end in
test/components/retrievers/test_sentence_window_retriever.py,test_run_after_cleaning_does_not_glue_chunks_together, which builds the window from cleaned chunks and asserts it matches the cleaned text. Three existing tests in that file and the three matching ones intest_sentence_window_retriever_async.pyasserted the glued output ('Sentence 1.Sentence 2....','a1.a2.a3.','1.2.3.') as expected: they build documents withoutsplit_idx_starton purpose, and now expect the space separated text.Before the change: 6 failed, 61 passed on
test/components/preprocessors/test_document_cleaner.pyandtest/components/retrievers/test_sentence_window_retriever.py(the 4 new tests plus the 3 updated ones, which fail on their new expectations).After:
pytest test/components/preprocessors test/components/retrievers -p no:recordinggives 886 passed, 21 skipped.ruff checkandruff format --checkare clean on the touched files.Notes for the reviewer
DocumentSplittermisplacing repeated literal chunks and fix: locate repeated overlapping custom splits in source order #13807 is already on it, so that one is out of scope here. DocumentPreprocessor: split metadata (split_idx_start, _split_overlap) does not match the cleaned chunks, so SentenceWindowRetriever builds wrong windows #13480 and fix!: clean before splitting in DocumentPreprocessor #13481 covered the same stale metadata forDocumentPreprocessorand were closed, so I kept this one to the metadata and left the content of the chunks alone.Checklist
fix:,feat:,build:,chore:,ci:,docs:,style:,refactor:,perf:,test:and added!in case the PR includes breaking changes.