Repository navigation
Conversation
_create_id hashed the metadata with json.dumps(..., default=str), which loses the type of
values that JSON can't represent: a tuple is written as a list, and a set goes through
str(). Documents with meta {"tags": [1, 2]} and {"tags": (1, 2)} therefore ended up with
the same ID, so writing both raised DuplicateDocumentError or, with OVERWRITE, silently
dropped one.
Tuples and sets are now marked before hashing, and sets are sorted by the JSON of their
elements so iteration order doesn't change the ID. Metadata that only uses JSON types is
passed through untouched, so existing IDs stay the same.
Contributor
|
@sunhouy is attempting to deploy a commit to the deepset Team on Vercel. A member of the Team first needs to authorize it. |
Contributor
Coverage reportClick to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||||||||||||||
1AbdullahButt
left a comment
There was a problem hiding this comment.
I compared Document(content="x", meta=a).id == Document(content="x", meta=b).id on main and on this branch.
meta a |
meta b |
same ID on main | same ID on this PR |
|---|---|---|---|
{"t": [1, 2]} |
{"t": (1, 2)} |
yes | no |
{"t": {1, 2}} |
{"t": "{1, 2}"} |
yes | no |
{"t": {1, 2, 3}} |
{"t": {3, 2, 1}} |
yes | yes |
{"a": 1, "b": [1, "x"]} |
same keys, other order | yes | yes |
{"t": (1, 2)} |
{"t": {"__tuple__": [1, 2]}} |
no | yes |
{"t": {1, 2}} |
{"t": {"__set__": [1, 2]}} |
no | yes |
{"t": {1, 2}} |
{"t": frozenset({1, 2})} |
no | yes |
The reported collisions are fixed. IDs for plain JSON metadata are unchanged (I checked one: 971eba39… on both). test/dataclasses/test_document.py passes (62 passed).
Things I found:
- New collisions with the marker keys. A tuple or set now hashes the same as a dict that uses the marker key, so the last three rows collide on this branch and did not on
main. A real dict with a__tuple__key is unlikely in metadata. But the PR is about type-based collisions, so it may be worth a marker that a plain JSON value cannot produce. Or the trade-off could be written down in the docstring.setvsfrozensetis probably fine to treat as equal, but that is also a change frommain. - IDs change for existing documents with tuple or set metadata. For example,
meta={"t": (1, 2)}gives9ef432f5…onmainand22818c13…here. The release note says that IDs of JSON-only metadata are unchanged. It could also say that documents with tuple or set metadata get new IDs. Then users who re-index withDuplicatePolicy.SKIPwould expect new copies instead of skips.
(I made this review with help from Claude Code. I ran the table above on main and on this branch, and pytest test/dataclasses/test_document.py on this branch. I did not run document store integration tests.)
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issues
No issue for this, I hit it while working on something else.
Proposed Changes:
Document._create_idbuilds the hash fromjson.dumps(self.meta, sort_keys=True, default=str). Two things in there drop the type of a value:json.dumpswrites(1, 2)as[1, 2], the same as a list.default=strand becomes the string"{1, 2}".Both cases make unrelated documents collide:
InMemoryDocumentStore.write_documents([...])then raisesDuplicateDocumentError: ID 'de0fa6eb5c49...' already exists, and withDuplicatePolicy.OVERWRITEthe second document is silently dropped. Indexing a JSONL where the same field is sometimes a list and sometimes a tuple from the source format is enough to hit it.The fix walks
metabefore hashing and marks the two problematic types:{"__tuple__": [...]}, keeping the element order{"__set__": [...]}, sorted by the JSON of each element so that iteration order doesn't leak into the IDEverything else is passed through untouched, which is what keeps existing IDs stable:
json.dumpsnever sees the marker dicts unless the metadata actually contained a tuple or a set, so documents whose metadata only uses JSON types hash exactly as before. The hardcoded IDs intest_init_with_legacy_fieldsandtest_initstill pass, andtest_to_dictstill assertsdocument["id"] == doc._create_id().I did not touch the empty vs
Nonecontent case, there are a few closed issues and PRs about that and I didn't want to reopen a settled discussion here.How did you test it?
New tests in
test/dataclasses/test_document.py:test_id_distinguishes_meta_value_types, parametrized over list vs tuple, set vs itsstr(), set vs list, nested lists vs nested tuples, and two different nestings of the same tuple.test_id_of_set_meta_is_independent_of_iteration_order:{1, 2, 3}and{3, 2, 1}give the same ID, which is what the sorting is for.test_id_of_tuple_meta_depends_on_element_order:(1, 2)and(2, 1)don't.test_documents_with_different_meta_types_can_be_stored_together: writes the list and the tuple document into anInMemoryDocumentStoreand gets 2 documents back.Before:
pytest test/dataclasses/test_document.pygives 4 failed, 58 passed. After: 62 passed.pytest test/dataclasses test/document_stores/test_in_memory.pygives 794 passed, 8 skipped. The one failure in that run,test_image_content_from_url_wrong_mime_type, fails on a clean main too (it fetches a URL).ruff checkandruff format --checkare clean.Notes for the reviewer
Document.to_dict()andDocument.metaare unchanged, so round trips and serialization are not affected._make_metadata_value_hashableinutils/misc.pyhas a similar blind spot for sets, but it's used by the evaluators for grouping and not for IDs. I left it alone to keep this PR focused.Checklist
fix:,feat:,build:,chore:,ci:,docs:,style:,refactor:,perf:,test:and added!in case the PR includes breaking changes.