Skip to content

fix: keep the type of meta values in the Document ID hash - #13847

Open
sunhouy wants to merge 1 commit into
deepset-ai:mainfrom
sunhouy:fix/document-id-meta-type-collisions
Open

sunhouy wants to merge 1 commit into
deepset-ai:mainfrom
sunhouy:fix/document-id-meta-type-collisions

Conversation

@sunhouy

@sunhouy sunhouy commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

Related Issues

No issue for this, I hit it while working on something else.

Proposed Changes:

Document._create_id builds the hash from json.dumps(self.meta, sort_keys=True, default=str). Two things in there drop the type of a value:

  • JSON has no tuple, so json.dumps writes (1, 2) as [1, 2], the same as a list.
  • A set isn't JSON serializable at all, so it goes through default=str and becomes the string "{1, 2}".

Both cases make unrelated documents collide:

Document(content="x", meta={"tags": [1, 2]}).id == Document(content="x", meta={"tags": (1, 2)}).id  # True
Document(content="x", meta={"tags": {1, 2}}).id == Document(content="x", meta={"tags": "{1, 2}"}).id  # True

InMemoryDocumentStore.write_documents([...]) then raises DuplicateDocumentError: ID 'de0fa6eb5c49...' already exists, and with DuplicatePolicy.OVERWRITE the second document is silently dropped. Indexing a JSONL where the same field is sometimes a list and sometimes a tuple from the source format is enough to hit it.

The fix walks meta before hashing and marks the two problematic types:

  • a tuple becomes {"__tuple__": [...]}, keeping the element order
  • a set becomes {"__set__": [...]}, sorted by the JSON of each element so that iteration order doesn't leak into the ID

Everything else is passed through untouched, which is what keeps existing IDs stable: json.dumps never sees the marker dicts unless the metadata actually contained a tuple or a set, so documents whose metadata only uses JSON types hash exactly as before. The hardcoded IDs in test_init_with_legacy_fields and test_init still pass, and test_to_dict still asserts document["id"] == doc._create_id().

I did not touch the empty vs None content case, there are a few closed issues and PRs about that and I didn't want to reopen a settled discussion here.

How did you test it?

New tests in test/dataclasses/test_document.py:

  • test_id_distinguishes_meta_value_types, parametrized over list vs tuple, set vs its str(), set vs list, nested lists vs nested tuples, and two different nestings of the same tuple.
  • test_id_of_set_meta_is_independent_of_iteration_order: {1, 2, 3} and {3, 2, 1} give the same ID, which is what the sorting is for.
  • test_id_of_tuple_meta_depends_on_element_order: (1, 2) and (2, 1) don't.
  • test_documents_with_different_meta_types_can_be_stored_together: writes the list and the tuple document into an InMemoryDocumentStore and gets 2 documents back.

Before: pytest test/dataclasses/test_document.py gives 4 failed, 58 passed. After: 62 passed.
pytest test/dataclasses test/document_stores/test_in_memory.py gives 794 passed, 8 skipped. The one failure in that run, test_image_content_from_url_wrong_mime_type, fails on a clean main too (it fetches a URL). ruff check and ruff format --check are clean.

Notes for the reviewer

  • The marker dicts live in the hash input only. Document.to_dict() and Document.meta are unchanged, so round trips and serialization are not affected.
  • Someone who already relied on a tuple and a list collapsing to the same document will now get two documents. That's the point of the change, but it is a visible difference, hence the release note.
  • _make_metadata_value_hashable in utils/misc.py has a similar blind spot for sets, but it's used by the evaluators for grouping and not for IDs. I left it alone to keep this PR focused.

Checklist

  • I have read the contributors guidelines and the code of conduct.
  • I have updated the related issue with new insights and changes.
  • I have added unit tests and updated the docstrings.
  • I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
  • I have documented my code.
  • I have added a release note file, following the contributors guidelines.
  • I have run pre-commit hooks and fixed any issue.

_create_id hashed the metadata with json.dumps(..., default=str), which loses the type of
values that JSON can't represent: a tuple is written as a list, and a set goes through
str(). Documents with meta {"tags": [1, 2]} and {"tags": (1, 2)} therefore ended up with
the same ID, so writing both raised DuplicateDocumentError or, with OVERWRITE, silently
dropped one.

Tuples and sets are now marked before hashing, and sets are sorted by the JSON of their
elements so iteration order doesn't change the ID. Metadata that only uses JSON types is
passed through untouched, so existing IDs stay the same.
@sunhouy
sunhouy requested a review from a team as a code owner November 20, 2026 18:46
@sunhouy
sunhouy requested review from sjrl and removed request for a team November 20, 2026 18:46
@vercel

vercel Bot commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

@sunhouy is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

@github-actions github-actions Bot added topic:tests type:documentation Improvements on the docs labels Nov 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  haystack/core/pipeline
  pipeline.py
  haystack/dataclasses
  document.py
Project Total  

This report was generated by python-coverage-comment-action

@1AbdullahButt 1AbdullahButt left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I compared Document(content="x", meta=a).id == Document(content="x", meta=b).id on main and on this branch.

meta a meta b same ID on main same ID on this PR
{"t": [1, 2]} {"t": (1, 2)} yes no
{"t": {1, 2}} {"t": "{1, 2}"} yes no
{"t": {1, 2, 3}} {"t": {3, 2, 1}} yes yes
{"a": 1, "b": [1, "x"]} same keys, other order yes yes
{"t": (1, 2)} {"t": {"__tuple__": [1, 2]}} no yes
{"t": {1, 2}} {"t": {"__set__": [1, 2]}} no yes
{"t": {1, 2}} {"t": frozenset({1, 2})} no yes

The reported collisions are fixed. IDs for plain JSON metadata are unchanged (I checked one: 971eba39… on both). test/dataclasses/test_document.py passes (62 passed).

Things I found:

  1. New collisions with the marker keys. A tuple or set now hashes the same as a dict that uses the marker key, so the last three rows collide on this branch and did not on main. A real dict with a __tuple__ key is unlikely in metadata. But the PR is about type-based collisions, so it may be worth a marker that a plain JSON value cannot produce. Or the trade-off could be written down in the docstring. set vs frozenset is probably fine to treat as equal, but that is also a change from main.
  2. IDs change for existing documents with tuple or set metadata. For example, meta={"t": (1, 2)} gives 9ef432f5… on main and 22818c13… here. The release note says that IDs of JSON-only metadata are unchanged. It could also say that documents with tuple or set metadata get new IDs. Then users who re-index with DuplicatePolicy.SKIP would expect new copies instead of skips.

(I made this review with help from Claude Code. I ran the table above on main and on this branch, and pytest test/dataclasses/test_document.py on this branch. I did not run document store integration tests.)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

topic:tests type:documentation Improvements on the docs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants