Repository navigation
Replies: 4 comments
|
Hello! I agree with you that currently it is not possible to specify the ID when using a converter (specifically As a workaround, you can store this information in If we see high demand for this feature, we may decide to implement it in the future. |
|
Hi Stephano, Thanks. I will try that. btw, the reason for my question is that I ran into an issue with the ids generated by Haystack itself.. I had about 4000 queries, and it generated about 25-30 duplicates after I ran the JSONConverter. I need to do some preprocessing of the Documents before creating the indexing pipeline (with the document store). The documentation does say that there could be duplicates, but I did not expect an issue to arise with such few queries. If there is an alternative solution (for my situation) to the workaround you've proposed, please let me know. Thanks for your help. |
|
Hi @sanjayc2, following up on your second question — those 25–30 duplicate IDs are actually expected behaviour once you know how Haystack generates IDs, and there are a couple of clean ways to avoid them. Why the duplicates happen When you don't pass an data = f"{text}{dataframe}{blob!r}{mime_type}{meta}{embedding}{sparse_embedding}"
return hashlib.sha256(data.encode("utf-8")).hexdigest()where One practical consequence: document stores usually upsert by ID, so colliding documents silently overwrite each other — that's why your stored count comes out lower than the number of records. How to fix it
from haystack.components.converters import JSONConverter
converter = JSONConverter(
jq_schema=".items[]",
content_key="text",
extra_meta_fields={"record_id"}, # a field that is unique per JSON record
)
from haystack import Document
docs = converter.run(sources=["data.json"])["documents"]
for doc in docs:
doc = Document(content=doc.content, meta=doc.meta, id=f"my-doc-{doc.meta['record_id']}")(or simply assign
As @anakin87 said, converters don't take IDs directly, so option 1 or 2 is the supported path today. Hope this helps, and please let me know what the colliding records look like if neither option clears it up! |
|
Hi @sanjayc2, just following up — did the explanation of how Haystack generates IDs, and the workarounds, help with the duplicates you saw? If it resolved things, marking the answer would help others hitting the same. Happy to clarify anything that didn't fit. |
Uh oh!
There was an error while loading. Please reload this page.
Hi,
I am trying to find a way to specify custom ids for Documents when they are created using a Converter. In particular, I am unable to do this for a JSON Converter. Neither the init nor the run methods take ids as input. Could someone please help?
Thanks
All reactions