Repository navigation
Conversation
A structurally valid LLM response such as `{"relevant_statements": null}`
previously raised TypeError in _postprocess_results, bypassing the
raise_on_failure handling. Validate the value type and route malformed
responses through the existing error-row path, or raise ValueError when
raise_on_failure is True.
Contributor
|
@zhenqiangZ is attempting to deploy a commit to the deepset Team on Vercel. A member of the Team first needs to authorize it. |
chrikrah
reviewed
Nov 16, 2026
chrikrah
left a comment
Contributor
There was a problem hiding this comment.
@zhenqiangZ I'm approving. The guard also stops a string reply scoring 1.
non-blocking: worth a line in the release note.
# git archive f038192 (head), base 4b4346b, Python 3.12, haystack.__file__ from the export
# reply {"relevant_statements": "None of the statements are relevant."}, raise_on_failure=False
4b4346b score 1 status evaluated
f038192 score nan status error
$ python -m pytest -q test/components/evaluators/test_context_relevance_evaluator.py
18 passed, 2 skipped
# context_relevance.py from 4b4346b, tests kept
2 failed, 16 passed, 2 skipped # both new raise_on_failure tests
$ ruff check haystack test && ruff format --check haystack test # ruff 0.16.0
All checks passed!
Contributor
Author
anakin87
approved these changes
Nov 20, 2026
anakin87
left a comment
Member
There was a problem hiding this comment.
Thank you.
I pushed a minor simplification
anakin87
enabled auto-merge (squash)
November 20, 2026 16:52
Contributor
Coverage reportClick to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||||||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issues
Proposed Changes:
ContextRelevanceEvaluator._postprocess_resultscalledlen()directly onrelevant_statements, so a structurally valid but malformed LLM response such as{"relevant_statements": null}raisedTypeErrorafter parsing and bypassed theraise_on_failurehandling.The value is now validated to be a list:
raise_on_failure=False, the row is logged, marked"status": "error"with aNaNscore, and excluded from the aggregate score, consistent with other failed rows;raise_on_failure=True, aValueErroris raised instead of aTypeError.How did you test it?
test_run_invalid_statements_value_raise_on_failure_falseandtest_run_invalid_statements_value_raise_on_failure_true.hatch run pytest test/components/evaluators/test_context_relevance_evaluator.py→ 18 passed, 2 skipped (the skipped tests require an OpenAI API key).sentence-transformersdependency is not installed.hatch run pre-commit run --files ...passes on the changed files.Notes for the reviewer
The validation is intentionally scoped to
ContextRelevanceEvaluator.FaithfulnessEvaluatorhas a related output-shape validation gap that is tracked separately in #13423.Checklist
fix:and added!in case the PR includes breaking changes.