Skip to content

fix: handle non-list relevant_statements in ContextRelevanceEvaluator - #13758

Merged
anakin87 merged 5 commits into
deepset-ai:mainfrom
zhenqiangZ:fix/context-relevance-invalid-statements
Nov 20, 2026
Merged

anakin87 merged 5 commits into
deepset-ai:mainfrom
zhenqiangZ:fix/context-relevance-invalid-statements

Conversation

@zhenqiangZ

Copy link
Copy Markdown
Contributor

Related Issues

Proposed Changes:

ContextRelevanceEvaluator._postprocess_results called len() directly on relevant_statements, so a structurally valid but malformed LLM response such as {"relevant_statements": null} raised TypeError after parsing and bypassed the raise_on_failure handling.

The value is now validated to be a list:

  • with raise_on_failure=False, the row is logged, marked "status": "error" with a NaN score, and excluded from the aggregate score, consistent with other failed rows;
  • with raise_on_failure=True, a ValueError is raised instead of a TypeError.

How did you test it?

  • Added unit tests test_run_invalid_statements_value_raise_on_failure_false and test_run_invalid_statements_value_raise_on_failure_true.
  • hatch run pytest test/components/evaluators/test_context_relevance_evaluator.py → 18 passed, 2 skipped (the skipped tests require an OpenAI API key).
  • Full evaluator suite: 194 passed, 4 skipped. The SAS evaluator was excluded locally because its optional sentence-transformers dependency is not installed.
  • hatch run pre-commit run --files ... passes on the changed files.

Notes for the reviewer

The validation is intentionally scoped to ContextRelevanceEvaluator. FaithfulnessEvaluator has a related output-shape validation gap that is tracked separately in #13423.

Checklist

A structurally valid LLM response such as `{"relevant_statements": null}`
previously raised TypeError in _postprocess_results, bypassing the
raise_on_failure handling. Validate the value type and route malformed
responses through the existing error-row path, or raise ValueError when
raise_on_failure is True.
@zhenqiangZ
zhenqiangZ requested a review from a team as a code owner November 15, 2026 01:10
@zhenqiangZ
zhenqiangZ requested review from anakin87 and removed request for a team November 15, 2026 01:10
@vercel

vercel Bot commented Nov 15, 2026

Copy link
Copy Markdown
Contributor

@zhenqiangZ is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

@CLAassistant

CLAassistant commented Nov 15, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@chrikrah chrikrah left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zhenqiangZ I'm approving. The guard also stops a string reply scoring 1.

non-blocking: worth a line in the release note.

# git archive f038192 (head), base 4b4346b, Python 3.12, haystack.__file__ from the export
# reply {"relevant_statements": "None of the statements are relevant."}, raise_on_failure=False
4b4346b   score 1     status evaluated
f038192   score nan   status error
$ python -m pytest -q test/components/evaluators/test_context_relevance_evaluator.py
18 passed, 2 skipped
# context_relevance.py from 4b4346b, tests kept
2 failed, 16 passed, 2 skipped    # both new raise_on_failure tests
$ ruff check haystack test && ruff format --check haystack test    # ruff 0.16.0
All checks passed!

@zhenqiangZ

Copy link
Copy Markdown
Contributor Author

Thanks for the independent verification, @chrikrah! Good catch on the string case: I updated the release note to call out that a plain string was previously scored as 1 and is now reported as an error (pushed in 7f5fa53).

@github-actions github-actions Bot added the type:documentation Improvements on the docs label Nov 20, 2026

@anakin87 anakin87 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you.

I pushed a minor simplification

@anakin87
anakin87 enabled auto-merge (squash) November 20, 2026 16:52
@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  haystack/components/evaluators
  context_relevance.py
  haystack/core/pipeline
  pipeline.py
Project Total  

This report was generated by python-coverage-comment-action

@anakin87
anakin87 merged commit 85b8d4e into deepset-ai:main Nov 20, 2026
24 of 25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

topic:tests type:documentation Improvements on the docs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ContextRelevanceEvaluator raises TypeError when relevant_statements is null

4 participants