Repository navigation
tests: compare save/load scores directly for pytest 9.2 - #70
Merged
Merged
Conversation
pytest 9.2 rejects nested data in pytest.approx, which broke this test. Compare labels and scores separately. This also checks the scores for real: the old fallback passed whenever the labels matched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
pytest 9.2.0 (released Oct 27) rejects nested data in
pytest.approx, so the weekly CI run started failing ontests/test_failure_modes.py::test_a_clean_save_loads_and_predicts_the_same:TypeError: pytest.approx() does not support nested data structuresThe test passed
[(label, score)]pairs topytest.approx, with a fallback that also passed when only the labels matched. This change compares labels and scores separately, so the test now also fails if scores change after a save and load.Checked locally on Python 3.12 with pytest 9.2.0: the old test fails, and the new one passes on 5 seeds along with the rest of
tests/test_failure_modes.py. Scores differ by about 6e-12 after a reload, well inside the test's 1e-5 tolerance, so the library itself is fine.