Repository navigation
Exclude contaminated samples from JudgeArena's win-rate tasks - #174
Merged
haideraltahan merged 1 commit intoNov 11, 2026
Merged
Conversation
haideraltahan
approved these changes
Nov 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
collect --exclude_samplesnow reports clean scores for JudgeArena's AlpacaEval, arena-hard and m-ArenaHard.How it works
<name>-annotations.csvholds each battle's question andresults-<name>.jsonthe judge's preference in it, in the same order. Withswap_mode: both(arena-hard), every question is judged twice: all direct battles first, then the reversed ones in the same order.Blocklist rows are matched to questions as follows (
sample-exclusion.yaml):instruction, since JudgeArena numbers its questions by position.question_id. arena-hard's uid is m-ArenaHard'squestion_id, and m-ArenaHard's ids are<question_id>-<language>, so one row excludes the question from arena-hard and from every m-ArenaHard language, including the-EUand all-language tasks.collectthen drops the excluded questions' battles, in every language and both orders, and recomputes:AlpacaEval's length-controlled win rate is a regression over all battles and arena-hard's confidence intervals are bootstrapped, so both are left out with a warning. The ELO tasks fit ratings over all battles at once and get no clean rows;
collectlogs that once.How this was tested
Unit tests with fake JudgeArena runs: each judging order, a question excluded in every language, unparsed verdicts, the reproduce check, the decisive weight, ELO and unsupported metrics refused, and the skipped cases.
Every row of the AlpacaEval and m-ArenaHard blocklist files matches exactly one question of each mapped task, in every language, when the questions are loaded with JudgeArena's own loaders.
On LUMI, 20-question runs of alpaca-eval, arena-hard-v0.1, arena-hard-v2.0 and m-arena-hard-v0.1-pl (SmolLM2-135M-Instruct judged by gemma-4-31b-it) all read correctly and reproduce their reported scores. Excluding a question drops one battle, or two in arena-hard.
Added a line to
CHANGELOG.mdunder## [Unreleased](not needed for CI, test or refactoring changes)[ ] If this can change evaluation results (tasks, languages, shots, metrics, prompts, datasets, scoring), the line is under "Changed results"- this leaves existing scores unchanged, unless there was contamination