Skip to content

Exclude contaminated samples from JudgeArena's win-rate tasks - #174

Merged
haideraltahan merged 1 commit into
OpenEuroLLM:mainfrom
HannanJaved:hannan/exclude-judgearena
Nov 11, 2026
Merged

haideraltahan merged 1 commit into
OpenEuroLLM:mainfrom
HannanJaved:hannan/exclude-judgearena

Conversation

@HannanJaved

@HannanJaved HannanJaved commented Nov 7, 2026 •

Copy link
Copy Markdown
Member

collect --exclude_samples now reports clean scores for JudgeArena's AlpacaEval, arena-hard and m-ArenaHard.

How it works

<name>-annotations.csv holds each battle's question and results-<name>.json the judge's preference in it, in the same order. With swap_mode: both (arena-hard), every question is judged twice: all direct battles first, then the reversed ones in the same order.

Blocklist rows are matched to questions as follows (sample-exclusion.yaml):

  • AlpacaEval by instruction, since JudgeArena numbers its questions by position.
  • arena-hard and m-ArenaHard by question_id. arena-hard's uid is m-ArenaHard's question_id, and m-ArenaHard's ids are <question_id>-<language>, so one row excludes the question from arena-hard and from every m-ArenaHard language, including the -EU and all-language tasks.

collect then drops the excluded questions' battles, in every language and both orders, and recomputes:

  • the plain win rate, (wins + ties / 2) / judged battles, with its counts
  • AlpacaEval's raw win rate (the mean probability that the model wins)
  • arena-hard's weighted win rate, in which a decisive verdict counts three times

AlpacaEval's length-controlled win rate is a regression over all battles and arena-hard's confidence intervals are bootstrapped, so both are left out with a warning. The ELO tasks fit ratings over all battles at once and get no clean rows; collect logs that once.

How this was tested

  • Unit tests with fake JudgeArena runs: each judging order, a question excluded in every language, unparsed verdicts, the reproduce check, the decisive weight, ELO and unsupported metrics refused, and the skipped cases.

  • Every row of the AlpacaEval and m-ArenaHard blocklist files matches exactly one question of each mapped task, in every language, when the questions are loaded with JudgeArena's own loaders.

  • On LUMI, 20-question runs of alpaca-eval, arena-hard-v0.1, arena-hard-v2.0 and m-arena-hard-v0.1-pl (SmolLM2-135M-Instruct judged by gemma-4-31b-it) all read correctly and reproduce their reported scores. Excluding a question drops one battle, or two in arena-hard.

  • Added a line to CHANGELOG.md under ## [Unreleased] (not needed for CI, test or refactoring changes)

  • [ ] If this can change evaluation results (tasks, languages, shots, metrics, prompts, datasets, scoring), the line is under "Changed results" - this leaves existing scores unchanged, unless there was contamination

@haideraltahan
haideraltahan merged commit 06edbe2 into OpenEuroLLM:main Nov 11, 2026
3 checks passed
@HannanJaved
HannanJaved deleted the hannan/exclude-judgearena branch November 11, 2026 13:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants