Skip to content

Exclude contaminated samples from more lm-eval benchmarks - #167

Merged
HannanJaved merged 7 commits into
OpenEuroLLM:mainfrom
HannanJaved:hannan/exclude-more-lm-eval
Nov 4, 2026
Merged

HannanJaved merged 7 commits into
OpenEuroLLM:mainfrom
HannanJaved:hannan/exclude-more-lm-eval

Conversation

@HannanJaved

Copy link
Copy Markdown
Member

This builds up on a previous PR (#159) and fixes #166.

The agenttrove-0.0 blocklist flags rows in 9 benchmarks that lm-eval has tasks for but #159 left out, because no oellm-eval task group ran them (GSM8K-Platinum, MATH, MathQA, MMLU-Redux, MMLU-Pro, TriviaQA, MBPP+, HumanEval and HumanEval+). This PR covers these missing benchmarks.

What has changed

  • sample-exclusion.yaml maps the 9 benchmarks. A flagged row is also dropped from its copy in the paired benchmark such as: HumanEval/HumanEval+, MBPP/MBPP+, GSM8K/GSM8K-Platinum, MMLU/MMLU-Redux and MATH-500/MATH.
  • lm-eval gets --confirm_run_unsafe_code and HF_ALLOW_CODE_EVAL=1, so mbpp and humaneval can run through oellm
  • collect recomputes humaneval's pass@1, which lm-eval reports under a different name from its metric function (pass_at_k).

How the changes were tested

  • Unit and container tests pass on LUMI. Every flagged row matches a document, and build_decontaminated_samples.py --check reproduces the bundled list.
  • A trial run using Qwen2.5-1.5B on MareNostrum 5, both methods, on humaneval, mbpp_plus, mathqa and one MATH and one MMLU-Redux subject. The two methods give equal scores, and every n_excluded is as expected.

Not covered

  • No task in any suite: DS1000, MT-MBPP, GSM-Symbolic, SuperGPQA are reported by the decontamination pipeline but not supported by oellm-eval
  • Benchmarks that are run only by evalchemy or JudgeArena, with no per-question results like HLE, MultiPL-E, LiveCodeBench, AlpacaEval and the arena benchmarks.

Checklist

  • Added a line to CHANGELOG.md under ## [Unreleased] (not needed for CI, test or refactoring changes)
  • If this can change evaluation results (tasks, languages, shots, metrics, prompts, datasets, scoring), the line is under "Changed results" (existing scores don't change; the decontaminated group runs more tasks, and mbpp/humaneval can now run)

@HannanJaved
HannanJaved requested a review from kargibora November 3, 2026 14:29
@HannanJaved
HannanJaved merged commit 8dea2d9 into OpenEuroLLM:main Nov 4, 2026
3 checks passed
@HannanJaved
HannanJaved deleted the hannan/exclude-more-lm-eval branch November 4, 2026 13:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Exclusion of contaminated samples from benchmarks

2 participants