Skip to content

Add xquad-eu task group (XQuAD in de, el, en, es, ro, tr) - #175

Open
Devesh-Maheshwari wants to merge 2 commits into
OpenEuroLLM:mainfrom
Devesh-Maheshwari:add-xquad-eu
Open

Devesh-Maheshwari wants to merge 2 commits into
OpenEuroLLM:mainfrom
Devesh-Maheshwari:add-xquad-eu

Conversation

@Devesh-Maheshwari

@Devesh-Maheshwari Devesh-Maheshwari commented Nov 11, 2026 •

Copy link
Copy Markdown

This adds XQuAD as an xquad-eu task group. XQuAD is the 1190 SQuAD v1.1 dev questions, professionally translated into 11 languages, and six of those are OpenEuroLLM target languages (de, el, en, es, ro, tr). I couldn't find any short-answer reading comprehension in the multilingual groups yet: Belebele is multiple choice, and squadv2 and coqa are English only.

What's in it

  • The xquad-eu group, using lm-eval's built-in xquad_{lang} tasks through a valid_langs template, 2-shot, scored with f1.
  • Six xquad.<lang> entries in _LANG_ALIAS, like the x-csqa-* ones. The HF configs are called xquad.de and so on, and without these test_templated_tasks_all_resolve_to_a_language fails.
  • A small test, a README line and a CHANGELOG entry.

Why 2 shots

Before picking the shot count I ran it on two OpenEuroLLM checkpoints: 240 questions per language (one per paragraph, the same rows in every language via --samples), with the same lm_eval flags as template.sbatch except device and batch size. F1 × 100:

Model Shots en de es el ro tr Mean
datamix-flag-top_level-1.7b-option_5-100bt 0 31.5 21.0 21.8 7.1 4.5 4.8 15.1
2 69.5 47.0 51.3 21.9 37.0 19.0 40.9
5 66.2 48.0 55.2 26.0 40.8 23.2 43.2
datamix-2b-80-20 0 41.4 32.2 29.1 20.0 32.3 27.6 30.4
2 69.8 53.6 56.2 47.4 58.5 46.0 55.2

What I took from it:

  • 0-shot doesn't work well for base models. They answer with a whole sentence (median 17 words for the 1.7B, while the gold answers are 2), and F1 punishes that. With two examples they answer in two words.
  • 5 shots is only about 2 points better than 2 overall (bootstrap 95% CI +0.5 to +4.1), and per language that's within the noise. Rerunning 5-shot with a different few-shot seed moved single languages by up to 2.5 points.
  • 5 shots doesn't fit in 2048 tokens. lm-eval cuts prompts to max_length - 256 from the left, so on datamix-2b-80-20 231 of the 240 Greek prompts and 83 Romanian ones lose some or all of their examples. At 2 shots it's 12 Greek and 2 Romanian. Nothing gets cut on the 1.7B, which has 4096.
  • It does pick up differences. The 2B is far ahead of the 1.7B in Greek and Turkish, and at 5 shots the 1.7B goes from 23.3 at iter_0012000 to 28.4 at iter_0024000 and 43.2 at main.

If you'd rather have 5 shots to match belebele-eu-5-shot, it's a one-line change.

Testing

  • pytest tests/ --ignore=tests/integration/: 745 passed, 40 skipped. ruff format and ruff check pass, and ty and yamllint give the same output as on main.
  • oellm-eval schedule --models sshleifer/tiny-gpt2 --task_groups xquad-eu --local true --limit 2, then collect, gives the six xquad_* rows with f1 at 2 shots.

Good to know

  • lm-eval's xquad utils don't import on transformers 5.20+ (the reason for the transformers<5.20 pin in containers/lumi.def). LUMI is fine, but a fresh venv needs transformers<5.20 until Fix xquad, coqa, record and other QA tasks on transformers 5.20 (squad_metrics was removed) EleutherAI/lm-evaluation-harness#4440 is merged. I ran the numbers above with that fix on transformers 5.22, and it doesn't change any scores.

  • normalize_answer only strips English articles, so it's better to compare models or checkpoints within one language than languages against each other.

  • 817 of the 1190 English questions are also in squadv2. I kept en anyway as the reference for the other five.

  • Added a line to CHANGELOG.md under ## [Unreleased] (not needed for CI, test or refactoring changes)

  • If this can change evaluation results (tasks, languages, shots, metrics, prompts, datasets, scoring), the line is under "Changed results" (it's a new group, so no existing results change)

XQuAD (google/xquad) is the 1190 SQuAD v1.1 dev questions translated into
11 languages, and six of them are OpenEuroLLM target languages: German,
Greek, English, Spanish, Romanian and Turkish. The group runs lm-eval's
built-in xquad_{lang} tasks at 2 shots and reports F1.

The HF configs are named xquad.<lang>, so they get entries in _LANG_ALIAS,
the same way the X-CSQA configs do.
@Devesh-Maheshwari

Copy link
Copy Markdown
Author

@haideraltahan you reviewed the XNLI and PAWS-X groups, so I thought you might be the right person for this one. No rush, and I'm happy to rerun it on other checkpoints if that helps.

@Devesh-Maheshwari

Copy link
Copy Markdown
Author

Hi @haideraltahan, following up on this PR from last week. Would you have time to review it, or could you point me to the right reviewer?

The latest CI run shows action_required with no jobs started. Could a maintainer check whether it needs workflow approval?

Happy to address feedback or run additional evaluations. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant