Repository navigation
Add xquad-eu task group (XQuAD in de, el, en, es, ro, tr) - #175
Open
Devesh-Maheshwari wants to merge 2 commits into
Open
Devesh-Maheshwari wants to merge 2 commits into
Devesh-Maheshwari wants to merge 2 commits into
Conversation
XQuAD (google/xquad) is the 1190 SQuAD v1.1 dev questions translated into
11 languages, and six of them are OpenEuroLLM target languages: German,
Greek, English, Spanish, Romanian and Turkish. The group runs lm-eval's
built-in xquad_{lang} tasks at 2 shots and reports F1.
The HF configs are named xquad.<lang>, so they get entries in _LANG_ALIAS,
the same way the X-CSQA configs do.
Author
|
@haideraltahan you reviewed the XNLI and PAWS-X groups, so I thought you might be the right person for this one. No rush, and I'm happy to rerun it on other checkpoints if that helps. |
Author
|
Hi @haideraltahan, following up on this PR from last week. Would you have time to review it, or could you point me to the right reviewer? The latest CI run shows Happy to address feedback or run additional evaluations. Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds XQuAD as an
xquad-eutask group. XQuAD is the 1190 SQuAD v1.1 dev questions, professionally translated into 11 languages, and six of those are OpenEuroLLM target languages (de, el, en, es, ro, tr). I couldn't find any short-answer reading comprehension in the multilingual groups yet: Belebele is multiple choice, andsquadv2andcoqaare English only.What's in it
xquad-eugroup, using lm-eval's built-inxquad_{lang}tasks through avalid_langstemplate, 2-shot, scored withf1.xquad.<lang>entries in_LANG_ALIAS, like thex-csqa-*ones. The HF configs are calledxquad.deand so on, and without thesetest_templated_tasks_all_resolve_to_a_languagefails.Why 2 shots
Before picking the shot count I ran it on two OpenEuroLLM checkpoints: 240 questions per language (one per paragraph, the same rows in every language via
--samples), with the same lm_eval flags astemplate.sbatchexcept device and batch size. F1 × 100:datamix-flag-top_level-1.7b-option_5-100btdatamix-2b-80-20What I took from it:
max_length - 256from the left, so ondatamix-2b-80-20231 of the 240 Greek prompts and 83 Romanian ones lose some or all of their examples. At 2 shots it's 12 Greek and 2 Romanian. Nothing gets cut on the 1.7B, which has 4096.iter_0012000to 28.4 atiter_0024000and 43.2 atmain.If you'd rather have 5 shots to match
belebele-eu-5-shot, it's a one-line change.Testing
pytest tests/ --ignore=tests/integration/: 745 passed, 40 skipped.ruff formatandruff checkpass, andtyandyamllintgive the same output as on main.oellm-eval schedule --models sshleifer/tiny-gpt2 --task_groups xquad-eu --local true --limit 2, thencollect, gives the sixxquad_*rows withf1at 2 shots.Good to know
lm-eval's xquad utils don't import on transformers 5.20+ (the reason for the
transformers<5.20pin incontainers/lumi.def). LUMI is fine, but a fresh venv needstransformers<5.20until Fix xquad, coqa, record and other QA tasks on transformers 5.20 (squad_metrics was removed) EleutherAI/lm-evaluation-harness#4440 is merged. I ran the numbers above with that fix on transformers 5.22, and it doesn't change any scores.normalize_answeronly strips English articles, so it's better to compare models or checkpoints within one language than languages against each other.817 of the 1190 English questions are also in
squadv2. I keptenanyway as the reference for the other five.Added a line to
CHANGELOG.mdunder## [Unreleased](not needed for CI, test or refactoring changes)If this can change evaluation results (tasks, languages, shots, metrics, prompts, datasets, scoring), the line is under "Changed results" (it's a new group, so no existing results change)