Skip to content

Fix Slurm parsing of model revisions in evaluation jobs - #162

Open
tire0011 wants to merge 4 commits into
OpenEuroLLM:mainfrom
tire0011:fix/132-model-revision-csv
Open

tire0011 wants to merge 4 commits into
OpenEuroLLM:mainfrom
tire0011:fix/132-model-revision-csv

Conversation

@tire0011

@tire0011 tire0011 commented Nov 1, 2026

Copy link
Copy Markdown

Summary

Fixes the Slurm scheduling failure described in #132 when a Hugging Face model descriptor includes a comma, such as openeurollm/prelude,revision=ter_0953312.

  • Keep the existing jobs.csv for reporting and result collection.
  • Write a separate tab-separated jobs.tsv execution manifest and use it in the generated Slurm script.
  • Parse the execution manifest with tab-delimited awk so commas in model options do not shift task and shot columns.
  • Add a regression test exercising --eval_csv_path with a quoted model revision and checking the generated Slurm manifest.

Verification

HF_HOME=/tmp/oellm-test-hf uv run --extra dev pytest tests/test_schedule_evals.py tests/test_task_batching.py tests/test_collect_results.py -q

118 passed locally. No live Slurm cluster run was performed.

Note: This addresses the documented --eval_csv_path route. The separate --models comma-splitting behavior is unchanged.

tire0011 commented Nov 4, 2026

Copy link
Copy Markdown
Author

Hi @kerkathy, would you have a chance to review this fix for #132? The PR adds a tab-separated execution manifest to preserve model revision parameters containing commas, while keeping jobs.csv unchanged for reporting. The regression test and 118 relevant tests pass locally; a live Slurm run is still pending. I'd appreciate any feedback, especially on compatibility with the cluster workflow. Thanks!

@haideraltahan haideraltahan self-assigned this Nov 4, 2026
@haideraltahan

Copy link
Copy Markdown
Collaborator

Hi @tire0011,

Thank you for this PR!
It looks like this PR merges #135, #143. I'll check all these and try to push them into one

tire0011 commented Nov 7, 2026

Copy link
Copy Markdown
Author

Hi @haideraltahan,

I compared #135, #143, and #162. My recommendation is to combine the model-argument parsing and revision-aware result collection from #135 with the separate TSV execution manifest from #162.

That would address all three parts of #132 while keeping the existing jobs.csv reporting interface and compatibility with historical runs.

flowchart TD
    A["CLI models + revisions"] --> B["Parser from #135"]
    B --> C["EvaluationJob objects"]
    C --> D["jobs.csv for reporting"]
    C --> E["jobs.tsv for Slurm from #162"]
    E --> F["Run evaluations"]
    F --> G["Revision-aware collection from #135"]
Loading

I'd keep the full CSV-to-TSV public-interface migration from #143 separate because it changes existing CLI options, output files, and historical-results workflows. We could first add regression tests for multiple revisions of the same model, mixed model lists, Slurm batching, and collect --check compatibility.

Happy to help integrate and test this if useful. What do you think?

@geoalgo

geoalgo commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator

@haideraltahan could you answer @tire0011 or let us know if someone should take the discussion over?
Many thanks!

@tire0011

Copy link
Copy Markdown
Author

@geoalgo there is no response how do we proceed?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants