Repository navigation
Conversation
|
Hi @kerkathy, would you have a chance to review this fix for #132? The PR adds a tab-separated execution manifest to preserve model revision parameters containing commas, while keeping |
|
Hi @haideraltahan, I compared #135, #143, and #162. My recommendation is to combine the model-argument parsing and revision-aware result collection from #135 with the separate TSV execution manifest from #162. That would address all three parts of #132 while keeping the existing flowchart TD
A["CLI models + revisions"] --> B["Parser from #135"]
B --> C["EvaluationJob objects"]
C --> D["jobs.csv for reporting"]
C --> E["jobs.tsv for Slurm from #162"]
E --> F["Run evaluations"]
F --> G["Revision-aware collection from #135"]
I'd keep the full CSV-to-TSV public-interface migration from #143 separate because it changes existing CLI options, output files, and historical-results workflows. We could first add regression tests for multiple revisions of the same model, mixed model lists, Slurm batching, and Happy to help integrate and test this if useful. What do you think? |
|
@haideraltahan could you answer @tire0011 or let us know if someone should take the discussion over? |
|
@geoalgo there is no response how do we proceed? |
Summary
Fixes the Slurm scheduling failure described in #132 when a Hugging Face model descriptor includes a comma, such as
openeurollm/prelude,revision=ter_0953312.jobs.csvfor reporting and result collection.jobs.tsvexecution manifest and use it in the generated Slurm script.awkso commas in model options do not shift task and shot columns.--eval_csv_pathwith a quoted model revision and checking the generated Slurm manifest.Verification
HF_HOME=/tmp/oellm-test-hf uv run --extra dev pytest tests/test_schedule_evals.py tests/test_task_batching.py tests/test_collect_results.py -q118 passed locally. No live Slurm cluster run was performed.
Note: This addresses the documented
--eval_csv_pathroute. The separate--modelscomma-splitting behavior is unchanged.