Skip to content

[MERGE WITH ELLIOT][CORE CHANGES][1/6] Add lmms-eval routing for image, video and audio models - #128

Open
islobozhan wants to merge 2 commits into
OpenEuroLLM:mainfrom
elliot-project:upstream-core-1-lmms-routing
Open

islobozhan wants to merge 2 commits into
OpenEuroLLM:mainfrom
elliot-project:upstream-core-1-lmms-routing

Conversation

@islobozhan

Copy link
Copy Markdown

Adds an lmms-eval suite to the job script, so image, video and audio models can be evaluated through lmms-eval.

This is MR a part of series smaller MRs (see a plan):
https://docs.google.com/document/d/1dEkXFeXm85SLEjKsnV18kGs54R7R0c650sYTJROXHZs/edit?tab=t.0

@islobozhan islobozhan changed the title [MERGE WITH ELLIOT][CORE CHANGES][1/5] Add lmms-eval routing for image, video and audio models [MERGE WITH ELLIOT][CORE CHANGES][1/6] Add lmms-eval routing for image, video and audio models Oct 5, 2026

@geoalgo geoalgo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, mostly have small comments.

Comment thread docs/VENV.md
1. Install lmms-eval from a clone at the pinned commit:
```bash
git clone https://github.com/EvolvingLMMs-Lab/lmms-eval.git
git -C lmms-eval checkout 45c766f60b6f8c153e4c72d06ca636e2db0ebcdb # v0.7.2

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need a clone? (In the case of harsh integration he was able to pin a commit and provide a requirements.txt)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lmms-eval wheel doesn’t ship the _default_template_yaml files that many task configs include (MMMU for example). So a regular pinned install or a requirements.txt can’t load those tasks and we need a clone.

Comment thread docs/VENV.md
uv pip install --python /path/to/.venv/bin/python -e lmms-eval
```

2. Run with a jobs CSV that sets `eval_suite` to `lmms-eval`:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we rather give the example with oellm-eval schedule? (We do that for all other backends)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that’s coming in the next MR. Schedule gets the suite from task-groups.yaml and there are no lmms-eval groups there yet. I will add the example once the groups are in.

esac
if [ -z "$LMMS_EVAL_MODEL" ]; then
# first match wins, so specific names come first
case "$(echo "$model_path" | tr '[:upper:]' '[:lower:]')" in

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you add a comment to explain why we need this? (eg to explain that VLM needs model class as opposed to other frameworks)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated

@islobozhan
islobozhan force-pushed the upstream-core-1-lmms-routing branch from 64d4697 to d641bfe Compare October 26, 2026 08:40
@islobozhan

Copy link
Copy Markdown
Author

@geoalgo Thanks for the review! Covered your questions and also fixed latest conflicts due to recent updates. Please review again when you have time

@islobozhan
islobozhan force-pushed the upstream-core-1-lmms-routing branch 3 times, most recently from 38b7d46 to 6e01d8f Compare October 29, 2026 14:03
@islobozhan
islobozhan force-pushed the upstream-core-1-lmms-routing branch from 6e01d8f to 766a48c Compare November 2, 2026 14:43
@islobozhan
islobozhan force-pushed the upstream-core-1-lmms-routing branch from 766a48c to 52adc8b Compare November 9, 2026 08:58
@joaquinvanschoren

Copy link
Copy Markdown

@islobozhan is this PR ready for re-review?

@islobozhan

islobozhan commented Nov 13, 2026 •

Copy link
Copy Markdown
Author

@joaquinvanschoren I commented that it is ready around 3 weeks ago

@joaquinvanschoren

Copy link
Copy Markdown

@geoalgo kind reminder: can this be merged? It seems that a re-review is needed.

@joaquinvanschoren

Copy link
Copy Markdown

@joaquinvanschoren I commented that it is ready around 3 weeks ago

I just requested David to re-review it. That's required according to the branch protection rules.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants