Repository navigation
[MERGE WITH ELLIOT][CORE CHANGES][1/6] Add lmms-eval routing for image, video and audio models - #128
islobozhan wants to merge 2 commits into
Conversation
geoalgo
left a comment
There was a problem hiding this comment.
LGTM, mostly have small comments.
| 1. Install lmms-eval from a clone at the pinned commit: | ||
| ```bash | ||
| git clone https://github.com/EvolvingLMMs-Lab/lmms-eval.git | ||
| git -C lmms-eval checkout 45c766f60b6f8c153e4c72d06ca636e2db0ebcdb # v0.7.2 |
There was a problem hiding this comment.
why do we need a clone? (In the case of harsh integration he was able to pin a commit and provide a requirements.txt)
There was a problem hiding this comment.
lmms-eval wheel doesn’t ship the _default_template_yaml files that many task configs include (MMMU for example). So a regular pinned install or a requirements.txt can’t load those tasks and we need a clone.
| uv pip install --python /path/to/.venv/bin/python -e lmms-eval | ||
| ``` | ||
|
|
||
| 2. Run with a jobs CSV that sets `eval_suite` to `lmms-eval`: |
There was a problem hiding this comment.
Can we rather give the example with oellm-eval schedule? (We do that for all other backends)
There was a problem hiding this comment.
Yes, that’s coming in the next MR. Schedule gets the suite from task-groups.yaml and there are no lmms-eval groups there yet. I will add the example once the groups are in.
| esac | ||
| if [ -z "$LMMS_EVAL_MODEL" ]; then | ||
| # first match wins, so specific names come first | ||
| case "$(echo "$model_path" | tr '[:upper:]' '[:lower:]')" in |
There was a problem hiding this comment.
could you add a comment to explain why we need this? (eg to explain that VLM needs model class as opposed to other frameworks)
64d4697 to
d641bfe
Compare
|
@geoalgo Thanks for the review! Covered your questions and also fixed latest conflicts due to recent updates. Please review again when you have time |
38b7d46 to
6e01d8f
Compare
6e01d8f to
766a48c
Compare
766a48c to
52adc8b
Compare
|
@islobozhan is this PR ready for re-review? |
|
@joaquinvanschoren I commented that it is ready around 3 weeks ago |
|
@geoalgo kind reminder: can this be merged? It seems that a re-review is needed. |
I just requested David to re-review it. That's required according to the branch protection rules. |
Adds an lmms-eval suite to the job script, so image, video and audio models can be evaluated through lmms-eval.
This is MR a part of series smaller MRs (see a plan):
https://docs.google.com/document/d/1dEkXFeXm85SLEjKsnV18kGs54R7R0c650sYTJROXHZs/edit?tab=t.0