Repository navigation
Tool usage using locally served LLMs #8675
Replies: 1 comment
|
Yes, a local server can be used, but serving ordinary OpenAI-compatible chat completions does not by itself establish tool-calling compatibility. For the current Haystack API,
Your older 2.x notebook needs its tool-execution pipeline connections retained when changing the generator endpoint. On current main's 3.x API, A raw response plus the exact checkpoint, SGLang and Haystack versions would locate which side is failing. I have checked the Haystack request/execution paths, not run your local model, so I cannot confirm that specific checkpoint/server combination. |
Uh oh!
There was an error while loading. Please reload this page.
I'm trying to follow though this notebook for using tools with Haystack: https://colab.research.google.com/github/deepset-ai/haystack-cookbook/blob/main/notebooks/tools_support.ipynb
I'm using a locally served LLM, Mistral 7B (served using SGLang) instead of OpenAI models. The pipelines work but the tools are not getting invoked as expected.
This may be a naive question but was wondering if I can get this example working with a locally served LLM?
Any inputs will be very helpful! Thanks a lot in advance!
All reactions