Skip to content

Latest commit

 

History

History
202 lines (141 loc) · 12.5 KB

models.md

File metadata and controls

202 lines (141 loc) · 12.5 KB

Models and backends

Vera always parses files, stores indexes (.vera/), runs BM25 keyword search, and applies code-aware ranking locally on your machine. Your backend choice only determines where embeddings (and optional reranking) run. For keyword, regex, and structural search with no model at all, build a keyword-only index.

Which backend to use

Most users should start with one of the first two rows below:

Recommended setup Backend Command Tradeoffs
Offline, zero API cost, any machine Potion Code on CPU (default) vera setup --potion-code Fast CPU indexing, ~32 MB one-time download, full local privacy, no GPU or ONNX Runtime needed.
Highest retrieval quality and agent context savings API mode (qwen/qwen3-embedding-8b + qwen/qwen3-reranker-8b) vera setup --api Highest measured nDCG@10 (0.8647 vs 0.8538 on the 320-task slice) and -48% prompt tokens in the Flask agent benchmark. Requires network access, provider billing, and sending chunk text and queries to configured endpoints.
Local neural embeddings on a GPU Jina ONNX on GPU vera setup --onnx-jina-cuda (or ROCm, CoreML, DirectML, OpenVINO) Runs Jina v5 nano, CodeRankEmbed, or custom ONNX models locally on supported accelerators.

Potion Code (minishlab/potion-code-16M-v2) is a static code embedding model from MinishLab. It runs directly on any supported CPU and works offline once cached.

API mode accepts any OpenAI-compatible /embeddings endpoint plus optional generic or Voyage-style /rerank endpoints. The built-in Qwen preset configures qwen/qwen3-embedding-8b and qwen/qwen3-reranker-8b through OpenRouter with a single API key, or you can point the embedding and reranker URLs at separate providers (for example, OpenRouter for embeddings and SiliconFlow for lower-cost reranking, or a local llama.cpp server). On the 320-task Semble subset, the Qwen pair scored 0.8647 nDCG@10 against 0.8538 for local Potion Code, lifting Recall@1 by three percentage points, and in the 4-arm Flask agent evaluation it cut total prompt tokens by 48% (see benchmarks).

Local backends download their assets into Vera's data directory (see Installation). Jina ONNX backends also download the matching ONNX Runtime library into lib/, and some GPU providers need drivers or Python packages; see Troubleshooting.

Local embedding models

Model Setup command Notes
Potion Code vera setup --potion-code Default. CPU only, static embeddings.
Jina embeddings v5 nano retrieval vera setup --onnx-jina-cuda or another --onnx-jina-* flag Opt-in, intended for GPUs. Vera prefixes queries with Query: and indexed text with Document:, as the model expects.
CodeRankEmbed vera setup --onnx-jina-cuda --code-rank-embed Optional preset on the Jina ONNX backend.

The --onnx-jina-* flags select the execution provider: cpu, cuda (NVIDIA), rocm (AMD, Linux), directml (Windows), coreml (Apple Silicon), and openvino (Intel, Linux).

Embedding screening

In historical model screening across Semble tasks and an independent repository set, potion-code-16M-v2, jina-embeddings-v5 nano, and CodeRankEmbed landed within 0.013 nDCG@10 of each other. Potion Code indexed fastest and runs on every machine, which is why it is the default. Detailed measurements are in the benchmark history.

Indexing speed

Reference points, each on a different corpus:

Backend Hardware Corpus Index time
Potion Code AMD Ryzen 7 9800X3D, CPU Flask, 227 files, 1,873 chunks 0.3 s
Potion Code AMD Ryzen 7 9800X3D, CPU Vera repository, 447 files, 8,109 chunks 0.9 s
Potion Code AMD Ryzen 7 9800X3D, CPU Semble suite, 63 repositories 89 s
Jina ONNX, CUDA NVIDIA RTX 4080 commons-lang, 11,429 chunks about 16 s
Jina ONNX, CPU AMD Ryzen 5 7600X3D, 6 cores Vera repository at the time, 239 files, about 3,100 chunks about 6 min

The two small Potion Code rows were measured with v2.1.3. Jina ONNX on CPU works but is slow; on a machine without a GPU, use Potion Code. After the first index, vera update . re-embeds only changed files on every backend.

On Apple Silicon, the CoreML backend treats half of system RAM as available GPU memory. Like the other GPU backends, it embeds at most 16 inputs per batch.

Reranking

A reranker rescores the top candidates after retrieval. It is off by default for local backends, and API setup turns it on when you configure an API reranker. To turn it on yourself:

vera config set retrieval.reranking_enabled true

With reranking on, Vera uses the API reranker if one is configured. Otherwise local backends download and run jinaai/jina-reranker-v2-base-multilingual through ONNX Runtime (on CPU for Potion Code).

In local mode you can leave it off. In the 2026-08-23 screening, no tested local cross-encoder beat the default ranking without a reranker (nDCG@10, 320-task subset / independent set):

Reranker nDCG@10
None (default) 0.8540 / 0.7644
mxbai-rerank-xsmall-v1 0.8497 / 0.7564
jina-reranker-v2-base-multilingual 0.8473 / 0.7557
gte-reranker-modernbert-base 0.8472 / 0.7525

The Qwen API reranker was measured separately and is part of the 0.8647 result above.

If you want a local reranker anyway, mxbai-rerank-xsmall-v1 came closest to the baseline. Select it with these environment variables:

export LOCAL_RERANKER_REPO=mixedbread-ai/mxbai-rerank-xsmall-v1
export LOCAL_RERANKER_REVISION=b5c6e9da73abc3711f593f705371cdbe9e0fe422
export LOCAL_RERANKER_ONNX_FILE=onnx/model_quantized.onnx
export LOCAL_RERANKER_TOKENIZER_FILE=tokenizer.json

Custom ONNX cross-encoders on Hugging Face work if they follow the expected contract: inputs named input_ids and attention_mask, with the first output returning a 1D or 2D float32 score tensor ([batch], or [batch, n] with the score in the first column). If you leave LOCAL_RERANKER_REVISION unset, Vera defaults to the main branch for custom repositories, while the built-in model keeps its pinned revision. A pinned revision is stored under models/<repo>/revisions/<revision>/, so different revisions do not overwrite each other.

API mode

Run the interactive setup:

vera setup --api

vera backend --api does the same without the agent and indexing steps. The menu offers these presets:

Preset Embedding model Reranker Provider
Qwen (recommended) qwen/qwen3-embedding-8b qwen/qwen3-reranker-8b OpenRouter
OpenAI text-embedding-3-small none built in OpenAI
Jina AI jina-embeddings-v3 jina-reranker-v2-base-multilingual Jina AI
Voyage AI voyage-code-3 rerank-2 Voyage AI
Custom any OpenAI-compatible endpoint optional your provider

Check pricing, quotas, and data retention policies with your chosen provider. Because embedding and reranker endpoints are configured independently, you can also split them across providers (for instance, using OpenRouter for qwen/qwen3-embedding-8b and SiliconFlow or another /rerank host for Qwen/Qwen3-Reranker-8B).

For a reranker, setup also asks for the wire protocol (auto, generic, or voyage), an optional endpoint path, and an optional task instruction. The Qwen preset sets the generic protocol for you and skips these questions. Switching to Qwen clears endpoint and task overrides left over from another provider.

When you configure a reranker, setup turns reranking on; setup without a reranker turns it off. Switching to a local backend later drops the saved API reranker. If one was saved, reranking turns off with it, unless your shell sets all three RERANKER_MODEL_* variables; then reranking stays on and uses that reranker. If you turned reranking on for a local backend and no API reranker was saved, it stays on.

Non-interactive setup

Export the endpoint values, then run setup with --yes. The three reranker variables must be all set or all unset.

export EMBEDDING_MODEL_BASE_URL=https://your-embedding-api/v1
export EMBEDDING_MODEL_ID=your-embedding-model
export EMBEDDING_MODEL_API_KEY=your-api-key

# Optional reranker
export RERANKER_MODEL_BASE_URL=https://your-reranker-api/v1
export RERANKER_MODEL_ID=your-reranker-model
export RERANKER_MODEL_API_KEY=your-api-key

vera setup --api --yes

# Optional reranker protocol settings
vera config set retrieval.reranker_protocol generic
vera config set retrieval.reranker_endpoint_path "/rerank"
vera config set retrieval.reranker_task_instruction "Given a query, retrieve relevant code"

For Qwen through OpenRouter:

export EMBEDDING_MODEL_BASE_URL=https://openrouter.ai/api/v1
export EMBEDDING_MODEL_ID=qwen/qwen3-embedding-8b
export EMBEDDING_MODEL_API_KEY=your-openrouter-key
export RERANKER_MODEL_BASE_URL=https://openrouter.ai/api/v1
export RERANKER_MODEL_ID=qwen/qwen3-reranker-8b
export RERANKER_MODEL_API_KEY=your-openrouter-key

vera setup --api --yes
vera config set retrieval.reranker_protocol generic

To run your own OpenAI-compatible endpoints locally, see llama.cpp setup.

What leaves your machine

API mode sends queries and chunk text to the configured endpoints. The index, BM25 search, and code-aware ranking stay local. If an API reranker is configured, it receives the query and the text of the top candidates. The daily update check is described in Installation.

Query and document prefixes

Some models expect a prefix on queries or documents. In API mode, Vera picks a prefix from the model ID for known models (Qwen3 embeddings, for example). Override it with EMBEDDING_QUERY_PREFIX and EMBEDDING_DOCUMENT_PREFIX, or the embedding.query_prefix and embedding.document_prefix config keys; see Configuration.

Custom local embedding models

The Jina ONNX backend can load another ONNX embedding model. These flags do not affect Potion Code or API mode.

From a Hugging Face repository (a full https://huggingface.co/... URL also works):

vera setup --onnx-jina-cuda \
  --embedding-repo Zenabius/CodeRankEmbed-onnx \
  --embedding-pooling cls \
  --embedding-no-onnx-data \
  --embedding-query-prefix "Represent this query for searching relevant code:"

From a directory you downloaded or exported yourself:

vera setup --onnx-jina-cuda \
  --embedding-dir /path/to/model-dir \
  --embedding-onnx-file onnx/model_quantized.onnx \
  --embedding-tokenizer-file tokenizer.json \
  --embedding-dim 768
Flag Meaning
--code-rank-embed Use the built-in CodeRankEmbed preset
--embedding-repo <repo-or-url> Download the model from Hugging Face
--embedding-dir <dir> Load the model from a local directory
--embedding-onnx-file <path> ONNX file inside the repository or directory (default onnx/model_quantized.onnx)
--embedding-onnx-data-file <path> ONNX external data file (default onnx/model_quantized.onnx_data)
--embedding-no-onnx-data The model has no external data file
--embedding-tokenizer-file <path> Tokenizer file (default tokenizer.json)
--embedding-dim <n> Embedding dimension the model returns
--embedding-pooling <mode> mean, cls, or last-token
--embedding-max-length <n> Tokenizer truncation length
--embedding-query-prefix <text> Prefix added to queries
--embedding-document-prefix <text> Prefix added to indexed text

Changing models

The index records which embedding model built it, including every --embedding-* setting, so changing the model, pooling, or a prefix requires a rebuild with vera index . --replace.

If two model names produce identical embeddings (the same model behind two endpoints, for example), declare them as aliases to skip the rebuild. Set embedding.model_aliases or export VERA_EMBEDDING_MODEL_ALIASES: groups separated by semicolons, names within a group separated by commas, at least two names per group.

export VERA_EMBEDDING_MODEL_ALIASES="jina-embeddings-v3,jina-v3;text-embedding-3-small,openai-small"

If your network blocks downloads from the command line, see Manual install.