Skip to content

About

Hyper3-CLIP: Hierarchy-Conditioned Hyperbolic Vision-Language Training

Resources

Stars

11 stars

Watchers

0 watching

Forks

Latest commit

 

History

9 Commits

Folders and files

Repository files navigation

hyper³labs logo

hyper³labs

Hyper3-CLIP

Paper · Paper model · Hyper3-CLIP V1 · Website · Discord

Hyper3-CLIP is a hyperbolic version of CLIP. It combines query-conditioned visual pooling with hyperbolic hierarchical representation learning.

Research code for the ECCV 2026 Beyond Euclidean workshop paper, Hyper3-CLIP: Hierarchy-Conditioned Hyperbolic Vision-Language Training.

How it works

To train the model, we construct lightweight query hierarchies from text, comprising full captions, sentence fragments, localized part descriptions, and extracted phrases. Each query conditions the pooling of visual patches, and the resulting representations support image-text, whole-part, and parent-child entailment losses.

Hyper3-CLIP overview: caption queries attend to image patches and form visual and text nodes in hyperbolic space.

Query-conditioned pooling is active only during training; at inference, Hyper3-CLIP is a standard dual encoder, and the pooling module adds no inference-time computation.

Paper results

All downstream results are zero-shot evaluations of frozen checkpoints. The main results from paper Table 2 are shown below; expand the remaining tables for classification, ablations and prompt comparisons.

Image–text retrieval

Hyper3-CLIP improves over UNCHA across all eight considered COCO/Flickr retrieval settings.

Paper Table 2, ViT-B/16. Published baseline values are from UNCHA Table 2; Hyper3-CLIP is evaluated with the same retrieval and hierarchy metrics. Each pair reports R@5 / R@10. Higher is better.

Image → text

Model COCO Flickr30K
CLIP 71.4 / 81.5 93.6 / 96.9
MERU 72.3 / 82.0 93.5 / 96.2
ATMG 62.9 / 74.0 85.1 / 92.2
HyCoCLIP 72.0 / 82.0 92.6 / 95.4
UNCHA 72.7 / 82.7 91.4 / 95.9
Hyper3-CLIP 75.7 / 84.3 95.4 / 97.6

Text → image

Model COCO Flickr30K
CLIP 57.4 / 68.5 83.5 / 89.9
MERU 57.4 / 68.6 84.0 / 90.0
ATMG 51.2 / 62.6 78.0 / 85.3
HyCoCLIP 58.4 / 69.3 84.9 / 90.3
UNCHA 60.0 / 71.0 84.9 / 91.2
Hyper3-CLIP 63.0 / 73.2 86.4 / 91.4

ImageNet hierarchy

Paper Table 2, ViT-B/16. Higher is better except TIE and LCA.

Model TIE ↓ LCA ↓ J ↑ H-P ↑ H-R ↑
CLIP 3.60 2.21 0.79 0.85 0.85
MERU 3.63 2.22 0.78 0.85 0.85
ATMG 4.19 2.48 0.75 0.83 0.83
HyCoCLIP 3.17 2.05 0.81 0.87 0.87
UNCHA 2.94 1.96 0.83 0.88 0.88
Hyper3-CLIP 3.16 2.08 0.82 0.87 0.88
Zero-shot classification (Table 3)

Paper Table 3. Values are mean-per-class accuracy. All models use ViT-B/16. Food-101, CUB, and Flowers, marked with *, use the common Photo prompt, a photo of a {}.; the remaining datasets use the baseline evaluation prompts.

Dataset HyCoCLIP UNCHA PHyCLIP Hyper3-CLIP
ImageNet 45.80 48.80 44.31 46.98
CIFAR-10 88.80 90.40 89.33 91.14
CIFAR-100 60.10 63.20 59.05 65.64
SUN397 57.20 57.70 55.32 59.39
Caltech-101 81.30 83.90 76.35 83.42
STL-10 95.00 95.70 94.84 96.56
Food-101* 61.14 64.10 62.40 67.31
CUB* 17.76 18.74 17.97 19.36
Cars 11.60 14.00 10.89 16.12
Aircraft 3.70 3.80 3.24 4.33
Pets 56.80 57.10 54.18 62.57
Flowers* 28.79 30.75 28.27 39.66
DTD 29.40 30.30 25.50 29.41
EuroSAT 35.80 41.30 36.29 28.52
RESISC45 45.60 52.70 48.22 48.94
Country211 6.50 6.10 5.56 6.06
Average 45.33 47.41 44.48 47.84
Multi-label classification (Table 4)

Under a shared evaluator, Hyper3-CLIP improves over UNCHA by 2.69 mAP on VOC and 3.35 mAP on COCO.

Paper Table 4, ViT-B/16. Values are mAP; Avg. is the mean of VOC and COCO. Higher is better.

Model VOC COCO Avg.
CLIP 68.55 44.89 56.72
MERU 71.29 45.33 58.31
HyCoCLIP 74.89 50.73 62.81
UNCHA 75.51 50.93 63.22
PHyCLIP 73.29 49.64 61.47
Hyper3-CLIP 78.20 54.28 66.24
Query-conditioned objective ablation (Table 5)

To isolate their effects, each variant is first trained with only the base objective for 80k steps, then continues training to 100k steps with the query entailment terms specified in the table. Data, batch size, optimizer schedule, and seed are fixed; the control rows keep all query entailment terms enabled and modify only the named control.

Variant Image-text Visual hierarchy Text hierarchy AP ΔAP AUROC ΔR@10
No query losses – – – 76.21 0.00 98.34 0.00
Query image-text only ✓ – – 75.34 -0.87 98.24 -0.46
Visual hierarchy only – ✓ – 71.85 -4.36 98.01 -0.77
Text hierarchy only – – ✓ 78.86 +2.65 98.44 -0.09
Full objective ✓ ✓ ✓ 79.85 +3.64 98.55 +0.06
Mean-pooled visual nodes ✓ ✓ ✓ 75.97 -0.25 98.32 +0.15
Reversed visual order (query → image) ✓ ✓ ✓ 76.06 -0.15 98.29 -0.09
Shuffled text parents ✓ ✓ ✓ 77.22 +1.00 98.40 -0.93

We report Average Precision (AP) and AUROC over 4,000 positive and 100,000 negative pairs from 1,000 images. Avg. R@10 is the mean of the four COCO/Flickr text- and image-retrieval R@10 values. ΔAP and ΔR@10 are measured relative to the no-query baseline.

Objective components and Table 5.

Prompt comparison (Table 6)

The 16-dataset average changes only the three prompt-sensitive columns; all other zero-shot columns are held fixed. Values are mean-per-class accuracy.

Food-101 / CUB / Flowers average

Model Official Photo Δ
HyCoCLIP 33.09 35.90 +2.81
UNCHA 33.90 37.86 +3.96
Hyper3-CLIP 3.26 42.11 +38.85

16-dataset average

Model Official Photo Δ
HyCoCLIP 44.82 45.33 +0.51
UNCHA 46.69 47.41 +0.72
Hyper3-CLIP 40.55 47.84 +7.29

Prompt templates and Table 6.

Quick start

Install the research code with Python 3.10+:

git clone https://github.com/Hyper3Labs/hyper3-clip.git
cd hyper3-clip
uv sync --extra dev
uv run python scripts/evaluate.py --help

The released training implementation has not yet reproduced the paper results. The paper model is available as a raw PyTorch checkpoint; direct Hub loading through this repository's Hyper3CLIP.from_pretrained requires a compatible safetensors export.

Authors and citation

Matin Mahmood, Antonio Rueda-Toicen, Mohamed ElBassat, Seifeldin Elkerdany, Weixing Wang and Gerard de Melo.

@inproceedings{mahmood2026hyper3clip,
  title     = {Hyper3-CLIP: Hierarchy-Conditioned Hyperbolic Vision-Language Training},
  author    = {Mahmood, Matin and Rueda-Toicen, Antonio and ElBassat, Mohamed and Elkerdany, Seifeldin and Wang, Weixing and de Melo, Gerard},
  booktitle = {Proceedings of the ECCV 2026 Workshop on Beyond Euclidean: Hyperbolic Deep Learning for Computer Vision},
  year      = {2026},
  eprint    = {2608.29313},
  archivePrefix = {arXiv},
  primaryClass = {cs.CV}
}

Code: MIT. Paper model: OpenMDW-1.0.

About

Hyper3-CLIP: Hierarchy-Conditioned Hyperbolic Vision-Language Training

Resources

Stars

11 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages