Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 24 additions & 13 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

This repository builds an AI-safety literature exploration system with two major parts:

1. A **Python pipeline** that harvests arXiv metadata, stores it in PostgreSQL, computes embeddings, filters papers, tags them against a fixed topic taxonomy (`taxonomy.py`) with an LLM, and computes a 2-D graph layout.
1. A **Python pipeline** that harvests arXiv metadata, stores it in PostgreSQL, computes embeddings, filters papers, tags them against a fixed topic taxonomy (`taxonomy.py`) with an LLM, derives peer-review status, computes a 2-D graph layout, and precomputes related papers.
2. A **React/Vite UI** that loads all data from a live FastAPI backend.

The active and only supported deployment mode is **API mode**: pipeline → PostgreSQL + pgvector → FastAPI → frontend. See top-level `CLAUDE.md`.
Expand All @@ -22,8 +22,10 @@ arXiv OAI-PMH
-> SPECTER2 embeddings (papers.embedding, vector(768))
-> stage-2 semantic keep / reject decisions (ai_stage2_keep)
-> LLM relevance + taxonomy classification (llm_relevant, llm_tags)
-> peer-review status (peer_reviewed, venue, venue_type) from arXiv fields, then OpenAlex
-> BGE topic embeddings (papers.embedding_topic, vector(768)) for relevant papers
-> graph coords (graph_x, graph_y on papers)
-> related papers (paper_related, SPECTER2 nearest neighbours)
-> FastAPI backend -> React frontend
```

Expand All @@ -41,10 +43,11 @@ aisafety-pipeline venue # peer-review status from arXiv journal-ref
aisafety-pipeline venue-enrich # fills unknowns from OpenAlex, rechecks recent ones monthly
aisafety-pipeline embed-topic # BGE topic embeddings (needed by compute-layout and semantic search)
aisafety-pipeline compute-layout # persists graph_x/y to DB
aisafety-pipeline compute-related # precomputes related papers into paper_related
aisafety-pipeline serve # start FastAPI
```

`aisafety-pipeline run-all` chains every stage above (except `serve`) in this order, stopping at the first failure; it is what the scheduled Railway cron job runs. All stages are incremental by default — they only process rows that are new or missing their output (see `src/aisafety_pipeline/ARCHITECTURE.md`).
`aisafety-pipeline run-all` chains every stage above (except `serve`) in this order, stopping at the first failure; it is what the scheduled Railway cron job runs. The exception is `venue-enrich`, whose OpenAlex errors never stop the run (a permanent one makes `run-all` exit non-zero after the last stage). All stages are incremental by default — they only process rows that are new or missing their output (see `src/aisafety_pipeline/ARCHITECTURE.md`).

---

Expand All @@ -57,10 +60,12 @@ Primary backend package. Owns:
- arXiv/OAI harvesting
- Persistence (PostgreSQL + pgvector via `PgConnection`)
- Embedding generation and storage
- Filtering (stage-1 regex + stage-2 semantic centroid / multi-centroid)
- Filtering (stage-1 regex + stage-2 semantic seed-centroid filter; `centroid-multi` is an experimental per-subtopic variant, not the default)
- LLM-based relevance + multi-label taxonomy classification (via OpenAI, sync or Batch API) — the live tag source
- Peer-review status and venue (`venue.py`): rules over arXiv journal-ref/DOI/comments, then OpenAlex lookups for papers left unknown
- Graph layout computation (incremental kNN placement by default; full UMAP/PCA refit with `--full`) and coordinate persistence to PostgreSQL
- Pipeline state in the `pipeline_state` table (harvest/stage-1 watermarks, in-flight LLM batches, latest-batch summary)
- Related-paper precomputation (`related.py`): top-20 SPECTER2 neighbours per relevant paper into `paper_related`
- Pipeline state in the `pipeline_state` table (harvest/stage-1/venue watermarks, in-flight LLM batches, related-papers input fingerprint, latest-batch summary)

See `src/aisafety_pipeline/ARCHITECTURE.md` for module-level details.

Expand All @@ -71,8 +76,9 @@ FastAPI backend. Owns:
- REST API serving live data from PostgreSQL
- Paginated paper listing with server-side filtering (keyword, date, multi-value tags and domain) and single paper detail
- Graph building on demand, by paper ids (`POST /api/graph/subset`) or by date range (`GET /api/graph/range`)
- On-demand related papers via pgvector HNSW nearest-neighbor over SPECTER2 embeddings (`GET /api/papers/related`)
- Related papers from the precomputed `paper_related` table, falling back to a live pgvector HNSW search over SPECTER2 embeddings for papers not yet precomputed (`GET /api/papers/related`)
- Semantic search via pgvector ANN over BGE topic embeddings (`POST /api/search`; disabled unless `ENABLE_SEMANTIC_SEARCH=true`)
- Author-name suggestions for the graph-recipe author filter (`GET /api/authors`)
- Tag catalog (`GET /api/tags`) and latest-batch / coverage metadata (`GET /api/batch/latest`, `GET /api/batch/coverage`)
- Per-IP rate limiting, cookieless search logging, and CORS for the Netlify frontend

Expand All @@ -86,10 +92,10 @@ Frontend application built with Vite/React. Owns:
- Desktop graph rendering: the last month of papers by default (`GET /api/graph/range`) or a saved graph (`POST /api/graph/subset`), with keyword/semantic search and ghost nodes
- Saved graphs (user-curated paper sets) kept in browser `localStorage`
- Server-side filtered, paginated paper browsing for the list view (`GET /api/papers`)
- On-demand related papers per paper selection (`GET /api/papers/related`)
- Related papers fetched per paper selection (`GET /api/papers/related`)
- Cookieless product analytics via Umami Cloud (`ui/src/lib/analytics.ts`), loaded only in production builds when `VITE_UMAMI_WEBSITE_ID` is set

(`MobileView` was removed; mobile now uses `StatsView`.) See `ui/ARCHITECTURE.md` for frontend details.
See `ui/ARCHITECTURE.md` for frontend details.

### `migrations/`

Expand All @@ -99,6 +105,8 @@ One-off SQL files, run manually (e.g. in the Supabase SQL editor). The schema it
- `add_search_log.sql`: creates the `search_log` analytics table for databases created before `init_db()` did so, and enables RLS on it.
- `drop_paper_tags.sql`: drops the legacy zero-shot `paper_tags` table.
- `drop_cluster_columns.sql`: drops the legacy `agg_cluster` / `hdbscan_cluster` columns from `papers`.
- `add_arxiv_venue_fields.sql`, `add_peer_review_columns.sql`, `add_openalex_checked_at.sql`: the peer-review columns on `papers_raw` (`journal_ref`, `doi`, `comments`) and `papers` (`peer_reviewed`, `venue`, …, `openalex_checked_at`).
- `add_recipe_trigram_indexes.sql`, `add_related_input_index.sql`, `add_relevant_published_index.sql`, `add_unclassified_index.sql`: indexes for existing databases, mirroring `init_db()`.

### `data/`

Expand Down Expand Up @@ -127,6 +135,7 @@ The public orchestration surface is defined in `src/aisafety_pipeline/utils.py`:
- `venue-enrich` — looks up papers `venue` left unknown in OpenAlex for a published version; rechecks recent ones monthly (`--limit`, default `OPENALEX_MAX_PER_RUN`)
- `embed-topic` — BGE topic embeddings (`papers.embedding_topic`) for LLM-relevant papers
- `compute-layout` — persists `graph_x/y` to DB (incremental kNN placement; `--full` refits UMAP/PCA)
- `compute-related` — precomputes SPECTER2 neighbours into `paper_related` (skipped when its input is unchanged; `--force` after re-embedding)
- `run-all` — every stage above in order, for the scheduled job (`--skip <stage>` to omit stages); records a latest-batch summary in `pipeline_state`
- `serve` — start FastAPI with uvicorn (`--host`, `--port`, `--reload`; defaults from `API_HOST`/`API_PORT`)

Expand All @@ -140,9 +149,10 @@ Ad-hoc tooling (backfills, eval-set builders, filter/classifier evaluation, moji

Tables:

- `papers_raw` — every harvested record (includes `harvested_at`, the stage-1 watermark column)
- `papers` — stage-1 survivors and all pipeline outputs: `embedding vector(768)` (SPECTER2), `embedding_topic vector(768)` (BGE), stage-2 columns, the LLM classification columns `llm_relevant`/`llm_tags`/etc., and `graph_x`/`graph_y`
- `pipeline_state` — key/JSON-value store: `harvest_watermark`, `stage1_watermark`, `llm_batches` (in-flight Batch API jobs), `latest_batch` (shown in the UI)
- `papers_raw` — every harvested record (includes `harvested_at`, the stage-1/venue watermark column, and arXiv's `journal_ref`/`doi`/`comments`)
- `papers` — stage-1 survivors and all pipeline outputs: `embedding vector(768)` (SPECTER2), `embedding_topic vector(768)` (BGE), stage-2 columns, the LLM classification columns `llm_relevant`/`llm_tags`/etc., the peer-review columns `peer_reviewed`/`venue`/`venue_type`/etc., and `graph_x`/`graph_y`
- `paper_related` — precomputed related papers (neighbour ids + sims per relevant paper), written by `compute-related`
- `pipeline_state` — key/JSON-value store: `harvest_watermark`, `stage1_watermark`, `venue_watermark`, `llm_batches` (in-flight Batch API jobs), `related_fingerprint` (compute-related's last input), `latest_batch` (shown in the UI)
- `search_log` — cookieless search analytics written by the API

Requires the `DATABASE_URL` env var to be set to a valid PostgreSQL DSN — `db.connect()` raises `RuntimeError` otherwise. Vector search uses two partial HNSW indexes (`vector_cosine_ops`), one on `embedding` and one on `embedding_topic`, both restricted to `llm_relevant` rows. Run `aisafety-pipeline init-db` (or `harvest`, which calls `init_db()` itself) against a fresh database to create tables/extensions; the other stages assume the schema exists.
Expand Down Expand Up @@ -188,7 +198,7 @@ AI should be careful around:

- PostgreSQL schema changes
- Paper id normalization
- Compact graph field names (`id`, `aid`, `t`, `au`, `pd`, `dm`, `ln`, `tags`)
- Compact graph field names (`id`, `aid`, `t`, `au`, `pd`, `dm`, `ln`, `tags`, `pr`, `vn`)
- pgvector operator syntax (`<=>`)

---
Expand All @@ -198,8 +208,9 @@ AI should be careful around:
1. **Identifier mismatch**: arXiv abs URL format must remain stable across pipeline and UI. The frontend assigns ephemeral numeric `id` values by index; `aid` (the arXiv URL) is the durable key.
2. **Layout staleness**: `graph_x`/`graph_y` only reflect the last `compute-layout` run — new papers added to `papers` after that (via later harvest/filter runs) have no coords until `compute-layout` runs again.
3. **API dependency in UI**: every view fetches from the API. With `VITE_API_URL` unset, `api.ts` falls back to same-origin relative paths, which only works behind a proxy — in practice it must be set.
4. **Two embedding spaces**: related papers and stage-2 filtering use SPECTER2 (`embedding`); semantic search and the layout use BGE (`embedding_topic`). Papers without `embedding_topic` (e.g. `embed-topic` not yet run) are absent from search and get no coordinates.
5. **Graph coordinate alignment**: the desktop graph places search/related "ghost" nodes by re-applying the loaded subset's normalisation to each paper's raw coords. This is a contract spanning `graph.py`/`papers.py` (which expose `meta.coords.bounds` and raw `rx`/`ry`) and `GraphView.tsx`'s `ghostCoord()` (which mirrors the canvas-mapping formula in `graph.py`'s `_graph_from_rows`). Changing the normalisation on either side without the other misplaces ghost nodes.
4. **Related-papers staleness**: `paper_related` only reflects the last `compute-related` run. Papers newer than that have no stored row and are served by the live HNSW fallback, while existing papers' stored neighbour lists won't include those newer papers until `compute-related` runs again.
5. **Two embedding spaces**: related papers and stage-2 filtering use SPECTER2 (`embedding`); semantic search and the layout use BGE (`embedding_topic`). Papers without `embedding_topic` (e.g. `embed-topic` not yet run) are absent from search and get no coordinates.
6. **Graph coordinate alignment**: the desktop graph places search/related "ghost" nodes by re-applying the loaded subset's normalisation to each paper's raw coords. This is a contract spanning `graph.py`/`papers.py` (which expose `meta.coords.bounds` and raw `rx`/`ry`) and `GraphView.tsx`'s `ghostCoord()` (which mirrors the canvas-mapping formula in `graph.py`'s `_graph_from_rows`). Changing the normalisation on either side without the other misplaces ghost nodes.

---

Expand Down
42 changes: 34 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# AI Safety Pipeline & Visualisation

A staged pipeline for harvesting **arXiv** papers → storing in **PostgreSQL + pgvector** → generating **SPECTER2 embeddings** → applying filters → LLM topic tagging → BGE topic embeddings + 2-D layout → serving live via **FastAPI**.
A staged pipeline for harvesting **arXiv** papers → storing in **PostgreSQL + pgvector** → generating **SPECTER2 embeddings** → applying filters → LLM topic tagging → peer-review detection (arXiv fields + **OpenAlex**) → BGE topic embeddings + 2-D layout → precomputed related papers → serving live via **FastAPI**.

[Live Web App](https://alignmentgraph.com/)

Expand All @@ -9,10 +9,12 @@ A staged pipeline for harvesting **arXiv** papers → storing in **PostgreSQL +
- 🔄 OAI-PMH harvest from arXiv (`cs`, `stat`, `econ`, `eess.SY`)
- 🗄️ Storage: **PostgreSQL + pgvector**
- 🧠 Embeddings via [SPECTER2](https://huggingface.co/allenai/specter2) (filtering, related papers) and [BGE](https://huggingface.co/BAAI/bge-base-en-v1.5) (semantic search, layout)
- 🧹 Two-stage filtering: regex + semantic centroid / multi-centroid
- 🧹 Two-stage filtering: regex + semantic seed-centroid filter (plus an experimental per-subtopic `centroid-multi` variant)
- 🏷️ LLM-based relevance + multi-label taxonomy tagging (OpenAI, sync or Batch API) — the live tag source served by the API
- 🎓 Peer-review status + venue, derived from arXiv journal-ref/DOI/comments and enriched via [OpenAlex](https://openalex.org)
- 🔗 Related papers precomputed from SPECTER2 nearest neighbours
- 🔍 Semantic search via pgvector ANN (self-hosted only; `ENABLE_SEMANTIC_SEARCH=true`)
- 🚀 FastAPI backend with live paper listing, detail, and semantic search endpoints
- 🚀 FastAPI backend with live paper listing/detail, graph, related papers, authors, tags and semantic search endpoints
- 🗺️ 2D graph layout coordinates (UMAP/PCA, then incremental kNN placement) precomputed and served live from Postgres

---
Expand Down Expand Up @@ -113,25 +115,49 @@ Only scores papers that have no score yet (`ai_sem_sim IS NULL`). Pass `--full`
aisafety-pipeline llm-classify-run
```

This submits, waits for, and collects OpenAI Batch API jobs until every unclassified paper has `llm_relevant`/`llm_tags` set — this is what the API serves.
This submits, waits for, and collects OpenAI Batch API jobs until every unclassified paper has `llm_relevant`/`llm_tags` set — this is what the API serves. (`llm-classify` runs synchronously instead; `llm-classify-submit` / `llm-classify-collect` split the batch flow into manual steps.)

**6. Generate BGE topic embeddings for relevant papers**
**6. Derive peer-review status from arXiv fields**

```bash
aisafety-pipeline venue
```

Sets `peer_reviewed`, `venue` and `venue_type` for `llm_relevant` papers from their arXiv journal-ref, publisher DOI and comments (e.g. "accepted at ICLR 2026"). No signal means *unknown* (`NULL`), not "not reviewed". Incremental; pass `--full` to re-derive everything after changing the rules in `venue.py`.

**7. Enrich unknowns from OpenAlex**

```bash
aisafety-pipeline venue-enrich
```

Looks up papers `venue` left unknown in OpenAlex, which links many arXiv preprints to their published journal/conference version. Papers never checked go first; unknown papers from the last ~18 months are rechecked monthly, since acceptance often comes months after posting. Capped at `--limit` (default `OPENALEX_MAX_PER_RUN`, 5000) per run. `OPENALEX_MAILTO` and `OPENALEX_API_KEY` are optional (see `.env.example`). After changing the venue-naming rules, run `venue-enrich --refresh-names` once.

**8. Generate BGE topic embeddings for relevant papers**

```bash
aisafety-pipeline embed-topic --device auto
```

Only embeds `llm_relevant` rows — run this *after* `llm-classify-run`, not alongside `embed`. These are what `compute-layout` and semantic search use; skipping this step is why relevant papers can be missing graph coordinates.

**7a. Compute graph layout**
**9. Compute graph layout**

```bash
aisafety-pipeline compute-layout --coords umap
```

By default this only places relevant papers that have no coordinates yet, each at the mean position of its nearest already-placed neighbours (`--knn-k`, default 10), so existing points don't move. Pass `--full` to refit the whole UMAP layout, e.g. monthly or after a large backfill, so new topics form their own clusters. With no layout at all yet it does a full fit automatically.

**7b. Start the API**
**10. Precompute related papers**

```bash
aisafety-pipeline compute-related
```

Stores the top 20 SPECTER2 neighbours of every relevant paper in `paper_related`, which `GET /api/papers/related` serves (falling back to a live pgvector search for papers not yet precomputed). Always a full recompute, but skipped entirely when the set of relevant embedded papers hasn't changed; pass `--force` after re-embedding papers in place.

**11. Start the API**

```bash
aisafety-pipeline serve --reload
Expand All @@ -147,7 +173,7 @@ API docs available at `http://localhost:8000/docs`.
aisafety-pipeline run-all
```

Chains harvest → stage1 → embed → filter → llm-classify-run → embed-topic → compute-layout in that fixed order, stopping immediately on the first stage's failure. This is what the scheduled Railway cron job (`Dockerfile.pipeline`) runs daily — see that file; the Railway service itself (Dockerfile path, cron schedule, restart policy) is configured in the Railway dashboard, not a config file. The harvest watermark and in-flight OpenAI batch tracking are stored in the `pipeline_state` table (not local files), so this is safe to run from a fresh container on every invocation. Pass `--skip <stage>` (repeatable) to omit a stage for manual recovery/debugging.
Chains harvest → stage1 → embed → filter → llm-classify (`llm-classify-run`) → venue → venue-enrich → embed-topic → compute-layout → compute-related in that fixed order, stopping immediately on the first stage's failure. The exception is `venue-enrich`: OpenAlex is optional enrichment, so OpenAlex request errors never stop the run, but a permanent one (e.g. a bad API key) makes `run-all` exit non-zero after the last stage. This is what the scheduled Railway cron job (`Dockerfile.pipeline`) runs daily — see that file; the Railway service itself (Dockerfile path, cron schedule, restart policy) is configured in the Railway dashboard, not a config file. The harvest watermark and in-flight OpenAI batch tracking are stored in the `pipeline_state` table (not local files), so this is safe to run from a fresh container on every invocation. Pass `--skip <stage>` (repeatable, using the stage names above, e.g. `--skip llm-classify`) to omit a stage for manual recovery/debugging.

### Configure the frontend

Expand Down
Loading