🦟 SNT — Subnational Tailoring pipelines

🚧 Migration in progress. This repo is a fresh snapshot (no history) of four earlier repos; see Provenance. The text below still comes from snt_development and describes its old layout and deploy routes. Only the layout table and Provenance have been updated so far; the rest is rewritten in Phase 2 of docs/MIGRATION_PLAN.md. Live country workspaces still run from the old repos.

Data pipelines that turn a country's routine health data into the evidence base for subnational tailoring (SNT) of malaria interventions — deciding which interventions to deploy where, at a sub-national level.

They take monthly malaria case data from a country's DHIS2 health information system and combine it with rainfall, population, malaria-prevalence and survey data from external sources. Each pipeline produces one or more data layers: tables of indicators — incidence, reporting completeness, seasonality, quality of care — for every administrative unit in the country, plus the plots and HTML reports that go with them. Which administrative level that is varies by country; throughout this repository it is called ADM2, and what it means on the ground (district, health zone, commune…) depends on the country's own hierarchy.

The layers are the deliverable. They can be used as they are, or loaded into the SNT Explorer (docs), a separate tool where national malaria programmes explore the results and build intervention scenarios.

Everything here runs on OpenHEXA, Bluesquare's open-source data platform. This repository is the source for ~20 pipelines; OpenHEXA publishes them as a template that each country's workspace subscribes to.

---

🗺️ How it works, in one picture

flowchart TD
    DHIS2["DHIS2 API<br/>routine malaria data"] --> A
    A["A. snt_dhis2_extract<br/>download raw data"] --> B
    B["B. snt_dhis2_formatting<br/>raw into SNT schema<br/>the hinge of the system"]
    B --> C["C. clean and adjust<br/>outlier imputation x5<br/>population transformation"]
    B -->|ADM2 shapes| AP["A2. external sources<br/>ERA5 rainfall, WorldPop, MAP<br/>DHS surveys, healthcare access"]
    SRC["Copernicus, WorldPop<br/>Malaria Atlas Project, DHS"] --> AP
    B --> D
    C --> D["D. compute indicators<br/>incidence, reporting rate x2<br/>quality of care, seasonality x2"]
    AP --> D
    D --> OUT["data layers, one or more tables per pipeline<br/>SNT Explorer, exported tables, plots, reports"]

Loading

Each box is one or more standalone pipelines, launched by hand from the OpenHEXA UI. They pass data to each other through OpenHEXA datasets — never by calling each other directly. Stages C and A2 publish usable layers of their own, not just intermediates, and the Explorer reads those datasets directly. (A former stage E, snt_assemble_results, flattened everything into one table per ADM2; it is retired and kept under deprecated/ in snt_development.)

Some pipelines also create blank CSV templates in the workspace uploads/ folder, for an operator to fill in and upload to a later pipeline. snt_dhis2_formatting, for example, writes population templates with one row per ADM2. These files are not published to any dataset. See that pipeline's readme.md.

The full pipeline-by-pipeline inventory is in docs/DATA_ARCHITECTURE.md §1.1, where the stage letters above are used throughout (stage A2 here is written A′ there).

---

📂 What's in this repo

Path	What it is
pipelines/<name>/	One folder per pipeline, folder name = @pipeline id: pipeline.py, requirements.txt, readme.md (the pipeline's user-facing contract), and its R half in code/, reporting/, utils/. Kinds (SNT, support, bootstrap) are listed in pipelines/registry.json. No CI deploys anything yet.
shared/r/	Shared R library used across pipelines (snt_utils, snt_report, snt_palettes); was code/.
webapps/	The orchestrator (orchestrator/app/ is the deployable bundle) and the config editor (spec stage, app/ empty), each with its own docs/ and design/.
contracts/	Every cross-unit contract: the SNT_config.json skeleton, the SNT_metadata.json schema, the release manifest and status report schemas, the orchestrator's schemas. The live config lives in each workspace.
catalogs/	Universal references shipped with each release (SNT_metadata_master.json).
docs/	Docs that span several units: architecture, glossary, standards, ownership, backlog, decisions, the migration plan; docs/bootstrap/ for the three bootstrap pipelines.
reference/openhexa/	The OpenHEXA GraphQL schema (grep it, don't read it whole).
tests/, tools/	pytest suite and helper scripts (from the bootstrap repo).
dev/	Optional local Python tooling. Never runs on OpenHEXA.
.claude/	Guardrails for AI coding agents. Active on clone.

---

🧭 Start here

Pick the row that matches what you came for.

I want to…	Read
understand the system in 10 minutes	docs/ARCHITECTURE.md — the units, how they connect, how they reach a workspace; Part 2 explains OpenHEXA
know what a term means — N1, PfPR, CSB, ADM2, "Magic Glasses"	docs/GLOSSARY.md
run a pipeline, or know exactly what it produces	that pipeline's own readme.md
follow the data end to end	docs/DATA_ARCHITECTURE.md §3
read the epidemiological method	the R notebooks in pipelines/<name>/code/*.ipynb, with docs/GLOSSARY.md §4 open beside them
change any code	CLAUDE.md — start at the rule Register
write or update a pipeline readme.md	docs/PIPELINE_README_STANDARD.md
add or edit an SNT Explorer data layer in SNT_metadata.json	contracts/snt_metadata_json/README.md
know who to ask about a pipeline	docs/OWNERSHIP.md
set up locally	dev/README.md

---

⚠️ Non-obvious things

Facts that are not visible from the code and that newcomers reliably get wrong. Each is deliberate; none is a bug to fix.

1. These pipelines are not a DAG. Nothing schedules them; a person presses Run, one at a time, so a results table can mix data vintages. → §4
2. Merging here deploys nothing. Live countries still run the old snt_development repo, and Pull scripts = ON pulls from there; the routes from this repo are still being built. → CLAUDE.md rule 2
3. A country may not be running the notebook you are looking at. A <generic_name>_<CC>.ipynb variant runs instead, in that one country's workspace, and no mechanism propagates it. → pipelines/CLAUDE.md
4. OpenHEXA datasets are the contract, not the filesystem. A file not registered with add_files_to_dataset(...) is invisible downstream. → R4
5. Some pipelines deliberately overwrite each other — alternative methods, last run wins. → Pipeline traps
6. There is no CI, and tests cover only the bootstrap pipelines and contracts. Nothing runs ruff or pytest for you. → CLAUDE.md

---

🛠️ Working on this repo

🔑 Before you start

• Access to the relevant OpenHEXA workspace with the Admin role — ask your team lead. Everything real runs there: the data, the datasets, the R kernel.
• A local clone of this repo. Nothing else is required to read it.
• Optional but recommended: the local Python tooling (ruff, nbstripout) — see dev/README.md.

🗓️ Your first day

1. Read docs/ARCHITECTURE.md, then docs/DATA_ARCHITECTURE.md §1, then skim docs/GLOSSARY.md.
2. Read CLAUDE.md — the Register is the whole rulebook in one table. It applies to humans and AI agents alike.
3. Open the OpenHEXA workspace and run one pipeline end to end, reading its readme.md first.
4. Set up locally: conda env create -f dev/environment.yml.
5. Take a small ticket. Ask your pipeline's contact in docs/OWNERSHIP.md before changing anything in pipelines/*/code/.

🤖 Working with an AI coding assistant

Assumed, not merely tolerated — most of the documentation in this repo is written to be read by a person or an agent.

• CLAUDE.md is the rulebook, and it is not Claude-specific. Claude Code loads it automatically; with any other assistant (Cursor, Copilot, Gemini, …) paste it in or point the tool at it. The Register is all 26 repo-wide rules in one table — the cheapest thing to give a model that is about to touch this repo.
• .claude/ ships live guardrails. They are active the moment you clone: a hook that blocks destructive and history-rewriting git commands, plus a matching deny list. If an agent gets blocked, that is the guardrail working — don't route around it. → .claude/README.md
• .claude/skills/ holds three task skills — handover (runs the handover checklist, including a script that checks injected notebook parameters against their exists() fallbacks), write-pipeline-readme and add-explorer-layer (with the metadata validator). Claude Code offers them automatically; with another assistant, point it at the SKILL.md. The scripts are plain Python and work without any assistant.
• pipelines/CLAUDE.md holds the notebook-only rules. Claude Code loads it only when working under pipelines/; with another assistant, give it that file too when the task touches a notebook.
• docs/PIPELINE_README_STANDARD.md is deliberately tool-agnostic — it works as instructions for a human or as a prompt for any model.
• An agent's output still needs the domain review below. An assistant cannot tell you that an epidemiological number moved in the wrong direction, and neither can anything else in this repo. Nothing here changes who has to approve a change to the analytics.

🔀 How changes get in

Every change to main goes through a pull request, and every PR should be reviewed by a team member — for analytics changes, by the person listed for that pipeline in docs/OWNERSHIP.md. Core data-processing logic (the R notebooks and code/) is not something to change unreviewed, however small the diff looks: nothing downstream will tell you that a number moved.

Four things that are never OK:

• Committing country data. No .csv / .xlsx / .parquet / .rds / .geojson with real values. This is health data for real districts. .gitignore blocks most of it — never git add -f past it.
• Committing notebook outputs. Strip them first (nbstripout).
• Putting business logic in pipeline.py. It orchestrates; the analytics live in R.
• Publishing from a workspace other than snt-development. It creates a duplicate template nobody is subscribed to. → why

Before opening a PR, walk the handover checklist — it is short and it maps to the rules.

---

📦 Release management

Each GitHub release of this repo gets a release_manifest.json, built by .github/workflows/generate_manifest.yaml: every file the release ships, with its hash. The tools that deploy a release into a workspace and check it against one live in their own repo, BLSQ/snt_workspace_bootstrap.

The manifest's shape is a contract between the two repos, defined by release_manifest.schema.json there. Do not change the shape without updating that schema in the same change. → R22

---

🚧 Status and known limitations

Stated plainly, so nobody spends a day rediscovering them:

No automated tests	ruff is the only check, and no CI job runs it.
Dependencies are unpinned	Both entries in every requirements.txt install from a Git branch, so two deploys of identical code can produce different runtimes. → detail
No full offline development	R notebooks are edited locally but execute on the workspace kernel. → the loop
Run order is not written down yet	Tracked in docs/DATA_ARCHITECTURE.md §4.2.

Everything still open — tasks, questions, pending decisions — is in docs/BACKLOG.md. Why things are the way they are: docs/decisions/.

---

📜 Provenance

This repo starts as a clean snapshot without history (D-1 in docs/MIGRATION_PLAN.md), imported on 2026-11-05. Each tree was exported with git archive at the commit below and moved to this layout following the mapping table in the plan (Phase 1). The import commit holds the files as they were. The redactions listed below were applied before that commit, and every later change is its own commit. The source repos stay readable at these commits.

Source repo	Commit	Imported	Left out
BLSQ/snt_development	94cfc383c0f8804d2eba264f26a4bd3d372d94da	210 files	82: deprecated/ (48), .github/ (21), .claude/ (8, already here), 4 country notebooks (D-13), .gitignore
BLSQ/snt_webapp_orchestrator	603a831f8d996b99908819f27afa4a241dc941b9	43 files	6: .claude/ (2), .github/ (1), .gitignore, LICENSE, a research questionnaire with personal answers
BLSQ/snt_workspace_bootstrap	abbe8c0ebdbfdda481adbb0156d99f9ce04c5924	30 files	6: .claude/ (2), .gitignore, LICENSE, pyproject.toml and dev/environment.yml (merged into this repo's own)
BLSQ/openhexa-webapps-edit-snt-config-json	bd0cbf99bd426e7cce5a7f424636899966cd3ade	6 files	3: .gitignore, LICENSE, an older SNT_config.skeleton.json (D-11)

Changed before import. Every other imported file is byte-identical to its source.

• Redacted, because this repo is public:

  • 15 DHIS2 UIDs, replaced by REDACTEDUID;
  • 2 live country workspace slugs, replaced by xxx-snt-process;
  • 1 bucket name, replaced by snt-xxx-files.

  Affected files: contracts/snt_config/config_analysis_notes.md, contracts/snt_config/config_field_matrix.md, docs/MIGRATION_PLAN.md, pipelines/snt_dhis2_outliers_imputation_path/code/snt_dhis2_outliers_imputation_path.ipynb, pipelines/snt_dhis2_reporting_rate_dataset/code/snt_dhis2_reporting_rate_dataset.ipynb, pipelines/snt_metadata_json_trim/readme.md, webapps/config_editor/design/index.html, webapps/config_editor/docs/PRODUCT_SPEC.md, webapps/orchestrator/design/wireframes/orchestrator_wireframe_cockpit_narrative.html, webapps/orchestrator/docs/BACKLOG.md.

• Removed: the Niger-only helper apply_ner_manual_geometry_fixes() (facility UIDs and coordinates, no caller) from pipelines/snt_dhis2_extract/utils/snt_dhis2_extract_report.r.

• Stripped: a stray execution count from pipelines/snt_seasonality_rainfall/reporting/snt_seasonality_rainfall_report.ipynb.

Country-specific notebooks (*_<CC>.ipynb) are not imported (D-13). They remain in snt_development at the commit above, and in the country workspaces.

---

⚖️ Licence

MIT — provisional, pending confirmation by the team (BACKLOG DOC-5).

This repository contains code and documentation only. No country health data is stored here, and none ever should be.