██████╗ ██████╗ ███████╗███╗ ██╗███████╗███████╗ █████╗ ██████╗ ██████╗██╗ ██╗
██╔═══██╗██╔══██╗██╔════╝████╗ ██║██╔════╝██╔════╝██╔══██╗██╔══██╗██╔════╝██║ ██║
██║ ██║██████╔╝█████╗ ██╔██╗ ██║███████╗█████╗ ███████║██████╔╝██║ ███████║
██║ ██║██╔═══╝ ██╔══╝ ██║╚██╗██║╚════██║██╔══╝ ██╔══██║██╔══██╗██║ ██╔══██║
╚██████╔╝██║ ███████╗██║ ╚████║███████║███████╗██║ ██║██║ ██║╚██████╗██║ ██║
╚═════╝ ╚═╝ ╚══════╝╚═╝ ╚═══╝╚══════╝╚══════╝╚═╝ ╚═╝╚═╝ ╚═╝ ╚═════╝╚═╝ ╚═╝
A privacy-first, self-hostable search engine built from scratch — no tracking, no profiling, no compromise.
OpenSearch is a completely independent, open-source web search engine engineered with a single principle: your search queries belong to you, not to a corporation.
Unlike every major search engine in existence, OpenSearch:
- Runs its own crawler — no data purchased from third parties
- Maintains its own inverted index — no calls to Google, Bing, or Elastic under the hood
- Stores zero user data — no session IDs, no cookies, no query logs
- Can be self-hosted by anyone on a free tier with no paid cloud dependencies
- Exposes a live hybrid search — merging its local BM25-ranked index with real-time web results for maximum relevance
This is not a proxy. This is not a meta-search engine wrapper. This is a fully functional search engine, built ground-up in TypeScript with a polite crawler, custom indexer, ranking engine, REST API, and a terminal-aesthetic web frontend.
Note
Product Identity & Non-Affiliation Notice:
OpenSearch Web is an independent, sovereign open-source search engine project. It is not affiliated with, funded by, or endorsed by Amazon Web Services, Inc. (AWS) or the Amazon OpenSearch project. All trademarks belong to their respective owners, and no claim of legal trademark clearance is made. It is compatible with the OpenSearch 1.1 browser description standard. See docs/product-identity-and-naming.md for full architectural naming details.
OpenSearch v1.5.0 delivers a major evolution across the search engine lifecycle, hardening privacy guarantees, incorporating graph-based authority scoring, enhancing developer tooling, and introducing zero-secret container deployments:
- 🛡️ Cryptographic Privacy Receipts & Local-Only Mode: Run completely offline (
LOCAL_ONLY=true) or toggle on-demand via the Web UI. Every search response includes an auditable Privacy Receipt documenting data-flow minimization and zero query logging. - 🌐 Link Graph & Power-Iteration PageRank: Real graph-based link analysis computed over crawler link adjacency matrices (
$d=0.85$ ,$L_1$ convergence$\le 10^{-6}$ ), integrated with BM25 and freshness into a multi-tier ranker with full score explainability (?explain=true). - 🎯 Tracker-Aware Ranking & Focus Lenses: Automatic score penalties for domains with heavy surveillance trackers (via DuckDuckGo Tracker Radar), plus curated domain lenses (
Developer,Academic,News,Privacy). - ⚡ Deterministic Extractive Quick Answers: Highlights cited passages extracted directly from top matching documents without LLM hallucinations, accompanied by a modular instant-answer toolkit (weather, currency FX, crypto, unit conversion, LaTeX math).
- 🔌 OpenAPI 3.1, Browser Search & MCP Integration: Interactive Swagger UI documentation (
/docs), OpenSearch 1.1 browser discovery (/opensearch.xml), and native Model Context Protocol (MCP) server support for AI agents. - 🐳 Alpine Dockerization & Volume Durability: Multi-stage
node:20-alpinecontainer running under unprivilegedappuser:1001with Docker Compose, automated schema migration snapshots, and rollback runbooks. - 🚀 Automated CI/CD Quality Gates: Least-privilege GitHub Actions workflows, frozen reproducible dependency builds, and 100% pass rate across 115 test suites and 1,339 tests.
┌────────────────────────────────────────────────────────────────────────┐
│ OPENSEARCH // INDEX MAP │
├──────────────────────────┬─────────────────────────┬───────────────────┤
│ 01. CORE & CAPABILITIES │ 02. SPECS & BENCHMARKS │ 03. RUNTIME & OPS│
│ ────────────────────── │ ───────────────────── │ ──────────────── │
│ ▸ Features │ ▸ Architecture │ ▸ Getting Started│
│ ▸ How It Works │ ▸ Competitive Analysis │ ▸ Running Stack │
│ ▸ Seed Corpus [113] │ ▸ Tech Stack │ ▸ Dev Scripts │
│ ▸ Privacy Guarantee │ ▸ Project Roadmap │ ▸ Configuration │
└──────────────────────────┴─────────────────────────┴───────────────────┘
| Sector | Module | Quick Jump | Focus |
|---|---|---|---|
| 01 // Core Engine | feat:v1.5.0 |
✨ What's New in v1.5.0 | Privacy receipts, PageRank, Focus Lenses, MCP & Docker |
feat:capabilities |
⚡ System Features | BM25 ranking, zero cookies, LRU query caching & live fallback | |
flow:lifecycle |
🧬 How It Works | Query parsing, candidate retrieval & multi-tier hybrid scoring | |
data:corpus |
🌱 Seed Corpus (113) | 19 curated domain categories across tech, AI, news & reference | |
sec:zero-log |
🛡️ Privacy Guarantee | Strict data minimization, zero tracking & IP anonymization | |
| 02 // Deep Dive | arch:boundaries |
🏛️ Architecture Tree | Modular monorepo (apps/*, packages/*) & strict boundaries |
cmp:matrix |
⚔️ Competitive Matrix | Side-by-side comparison with Google, DuckDuckGo, Brave & SearXNG | |
env:stack |
🛠️ Technology Stack | Native TypeScript 5.7+, Node.js 20+, Vitest & zero-dep UI | |
status:v1.5.0 |
🏁 Architecture Verification | Production verification ledger (115 suites, 1,339 tests passing) | |
| 03 // Operations | ops:bootstrap |
🚀 Getting Started | Repository clone, workspace installation & index seeding |
ops:runtime |
💻 Running the Stack | Dual-daemon launch (api:3001 + web:3000) & curl healthchecks |
|
ops:tooling |
📜 Development Scripts | Monorepo build, linting, formatting & automated test runners | |
ops:cicd |
🔄 CI/CD Automation | Multi-node matrix test, verification gate & artifact delivery | |
cfg:environment |
⚙️ Configuration (.env) | Ports, index storage paths, rate limits & logging levels | |
collab:git |
🤝 Contributing Guide | Contribution workflow, PR guidelines & code standards | |
legal:mit |
⚖️ Open Source License | MIT permissive license details & commercial usage rights | |
comm:conduct |
📜 Code of Conduct | Community guidelines, pledge, standards & enforcement |
┌────────────────────────────────────────────────────────────────────────────────┐
│ CORE ENGINE CAPABILITIES │
├──────────────────────────────────────┬─────────────────────────────────────────┤
│ [✓] 00. PRIVACY : ZERO-TELEMETRY │ [✓] 01. ENGINE : HYBRID BM25 + WEB │
│ [✓] 02. CRAWLER : POLITE RFC 3986 │ [✓] 03. RANKING: MULTI-FIELD WEIGHTED │
│ [✓] 04. SPEED : < 10MS LRU CACHE │ [✓] 05. SEC : SSRF GUARD & CSP │
│ [✓] 06. FRONTEND: CLI TUI EMULATOR │ [✓] 07. TESTS : 360 VITEST SUITES │
└──────────────────────────────────────┴─────────────────────────────────────────┘
Zero telemetry. Zero tracking. Mathematically minimized footprints.
$ status --privacy
[ok] identity_isolation : true (zero user accounts, zero profiles)
[ok] session_tracking : disabled (zero cookies, zero localStorage tokens)
[ok] query_persistence : ephemeral (zero search logs stored on disk)
[ok] ip_masking : active (IPv4 truncated to /24, IPv6 to /48)
[ok] outbound_leaks : blocked (rel="noopener noreferrer" + strict referrer policy)
[ok] third_party_deps : none (zero external CDNs, fonts, or tracking beacons)- Air-Gapped Telemetry: Searches are processed purely in-memory and discarded post-response.
- Auditable Security: Full guarantees and architecture codified in
docs/PRIVACY.md.
Best of both worlds: local curated sovereign indexing backed by live multi-engine web fallback.
graph LR
Q[User Query] --> L{Local BM25 Index}
L -->|Found >= Limit| R[Rank & Format Results]
L -->|Tail Deficit| F[Live Web Engine]
F --> G[Google Suggest API]
F --> D[DuckDuckGo Organic Scraper]
F --> A[DDG Instant Answers]
G & D & A --> M[De-duplicate & Merge]
M --> R
- Sovereign Local Index: Instantaneous retrieval across your locally indexed corpus with multi-field scoring.
- Zero-Dead-End Fallback: Automatically activates real-time web scrapers when local results are scarce:
- Google Suggest & Knowledge Engine: Fetches direct domain destinations without tracking redirects.
- DuckDuckGo Organic Scraper: Harvests actual web destinations (GitHub, Wikipedia, official docs, news).
- Instant Answers API: Inline topic abstracts and direct encyclopedic definitions.
- Intelligent Fusion: Curated local documents always receive tier-one rank priority, while verified web results seamlessly enrich the tail.
Safe, polite web extraction built strictly according to web standards.
| Guardrail | Mechanism | Specification |
|---|---|---|
| Robots Compliance | RobotsEvaluator |
Respects User-agent directives, path rules & Crawl-delay |
| SSRF Shield | DnsResolverGuard |
Rejects private CIDRs (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) at DNS level |
| Loop Prevention | RedirectTracker |
Circular redirect detection with hard hop limit (max 5 hops) |
| Resource Quota | StreamLimiter |
Streaming body size cap (maxPageBytes: 2MB), MIME-type filtering |
| Seed Corpus | 113 Seeds |
Pre-calibrated over 19 categories (Tech, AI, Docs, News, Finance, Open Source) |
No opaque black-box neural ranking. Pure explainable BM25 lexical relevance with structural boosts.
-
Core Parameters:
$k_1 = 1.2$ ,$b = 0.75$ with document length normalization. -
Structural Field Boosts:
-
Title: 3.0× weight multiplier -
Headings (H1-H3): 2.0× weight multiplier -
Meta Description: 1.5× weight multiplier -
Visible Body: 1.0× baseline weight
-
-
Contextual Modifiers:
- 🎯 Quoted Phrase Match: 1.5× boost for exact sequence matches (e.g.
"distributed systems") - 🧩 Conjunctive Multiplier: 1.25× multiplier when a document contains all query tokens
- 🌐 URL Slug / Hostname Bonus: +0.35 lexical affinity boost for domain/path matches
- 🛡️ Clustering Dampener: 0.8× dampener on consecutive results from the same root domain
- 🎯 Quoted Phrase Match: 1.5× boost for exact sequence matches (e.g.
Engineered for instant typing feedback and deterministic memory bounds.
Query Cache Telemetry (Hit Ratio: ~94% on warm traffic)
┌────────────────────────────────────────────────────────┐
│ WARM CACHE LATENCY : 4ms - 8ms │
│ COLD SEARCH LATENCY : 18ms - 35ms │
│ MEMORY RESIDENCY : Bounded LRU (500 entries) │
│ CLIENT CONCURRENCY : AbortController Typist Debounce │
└────────────────────────────────────────────────────────┘
- Sub-10ms Warm Hits: In-memory bounded LRU cache with transparent
X-Cache: HIT / MISSresponse headers. - Typing Cancellation: Frontend queries use
AbortControllerto cancel in-flight HTTP requests during rapid keypresses. - Telemetry Probing: Live memory, cache hits, and process metrics exposed at
/health.
Defensive security applied at every network and parsing boundary.
- 🔒 Defensive Headers:
Content-Security-Policy,X-Frame-Options: DENY,X-Content-Type-Options: nosniff,Referrer-Policy: no-referrer. - ⏱️ Rate Limiter: In-memory sliding-window request throttling per IP with
Retry-Afterheaders. - 🛑 Strict Boundaries: Max payload capped at 64KB (
413 Payload Too Large), search queries bounded to 200 chars. - 🛡️ Zero Stack Leaks: Internal exceptions and database errors are redacted before reaching client responses.
A distraction-free, keyboard-first search terminal with multi-theme support.
- 🟢 5 Retro CRT Themes: Phosphor Matrix Green (
#00ff66), VT100 Amber Monochrome, Synthwave '84 Neon, Solarized Cyber Dark, and Nordic Ice Frost. - ⌨️ Vim Keyboard Navigation:
j/knavigate down and up through search results with active focus indicatorsEnter/oopen highlighted search result in current/new browser tabr/vtoggle Reader Mode and Markdown snapshotsccopy result URL directly to clipboard/focus search input prompt;?opens the keyboard shortcut cheat sheet
- 📁 Search Workspaces: Built-in mode filters (
/devfor GitHub/RFCs/MDN,/newsfor tech feeds,/papersfor arXiv/academic preprints).
Instant runtime answers for engineers, mathematicians, and sysadmins.
- 🔐 Cryptographic Toolkit (Phase 56): Instant UUID v4 generation, strong password generators, Base64/Hex/URL encoders, JWT claims decoding, SHA-256/SHA-512/MD5 hashing.
- 🌐 Network & Cron Diagnostics (Phase 57): Full CIDR IPv4 subnet calculators (
192.168.1.0/24), 5-field cron explainer with next execution timestamps, and IANA port reference lookups. - 🛡️ Zero-Trace Privacy & Security Auditor (Phase 58 & 59): Domain privacy grading (A+ to F), TLS/HSTS/CSP checks, and zero-trace distraction-free Markdown reader mode.
- 💱 Fiat FX, Crypto & Scientific Scale Converter (Phase 60): Real-time fiat exchange rates (USD, EUR, GBP, JPY, INR, CAD, AUD, CHF), crypto rates (BTC, ETH, SOL) with satoshi/gwei, and digital storage (IEC 1024).
- 📐 Symbolic Math & LaTeX Renderer (Phase 61): Linear algebraic equation solver (
solve 3x + 12 = 45), calculus symbolic derivatives ($d/dx\ x^n$ ), and pure CSS/SVG LaTeX formula rendering without heavy external libraries.
Search anywhere, even in airplane mode.
- ⚡ Offline Service Worker Cache: Caches all instant answer engines, unit converters, timezone tools, and cheatsheets for instant zero-latency offline execution.
- 📦 Web App Manifest: Full standalone desktop and mobile installation with custom phosphor theme colors and app icons.
- 💾 Local Inverted Index Export: Export and import compressed
.opensearchbinary inverted index files.
Production confidence backed by a rigorous 65-phase test harness.
- 521+ Automated Tests: 100% pass rate across 50 Vitest test suites.
- Zero External Runtime Dependencies: Preserved across all packages.
- Sub-10ms P99 Latency SLA: Instant answer evaluation in < 1ms.
╔═══════════════════════════════╗
║ CLIENT SEARCH QUERY ║
╚═══════════════════════════════╝
│
▼
┌────────────────────────────────────────────────────────────────────────────────────────┐
│ STAGE 01 // INGESTION & BOUNDARY SANITIZATION │
│ • Max length guard (200 chars) • Script/HTML tag strip • IPv4/IPv6 client masking │
└──────────────────────────────────────────┬─────────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────────────────────────────────┐
│ STAGE 02 // DETERMINISTIC QUERY PARSER (Phase 12) │
│ • Unicode NFC Normalization • Accent / Case folding • Quoted phrase extraction │
│ • Negation syntax (-term, NOT) • Stopword handling • CJK / Cyrillic / Arabic │
└──────────────────────────────────────────┬─────────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────────────────────────────────┐
│ STAGE 03 // IN-MEMORY LRU QUERY CACHE (Phase 28) │
│ • Hash key computation • TTL & evictions check • Sub-10ms fast path return │
└──────────────────────┬───────────────────────────────────────────────────┬─────────────┘
│ [Cache HIT] │ [Cache MISS]
▼ ▼
╔═════════════════════════════╗ ┌─────────────────────────────────────┐
║ FAST EXIT: HTTP 200 ║ │ STAGE 04 // CANDIDATE RETRIEVAL │
║ Header: X-Cache: HIT (4ms) ║ │ • Inverted index posting lookups │
╚═════════════════════════════╝ │ • Boolean union / intersection mode │
│ • Positional phrase verification │
└──────────────────┬──────────────────┘
│
▼
┌─────────────────────────────────────┐
│ STAGE 05 // BM25 RANKING ENGINE │
│ • Multi-field TF-IDF (k1=1.2, b=.75)│
│ • Title (3.0x), Heading (2.0x) │
│ • Quoted phrase boost (1.5x) │
│ • Domain diversity dampening (0.8x) │
└──────────────────┬──────────────────┘
│
┌──────────────────┴──────────────────┐
│ EVALUATION: Sufficient local hits? │
└──────────┬──────────────────┬───────┘
│ [No: Deficit] │ [Yes: >= limit]
▼ │
┌──────────────────────────────────────────────────────────────────┐ │
│ STAGE 06 // LIVE HYBRID FALLBACK ENGINE (apps/api) │ │
│ 1. Google Suggest / Knowledge Engine (Direct authority domains) │ │
│ 2. DuckDuckGo Organic HTML Parser (Direct real-world web URLs) │ │
│ 3. Instant Answers API (Definitions & entity overviews) │ │
└─────────────────────────────────┬────────────────────────────────┘ │
│ │
▼ ▼
┌────────────────────────────────────────────────────────────────────────────────────────┐
│ STAGE 07 // DE-DUPLICATION & RESULT SYNTHESIS │
│ • Domain collision suppression • Cross-provider deduplication • Score calibration │
└──────────────────────────────────────────┬─────────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────────────────────────────────┐
│ STAGE 08 // CONTEXTUAL HIGHLIGHTING & SNIPPET GENERATION (Phase 15) │
│ • Keyword proximity clustering • Word boundary truncation • Safe XSS sanitization │
│ • `<mark>` term injection • Display URL formatting • Pagination metadata │
└──────────────────────────────────────────┬─────────────────────────────────────────────┘
│
▼
╔═══════════════════════════════╗
║ TERMINAL JSON RESPONSE ║
║ Header: X-Cache: MISS (22ms) ║
╚═══════════════════════════════╝
| Stage | Module | Key Operations | Fallback / Behavior |
|---|---|---|---|
01. Ingestion |
apps/api/middlewares |
Input sanitization, IP anonymization (/24 or /48), size checks |
Capped at 200 chars; oversized queries return 400 Bad Request |
02. Parsing |
packages/indexer/query |
Unicode NFC normalization, exact quotes "...", negation -term |
Safe token stream generation without regex backtracking risks |
03. Caching |
apps/api/cache |
High-concurrency bounded LRU cache (500 entries) | Instantaneous 4ms response with X-Cache: HIT telemetry |
04. Retrieval |
packages/indexer/index |
Reading disk-persisted inverted postings, positional matches | Adaptive Boolean mode (switches from AND to OR if zero hits) |
05. Ranking |
packages/ranking/bm25 |
Length-normalized BM25 scoring with 4 structural field weights | Deterministic float score computed per candidate document |
06. Fallback |
apps/api/external-search |
Ephemeral queries to Google Suggest & DuckDuckGo organic scraping | Only activated if local candidates are fewer than the query limit |
07. Synthesis |
apps/api/routes |
Merges local and live sets; deduplicates identical destination domains | Local crawled knowledge always preserves rank precedence |
08. Formatting |
packages/ranking/results |
Snippet generation around densest hit clusters, <mark> highlights |
Strict HTML escaping prevents script injection or DOM clobbering |
OpenSearch is engineered as a strictly typed, zero-circular-dependency TypeScript monorepo using npm workspaces and TypeScript composite project references (
tsc -b).
┌─────────────────────────────────────────────────────────────────────────────────────────┐
│ PRESENTATION LAYER │
│ │
│ ┌─────────────────────────────────────────┐ ┌───────────────────────────────────┐ │
│ │ apps/web (Port 3000) │ │ apps/api (Port 3001) │ │
│ │ • Pure HTML5/CSS3 TUI Frontend │ │ • Node.js Native HTTP Server │ │
│ │ • JetBrains Mono Terminal Aesthetic │ │ • Sliding-Window Rate Limiter │ │
│ │ • Keyboard Shortcuts & Zero CDNs │ │ • External Search Scrapers (Live)│ │
│ └────────────────────┬────────────────────┘ └─────────────────┬─────────────────┘ │
└────────────────────────┼──────────────────────────────────────────┼─────────────────────┘
│ │
│ HTTP JSON Calls │ Internal Module API
▼ ▼
┌─────────────────────────────────────────────────────────────────────────────────────────┐
│ CORE ENGINE PACKAGES │
│ │
│ ┌─────────────────────┐ ┌─────────────────────┐ ┌─────────────────────────────┐ │
│ │ packages/ranking │ │ packages/indexer │ │ packages/crawler │ │
│ │ • BM25Scorer │◄──┼─ Inverted Index │◄──┼─ Crawl Orchestrator │ │
│ │ • Relevance Engine │ │• Positional Postings│ │• SSRF Guard HTTP Fetcher │ │
│ │ • Result Generator │ │• Tokenizer/Normalizer │• robots.txt Policy Engine │ │
│ └──────────┬──────────┘ └──────────┬──────────┘ └──────────────┬──────────────┘ │
└──────────────┼─────────────────────────┼─────────────────────────────┼──────────────────┘
│ │ │
▼ ▼ ▼
┌─────────────────────────────────────────────────────────────────────────────────────────┐
│ FOUNDATIONAL SUBSTRATE │
│ │
│ ┌─────────────────────────────────────────┐ ┌───────────────────────────────────┐ │
│ │ packages/storage │ │ packages/shared │ │
│ │ • JsonStorageAdapter (Atomic Locking) │ │ • Global Types & Data Models │ │
│ │ • DocumentRepository & UrlRepository │ │ • Structured JSON Logger (Redact)│ │
│ │ • ACID file system persistence │ │ • Error Hierarchy & Crypto Utils │ │
│ └─────────────────────────────────────────┘ └───────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────────────────┘
| Workspace | Package / Path | Role & Responsibilities | Key Dependencies |
|---|---|---|---|
apps/api |
@opensearch/apiapps/api/ |
Exposes REST endpoints (/api/v1/search, /health), sliding-window rate limiting per IP, request bounds (64KB payload, 200-char query), and live multi-provider web fallback (external-search.ts). |
@opensearch/ranking@opensearch/shared |
apps/web |
@opensearch/webapps/web/ |
Self-contained, zero-dependency browser client mimicking a real developer CLI. Includes phosphor-green styling, skip-links, ARIA live-announcements, and keybinds. | Native Browser APIs |
crawler |
@opensearch/crawlerpackages/crawler/ |
Polite autonomous web crawler. Implements robots.txt compliance, DNS-level SSRF defense, streaming body limits, and houses the 113-URL curated seed corpus. |
@opensearch/storage@opensearch/shared |
indexer |
@opensearch/indexerpackages/indexer/ |
Multilingual text normalization, positional posting lists, document frequency tracking, and disk-persisted inverted index compilation with atomic rebuilds. | @opensearch/storage@opensearch/shared |
ranking |
@opensearch/rankingpackages/ranking/ |
Mathematical BM25 scoring with multi-field structural weights (Title 3.0×, Headings 2.0×, Quoted phrase 1.5×), domain clustering dampening, and safe snippet formatting. | @opensearch/shared |
storage |
@opensearch/storagepackages/storage/ |
Zero-external-dependency persistence substrate. Provides transactional JSON repositories with POSIX/Windows atomic file-locking to prevent concurrent write corruption. | @opensearch/shared |
shared |
@opensearch/sharedpackages/shared/ |
Shared domain primitives, configuration validators (AppConfig), centralized error definitions, cryptographically secure ID generators, and structured privacy loggers. |
None (Root leaf) |
opensearch/
├── apps/
│ ├── api/ # Search REST API & Live Fallback Gateway
│ │ ├── src/
│ │ │ ├── external-search.ts # Real-time web engine (Google + DuckDuckGo)
│ │ │ ├── middlewares.ts # Rate limiters, security headers, CORS
│ │ │ ├── routes.ts # /api/v1/search & /health endpoint handlers
│ │ │ └── server.ts # Production HTTP server bootstrap
│ │ └── tests/ # API, security audit, and deployment tests
│ └── web/ # Terminal Web UI Emulator
│ ├── src/
│ │ ├── app.js # Client controller with AbortController debounce
│ │ ├── html-template.ts # Semantic, accessible HTML5 application shell
│ │ ├── server.ts # Lightweight static server
│ │ └── style.css # JetBrains Mono phosphor-green design tokens
│ └── tests/ # Cross-environment rendering & a11y tests
│
├── packages/
│ ├── crawler/ # RFC 3986 Polite Web Spider
│ │ └── src/
│ │ ├── fetcher/ # HTTP fetcher armed with DNS-level SSRF guards
│ │ ├── orchestrator/ # Crawl scheduler, memory limiter & budget tracker
│ │ │ └── seed-corpus.ts # 113 verified seeds across 19 categories
│ │ └── robots/ # RFC-compliant robots.txt parser & evaluator
│ │
│ ├── indexer/ # Inverted Index & Token Processing Engine
│ │ └── src/
│ │ ├── builder/ # Atomic staging builder (<indexDir>/builds/<id>)
│ │ ├── index/ # Term dictionary, inverted posting lists
│ │ └── processor/ # NFC Unicode normalization, folding & stop-words
│ │
│ ├── ranking/ # BM25 Relevance & Context Extraction
│ │ └── src/
│ │ ├── bm25/ # BM25Scorer implementation (k1=1.2, b=0.75)
│ │ ├── engine/ # Multi-field weighting & domain diversity dampening
│ │ └── results/ # XSS-safe snippet generator with <mark> highlighting
│ │
│ ├── storage/ # Atomic JSON Storage Layer
│ │ └── src/
│ │ ├── adapters/ # Atomic file-locking adapter with fsync guards
│ │ └── repositories/ # DocumentRepository, UrlRepository
│ │
│ └── shared/ # Universal System Substrate
│ └── src/
│ ├── config/ # Environment schema & validation
│ ├── errors/ # Centralized typed error hierarchy
│ ├── logger/ # PII-scrubbing structured JSON logger
│ └── types/ # Core contracts, search types, document models
│
├── scripts/ # Lifecycle scripts (feed-corpus, deploy-production, verifications)
├── docs/ # PRIVACY.md, DEPLOYMENT.md, SECURITY.md
├── report/ # Verification audit reports and evaluation metrics
├── PROJECT.txt # Full technical specification and constraints
└── PHASE.txt # Detailed engineering roadmap and verification ledger
Most "private" search engines protect you from advertisers, but leave you beholden to Microsoft Bing or third-party query syndicates. OpenSearch restores genuine technological sovereignty.
┌────────────────────────────────────────────────────────────────────────────────────────┐
│ COMPETITIVE CAPABILITY COMPARISON │
├──────────────────────────┬────────────┬───────────┬────────────┬───────────┬───────────┤
│ ARCHITECTURAL VECTOR │ OPENSEARCH │ GOOGLE │ DUCKDUCKGO │ BRAVE │ SEARXNG │
├──────────────────────────┼────────────┼───────────┼────────────┼───────────┼───────────┤
│ Autonomous Web Crawler │ [YES] │ [YES] │ [NO] │ [HYBRID] │ [NO] │
│ Sovereign Inverted Index │ [YES] │ [YES] │ [NO] │ [PARTIAL]│ [NO] │
│ Zero User Profiling / PII│ [YES] │ [NO] │ [YES] │ [YES] │ [YES] │
│ 100% Self-Hostable Node │ [YES] │ [NO] │ [NO] │ [NO] │ [YES] │
│ Open Explainable Ranking │ [YES] │ [NO] │ [NO] │ [NO] │ [HYBRID] │
│ Zero-Cost Infrastructure │ [YES] │ [NO] │ [NO] │ [NO] │ [VARIES] │
│ Ground-Up Implementation │ [YES] │ [YES] │ [NO] │ [HYBRID] │ [NO] │
│ Ephemeral Web Fallback │ [YES] │ [N/A] │ [CORE] │ [N/A] │ [CORE] │
└──────────────────────────┴────────────┴───────────┴────────────┴───────────┴───────────┘
┌───────────────────────────────────────────────────────────────────────────────┐
│ THE "PRIVATE SEARCH" ILLUSION │
├───────────────────────────────────────────────────────────────────────────────┤
│ │
│ [User Query] ──► [DuckDuckGo / Ecosia] ──► [Microsoft Bing API Backend] │
│ └── (Zero ads on UI) └── (Full query logged at source) │
│ │
│ [User Query] ──► [SearXNG Instance] ──► [Scrapes Google / Bing / Yahoo] │
│ └── (Open source wrapper) └── (Aggregator, no own index) │
│ │
│ [User Query] ──► [OPENSEARCH NODE] ──► [Local Inverted Index (BM25)] │
│ ├── In-Memory LRU Cache ├── Zero Outbound Network Leaks │
│ └── Ephemeral Web Bridge └── Complete Machine Sovereignty │
│ │
└───────────────────────────────────────────────────────────────────────────────┘
Virtually every privacy search brand (DuckDuckGo, Yahoo, Ecosia, Qwant) is a syndication customer of Microsoft Bing. If Bing alters its index, downranks open knowledge, or blocks an endpoint, these services follow suit. OpenSearch operates an independent crawler (@opensearch/crawler) and inverted index (@opensearch/indexer). Your search index belongs to you.
Aggregators like SearXNG relay your unencrypted queries to 10+ third-party search engines simultaneously. While your client IP may be masked, your intent profile is broadcast across multiple Big Tech properties. OpenSearch evaluates your query locally first. Only when local knowledge is depleted does it fire an isolated, ephemeral web probe with zero persistent identifiers.
Commercial search algorithms are closed commercial secrets optimized for ad impressions and user engagement time. OpenSearch uses pure, open-source BM25 lexical scoring (
- Full transparency: inspect exactly why a page ranked where it did via score decomposition.
- Zero engagement traps: no algorithmic manipulation, clickbait bias, or sponsored result injections.
OpenSearch has zero paid cloud dependencies:
- No Elasticsearch cluster bills.
- No Pinecone / OpenAI vector API token fees.
- No Redis cloud subscription or MongoDB instances.
- Runs entirely within a lightweight Node.js runtime backed by ACID-safe file storage. Deploy it on a Raspberry Pi, a $4 VPS, or your local development machine for free.
You decide the worldview of your search engine. By modifying seed-corpus.ts, you can curate an engine laser-focused on internal engineering docs, biomedical research, open-source codebases, or the general public web. Rebuild with a single command:
node scripts/feed-corpus.mjsEngineered for zero operational cost, maximal determinism, and zero external binary dependencies.
┌────────────────────────────────────────────────────────────────────────────────────────┐
│ TECHNOLOGY BLUEPRINT │
├─────────────────────────┬──────────────────────────────────┬───────────────────────────┤
│ ARCHITECTURAL TIER │ IMPLEMENTATION / TOOL │ DESIGN RATIONALE │
├─────────────────────────┼──────────────────────────────────┼───────────────────────────┤
│ Language & Typing │ TypeScript 5.7+ (Strict Mode) │ End-to-end type safety │
│ Runtime Platform │ Node.js 20+ LTS │ Native fetch, abort signals│
│ Monorepo Orchestration │ npm Workspaces + tsc -b │ Isolated composite builds │
│ Inverted Index Storage │ Flat JSON + Atomic File-Locking │ Zero DB daemon footprint │
│ Ranking & Scoring │ Custom BM25 Relevance Engine │ Auditable, open scoring │
│ HTTP API Transport │ Native Node.js `http` Module │ Zero-framework overhead │
│ Client Presentation │ Semantic HTML5 + Vanilla CSS3/JS │ Zero bloat, zero trackers │
│ Verification & Test │ Vitest 2.1+ (360 Passing Tests) │ Instant multi-thread runs │
│ Quality & Code Health │ ESLint 9 (Flat Config) + Prettier│ Uniform code ergonomics │
│ Live Web Scrapers │ Ephemeral DuckDuckGo + Google │ Fallback without identity │
└─────────────────────────┴──────────────────────────────────┴───────────────────────────┘
$ node -v && npm -v && git --version
v20.18.0 # Node.js >= 20.0.0 LTS required
10.8.2 # npm >= 10.0.0 required
git 2.45.0 # Git required# Clone the repository
git clone https://github.com/Ashish6298/opensearch.git
cd opensearch
# Install all monorepo dependencies across workspaces
npm install
# Compile TypeScript composite projects
npm run build# Crawl the 113-URL seed corpus and build the local inverted index
node scripts/feed-corpus.mjs[feed-corpus] Starting corpus ingestion pipeline...
[feed-corpus] Processed 113 seeds across 19 categories.
[feed-corpus] Document store updated: 113 documents persisted (ACID locked).
[feed-corpus] Inverted index built: data/index (BM25 postings active).
[feed-corpus] Pipeline complete in 3.42s.OpenSearch Web enforces formal deployment tiers, persistence invariants, and zero-paid infrastructure guarantees (see docs/deployment-architecture.md):
-
local-first(Tier 1 — Primary & Fully Supported):- Target: Developer laptops, desktop workstations, single-user offline environments.
- Characteristics: Zero cloud dependencies, zero secrets required, SQLite WAL durability on local disk, 100% offline-capable with built-in demo corpus.
- Execution:
npm run devornpm start.
-
self-hosted(Tier 1 — Primary & Fully Supported):- Target: Single Linux VPS (Ubuntu/Debian), home servers (Raspberry Pi/NUC), or Docker containers.
- Characteristics: Persistent volume mounted at
/app/data, continuous crawler execution, PageRank pipelines, in-memory or file-backed rate limiting, reverse proxy friendly (Nginx/Caddy).
-
serverless(Tier 2 — Restricted / Read-Only Advisory):- Target: AWS Lambda, Vercel Serverless, Netlify Functions.
- Explicit Non-Assumptions: Serverless filesystem writes (
/tmp) are ephemeral and wiped between invocations; background crawlers cannot run due to invocation timeouts; process-memory rate limits are instance-local and do NOT provide global cluster guarantees. Supported strictly as a read-only query node.
-
cluster(Tier 2 — Multi-Instance Clustered):- Target: Multi-process Node instances behind an ingress reverse proxy or load balancer.
- Requirement: Ingress rate limiting (Nginx
limit_req_zoneor Caddyrate_limit) required for cluster-wide enforcement.
- Zero Paid Domains: Operates out-of-the-box on
localhost,127.0.0.1, or private network IPs. - Zero Paid Services: Requires no commercial databases, no Redis clusters, and no third-party search APIs.
- Zero Mandatory Secrets: Starts cleanly with empty environment variables or without a
.envfile. - Runtime Introspection: Query
GET /api/v1/deploymentorGET /api/v1/statusfor real-time capability and persistence verification.
OpenSearch Web provides an unprivileged, non-root multi-stage container on node:20-alpine with explicit persistent volume storage and native health checks:
# Launch sovereign container via Docker Compose (Single Service, Zero Secrets)
docker compose up -d
# Verify container health
docker compose ps
# -> STATUS: Up (healthy)
# View unified search web interface
open http://localhost:3000- Persistent Volume: Stored in named volume
opensearch_data(/app/data). Search indexes and crawled documents survive container recreation. - Operations & Backups Guide: See
docs/docker-and-compose.mdfor volume snapshots, zero-downtime upgrades, and reset procedures.
OpenSearch Web release workflows prioritize determinism, zero accidental data loss, and zero unvetted package exposure:
- Traceable Release Metadata: Inspect the active build revision, node runtime, build date, and safe configuration defaults via
GET /api/v1/release. - Pre-Migration Automated Backups: All schema migrations execute a non-destructive pre-flight integrity check and create an atomic snapshot in
<storageDir>/backups/pre-migration-v<version>-<timestamp>. - Automatic Failure Recovery: If a migration script or post-validation fails, the system automatically restores from the pre-migration snapshot, guaranteeing that broken upgrades never destroy existing indexes or crawl data.
- Safe Environment Defaults: 100% of environment variables feature explicit zero-secret fallbacks. The system boots cleanly without any
.envfile or credentials. Seedocs/environment-variables.md. - Publication Safety Guard: All 9 packages across
packages/*,apps/*, and root feature"private": true.scripts/publish-guard.mjsis bound viaprepublishOnlyto prevent accidental publication without explicitALLOW_PUBLISH=true. - Comprehensive Rollback Runbook: Step-by-step procedures for local binary rollback and Docker container rollbacks are detailed in
docs/release-and-rollback.md.
Automated continuous integration and release workflows enforce reproducibility, zero-secret operations, and least-privilege security:
- Least-Privilege Security: Workflows enforce top-level
permissions: contents: read(elevated tocontents: writeexclusively for publishing tagged release assets). - PR Quality Gate (
.github/workflows/ci.yml):- Frozen dependency installation via
npm ciadhering topackage-lock.json. - TypeScript typechecking (
npm run typecheck), ESLint zero-error validation (npm run lint), and composite package builds (npm run build). - Unit and integration tests across Node.js 20 LTS and Node.js 22 LTS.
- Security regressions, network data flow privacy audits, local-only mode enforcement, and OpenAPI 3.1.0 contract verification.
- Multi-stage Docker container build verification.
- Frozen dependency installation via
- Main & Release Packaging (
.github/workflows/release.yml):- Clean checkout builds triggered on version tags (
v*.*.*) or default branch pushes. - Automated release artifact bundling (
tar.gz) with SHA256 checksum provenance generation. - Accidental package registry publication prevention via
scripts/publish-guard.mjs.
- Clean checkout builds triggered on version tags (
- Zero Secrets Invariant: All CI testing and build gates execute completely without external API keys, tokens, or paid services.
- Verification Runbook: See dedicated test suite
apps/api/tests/final-release-gate.test.ts.
Launch both daemon processes in separate terminal instances:
# Terminal 1: Search REST API & Live Fallback Gateway (Port 3001)
npm run start:api# Terminal 2: Distraction-Free Terminal Web Interface (Port 3000)
npm run start:webOpen http://localhost:3000 to access the search terminal.
Search directly from your command line without opening a browser:
# Search directly via npx or local binary
npm run opensearch "typescript async await"
# Limit output results
npm run opensearch "bm25 ranking" -- --limit 5
# Pipe raw JSON into jq
npm run opensearch "database" -- --json | jq .results[0]
# Pipe queries through stdin
echo "privacy policy" | npm run opensearch
# Check cluster health diagnostics
npm run opensearch -- --health# 1. Execute a hybrid search query via cURL
curl -s "http://localhost:3001/api/v1/search?q=typescript+compiler" | jq .
# 2. Inspect runtime health & LRU cache telemetry
curl -s "http://localhost:3001/health" | jq .{
"status": "healthy",
"uptimeSeconds": 1420,
"memory": {
"rssBytes": 68423680,
"heapUsedBytes": 38914560
},
"index": {
"totalDocuments": 113,
"status": "ready"
},
"cache": {
"hits": 42,
"misses": 8,
"hitRatio": 0.84
}
}┌────────────────────────────────────────────────────────────────────────────────┐
│ DEVELOPER COMMAND REGISTRY │
├───────────────────────────────┬────────────────────────────────────────────────┤
│ COMMAND │ OPERATIONAL ACTION │
├───────────────────────────────┼────────────────────────────────────────────────┤
│ npm run build │ Transpile all packages via composite tsc -b │
│ npm run typecheck │ Type-check monorepo without emitting artifacts │
│ npm run test │ Run full Vitest suite (360 unit/integration) │
│ npm run lint │ Scan all TS/JS files with ESLint 9 flat config │
│ npm run format │ Enforce Prettier rules across all workspaces │
│ npm run clean │ Purge all compiled dist/ build outputs │
│ npm run start:api │ Boot standalone search API server daemon │
│ npm run start:web │ Boot standalone terminal web emulator daemon │
│ node scripts/feed-corpus.mjs │ Ingest 113 seeds and compile BM25 search index │
│ npm run verify:phase1..34 │ Run automated verification audit harness │
└───────────────────────────────┴────────────────────────────────────────────────┘
OpenSearch utilizes automated GitHub Actions workflows (.github/workflows/ci.yml) to guarantee type integrity, test coverage, code health, and production build readiness on every push and pull_request to main and ashish.
┌────────────────────────────────────────────────────────────────────────────────────────┐
│ GITHUB ACTIONS CI/CD PIPELINE │
├──────────────────────────┬──────────────────────┬──────────────────────────────────────┤
│ PIPELINE STAGE │ RUNNER / MATRIX │ CRITICAL GATES │
├──────────────────────────┼──────────────────────┼──────────────────────────────────────┤
│ 1. 🔍 Lint, Format & Types│ Ubuntu (Node 20 & 22)│ ESLint 9, Prettier, tsc -b composite │
│ 2. 🧪 Automated Tests │ Ubuntu (Node 20 & 22)│ 115 Vitest Suites (1,339 Tests) │
│ 3. 🛡️ Security & Privacy │ Ubuntu (Node 20 LTS) │ Zero-PII, SSRF guard & leak audits │
│ 4. 🐳 Docker & Packaging │ Ubuntu (Node 20 LTS) │ Multi-stage build & SHA256 archive │
└──────────────────────────┴──────────────────────┴──────────────────────────────────────┘
flowchart TD
A[Git Push / PR Event] --> B[Job 1: Lint, Format & Types<br/>Node 20.x & 22.x Matrix]
B --> C[Job 2: Vitest Automated Test Suite<br/>115 Suites & 1,339 Passing Tests]
B --> D[Job 3: Security & Privacy Gate<br/>Zero-PII & Network Data Flow Audits]
B --> E[Job 4: Docker Container Build<br/>Alpine Multi-Stage Container Check]
- 🔄 Concurrency Controls: Active workflows are automatically grouped and cancelled if superseded by newer commits (
cancel-in-progress: true). - ⚡ Caching Strategy: Automatic
npmcache keys for fast, deterministic installs vianpm ci. - 📦 Artifact Retention: Builds generate versioned distribution packages stored for 7 days.
OpenSearch uses zero-secret, transparent environment files. Copy the blueprint to start:
cp .env.example .env| Environment Key | Default Value | Purpose / Impact |
|---|---|---|
API_PORT |
3001 |
HTTP port for the Search REST API & Live Fallback service |
WEB_PORT |
3000 |
HTTP port for the standalone web terminal interface |
INDEX_DIR |
./data/index |
Directory path hosting disk-persisted inverted index postings |
STORAGE_DIR |
./data/storage |
Path for atomic document and URL metadata JSON stores |
LOG_LEVEL |
info |
Logging verbosity (debug, info, warn, error) |
RATE_LIMIT_MAX |
100 |
Sliding-window maximum request threshold per IP |
CACHE_MAX_ENTRIES |
500 |
In-memory LRU query cache capacity limit |
🔒 Security Notice:
.envfiles are ignored by version control via.gitignore. Never commit production secrets or host keys.
The baseline index comes pre-populated with high-utility knowledge repositories:
| Sector | Target Count | Curated Domain Examples | Primary Focus |
|---|---|---|---|
| Tech Documentation | 12 | MDN, TypeScript, Node.js, Python, Rust, Go, Docker | Language specs, APIs, standard libraries |
| AI & Machine Learning | 8 | OpenAI, HuggingFace, PyTorch, TensorFlow, Kaggle | Foundation models, datasets, research |
| Developer Ecosystem | 8 | GitHub, Stack Overflow, npm, crates.io, VS Code | Repositories, package registries, troubleshooting |
| News & Global Events | 9 | BBC, Reuters, The Guardian, TechCrunch, Hacker News | Real-time global reporting, tech journalism |
| Finance & Markets | 6 | Yahoo Finance, CoinMarketCap, Investopedia, Bloomberg | Market tracking, economic definitions |
| Health & Medicine | 6 | WHO, NIH, WebMD, Mayo Clinic, CDC | Peer-reviewed medical guidance |
| Science & Research | 6 | NASA, arXiv, Nature, PubMed, Science.org | Preprints, astronomical data, academic papers |
| Open Source & Linux | 7 | Linux Kernel, Apache, CNCF, Free Software Foundation | Kernel documentation, cloud native governance |
| Privacy & Security | 6 | EFF, Privacy Guides, Tor Project, Have I Been Pwned | Threat intelligence, cryptographic protocols |
| Education & Encyclopedias | 6 | Wikipedia, Khan Academy, MIT OpenCourseWare, edX | General open knowledge & academic courses |
| Food & Culinary | 5 | Allrecipes, Food Network, Serious Eats, Tasty | Curated recipes & culinary technique |
| Travel & Transit | 5 | OpenStreetMap, Google Flights, Booking.com, Lonely Planet | Global cartography, booking references |
| Entertainment & Media | 6 | IMDb, Rotten Tomatoes, Spotify, YouTube | Film indexes, streaming discographies |
| Sports & Athletics | 5 | ESPN, BBC Sport, NBA, FIFA, Formula 1 | League statistics, standings, match reports |
| Community & Forums | 5 | Reddit (selected subreddits), Hacker News, Lobsters | Technical discussions & peer debates |
| Government & Public Services | 4 | USA.gov, UK Gov, European Union Portal | Official public registries & policy documents |
| Weather & Meteorology | 3 | National Weather Service, Weather.com | Atmospheric data, meteorological forecasts |
| Commerce & Marketplaces | 3 | Amazon, eBay (structured meta-extracts only) | Catalog definitions (zero tracker cookies) |
| Reference & Linguistics | 3 | Merriam-Webster, Wiktionary, Stanford Phil. | Lexicons, philosophical indexes, etymology |
Privacy is not a setting or toggle in OpenSearch — it is an immutable architectural invariant.
┌────────────────────────────────────────────────────────────────────────────────────────┐
│ IN-MEMORY PRIVACY AUDIT CONSOLE │
├──────────────────────────┬──────────────────────┬──────────────────────────────────────┤
│ PRIVACY VECTOR │ ENFORCEMENT STATE │ ARCHITECTURAL IMPLEMENTATION │
├──────────────────────────┼──────────────────────┼──────────────────────────────────────┤
│ User Accounts & Auth │ [NEVER IMPLEMENTED] │ Zero account schemas or credentials │
│ Browser Cookies │ [STRICTLY FORBIDDEN] │ Set-Cookie headers dropped by server │
│ Query Retention Log │ [DISCARDED ON REPLY] │ Processed in-RAM; 0 disk writes │
│ Third-Party Trackers │ [AIR-GAPPED / ZERO] │ No Google Tag, Mixpanel, or PostHog │
│ External CDN Assets │ [100% SELF-CONTAINED]│ Local fonts & zero remote stylesheets│
│ Client IP Anonymization │ [ACTIVE BY DEFAULT] │ Truncated: IPv4 /24 & IPv6 /48 │
│ Outbound Referrer Leaks │ [SANITIZED / BLOCKED]│ rel="noopener noreferrer" + No-Ref │
│ Ephemeral Fallback State │ [STATELESS SCRAPE] │ Scrapers carry zero sessions/tokens │
└──────────────────────────┴──────────────────────┴──────────────────────────────────────┘
$ audit --privacy-boundaries
[PASS] TCP_PAYLOAD_CHECK : 0 tracking pixels or beacon endpoints detected
[PASS] CLIENT_STORAGE_CHECK : 0 cookies, 0 indexedDB databases, 0 localStorage items
[PASS] DISK_PERSISTENCE_AUDIT: query tokens never written to data/storage
[PASS] IP_MASKING_AUDIT : 192.168.1.105 -> 192.168.1.0 (masked in memory)
[PASS] REFERRER_LEAK_AUDIT : Referrer-Policy: no-referrer active on all HTTP responses
[PASS] OUTBOUND_LINK_AUDIT : target="_blank" rel="noopener noreferrer" enforced| Privacy Vector | Big Tech Standard | OpenSearch Sovereign Standard | Technical Guarantee |
|---|---|---|---|
| Search Profiling | Builds behavioral profiles across weeks & months | Never stored | Search terms exist only in stack memory during request processing and are freed immediately upon response dispatch. |
| Session Identification | Persistent tracking cookies, device fingerprinting, ETags | Zero identity state | The server emits zero Set-Cookie headers. No session identifiers or unique device fingerprints are ever generated or evaluated. |
| Client IP Addresses | Full IPv4/IPv6 addresses logged with timestamps | Subnet truncation | IPv4 addresses have their final octet zeroed (192.0.2.0/24) and IPv6 are masked to /48 before entering loggers. |
| Outbound Leaks | Search queries leaked via HTTP Referer headers |
Referrer isolation | Outbound links enforce rel="noopener noreferrer" and global responses carry Referrer-Policy: no-referrer. |
| Analytics & Telemetry | Google Analytics, Sentry, Mixpanel, Datadog tracking | Zero telemetry | 100% self-hosted Node.js processes without external analytics beacons, third-party CDNs, or remote font loads. |
📖 Formal Privacy Specifications: Review the complete data minimization principles, network guarantees, and threat models codified in
docs/PRIVACY.md.
┌────────────────────────────────────────────────────────────────────────────────────────┐
│ ARCHITECTURE & VERIFICATION LEDGER │
├──────────────────────────────────────┬─────────────────────────────────────────────────┤
│ TARGET BUILD : v1.5.0-STABLE │ ARCHITECTURE : MODULAR TYPESCRIPT MONOREPO │
│ TEST SUITE : 115 SUITES (1,339) │ AUDIT STATUS : 100% QUALITY GATE VERIFIED │
└──────────────────────────────────────┴─────────────────────────────────────────────────┘
| Subsystem / Capability Layer | Architecture & Engineering Focus | Delivery Verification & Artifacts | Status |
|---|---|---|---|
| Monorepo & Storage Substrate | Scalable monorepo architecture with strict TypeScript composite project references and atomic persistence | TS 5.7+ composite configs, JSON storage adapter, ACID file locks | [VERIFIED] |
| Polite Crawler & Fetching Engine | RFC 3986 URL canonicalization, DNS-level SSRF resolution guards, robots.txt politeness parser | SSRF DNS guards, robots.txt parser, HTML streaming extractor, link discovery | [VERIFIED] |
| Search Index & Normalization | Positional postings inverted index, multilingual tokenization, Unicode normalization, incremental recovery | Multilingual tokenization, positional postings, incremental indexer | [VERIFIED] |
| Query Processing & BM25 Scoring | Lexical query parser, quoted phrase search, negation operators, multi-field BM25 ( |
Quoted phrases, negation, multi-field BM25 ranking, dynamic contextual snippets | [VERIFIED] |
| HTTP REST API & Security Gate | Pure node:http router, sliding-window rate limiter, payload bounds, CORS and header security |
Pure node:http router, sliding-window rate limiter, payload bounds, input validation |
[VERIFIED] |
| Public Terminal Web Interface | Responsive terminal user interface, JetBrains Mono phosphor styling, full keyboard accessibility | Responsive TUI, JetBrains Mono phosphor styling, full WCAG a11y, zero remote CDN assets | [VERIFIED] |
| End-to-End Search Pipeline | Cold-start crawl loop, 113 curated seed corpus, crawler-to-indexer pipeline orchestration | Cold-start crawl loop, 113 curated seed corpus, IR MRR quality benchmark | [VERIFIED] |
| Security Hardening & Zero-Logs | Strict CSP enforcement, anti-SSRF protections, circular redirect traps, IP subnet masking (/24 & /48) | Strict CSP enforcement, circular redirect traps, IP subnet masking, zero surveillance logs | [VERIFIED] |
| High-Concurrency Performance | In-memory LRU query cache (<10ms warm hits), crawler memory backpressure, bounded heap allocations | In-memory LRU query cache, crawler memory backpressure, zero unbounded memory leaks | [VERIFIED] |
| Production Release & Live Fallback | Resilience fallback integrating Google Suggest and DuckDuckGo for thin local results | Google/DuckDuckGo live hybrid search, cross-env tests, referrer isolation | [VERIFIED] |
| Query Intelligence & Productivity | Autocomplete engine, Levenshtein distance typo tolerance, prefix matching, 50+ privacy bangs | Autocomplete, Levenshtein typo-tolerance, advanced operators, 50+ privacy-respecting bangs | [VERIFIED] |
| Terminal UI, Accessibility & Themes | Full Vim navigation keybindings (j/k/v///Esc), 5 retro CRT themes, local preference storage |
Vim/terminal keybinds, 5 retro CRT themes (Amber, Green, Cyberpunk, Matrix, Monokai) | [VERIFIED] |
| Crawler Efficiency & Sitemaps | XML and Gzip sitemap stream parsing, HTTP 304 Not Modified conditional fetching via ETags | Streaming/Gzip Sitemap ingestion, HTTP 304 fast path & ETag re-crawling | [VERIFIED] |
| Developer Platform & CLI Tooling | OpenAPI 3.1.0 specification with interactive Swagger UI (/docs), standalone @opensearch/cli
|
OpenAPI 3.1.0 spec (/docs), standalone @opensearch/cli terminal tool |
[VERIFIED] |
| Release Convergence & Benchmarking | Performance profiling, sub-5ms P99 latency SLA on warm indices, execution tracing | Full regression testing, sub-5ms P99 latency SLA, v1.2.0 verification | [VERIFIED] |
| Developer Utilities & Runtime Toolkit | Instant utilities for UUID generation, secure passwords, JWT inspection, cryptographic hashes, CIDR IPv4 subnets, cron explainer | UUID/Password/JWT/Hash crypto engine, CIDR IPv4 subnets, cron explainer | [VERIFIED] |
| Privacy Auditing & Zero-Trace Reader | Domain security grades (A+ to F), TLS/HSTS posture analyzer, zero-trace distraction-free Markdown document reader | Domain security grades, TLS/HSTS audit, zero-trace Markdown reader | [VERIFIED] |
| Scientific Math & Currency Engine | Real-time fiat FX, crypto rates (BTC/ETH/SOL), symbolic algebra computation, rendered LaTeX formatting | Real-time fiat FX, crypto rates, symbolic algebra & LaTeX rendering | [VERIFIED] |
| Hacker UX & Retro Themes | Vim result navigation (j/k/o/c/r), targeted search filters (/dev, /news, /papers, /code) |
Vim result navigation, clipboard URL copy, tabbed query filters, zero-latency switching | [VERIFIED] |
| Offline PWA & Service Worker | Progressive Web App manifest, ServiceWorker offline asset caching, offline fallback interface | ServiceWorker offline caching, PWA Manifest, offline search fallback, installation prompts | [VERIFIED] |
| Repository Audit & Sovereign Baseline | Monorepo architecture audit, baseline benchmark instrumentation, formal product and privacy contracts | Architecture inventory, baseline measurements, product & privacy contracts | [VERIFIED] |
| Search Providers & Source Reliability | Standardized SearchProvider interface, circuit breakers, timeout budgets, zero brittle scraping |
Standard provider interface, zero brittle scraping, resilience, attribution compliance | [VERIFIED] |
| Index Storage & Dataset Growth | Storage engine lifecycle audit, unified storage abstraction, reproducible 500+ document seed corpus | Lifecycle audit, storage abstraction, reproducible corpus expansion, incremental recovery | [VERIFIED] |
| Everyday-User Interface | Search-first responsive layout, structured result cards, light/dark themes, category tabs, safe search | Search-first responsive home, result cards & states, themes & preferences, categories, safe search controls | [VERIFIED] |
| Privacy Receipt & Local-Only Mode | Network and data-flow auditing, cryptographic privacy receipt generation, strict Local-Only mode | Network & data-flow audit, local-only enforcement, privacy receipt schema, leak tests | [VERIFIED] |
| Real PageRank & Ranking Explainability | Power-iteration PageRank algorithm, link graph adjacency matrix extraction, hybrid ranking | Link graph extraction, power-iteration PageRank, multi-tier ranking, explainability breakdown | [VERIFIED] |
| Search Quality Lab | Judged query evaluation harness, automated NDCG@10 and MAP scoring, ablation analysis | Graded judged dataset, evaluation metrics (NDCG/MAP), score ablation, quality regression gates | [VERIFIED] |
| Performance & Large-Scale Search | Benchmark harness, execution profiling, postings traversal optimizations, multi-segment indexing | Benchmark harness, execution profiling, postings traversal, segment indexing & scale metrics | [VERIFIED] |
| Tracker-Aware Search & Focus Lenses | Reviewable tracker registry (DuckDuckGo Tracker Radar), tracker penalty scoring, Focus Lens schema | Reviewable tracker registry, static subresource parsing, tracker-aware ranking signals, Focus Lens schema | [VERIFIED] |
| Instant Answers & Extractive Quick Answer | Modular instant-answer framework (weather, time, calculator, unit conversion), extractive Quick Answer | Modular instant-answer framework, provenance attribution, deterministic Quick Answer panel | [VERIFIED] |
| Privacy-Preserving Quality Feedback | User-consented search quality rating, strict query/IP redaction, 30-day retention with auto-purge | Restrained feedback rating, explicit user consent, strict prohibition of queries/IPs, 30-day retention | [VERIFIED] |
| API, Browser Search & MCP Integration | OpenAPI 3.1.0 contract fidelity, schema consistency, OpenSearch browser descriptor, MCP server | OpenAPI 3.1.0 contract fidelity, schema consistency, OpenSearch descriptor, MCP server | [VERIFIED] |
| Security & Operational Resilience | Multi-tier rate limiting, health/readiness probes, graceful shutdown connection draining, Prometheus metrics | SSRF defense audit, scheme allowlist, storage persistence audit, multi-tier rate limiting, Prometheus metrics | [VERIFIED] |
| Documentation, Hygiene & Product Identity | OpenSearch Web distinct product branding, trademark non-affiliation notices, accessibility compliance | Distinct product identity (OpenSearch Web), trademark disambiguation, user docs, accessibility audit | [VERIFIED] |
| Deployment & Container Support | Multi-stage Docker container (node:20-alpine), unprivileged non-root user (UID 1001), Docker Compose |
Multi-stage Docker build, unprivileged user execution, explicit volume durability (/app/data), rollback runbooks |
[VERIFIED] |
| GitHub Actions CI/CD & Automated Quality Gates | Least-privilege automated CI workflows, frozen dependency installation, security/privacy regression checks | Least-privilege CI workflow, pull-request verification gate, release packaging & provenance, 115 suites passed | [VERIFIED] |
$ npm test
✓ apps/api/tests/demo-experience-accessibility.test.ts (17 tests)
✓ apps/api/tests/readme-documentation.test.ts (5 tests)
✓ apps/api/tests/product-identity.test.ts (12 tests)
✓ apps/api/tests/observability-metrics.test.ts (14 tests)
✓ apps/api/tests/health-status-graceful-shutdown.test.ts (10 tests)
✓ apps/api/tests/rate-limiting-and-hosting.test.ts (22 tests)
✓ packages/crawler/tests/security-regression-audit.test.ts (25 tests)
✓ apps/api/tests/mcp-server.test.ts (26 tests)
✓ apps/api/tests/api.test.ts (38 tests)
✓ apps/api/tests/openapi-docs.test.ts (18 tests)
✓ apps/api/tests/openapi-contract.test.ts (12 tests)
✓ apps/api/tests/browser-search-descriptor.test.ts (11 tests)
✓ apps/web/tests/browser-search-descriptor.test.ts (6 tests)
✓ packages/ranking/tests/instant-answer-framework.test.ts (14 tests)
✓ packages/ranking/tests/quick-answer.test.ts (19 tests)
✓ packages/ranking/tests/privacy-conscious-feedback.test.ts (13 tests)
✓ packages/ranking/tests/feedback-review-workflow.test.ts (16 tests)
✓ apps/api/tests/feedback-review-api.test.ts (8 tests)
✓ apps/api/tests/lens-api.test.ts (7 tests)
✓ apps/api/tests/release-packaging-rollback.test.ts (10 tests)
✓ apps/api/tests/final-release-gate.test.ts (22 tests)
...
Test Files 115 passed (115)
Tests 1339 passed (1339)
Duration 24.55s -- 100% Passing Pass Rate
────────────────────────────────────────────────────────────
TOTAL RESULT: 1339 / 1339 TESTS PASSED (100.0%) -- V1.5.0 END-TO-END RELEASE GATE VERIFIEDOpenSearch is built by and for developers who believe that web search infrastructure should be decentralized, private, and open source.
┌────────────────────────────────────────────────────────────────────────────────┐
│ CONTRIBUTING PROTOCOL HUD │
├───────────────────────────────┬────────────────────────────────────────────────┤
│ 1. BRANCHING MODEL │ feat/<feature-name> or fix/<bug-name> │
│ 2. COMMIT DISCIPLINE │ Conventional Commits (feat:, fix:, docs:) │
│ 3. COVERAGE REQUIREMENT │ 100% test coverage for new ranking/index code │
│ 4. TYPE INTEGRITY │ Zero 'any' types; strict tsc composite checks │
│ 5. STYLE ENFORCEMENT │ Clean ESLint 9 + Prettier pass prior to PR │
└───────────────────────────────┴────────────────────────────────────────────────┘
# 1. Fork & clone the repository
git clone https://github.com/Ashish6298/opensearch.git
cd opensearch
# 2. Create your isolated feature branch
git checkout -b feat/add-new-seed-sector
# 3. Make changes and verify full suite
npm run typecheck
npm run lint
npm run test
# 4. Commit using Conventional Commits convention
git commit -m "feat(crawler): add 15 curated academic research seeds"
# 5. Push and open your Pull Request
git push origin feat/add-new-seed-sectorFor in-depth architectural boundaries, module contracts, and testing conventions, refer to CONTRIBUTING.md.
This project is licensed under the terms of the MIT License.
MIT License © 2026 Ashish (Ashish6298)
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so.
Review the complete legal text in LICENSE.md.
We are committed to providing a welcoming, inclusive, and harassment-free community for all contributors and maintainers. OpenSearch adheres strictly to the Contributor Covenant (Version 2.1).
$ cat CODE_OF_CONDUCT.md | grep -E "(Pledge|Standards|Enforcement)"
## Our Pledge: Make participation a harassment-free experience for everyone.
## Our Standards: Demonstrate empathy, respect differing viewpoints, accept feedback.
## Enforcement: Maintainers actively address inappropriate behavior fairly.- 🤝 Our Pledge: Fostering an open, welcoming, diverse, and healthy engineering environment.
- 🛡️ Zero Tolerance: Immediate moderation against abusive language, harassment, or bad-faith interactions.
- 📬 Reporting: Report unacceptable behavior privately via GitHub Issues.
Read the full community pledge in CODE_OF_CONDUCT.md.
┌────────────────────────────────────────────────────────────────────────────────┐
│ ARCHITECT PROFILE │
├────────────────────────────────────────────────────────────────────────────────┤
│ ARCHITECT : Ashish (Ashish6298) │
│ PROJECT : OpenSearch (V1.0.0 Production Release) │
│ GITHUB : https://github.com/Ashish6298 │
│ PLATFORM : Distributed Computing • Search Engine IR • Systems Engineering │
└────────────────────────────────────────────────────────────────────────────────┘
Crafted with obsession for privacy, distributed systems, and independent search.