An enterprise-grade, microservices-based backend for storing, serving, and analyzing "posts" (title/body records with per-character-frequency analytics and ML-driven sentiment classification). Built on a CloudWeGo stack (Kitex + Hertz + Sonic + Netpoll) with RPC and HTTP surfaces, ABAC authorization, three distinct message brokers, local S3-compatible object storage, an Nvidia Triton ML integration, full observability, and Kubernetes/Terraform deployment artifacts for AWS, Azure, OCI, and GCP.
It's fundamentally about post analysis β every piece of infrastructure here (Kafka event stream, RabbitMQ reanalysis queue, RocketMQ scheduled rechecks, Redis cache, Triton sentiment model) exists in direct service of that core CRUD-and-analyze use case, not as unrelated scaffolding.
- Screenshots
- Architecture
- Tech stack
- Messaging: why three brokers
- Quickstart
- Interacting with it: HTTP, RPC, CLI, REPL, UI
- API conventions: errors, envelopes, status codes
- Project structure
- Configuration reference
- Testing
- Deployment
- API documentation
- Makefile reference
- License
Everything below is the actual running app β captured at viewport size against the real docker-compose stack, not mockups. More detail (including the ABAC role model behind the dashboard's admin panel) is in ARCHITECTURE.md.
flowchart TB
Browser(["Browser<br/>UI + Dashboard"])
Nginx["nginx (edge LB)<br/>resolver-based upstream β<br/>scales with --scale gateway=N"]
Gateway["gateway (Hertz / Netpoll)<br/>server Β· CLI Β· REPL β same binary, 3 modes<br/>REST API Β· ABAC middleware Β· rate limit Β·<br/>panic recovery Β· metrics Β· CORS Β· compression"]
PostSvc["postsvc<br/>CRUD Β· analysis Β·<br/>Redis cache Β·<br/>Kafka + RocketMQ producers Β· Triton"]
AuthSvc["authsvc<br/>JWT issuance Β·<br/>ABAC policy decisions"]
PG[("Postgres<br/>(or file)")]
Redis[("Redis cache")]
Kafka[("Kafka<br/>events")]
RocketMQ[("RocketMQ<br/>delayed")]
Analytics["analytics-consumer"]
Notification["notification-consumer"]
RabbitMQ[("RabbitMQ<br/>reanalysis.jobs")]
ReanalysisWorker["reanalysis-worker"]
MinIO[("MinIO<br/>S3-compatible")]
Triton["Nvidia Triton<br/>ONNX sentiment"]
Prometheus["Prometheus"]
Grafana["Grafana"]
Browser -->|HTTP| Nginx --> Gateway
Gateway -->|Kitex RPC, Thrift, TTHeader, mux| PostSvc
Gateway -->|Kitex RPC| AuthSvc
PostSvc --> PG
PostSvc --> Redis
PostSvc --> Kafka --> Analytics
PostSvc --> RocketMQ --> Notification
Gateway -->|enqueue| RabbitMQ --> ReanalysisWorker -->|Kitex RPC| PostSvc
PostSvc <--> MinIO
PostSvc <--> Triton
Prometheus -.->|scrapes /metrics| Gateway
Prometheus -.-> PostSvc
Prometheus -.-> AuthSvc
Grafana --> Prometheus
Diagrams for the pieces that are easier to follow as a picture β the full component graph, the request lifecycle through ABAC/RPC, the post-creation event fan-out, the error-propagation path from a typed service error to an HTTP response, the ABAC decision flow, and how the same six service images map onto Docker Compose vs. Kubernetes vs. Terraform β are all in ARCHITECTURE.md.
Every service is independently deployable (its own Dockerfile stage, its own Kubernetes Deployment, its own /metrics and health endpoints) but all six binaries live in this one repo/module, built via the same shared Dockerfile (ARG SERVICE=...).
Not part of the local stack (no cloud credentials are used by this repo), but the Terraform is structured for it: the AWS module includes an optional CloudFront distribution (enable_cdn), the Azure module an optional Front Door profile (enable_cdn), and the GCP module an optional Cloud CDN-fronted load balancer (enable_cdn), all off by default since their origin is the ingress LoadBalancer hostname/IP that only exists after a real cluster is stood up and deployments/k8s/ is applied to it β a genuine two-phase deploy, not something fakeable ahead of time. All three cache static UI assets at the edge while explicitly excluding /api/* from caching, since those responses are per-user and ABAC-gated. OCI has no equivalent first-party resource in its Terraform provider; see the comment in deployments/terraform/modules/oci/main.tf for the recommended path (a third-party CDN in front of the OCI Load Balancer).
| Concern | Technology |
|---|---|
| RPC framework | Kitex β Thrift IDL, TTHeader transport, connection multiplexing |
| HTTP framework | Hertz β built on Netpoll |
| JSON | Sonic (via Hertz/Kitex) |
| IDL | Thrift (idl/thrift/*.thrift) for RPC contracts, Protobuf (idl/proto/events.proto) for message-bus payloads |
| Auth | JWT (golang-jwt/v5) + custom ABAC policy-decision-point (internal/abac), default-deny |
| Database | PostgreSQL (works against any local Postgres β see Configuration) or flat-file JSON store |
| Cache | Redis, with automatic in-memory fallback if unreachable |
| Messaging | Kafka, RabbitMQ, RocketMQ β see why three brokers |
| Object storage | MinIO (local, S3-API-compatible β no cloud storage dependency) |
| ML | Nvidia Triton Inference Server (CPU), serving a trained TF-IDF + LogisticRegression ONNX sentiment classifier |
| Edge / LB | nginx (resolver-based upstream so Compose scaling load-balances) |
| Observability | Prometheus + Grafana (provisioned dashboard), structured JSON logging, request IDs |
| API surface | REST (OpenAPI 3.0.3, api-docs.yaml) + Kitex RPC + CLI + REPL + a small JSON surface backing the built-in web UI |
| Containers | Docker, Docker Compose (17-container full stack) |
| Orchestration | Kubernetes β Kustomize base + dev/staging/prod/local-kind overlays, Ingress, Argo Rollouts (canary/blue-green), Istio mesh manifests |
| IaC | Terraform modules for AWS (EKS/RDS/ElastiCache/ECR/CloudFront), Azure (AKS/Postgres Flexible/Redis Cache/ACR/Front Door), OCI (OKE/PSQL/OCIR), and GCP (GKE/Cloud SQL/Memorystore/Artifact Registry/Cloud CDN) |
| CI | GitHub Actions β lint, unit + integration tests, per-service build matrix, per-service Docker image matrix, Trivy security scan |
Each broker was picked for a distinct pattern it's actually good at β not "one topic per broker" for its own sake:
- Kafka (
post.events) β a durable, replayable, multi-consumer event log.postsvcpublishes on every create/update/delete/analyze;analytics-consumeris one subscriber today, but the topic is designed so other consumers (a warehouse loader, a search indexer) could subscribe independently withoutpostsvcknowing or caring. - RabbitMQ (
reanalysis.jobs) β a classic work queue.POST /api/v1/posts/reanalyzeenqueues a job;reanalysis-workeris a competing consumer that acks/nacks-with-requeue. This is a task to be done exactly by one worker, not an event for many subscribers β the wrong fit for Kafka. - RocketMQ (
post-notifications) β native delayed/scheduled delivery. On post creation, a "recheck this post" notification is scheduled ~10s out using RocketMQ's delay-level messages, something Kafka and RabbitMQ don't support natively without extra machinery (a DLX-based delay queue, a scheduler polling a table, etc.).
See ARCHITECTURE.md for the full fan-out sequence diagram.
Requires Docker + Docker Compose. Nothing else needs to be pre-installed to run the full stack.
git clone https://github.com/hoangsonww/Post-Analyzer.git
cd Post-Analyzer
make docker-up # or: docker compose up -dThis starts all 17 containers: Postgres, Redis, Kafka, RabbitMQ, RocketMQ (nameserver + broker), MinIO, Nvidia Triton, nginx, Prometheus, Grafana, and all 6 app services (gateway, postsvc, authsvc, analytics-consumer, reanalysis-worker, notification-consumer). First run pulls the Triton image (~13GB) and can take a few minutes.
Once healthy:
| Surface | URL |
|---|---|
| Web UI | http://localhost/ |
| Analytics dashboard | http://localhost/dashboard |
| REST API | http://localhost/api/v1/ |
| Prometheus | http://localhost:9090 |
| Grafana | http://localhost:3000 (admin/admin) |
| MinIO console | http://localhost:9001 (minioadmin/minioadmin) |
| RabbitMQ management | http://localhost:15672 (guest/guest) |
Demo accounts (seeded on authsvc startup), three distinct ABAC roles β see internal/abac/policy.go:
| Username | Password | Role | Can |
|---|---|---|---|
admin |
admin123 |
admin | Everything, including /api/v1/admin/status |
editor |
editor123 |
editor | Create/update posts; delete requires X-MFA-Verified: true |
viewer |
viewer123 |
viewer | Read-only |
The default .env/Compose config uses its own Postgres container, but nothing here is hardcoded to that β point DB_HOST/DB_PORT/DB_USER/DB_PASSWORD/DB_NAME (see .env.example) at any Postgres instance you already have running locally and make run (or go run ./cmd/gateway) will create its schema there on first connect via internal/migrations.
The gateway binary supports all of these from the same executable:
go build -o post-analyzer ./cmd/gateway
./post-analyzer # server mode (default) β HTTP + drives RPC to postsvc/authsvc
./post-analyzer login admin admin123 # one-shot CLI
./post-analyzer posts list
./post-analyzer posts create --title "Hello" --body "World" --user-id 1
./post-analyzer sentiment "this is great"
./post-analyzer repl # interactive REPL β same command tree, one processRPC (Kitex/Thrift/TTHeader) is what the gateway itself speaks to postsvc/authsvc internally β see internal/rpcclient for the plain-Go interfaces that hide this from the rest of the gateway code.
Success (β green), errors (β red), table headers (bold), IDs (cyan), timestamps (dim), and sentiment labels/bars (green/yellow/red by class) β colored automatically when stdout is a real terminal, and automatically off when it isn't (piped to a file, redirected in a script, captured by | tee, running in most CI log viewers) or when NO_COLOR is set. --no-color forces it off explicitly. The REPL also opens with a small banner:
_____ __ __
| __ \ /\ \ / /
| |__) / \ \ /\ / /
| ___/ /\ \ \/ \/ /
| | / ____ \ /\ /
|_| /_/ \_\/ \/
connected to http://localhost:8080
Type "help" for commands, "exit" to quit.
post-analyzer>
Real captures (also in Screenshots):
CLI β login, list, create, sentiment, error handling ![]() |
REPL β banner, login, list, sentiment, help ![]() |
home.html is a small SPA-ish page (vanilla JS, no framework, no CDN dependencies β even the character-frequency bar charts are hand-rolled, not Chart.js) backed by a set of JSON endpoints under /web/*:
| Method | Path | Does |
|---|---|---|
GET |
/web/posts |
List, with ?search=&sortBy=&sortOrder=&page=&pageSize= β all pushed down to postsvc, not filtered client-side |
GET |
/web/posts/{id} |
Fetch one (prefills the edit modal) |
POST |
/web/posts |
Create |
PUT |
/web/posts/{id} |
Update |
DELETE |
/web/posts/{id} |
Delete |
These are deliberately separate from the ABAC-gated /api/v1 surface β same trust level as the pre-existing /add form post, a convenience surface for the bundled UI rather than a public API contract. External integrations should use /api/v1 (below), which requires a JWT and is subject to ABAC.
The dashboard (/dashboard) is a second, independent page: sign in with any of the three demo roles to see live analytics (character frequency, posts per user, posts by time of day, recent posts), with an additional system-status panel that only renders for admin (see the screenshots).
Every JSON response β success or error, /api/v1/* or /web/* β uses the same envelope shape:
{"data": {...}, "meta": {"requestId": "...", "timestamp": "..."}}
{"error": {"code": "NOT_FOUND", "message": "Post not found"}, "meta": {...}}Error messages are specific and safe to show a user, not a blanket "something went wrong" β code is a stable machine-readable string (NOT_FOUND, VALIDATION_FAILED, UNAUTHORIZED, FORBIDDEN, CONFLICT, SERVICE_UNAVAILABLE, INTERNAL_ERROR) and message is meant to be displayed as-is. Concretely:
- Wrong or unregistered login credentials β
401"invalid username or password"β not a generic "authentication required." GET/PUT/DELETEon a post ID that doesn't exist β404"Post not found"β not a500.- A missing/invalid JWT β
401"missing bearer token"or"invalid or expired token". - An ABAC denial β
403with the actual policy-evaluation reason (e.g."role 'viewer' has no write permission on resource 'post'"), not just "forbidden." - Anything genuinely unexpected (a database or transport failure) β
500"Internal server error", deliberately generic β the real cause is logged server-side, never echoed to the client.
This is enforced end-to-end, including across the Kitex RPC boundary: postsvc and authsvc classify their own errors with *errors.AppError (a Go type carrying an HTTP status, a machine code, and a safe message), internal/adapt.Err encodes that onto the Thrift BaseResp (status in StatusCode, message in StatusMessage, code in Extra["code"]), and internal/rpcclient decodes it back into the exact same typed error on the gateway side β see ARCHITECTURE.md for the full sequence diagram. A raw, unclassified error (a driver/network failure) is never echoed across that boundary β only logged.
flowchart LR
subgraph cmd["cmd/ β 6 entrypoints"]
gateway["gateway"]
postsvc["postsvc"]
authsvc["authsvc"]
ac["analytics-consumer"]
rw["reanalysis-worker"]
nc["notification-consumer"]
end
subgraph internal["internal/"]
direction TB
abac_["abac β policy engine, JWT, demo users"]
adapt_["adapt β Thrift β domain model + error mapping"]
api_["api β REST handlers + router"]
bootstrap_["bootstrap β storage/migration wiring"]
cache_["cache β Redis + in-memory"]
cli_["cli β Cobra CLI/REPL + serve.go"]
export_["export β JSON/CSV writers"]
handlers_["handlers β web UI + /web/* JSON API"]
messaging_["messaging β kafka/, rabbitmq/, rocketmq/"]
metrics_["metrics β Prometheus collectors"]
middleware_["middleware β ABAC, rate limit, compression, ..."]
ml_["ml/triton β KServe v2 HTTP client"]
objectstore_["objectstore β MinIO wrapper"]
rpcclient_["rpcclient β Kitex client β plain Go interfaces"]
service_["service β PostService business logic"]
storage_["storage β Postgres + file backends"]
end
subgraph idl["idl/"]
thrift_["thrift/ β PostService, AuthService"]
proto_["proto/ β Kafka/RocketMQ payloads"]
end
subgraph deploy["deployments/"]
k8s_["k8s/ β Kustomize + Rollouts + Istio"]
tf_["terraform/ β AWS Β· Azure Β· OCI Β· GCP"]
infra_["nginx/, grafana/, triton/, rocketmq/"]
end
cmd --> internal
internal --> idl
cmd -.-> deploy
kitex_gen/ generated Kitex/Thrift code (committed)
assets/ dashboard.html, home.html static assets, UI screenshots
postman/ Postman collection + environment
api-docs.yaml OpenAPI 3.0.3 spec
docker-compose.yml full local stack (17 containers)
Dockerfile single parameterized multi-stage build (ARG SERVICE=...)
.dockerignore keeps build contexts to "what a Go build needs" (no .git, no Terraform state)
index.html, sitemap.xml static marketing/landing page for the project (not served by the app itself)
Makefile
Everything is environment-variable driven β the full, authoritative list lives in .env.example. Every optional integration (Redis, each message broker, MinIO, Triton) degrades gracefully when disabled or unreachable: the service logs a warning and the dependent feature returns a clear error/503 instead of failing startup.
Server & environment
| Variable | Default | Purpose |
|---|---|---|
PORT / HOST |
8080 / 0.0.0.0 |
Gateway listen address |
ENVIRONMENT |
development |
development, staging, or production |
READ_TIMEOUT / WRITE_TIMEOUT / IDLE_TIMEOUT |
15s / 15s / 60s |
HTTP server timeouts |
SHUTDOWN_TIMEOUT |
30s |
Grace period for in-flight requests during shutdown |
Database
| Variable | Default | Purpose |
|---|---|---|
DB_TYPE |
file |
file (flat JSON, zero setup) or postgres |
DB_FILE_PATH |
posts.json |
Used when DB_TYPE=file |
DB_HOST / DB_PORT / DB_USER / DB_PASSWORD / DB_NAME |
localhost / 5432 / postgres / postgres / postanalyzer |
Used when DB_TYPE=postgres β point this at any local Postgres |
DB_SSL_MODE |
disable |
Postgres SSL mode |
DB_MAX_CONNS / DB_MIN_CONNS |
25 / 5 |
Connection pool bounds |
Security & auth
| Variable | Default | Purpose |
|---|---|---|
JWT_SECRET |
dev-only-change-me-in-production |
Set a real value in production. |
JWT_TTL |
24h |
Token lifetime |
ADMIN_USERNAME / ADMIN_PASSWORD |
admin / admin123 |
Seeded admin account (editor/viewer demo accounts always seeded too) |
RATE_LIMIT_REQUESTS / RATE_LIMIT_WINDOW |
100 / 1m |
Per-client rate limit |
MAX_BODY_SIZE |
1048576 (1MB) |
Request body size cap |
ALLOWED_ORIGINS |
* |
CORS β comma-separated or * |
TRUSTED_PROXIES |
(empty) | Comma-separated IPs trusted to set X-Forwarded-For |
Logging
| Variable | Default | Purpose |
|---|---|---|
LOG_LEVEL |
info |
debug, info, warn, error |
LOG_FORMAT |
json |
json or text |
LOG_OUTPUT |
stdout |
stdout or a file path |
LOG_TIME_FORMAT |
2006-01-02T15:04:05Z07:00 |
Go time-format layout |
RPC & external HTTP
| Variable | Default | Purpose |
|---|---|---|
POSTSVC_ADDR / AUTHSVC_ADDR |
127.0.0.1:9001 / 127.0.0.1:9002 |
Kitex RPC targets |
RPC_MUX |
true |
Kitex connection multiplexing (one TCP connection, many concurrent calls) |
JSONPLACEHOLDER_URL |
https://jsonplaceholder.typicode.com/posts |
Source for "Fetch Sample Posts" |
HTTP_TIMEOUT |
30s |
Timeout for that outbound call |
Redis, Kafka, RabbitMQ, RocketMQ, MinIO, Triton (all optional)
| Variable | Default | Purpose |
|---|---|---|
REDIS_ENABLED / REDIS_ADDR / REDIS_PASSWORD / REDIS_DB |
true / 127.0.0.1:6379 / (empty) / 0 |
Cache β falls back to in-memory automatically if unreachable |
KAFKA_ENABLED / KAFKA_BROKERS |
false / 127.0.0.1:9092 |
Event log producer |
RABBITMQ_ENABLED / RABBITMQ_URL |
false / amqp://guest:guest@127.0.0.1:5672/ |
Reanalysis work queue |
ROCKETMQ_ENABLED / ROCKETMQ_NAMESRV_ADDRS |
false / 127.0.0.1:9876 |
Delayed notification delivery |
MINIO_ENABLED / MINIO_ENDPOINT / MINIO_ACCESS_KEY / MINIO_SECRET_KEY / MINIO_USE_SSL |
false / 127.0.0.1:9000 / minioadmin / minioadmin / false |
Export persistence |
TRITON_ENABLED / TRITON_URL |
false / http://127.0.0.1:8000 |
ML sentiment classification |
make test # unit tests (race detector + coverage), no external dependencies
make test-integration # storage integration tests against a real local Postgres
make postman-run # Postman/newman collection against a running stack (BASE_URL=http://localhost by default)
make lint # golangci-lint (0 issues)
make check # vet + lint + test + gosec β full pre-commit gateRepresentative per-package unit-test coverage (go test ./... -cover), all with -race clean and golangci-lint at 0 issues:
| Package | Coverage | Notes |
|---|---|---|
internal/errors, internal/logger |
100% | Pure logic, fully deterministic |
internal/adapt |
~87% | Includes the typed-error β BaseResp round trip |
internal/handlers |
~84% | Web UI handlers + the /web/* JSON API |
internal/ml/triton |
~89% | Against a fake KServe v2 HTTP server |
internal/api |
~76% | REST handlers, via fake RPC clients |
internal/middleware |
~78% | Includes gzip round-trip correctness (compress β decompress β byte-identical) |
internal/rpcclient |
Partial | Pure-function error-mapping logic (rpcErr) is unit-tested; the Kitex-dialing code itself is exercised via the live docker-compose stack, not mocked |
Integration- and infrastructure-dependent code (internal/messaging/*, internal/objectstore, the RPC dial paths, cmd/* wiring) is deliberately verified against the real local stack (docker compose up) rather than mocked β see Testing commands above and the CI workflow for how that's automated.
- Container images β CI builds and pushes all six service images to GHCR on every push (
ghcr.io/hoangsonww/post-analyzer-{gateway,postsvc,authsvc,analytics-consumer,reanalysis-worker,notification-consumer}, tagged by commit SHA + branch,lateston the default branch). Pull withdocker pull ghcr.io/hoangsonww/post-analyzer-gateway:latest. - Docker Compose β
make docker-up/docker compose up -d, the full local stack. - Kubernetes β
deployments/k8s/base(Kustomize) withoverlays/{dev,staging,prod,local-kind}.make k8s-apply-devetc. Includes an Ingress, aprodoverlay with PodDisruptionBudget + HorizontalPodAutoscaler, Argo Rollouts manifests for canary/blue-green (deployments/k8s/rollout/), and Istio mesh manifests (deployments/k8s/mesh/). - Local kind cluster β
make kind-create && make kind-load && make k8s-apply-local-kind. - Terraform (AWS / Azure / OCI / GCP) β
deployments/terraform/environments/{aws,azure,oci,gcp}, each provisioning the managed equivalent of the local stack (managed Kubernetes, managed Postgres, managed Redis, container registry, optional CDN).make tf-validatevalidates all four against their real provider schemas. Not applied by default β no cloud credentials are used by this repo; see each environment's*.tfvars.example.
See ARCHITECTURE.md for how the same six service images flow into all four targets, and how the Kubernetes/Terraform artifacts were actually validated (a real local kind cluster, terraform validate against live provider schemas).
- OpenAPI 3.0.3:
api-docs.yaml(validated withopenapi-spec-validator) β covers/health,/readiness,/metrics,/api/v1/auth/login,/api/v1/posts(list/create),/api/v1/posts/{id}(get/update/delete),/api/v1/posts/bulk,/api/v1/posts/export,/api/v1/posts/analytics,/api/v1/posts/reanalyze,/api/v1/exports,/api/v1/exports/{key},/api/v1/ml/sentiment,/api/v1/admin/status. - Postman collection:
postman/Post-Analyzer.postman_collection.json+postman/Post-Analyzer.postman_environment.jsonβ 27 requests covering CRUD, bulk ops, export, analytics, ABAC role checks (viewer/editor/admin), ML sentiment, and object-storage export listing/download. Runs clean viamake postman-run. - Bruno collection:
bruno/β the same 27 requests as plain-text.brufiles, for Bruno (an open-source, offline Postman alternative). See POSTMAN_BRUNO.md for how to use either one, with example requests.
Run make help for the live list. Grouped summary:
| Group | Targets |
|---|---|
| Build | build, build-all, run, repl, dev |
| Codegen | thrift, proto, generate |
| Test & quality | test, test-integration, test-coverage, bench, lint, vet, format, security, check |
| API validation | postman-run, openapi-validate |
| Docker | docker-build, docker-up, docker-down, docker-logs, docker-restart, docker-ps, docker-scale-gateway |
| Kubernetes | k8s-apply-dev, k8s-apply-staging, k8s-apply-prod, k8s-apply-local-kind, k8s-delete, k8s-status, k8s-rollout-status, k8s-rollback |
| kind | kind-create, kind-load, kind-delete |
| Terraform | tf-init, tf-plan, tf-validate |
| Misc | migrate, db-shell, clean, version, install, install-tools, init |
Distributed under the MIT License. See LICENSE for details.
Created by Son Nguyen.











