A test harness for UI-driven integration: driving an app the way a person does, for systems that expose no API. Built by reverse-engineering the approach behind Codex Computer Use, not its binary. The harness comes first, on purpose: it tells us with numbers where the approach holds and where it breaks, before we build the full system or point it at a real EHR.
A generic integration platform: the API for software that has no API. Teach it any application's screen once, and it exposes that workflow as a callable API. Any system a person can operate -- a legacy ERP, a bank or insurer portal, a government site, a logistics tool, a CRM, an old terminal app -- becomes integrable, with no cooperation from the vendor.
The core is domain-neutral. It captures a screen, acts on it, learns a flow, replays it with no model, heals it on change, exposes it as an API, and measures its own reliability. Nothing in the core knows about any industry.
Domains are applications on top. Healthcare is the first: a pack that maps flow output to FHIR and redacts patient data, with EHR flow libraries. Finance, logistics, and others plug in the same way. Platform first, applications on top.
- API auth and allowlist. The HTTP and MCP servers refuse every call unless a
bearer token (
UIA_API_TOKEN) matches, and an optionalUIA_ALLOWrestricts which flows may run. Closed by default: with no token set, the server refuses rather than running open. Proven live: no token and wrong token return 401, the right token returns data. - Secrets vault. Credentials live in an encrypted local vault
(
engine/secrets.py), keyed byUIA_VAULT_KEY, and inject into a step via{{secret:name}}at the last moment. A secret never appears in a flow file, the audit log, or a trace. The stdlib cipher protects at rest; for production pointUIA_VAULT_KEYat a key from your secrets manager. - Trace and audit redaction. Before anything is written,
engine/redaction.pymasks known sensitive shapes (long digit runs, emails, id/SSN shapes, tokens) and any injected secret value. A domain pack adds its own field rules on top. This is a safety net, not a guarantee every sensitive value is caught.
The core, the browser (CDP) driver, the flow API, and the MCP server are pure Python and run on macOS, Windows, and Linux with no extra dependency. Desktop control (native app windows) is per-platform:
| Capability | macOS | Windows | Linux |
|---|---|---|---|
| Browser control (DOM over CDP) | yes | yes | yes |
| Flow-as-API + MCP server | yes | yes | yes |
| Learn / replay / heal / measure | yes | yes | yes |
| Desktop app control | yes (built in) | yes ([windows] extra) |
planned |
| Vision path (screenshot) | yes | planned | planned |
pip install nirmitee-control # core + browser + API + MCP (all platforms)
pip install nirmitee-control[windows] # adds the Windows desktop adapterOn Windows the desktop adapter uses Windows UI Automation via the uiautomation
package; the same Skyshot/Action contract, so flows, drivers, and the suite are
identical across platforms. See PUBLISHING.md to release to PyPI.
A learned flow becomes a single call. The engine is generic: a flow is just steps, so the same mechanism turns any no-API system into one that answers a call.
Single HTTP API:
python3 -m server.http_api --flows ./flows --target macos --port 8077
curl -s localhost:8077/flows/get_record -d '{"id":"123"}'
# -> {"name":"get_record","status":"ok","reads":[...],"steps":2}As an MCP tool (any agent can call it): each saved flow appears as flow.<name>.
python3 -m server.mcp_server --flows ./flows --target macosProven live on the fixture target: tools/list shows the flow, tools/call runs it
and returns structured reads. Parameters fill {{placeholders}} in a flow's steps.
git clone https://github.com/Nirmitee-tech/uia-bench && cd uia-bench
swiftc -O native/skyshot/Skyshot.swift -o native/skyshot/skyshot # macOS
python3 bench.py # 14 test classes, should be green
python3 scan.py "Finder" # read a live app's control treeTeach a flow with any model, then replay it forever with no model:
export OPENAI_API_KEY=... # or point UIA_LOCAL_BASE at a local model
python3 uia.py learn "Finder" "open Applications" --driver openai --out flow.json
python3 uia.py replay flow.json # no model, deterministic- Drive any app: see the control tree (accessibility), see pixels (vision), and act with the full mouse and keyboard, foreground or background.
- Drive any web page by its real DOM over Chrome DevTools Protocol. No browser framework dependency.
- Model-agnostic: OpenAI, Anthropic, a local model, vision, or auto. Swap freely.
- Learn once, replay with no model, self-heal when a screen changes.
- Open, auditable action-risk policy with a human-approval gate and an audit log.
- A run trace: every flow writes a JSON record and an HTML evidence report.
- A measurement suite: fourteen test classes, including a PHI-containment proof.
- Standard library only. MIT licensed.
Most healthcare value sits behind systems with no API: legacy EHRs, thick clients, state portals. The only door is the screen a human uses. The effective trick, confirmed from the teardown, is the skyshot: a screenshot fused with the accessibility tree (the OS's named list of on-screen controls), every control given a stable index. A model clicks "the Labs button," not a guessed pixel.
Because actions anchor on element identity (role + label), a flow recorded once replays with no model. The model returns only to repair a flow when a screen changes. That record / replay / heal loop is the production path and the moat.
native/skyshot/Skyshot.swift— compiles and runs. It walksAXUIElementon a live window and emits the control tree as JSON. Confirmed against Finder: 81 controls, 22 actionable. This is the real skyshot, read-only.bench.py— the seven test classes run green on a deterministic fixture.scan.py— runs perception against any live Mac app and reports AX coverage.
native/skyshot/Skyshot.swiftreads a live AX tree AND performs real input via CGEvent (act click|type|key). Confirmed: typed text into TextEdit end to end.engine/llm.pyis a real model client (stdlib only) speaking OpenAI-compatible chat and Anthropic messages. Point it at OpenAI, Codex, Anthropic, or a local server (Ollama/LM Studio/vLLM) by env. The reply parser refuses any control not on screen, so a model never clicks something that is not there.engine/driver.pymakes the teacher swappable data, not code: openai / anthropic / local drivers share one contract. The learn loop never names a model.uia.pyis the CLI: bench, scan, drivers, learn, replay.bench.pyruns nine test classes green, including a real parser-guard test.
The executor drives any application's keyboard and mouse. Verbs: move, click,
rclick (right), dclick (double), drag, scroll, type, and key with modifier chords
(e.g. a+cmd for select-all, Return, Tab, arrows). All through real CGEvent.
Two delivery modes:
- Foreground (default): events go to the shared input stream and drive whatever is frontmost, exactly like a person. Simple and universal.
- Background (
MacosAdapter(background=True), CLI--bg): events are delivered to the target app's process with CGEventPostToPid. The app receives the input without the hardware cursor moving and without stealing your focus. Proven: text typed into a background TextEdit while Chrome stayed frontmost.
Honest limit on isolation: background mode avoids moving your cursor and taking focus, and works for most Cocoa apps. It is not perfect isolation -- some apps only accept input when active, and a shared global shortcut can still fire. For a hard guarantee that automation never touches your live session, run the target in a separate space: a second macOS user (launchctl asuser), a VM (UTM/Parallels), or a virtual display. The framework runs the same in any of those. For an RDP or Citrix EHR this is the recommended setup: the remote window lives in its own VM or session, and the automation drives it there.
For web apps, the framework talks to Chrome's DevTools Protocol directly, so it acts on the page's real element tree rather than pixels -- the most reliable and fastest web path, the open-source equal of what Codex does with Playwright. Proven live: connected to a real web EHR, read 25 patient rows and a patient name straight from the DOM. Standard library only.
Launch Chrome once with remote debugging, then use engine/browser_cdp.py:
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" \
--remote-debugging-port=9222 --user-data-dir=/tmp/uia-chromeSee ROADMAP.md for the full feature plan derived from the Codex teardown.
For screens with no accessibility tree -- a remote desktop (RDP/Citrix), a canvas app, a game -- there are no named controls. The framework captures the window as an image and asks a vision model for a click coordinate, then clicks through the same CGEvent executor.
engine/vision.py: captures the window (screencapture + sips), normalises the image to point size so pixels map 1:1 to screen points, calls a vision model, and maps the returned pixel to a global screen point. Proven live: a 1512x982 window captured and normalised; coordinate math covered by the vision_parse test.AutoDriverfuses both: it uses element grounding when the tree is present and the vision path when it is empty. One driver reaches a native app AND a remote desktop. Drivers: openai, anthropic, local, vision, auto.
This is the piece that lets the system drive an RDP/Citrix EHR, where no API and no accessibility tree exist. It needs a vision-capable model (OpenAI-compatible, or a local one via UIA_LOCAL_BASE to keep PHI on the machine).
- A captured ScreenCaptureKit frame alongside the tree (vision-only path).
- Model-driven heal for genuinely ambiguous cases (today's healer is heuristic).
| Class | Checks | Bar (fixture control case) |
|---|---|---|
| perception | skyshot lists the target control | 100% recall of gold labels |
| grounding | resolves the right control, not a neighbor | 100% |
| task_success | 3 read-only tasks end to end | 20/20 runs |
| replay_determinism | record once, replay model-off | 20/20 identical |
| heal | rename a control, auto re-resolve and re-save | recovers |
| phi_containment | full task runs with zero outbound network | 0 egress |
| latency | time per step | < 50 ms/step |
Fixture bars are strict because the fixture is deterministic. Real EHR bars get set from the first measured baseline, not guessed.
python3 bench.py # the suite, green on the fixture
swiftc -O native/skyshot/Skyshot.swift -o native/skyshot/skyshot
python3 scan.py "Finder" # AX coverage of a live appNeeds Node-free Python 3.9+, and macOS Accessibility permission for the caller (System Settings > Privacy & Security > Accessibility) to scan a live app.
The phi_containment class proves the harness made no network call during a
task, so the local-first claim is testable, not a promise. It is not a kernel
firewall; a production deployment still needs an OS-level egress block. The test
documents that boundary rather than overclaiming. The customer owns HIPAA and
GDPR sign-off; this harness exists so that sign-off rests on measured behavior.
bench.py suite runner, exit 0 only if every class passes
scan.py live-app AX coverage probe (read-only)
engine/
skyshot.py Skyshot + Element (identity = role::label)
actions.py Action model + Executor (identity-first, pixel fallback)
recorder.py record once, replay model-off, stability check
healer.py re-resolve a broken target and rewrite the recording
adapters/
fixture.py deterministic in-memory app (the control case)
macos.py live skyshots via the Swift CLI; action apply stubbed
tests/
suite.py the seven classes and their pass bars
gold/ hand-labeled gold screens (masked, synthetic)
sandbox/
egress_guard.py blocks + records outbound network for the PHI test
native/skyshot/ Swift AX dumper (compiles to ./skyshot)