Repository navigation
feat(inspect-ai,harbor): managed sessions, integrations/common, trace export and Harbor environment - #364
Draft
tholop wants to merge 6 commits into
Draft
feat(inspect-ai,harbor): managed sessions, integrations/common, trace export and Harbor environment#364tholop wants to merge 6 commits into
tholop wants to merge 6 commits into
Conversation
…n.gz
Add `GET /vms/{id}/export/rows.ndjson.gz?tables=` to stream a session's
SQLite ledger rows as gzipped NDJSON (`application/gzip` attachment,
single-member gzip stream) alongside the existing `export.warc.gz` body
archive:
- Line 1 emits a `capsem.rows.v1` header (`type`, `format`, `format_version`,
`vm_id`, `exported_at`, `tables`).
- Snapshots per-table max row IDs up front via a single-SELECT
`LedgerQuery::TableWatermarks` query and bounds paged `LedgerQuery::TableRows`
reads by `max_id` (`MAX_LEDGER_QUERY_ROWS = 2_000`, flushing `page_buf` at
`64 KiB` via `GzipChunkEncoder` in `capsem-archive`).
- Wraps multi-query `execute_query_many` batches in a read transaction on the
reader worker connection.
- Defaults to the 16 non-security ledger tables (`DEFAULT_EXPORT_LEDGER_TABLES`)
when `?tables=` is omitted, and accepts any subset of the 18 exportable
tables (`EXPORTABLE_LEDGER_TABLES`, joining `security_rule_runs` and
`security_decision_runs` when requested).
- Retains stopped lease-owned managed VM session directories across
`POST /vms/{id}/stop` until `POST /vms/managed/{request_id}/close` (or
`DELETE /vms/{id}/delete`) so callers can export both `rows.ndjson.gz` and
`bodies/export.warc.gz` after stopping a managed VM and before closing the
lease.
- Registers `exportVmRows` in OpenAPI (`crates/capsem-api`,
`sdk/specification/openapi.json`) and regenerates the Python, TypeScript, and
Rust SDK operation bindings.
Add VM.export (quiesce, rows, warc, events, atif) and capsem.trace helpers
(parse_rows_ndjson_gz, load_rows_tables, WarcBodyReader, rows_to_atif,
enrich_atif_trajectory) backed by GET /vms/{id}/export/rows.ndjson.gz and
GET /vms/{id}/bodies/export.warc.gz without sqlite3 or local session.db:
- Stream-decompresses `.rows.ndjson.gz` line-by-line via `gzip.GzipFile` and
writes exported files atomically via sibling tempfiles and `os.replace`.
- Downloads `rows.ndjson.gz` and `bodies/export.warc.gz` sequentially after
quiescing.
- Stops managed VMs during `quiesce_vm_for_export` (skipping `persist` while
disarming background lease renewal via `on_stop` before `POST /vms/{id}/stop`)
so the service drain barrier flushes and seals session telemetry before export.
Start each sample VM through capsem.Hypervisor.managed() and keep the ManagedSession lease in SdkCapsemController, so the service reclaims the VM when the lease ends (sample_cleanup, cancellation, or a crashed evaluator) instead of the adapter tracking process-owned VMs and sweeping leftovers by label. Remove _lifecycle.py, _cleanup.py and the leftover-sweep configuration; task_cleanup(cleanup=False) now raises NotImplementedError because a lease cannot outlive its owner. Sample VMs keep their inspect-capsem-task and capsem-vm-prefix labels, truncated to 255 UTF-8 bytes by truncate_label_value. Update the README, the usage docs and the tests accordingly.
…ions/common Move the framework-agnostic container pipeline (capsem_integrations.containers), guest execution and user helpers (_exec.py, _users.py), staged tar transfer (_transfer.py), and ManagedSession controller (SdkCapsemController and truncate_label_value in _controller.py) out of inspect_capsem into a shared uv project under integrations/common (capsem-integrations / capsem_integrations, no PyPI publish). Update inspect_capsem to depend on capsem-integrations, support neutral CAPSEM_ALLOWED_HOST_PATHS with CAPSEM_INSPECT_ALLOWED_HOST_PATHS fallback, wire integrations/common through PythonPackageConfig (pytest_scratch), capsem-gate SDK checks, Citadel boundary/surface/lockfile audits, fast-gate prewarm, CI scope classification, and Codecov.
…d ATIF trace export
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## integration/0.7.1-pierre-sdk-20261016 #364 +/- ##
=========================================================================
+ Coverage 69.4% 76.3% +6.9%
=========================================================================
Files 1878 1498 -380
Lines 154231 125487 -28744
Branches 114221 115027 +806
=========================================================================
- Hits 107037 95850 -11187
+ Misses 40349 22621 -17728
- Partials 6845 7016 +171
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Based on
integration/0.7.1-pierre-sdk-20261016(#362, which already includes #336–#340), so the diff shows only the commits below. Each commit is self-contained: merge the branch, cherry-pick single commits into #362, or ask for any of them as separate PRs.Summary
Running Inspect and Harbor benchmarks on Capsem needs three things the 0.7.1 branch doesn't have yet: sandboxes whose VMs cannot leak when an eval dies, a way to export what happened inside a VM, and a Harbor environment. This branch adds them:
feat(service): stream ledger rows viaGET /vms/{id}/export/rows.ndjson.gz(all non-security session tables, gzipped NDJSON).feat(sdk):vm.exportand ATIF trajectory conversion over the rows plus the existing WARC export.feat(inspect-ai): run sample VMs as managed sessions (Hypervisor.managed()). Optional design change, worth a look before Inspect 0.1.0 ships; commits 4–6 build on it, so skipping it means reworking the shared controller. What it buys over the_lifecycle.py/_cleanup.py/ label-sweep cleanup that feat(sdk): integrate Pierre stack and Inspect for 0.7.1 #362 carries from feat(inspect-ai): add VM-mode Capsem SandboxEnvironment integration #338–feat(inspect-ai): add host-side Dockerfile and Compose build image support #340:SIGKILL/OOM, whereatexitand signal handlers never run) no longer leaves VMs behind for a later label sweep or a manualinspect sandbox cleanup capsem.The cost is
inspect eval --no-sandbox-cleanup(task_cleanup(cleanup=False)), which now raisesNotImplementedError. That answers my question on #338; the option can come back if the hypervisor gains a way to turn a leased VM into a normal one.feat(integrations): move the framework-agnostic controller, exec, file transfer and container builds into a sharedintegrations/commonpackage (capsem_integrations) used by both Inspect and Harbor.feat(inspect-ai): ATIF-v1.7 trace export for Inspect.evallogs.feat(harbor):harbor_capsem:CapsemEnvironment, a Harbor environment on Capsem VMs and OCI containers, with credential brokering and ATIF export.Other branches from the same benchmark runs
Not part of this PR. They are independent of each other and of this stack, and each merges cleanly onto the integration branch:
feat/container-exec:userandcwdon exec requests (instead ofcd … && su … -cwrapping), plus avm.container.exec(...)helper.feat/files-multipart: Pythonvm.files.upload/downloadthat splits transfers above the 10 MiB request limit.fix/mitm-sse-streaming-abort: stream SSE model responses instead of buffering to EOF (longgemini-cliturns hit the client's 60 s header timeout), and record calls the guest aborted mid-stream.fix/provision-credential-sync-launch-wait:POST /vms/createcould kill a healthy VM that took longer than 500 ms to bind its owner socket.feat/vertex-ai-mitm: recognise Vertex AI hosts and paths as model traffic and broker their tokens (credential-sensitive, worth its own look).Notes for Review
About 15k added lines: tests ~35%, lockfiles ~29% (
uv.lockfor the two newuvprojects,integrations/commonandintegrations/harbor), SDK Python ~12% (export and ATIF conversion), Harbor ~8%, Inspect ~7% (it also deletes ~700 lines of lifecycle/cleanup code), Rust service/logger ~4% (the rows endpoint and keeping a stopped managed session's data until close), sharedintegrations/common~2%, and the rest gate config, docs and generated SDK operations.Testing
CI covers unit, lint and gate checks; it has no KVM. Live runs on a KVM host at this exact head:
tests/ironbank/test_sdk_live.py: the Inspect and Harbor live acceptance tests pass for both the wheel and the sdist, covering VM, OCI container and host-built image sandboxes and managed quiesce → export → close.--sandbox capsemand trace export:inspect_evals/vimgolf_single_turn(3 samples) andinspect_harbor/terminal_bench_2(fix-git) complete with no errors; every exported trajectory passes Harbor'strajectory_validator, with every agent step linked to its LLM call.terminal_bench_2fix-git:oracleandterminus-2solve the task, with valid trajectories.gemini-cliover Vertex AI with brokered credentials needs the Vertex support fromfeat/vertex-ai-mitmin the service; with that branch (and the SSE fix) merged in, it solves the task with a valid trajectory.No VMs leaked in any run.