Skip to content

Integrate and qualify Inspect AI on current 0.7 (Pierre handoff) #342

Open

Description

@ebursztein

The user has transferred full Inspect AI integration to Pierre (@tholop). This work is removed from the current agent's active Sprinty scope. This issue owns the integration and its qualification; the partial carry is not a finished sandbox.

Base and source to reuse

Work against the shared 0.7 branch. Commit 86de47d77d5de45417b78c1955a2f056f4a4c337 merges current main b862c89d8363fb2ffa09a870378122aa3e8fb532 and all the committed OCI/client work through 224130ed1. Fetch and pull the latest shared branch before integration; the normal push to shared 0.7 has completed.

The repository already has these implementation branches. Inspect their ancestry/diffs and integrate them in this order, resolving against current 0.7 rather than blindly duplicating the old integration:

  1. feat(inspect-ai): add VM-mode Capsem SandboxEnvironment integration #338, feat/inspect-capsem, tip 6f9656f7b: VM SandboxEnvironment, controller/exec/files/lifecycle/transfer, tools, registration/package metadata, SDK/gate/CI/docs routing and tests.
  2. feat(inspect-ai): add OCI container execution mode and Compose parser #339, feat/inspect-capsem-containers, tip 99a088b39: OCI execution mode, Compose parser/fields, workload controller, file/exec targeting and acceptance.
  3. feat(inspect-ai): add host-side Dockerfile and Compose build image support #340, feat/inspect-capsem-host-build, tip da3aea14c: host Dockerfile/Compose image builder and OCI registry handoff candidate. This source is a candidate, not authorization for implicit host execution/credentials/network access.

Original full design/source: #291, commit bb61fc82d44bc42c978c4acf5435a1975600c75e. Preserve Pierre's source provenance when adapting it. Related tracking: #310 (VM/workload), #311 (full Dockerfile/Compose), parent #289.

What is already on 0.7

integrations/inspect-ai/ currently contains pure Dockerfile and Compose components, without an installed package/Inspect registration:

  • Dockerfile instruction scanning, heredoc handling/lowering, multistage CA patching, final-stage USER/WORKDIR metadata. Carry commit 1cec4f2c9.
  • Compose single-service field extraction, interpolation/.env behavior, build context/args/target, command/entrypoint/user/workdir, mounts/ports, healthcheck/init/memory/network intent. ComposeInputs supplies immutable evaluator-selected text/environment/directory facts; ComposeLimits bounds bytes/nodes/depth and YAML/merge/alias/interpolation expansion. Carry commit f7a697c26.
  • These overlaps must be reconciled with the newer branches above, retaining the input bounds and redaction instead of creating competing parser implementations.

The Python and TypeScript SDKs already expose Hypervisor.images.list/pull and explicit VM.exec targets. Python target commit 9c6a0b973, TS 224130ed1. Existing service APIs admit/pull immutable OCI images, create workloads and execute as the workload user/cwd/environment; explicit target=vm is for VM operations. Current file APIs are workspace-scoped with a 10 MiB request/file limit. Ordinary client VM.close() closes a connection; it does not destroy a VM.

Integration order

  1. Reconcile VM-mode controller, configuration, SandboxEnvironment registration and packaging from the VM branch against current generated SDK DTOs. Use the SDK/gateway, explicit IDs, and evaluator-selected inputs.
  2. Complete owned-session cleanup and target-correct exec/file operations. Attached/named/unrelated sessions must survive cleanup. Exercise timeout, cancelled creation/lost responses, nonzero execution and process death. Internal managed-ownership primitives exist, but no new claim/close endpoint or lease policy was approved.
  3. Integrate native OCI/Compose launch behavior against actual ContainerSpec and ExecRequest capabilities. Preserve real refusal/cancellation behavior; no retry into VM-root exec after workload denial.
  4. Complete the selected Dockerfile/Compose build path, with a reviewed execution/context/network/registry/credential authority boundary. Reuse the host-build candidate if appropriate, but do not infer permission from the presence of its source. For a guest builder, ordinary runc workloads currently deny namespace/mount delegation and do not ship Docker/BuildKit; installing an engine alone does not establish a supported builder.
  5. Connect the package, docs and existing CI/gate owners, then qualify the entire installed integration on both native architectures. Coordinate genuine missing native controls with the service owner and regenerate shared DTOs/SDKs together where required.

Confirmed image/cache scope from the user

images.list is the registry catalog: images potentially available from the configured registry. It may include truthful local disk status and a VM-use count. Cache verification, reuse/download and protecting data referenced by running VMs remain service-owned through existing pull/create paths.

Do not make Inspect integration depend on separate image-cache inventory/removal endpoints, cache heartbeats, or a public cache-only pull mode. The old combined four-route cache/managed-session proposal is withdrawn from this scope. A cached image never substitutes for admission of its source/resolved digest.

Required behavior and proof

Preserve the full integration rather than closing this with an image-only subset:

  • VM and OCI exec, files, tools, readiness, resource settings, owned cleanup and exact error/output semantics.
  • Actual Docker frontend build behavior: context/.dockerignore, build args/target, FROM/scratch/multistage/COPY --from, shell/JSON RUN and heredocs, COPY/ADD ownership/mode, USER/WORKDIR/ENV/ENTRYPOINT/CMD.
  • Compose precedence/interpolation and explicit overrides; bind files/directories, named volumes, protocol/port handling, healthcheck, init, memory and network modes. Preserve documented refusals for unsupported shapes.
  • CA bundle preservation, stage propagation, browser/NSS trust where applicable, CA-rotation/corruption cache invalidation, and build-result reuse.
  • Evaluator controls all host inputs and credential/network grants. No ambient secret discovery, global policy edits, host Docker socket mounts, private runc/bundle/cgroup rewrites, or deletion by guessed VM name.
  • Real installed Inspect task proof through the authenticated gateway and workload, with functional output, cleanup/resource evidence and ledger attribution. Parser tests and mocked controller tests prove their components only. Acceptance is hermetic, not an optional public-network/live flag.

During implementation use TDD and minimal focused local tests. No global machine lock or repeated broad workspace/VM/install/benchmark runs. Full qualification belongs at the major phase boundary and before release. Current 0.7 is development source: phase/runtime/release qualification is still outstanding.

Focused checks for the currently carried helpers:

python3 build_system/scripts/ci/run-bounded-command.py --timeout-seconds 60 -- \
  env PYTHONPATH=integrations/inspect-ai uv run --project build_system --frozen \
  python -m pytest -c build_system/pyproject.toml --rootdir . \
  integrations/inspect-ai/tests/containers -q

Expected closeout: one working installed integration on current 0.7, coherent source/provenance, native/packaged parity for the required features, and complete recorded end-to-end qualification.

Activity

  1. tholop commented on Oct 9, 2026

    @tholop
    Collaborator

    Update: all 6 PR branches in the stack are rebased onto current 0.7 (#291 is now closed in favor of #342 and this stack). The 3 service/SDK prerequisite PRs (#341, #336, #337) are ready for review first, while the 3 inspect-capsem integration PRs (#338, #339, #340) are pushed as Draft so the prerequisites can land on 0.7 first:

    1. Service/SDK prerequisites on 0.7 (ready for review, land first):
    2. Inspect AI integration stack (pushed as Draft, ready once feat(service,sdk): attach labels to VMs at create and expose them in list #336/feat(service,sdk): add typed ErrorCode, timeout errors, and gateway discovery #337 land):
    3. CI / platform notes:
      • linux/x86_64 installed-package qualification passed across wheel and sdist (SDK_IMAGE_PACKAGE_ACCEPTANCE_OK, INSPECT_CAPSEM_CONTAINER_ACCEPTANCE_OK, INSPECT_CAPSEM_HOST_BUILD_ACCEPTANCE_OK, INSPECT_CAPSEM_VM_ACCEPTANCE_OK, 0 leaked sessions).
      • darwin/arm64 qualification is queued on the macos-15 CI lane from this push.
      • Known red on 0.7: test-install fails across 0.7 PRs because the stable release manifest has no runtime entry yet.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions