Skip to content

feat(ci): route Linux API jobs and wheel builds to dedicated runner - #6806

Open
zhanglistar wants to merge 26 commits into
mainfrom
feat/ci-linux-pools-official
Open

zhanglistar wants to merge 26 commits into
mainfrom
feat/ci-linux-pools-official

Conversation

@zhanglistar

@zhanglistar zhanglistar commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Linux API jobs can stall behind exhausted hosted capacity while the dedicated runner is idle. Route the entire trusted Linux API dependency chain to iv-yezentk8ao3z47ge5tmm using the custom ci-linux-builds label, and alternate Linux x86_64 wheel builds between the two pools.

  • Linux wheels use the dedicated runner on even workflow run numbers. API change planning and native runtime preparation use it on every eligible run, independent of run number. Linux API/CLI, MySQL/Redis and cuVS tests inherit their same-run native producer's destination, so no trusted Linux API dependency waits for hosted capacity.
  • Only same-repository PRs and official branch pushes are eligible. Fork PRs, release/tag events, schedules and manual builds remain hosted. ARM, macOS, Windows, Studio, musl CLI and other test jobs keep their existing assignments.
  • Require CI_LINUX_BUILDS_ENABLED=true before routing these jobs to the dedicated runner. CI_DISABLE_SELF_HOSTED=true returns new workflow runs to hosted capacity; already queued jobs retain their labels.
  • Preserve Ubuntu 20.04/glibc 2.31 wheels and use Ubuntu 24.04 containers for API native preparation on both destinations. Self-hosted API consumers use disposable Ubuntu 24.04 containers, job-local HOME/Cargo/Rustup directories and ownership cleanup. Hosted consumers retain host execution. Database tests use service DNS in containers and loopback on hosts, with fresh MySQL 8.4 and Redis 6.2 services.
  • Download Ubuntu packages over verified HTTPS, bootstrapping minimal containers with the host's read-only CA bundle. CI_UBUNTU_MIRROR overrides package sources only on self-hosted builds/tests.
  • Add an explicit, five-minute readiness check to _build.yml that tests Docker, host/container connectivity and actual package installation without checking out repository code.
  • Stabilize the remote task cancellation contract by observing delivery separately from durable cleanup, with bounded waits. Keep cancellation, empty active-handle index and terminal task-state assertions; slow MySQL cleanup must not be mistaken for a cancellation failure.

Validation

  • 90 local runner-routing, packaging, release-selection and database CI contract tests passed, including inherited consumer routing, fork isolation, non-Linux assignments, container bootstrap/cleanup and service networking.
  • actionlint, Ruff and whitespace checks passed.
  • Dedicated runner package readiness: diagnostic run passed on iv-yezentk8ao3z47ge5tmm. Official Ubuntu endpoints timed out from both host and container; the configured HTTPS Aliyun mirror was reachable from both. Ubuntu 20.04 and 24.04 containers successfully downloaded indexes and installed Git and CA certificates. CI_UBUNTU_MIRROR=https://mirrors.aliyun.com/ubuntu and CI_LINUX_BUILDS_ENABLED=true are enabled.
  • Before the consumer routing extension, Linux x64 wheels and API native preparation both passed on the dedicated runner in PR checks and API tests.
  • Final implementation routing: even-numbered API run 11720 requested self-hosted / Linux / X64 / ci-linux-builds, and its prepare job started on iv-yezentk8ao3z47ge5tmm.
  • API/CLI, cuVS and all five wheel platforms passed at fb2956b25. Its only failure was a one-second timeout in remote task cancellation cleanup; an isolated real-MySQL rerun passed with 107 passing cases and two expected skips.
  • Cancellation test correction: 33 local Task/Work and concurrency cases passed (14 backend cases skipped locally). Injecting 1.2 seconds of durable-cleanup latency failed both original cancellation branches and passed both corrected branches. Ruff formatting/lint and whitespace checks passed.
  • New head 66f9433da: API CI and PR CI are running.

Local checks verify routing and compatibility contracts; they do not establish a build speedup.

…ools

Use prepared self-hosted hosts for ordinary Linux jobs with isolated build environments, shared pool scheduling without inventory credentials, and hosted fallback when selection is unavailable.
Preserve the Ubuntu 20.04 wheel baseline while routing x64 wheels and shared Studio/source builds through the Linux pool selector.
Default the selector to GitHub-hosted and retain an explicit runner-label override for prepared self-hosted machines.
Register setup-python libraries with the container loader so Hermes can clear its environment without losing the shared Python runtime.
The hosted image supplied zip implicitly, but archive installation tests also need it in the isolated test container.
Install jq in Linux test containers and check archive and marketplace tooling before running the plugin suites.
@zhanglistar zhanglistar changed the title feat(ci): route Linux jobs across available compute pools feat(ci): route Linux jobs to ov-ci-runner or GitHub-hosted Oct 30, 2026
@zhanglistar zhanglistar changed the title feat(ci): route Linux jobs to ov-ci-runner or GitHub-hosted feat(ci): split Linux jobs 2:1 across self-hosted and hosted Oct 30, 2026
Run independent tests and checks on hosted runners so they do not queue behind compilation on ov-ci-runner.
@zhanglistar zhanglistar changed the title feat(ci): split Linux jobs 2:1 across self-hosted and hosted feat(ci): route heavy Linux builds to self-hosted capacity Oct 30, 2026
Select runners directly for heavy builds and keep hosted tests unchanged, removing the selector, pool configuration and unrelated container migration.
@zhanglistar zhanglistar changed the title feat(ci): route heavy Linux builds to self-hosted capacity feat(ci): assign heavy Linux builds to ov-ci-runner Oct 30, 2026
Limit self-hosted builds to trusted PRs and branch pushes, move ARM64 musl to hosted capacity, and provide execution limits plus an emergency hosted override.
Keep Studio, API native runtime and musl CLI builds hosted so they do not serialize behind the wheel build.
@zhanglistar zhanglistar changed the title feat(ci): assign heavy Linux builds to ov-ci-runner feat(ci): reserve ov-ci-runner for Linux x64 wheels Oct 30, 2026
@zhanglistar zhanglistar changed the title feat(ci): reserve ov-ci-runner for Linux x64 wheels feat(ci): isolate compiler caches and reserve Linux wheel runners Oct 30, 2026
@zhanglistar zhanglistar changed the title feat(ci): isolate compiler caches and reserve Linux wheel runners feat(ci): reserve self-hosted runners for Linux wheels Oct 31, 2026
@zhanglistar
zhanglistar force-pushed the feat/ci-linux-pools-official branch from fade53a to dbed5ef Compare October 31, 2026 04:07
Record platform timings and runner cost tradeoffs for later CI optimization.
Keep runner routing compatible with current API and database CI workflows.
Alternate x64 wheel and API native build runs between hosted capacity and ci-linux-builds, with readiness checks and an enable switch before routing work to the new machine.
@zhanglistar zhanglistar changed the title feat(ci): reserve self-hosted runners for Linux wheels feat(ci): offload half of selected Linux builds to dedicated runner Nov 13, 2026
Synchronize with upstream main while retaining the Linux build distribution changes.
Keep the timing analysis without carrying the large CSV in the runner-routing PR.
Use the host CA bundle for minimal containers and fail on incomplete apt indexes; verify package downloads in runner readiness checks.
Keep hosted package sources unchanged and diagnose host versus container connectivity before enabling build routing.
Keep the runner-routing PR focused on CI configuration and current validation.
Keep API, database and cuVS tests from waiting for hosted capacity after a dedicated native build, with disposable test containers and service-aware networking.
Avoid blocking dedicated API builds and tests behind a hosted-only planning job.
@zhanglistar zhanglistar changed the title feat(ci): offload half of selected Linux builds to dedicated runner feat(ci): offload half of Linux builds and API tests Nov 13, 2026
Remove whole-run parity gating so planning, native preparation and Linux API consumers use the dedicated runner on every eligible run.
@zhanglistar zhanglistar changed the title feat(ci): offload half of Linux builds and API tests feat(ci): route Linux API jobs and wheel builds to dedicated runner Nov 13, 2026
Observe cancellation separately from worker completion so slower MySQL cleanup does not turn a successful cancellation into a one-second timeout.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

2 participants