Skip to content

test: give each test client its own SDK retry partition - #199

Merged
shikhar merged 1 commit into
mainfrom
cc/test-retry-partitions
Oct 28, 2026
Merged

shikhar merged 1 commit into
mainfrom
cc/test-retry-partitions

Conversation

@shikhar

@shikhar shikhar commented Oct 28, 2026

Copy link
Copy Markdown
Member

Fixes #196.

cargo test --lib failed on every run in simulation_rescue_preserves_availability_and_work_under_overload. That test (s3_express_overload_sdk_retries) passes when run alone, under --test-threads=1, and under nextest, which CI uses.

Cause

It isn't task scheduling order, as the issue suggests. Each test has its own paused runtime. The shared state is the AWS SDK's retry token bucket: aws-smithy-runtime keeps one in a process-wide StaticPartitionMap per retry partition. The partition defaults to s3-{region}, and every test client uses us-east-1. Simulations running in parallel in one process drew down the same retry quota, so SDK retries were denied depending on what else was running.

Fix

The simulator, the downloader's scripted test clients, and the service's mock client each take a fresh RetryPartition, so every test client gets its own quota, as a separate process would. Production is unchanged: the server builds one client, and sharing the quota across clients is the SDK's intended behavior.

The issue's suggested change, gating the attempt budget on overload alone, would mask the symptom without removing the shared state.

Validation

  • cargo test --locked --lib: 190 passed on three runs in a row, where main failed all three.
  • nextest: 204 passed. Clippy and fmt are clean.
  • The full simulation campaign is identical to main's. Sequential runs within one campaign never drew the bucket down enough to matter, so no reference results change.

🤖 Generated with Claude Code

The SDK shares a retry token bucket among a process's clients with the same
retry partition, which defaults to the service and region. Under cargo test,
simulations running in parallel drew down each other's retry quota, so
s3_express_overload_sdk_retries failed on every run while passing alone or
under nextest. Each test client now gets its own partition.

Fixes #196

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@greptile-apps

greptile-apps Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Low impact] The PR appears safe to merge; no actionable issues were found.

Summary

Gives each test client a separate AWS SDK retry quota so parallel tests do not consume each other's retries.

  • Parallel tests keep separate AWS SDK retry quotas.

Reviews (1) · Last reviewed commit: "test: give each test client its own SDK ..." · Reviewed by Greptile

@shikhar
shikhar merged commit 8e4fdae into main Oct 28, 2026
8 checks passed
@shikhar
shikhar deleted the cc/test-retry-partitions branch October 28, 2026 14:23
@github-actions github-actions Bot mentioned this pull request Oct 28, 2026
shikhar pushed a commit that referenced this pull request Oct 28, 2026
## 🤖 New release

* `cachey`: 0.12.1 -> 0.13.0

<details><summary><i><b>Changelog</b></i></summary><p>

<blockquote>

##
[0.13.0](0.12.1...0.13.0)
- 2026-10-28

### Added

- *(object_store)* report early hedge outcomes
([#194](#194))
- *(object_store)* [**breaking**] model latency by size band
([#192](#192))

### Fixed

- *(object_store)* recognise bare 404 and 416 responses
([#200](#200))

### Other

- use Chainguard Rust for cachey
([#195](#195))
- give each test client its own SDK retry partition
([#199](#199))
</blockquote>


</p></details>

---
This PR was generated with
[release-plz](https://github.com/release-plz/release-plz/).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Detail Bug] Object-store simulations become flaky under parallel cargo test due to order-sensitive overload retry budgeting

1 participant