Repository navigation
Add direct-to-cold CTST scenarios - #2558
Draft
francoisferrand wants to merge 5 commits into
Draft
francoisferrand wants to merge 5 commits into
francoisferrand wants to merge 5 commits into
Conversation
The dead letter consumer only read the azure archive location's topic, and the sorbetctl retry step was wired to that same location through the world parameters, which made both unusable for the other cold locations. Topics are named after the instance id and the location, so the tests compute them and the consumer covers every cold location's dead letter topic. The sorbetctl step is reworded accordingly, as it retries every failed job and not only restores. Issue: ZENKO-5292
Cover the oplog-driven transition end to end: an object written with a cold storage class (PUT, MPU, CopyObject, self-copy) is declared cold right away, stays readable while its data is hot, and is archived by backbeat from the oplog without a lifecycle scan. Also check that a bucket with an enabled transition rule does not trigger the archive twice, that a restore requested during the archive window is honoured once the archive completes, and that a failed transition can be retried via sorbetctl with the attempt counter bumped and the flag preserved. Backbeat only tails the oplog of buckets with a transition rule, so the scenarios set a disabled rule to the cold location and wait for the bucket to show up in the connector pipeline before writing. Issue: ZENKO-5292
Contributor
Hello francoisferrand,My role is to assist you with the merge of this Available options
Available commands
Status report is not available. |
Contributor
Incorrect fix versionThe
Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:
Please check the |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
The "archived once" direct-to-cold scenario sets an enabled transition rule and writes right away. If the Kafka Connect pipeline does not tail the bucket yet, the write is missed by the oplog path and the scenario either passes without exercising it or flakes. Both transition rule steps now wait for the bucket to show up in the connector pipeline, which also makes the existing restore scenarios more deterministic. Issue: ZENKO-5292
Picks up the suspended cold requests handling from scality/sorbet#243 (SOR-271 + SOR-272): requests to a paused location are kept aside and replayed once it is resumed, instead of being dropped or requeued in a loop. Needed to cover the paused-location case of direct-to-cold. Pinned to the PR head commit 86ea0a3a; to be replaced by the proper tag once it is released. Issue: ZENKO-5292
| dashboard: sorbet/sorbet-dashboards | ||
| image: sorbet | ||
| tag: v1.3.0 | ||
| tag: 86ea0a3a252c8e61afdfdea3f64cf872466a277e |
There was a problem hiding this comment.
Sorbet is pinned to a commit SHA build instead of a released tag. Release sorbet with the direct-to-cold support (BB-804 / SOR-*) and pin that tag before merging, so the 2.16 release doesn't ship a dev build.
With SOR-271/272, a direct-to-cold archive request reaching a paused location is kept aside and sent again once the location is resumed, instead of being dropped or bounced back through a transition retry. The scenario checks the object stays awaiting its transition while paused, is archived after the resume with a single replay of its request, and that a write made after the resume is not held back by the backlog. The lifecycle pause step now records the paused locations so they are resumed when the scenario ends: cold storage scenarios share the locations, and a failure before the resume would leave the next ones stuck. Issue: ZENKO-5292
| @Dmf | ||
| @ColdStorage | ||
| @DirectToCold | ||
| Scenario Outline: Direct-to-cold write while the cold location is paused |
There was a problem hiding this comment.
The PR description says the paused-location case is not covered because it depends on SOR-271, but this scenario is here and tagged @PreMerge. If the pinned Sorbet build doesn't include SOR-271, it will fail in CI. Either confirm the pinned Sorbet build includes SOR-271 (and update the description), or remove/untag this scenario until it lands.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
End-to-end coverage for direct-to-cold (CLDSRV-917 / BB-786 / CLDSRV-956 / BB-804), now that the feature flag is enabled in 2.16.
New
directToCold.featureexercises the full cloudserver → oplog → backbeat → Sorbet loop against the mock DMF:x-amz-storage-class: e2e-cold: the object reports the cold storage class and the transition-in-progress flag right away, stays readable while its data is still hot, then gets archived (flag cleared, data in DMF, GET →InvalidObjectState).scal-fail-archive-job): lands in the DLQ with the flag preserved,sorbetctlretry bumps the attempt counter and re-publishes via the oplog.Backbeat only tails the oplog of buckets with a transition rule (
listenAllBucketsis off), so the scenarios set a disabled rule to the cold location and wait for the bucket to show up in the Kafka Connect pipeline before writing.The first commit is a prerequisite cleanup: the DLQ consumer and the
sorbetctlretry step were hardwired to the azure archive location's topics through the world parameters. Topics are now derived from the instance id, so the consumer covers every cold location.Not covered: the paused-location case, which depends on SOR-271.
Validated with tsc, eslint, cucumber dry-run and
unused-steps; not yet run against a live cluster.Issue: ZENKO-5292