[codelipenghui]

Motivation

The client asks the service address for the shard assignments, and asks the same address again when the stream breaks, with a wait that grows to a minute (30 s in the retry of the connection). After the failure of a server the coordinator elects the new leaders within seconds, but a service that still routes to the dead server, or to one that lost the coordinator and refuses the clients, kept the client without the assignments. The wait before its next attempt had grown beyond the election, so the client learned the new leaders late and its operations ran into their deadline. The Go client had the same problem and was fixed in oxia-db/oxia#1892.

Found by chaos testing of the Go client. The Java client was not run through the chaos harness: this change comes from reading its code, which has the same two properties (always the service address, and the 60 s and 30 s maximum waits).

Modifications

• RpcProvider.getShardAssignments takes a Supplier<String> target, asked for each attempt (null is the service address). The 2-argument version is replaced.
• ShardManager keeps the leaders of the last assignments. Once the service address has failed, each attempt asks the next of them in turn, and an attempt that receives assignments sends the next one back to the service address.
• The wait between the attempts is capped at 1 s, jitter included, in the stream restart of ShardManager and in the retry of the connection in GrpcRpcProvider. Backoff never returns more than its maximum. When the minimum of the connection backoff is 1 s or more, the retry of the connection in GrpcRpcProvider waits 1 s each time: a Failsafe backoff needs its first delay below its maximum.

Shared state

ShardManager.servers is an immutable list that the shard manager replaces when the assignments change (volatile) and the attempts read. attempt is an AtomicInteger, reset when assignments arrive.

Testing

• ShardManagerTest.asksTheLeadersWhenTheServiceAddressFails: the targets are the service address, the service address, each leader, and the service address again after assignments arrived. Fails if the target is always the service address, and if the attempts are not reset.
• ShardManagerTest.retriesTheAssignmentsAtLeastOncePerSecond: 8 failing attempts finish within 8 s (about 4.5 s with the cap, 12.7 s without it).
• GrpcRpcProviderTest.getShardAssignmentsAsksTheAddressOfTheTarget and getShardAssignmentsRetriesAtLeastOncePerSecond, and BackoffTest.neverWaitsLongerThanTheMaximumIncludingTheJitter.
• Each of these fails when the matching part of the fix is removed (five mutations tried: no leader rotation, no attempt reset, a 60 s cap, no jitter clamp, no cap in the provider).
• GrpcRpcProviderTest.getShardAssignmentsRetriesOncePerSecondWhenTheMinimumIsNotBelowIt and ConnectionBackoffStartTest (a standalone client, and a client of a SharedResources pool), each with a connection backoff of (1 s, 30 s) and (2 s, 60 s): the client starts, and the retries of the assignments wait 1 s (two failed attempts, then the third receives them: each case takes about 2.1 s). Without the second commit they fail with Failsafe's IllegalArgumentException: delay must be < the maxDelay.
• GrpcRpcProviderTest.getShardAssignmentsKeepsAConnectionBackoffMaximumBelowOneSecond: with a connection backoff of (100 ms, 300 ms), the waits from the third attempt to the seventh take under 2 s (about 1.2 s; 3.2 s when the retry takes the 1 s cap instead of that maximum, which was tried). The first attempt is left out because it also opens the connection, which took 0.4 to 0.65 s here.
• Unit tests pass (677). Line and branch coverage of the changed lines is 100% from them (JaCoCo). The 8 *IT tests fail to initialize here (Testcontainers finds no Docker); they were not run.
• Not run: ./gradlew check as a whole (because of those tests) and a cluster test with the Java client.

AI Usage

Claude Code wrote the change, the tests and this description, and ran the tests locally. The contributor asked for it and for the PR to be opened ready for review; this description makes no claim about their review.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!