Repository navigation
[Issue 1543][consumer] Cancel in-flight ack requests when the broker closes the consumer - #1562
EdwardShen125 wants to merge 5 commits into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Cancellation races can still stall reconnection or retain completed-request entries, while missing consumer-level tests leave both ACK paths and broker-close handling insufficiently covered.
3 open issues
What changed in this PR
This PR addresses #1543 by adding cancellation of in-flight ACK receipt waits when the broker closes a consumer, aiming to let the consumer event loop reconnect without waiting for OperationTimeout.
Changes:
- Track the active ACK request and its connection for individual and list acknowledgements.
- Add connection-level request cancellation and pass the closing connection to consumer callbacks.
- Add cancellation tests and update connection test doubles.
| File | Description |
|---|---|
pulsar/producer_test.go |
Forward request cancellation through the connection mock. |
pulsar/internal/connection.go |
Add cancellation bookkeeping and connection-aware close callbacks. |
pulsar/internal/connection_close_request_test.go |
Test cancellation, cleanup, and response races. |
pulsar/consumer_partition.go |
Track in-flight ACKs and cancel them on matching broker-close notifications. |
pulsar/consumer_multitopic_test.go |
Add cancellation support to the dummy connection interface. |
Review effort: Balanced
| if closeConsumer != nil { | ||
| if ack := pc.inflightAck.Load(); ack != nil && ack.cnx == cnx { | ||
| cnx.CloseRequest(ack.requestID, errAckConsumerClosed) |
a5beb8a to
17a1e46
Compare
…the consumer The events loop waits synchronously for the response of acks sent with AckWithResponse. When the broker closes the consumer without answering in-flight acks (e.g. on topic unload), the loop stays blocked until the request times out, delaying reconnection by a full OperationTimeout. Fail the in-flight ack from ConnectionClosed instead, which runs on the connection read goroutine and is therefore not blocked by the events loop. Fixes apache#1543
17a1e46 to
3d6726b
Compare
|
@crossoverJie Addressed and force-pushed. |
| req.callback(nil, err) | ||
| return | ||
| } | ||
| if requestID > c.lastCompletedReqID { |
There was a problem hiding this comment.
The lastCompletedReqID guard can silently drop a valid cancellation. Request IDs are allocated monotonically per client, but on a shared connection responses complete out of order. An ACK's id can be allocated and then block in SendRequest because incomingRequestsCh (cap 10) is full; meanwhile a higher-id request is enqueued and completed, pushing lastCompletedReqID above the ACK's id. If CommandCloseConsumer arrives before that ACK is registered, CloseRequest sees requestID <= lastCompletedReqID, does not remember the cancellation, and the ACK is then registered/sent and waits the full OperationTimeout — exactly the stall this PR is meant to remove.
Since request ids are never reused, I'd drop the guard and remember the cancellation unconditionally (bounding the map if needed). Repro: complete id 1001, call CloseRequest(1000) while 1000 is not yet registered, then internalSendRequest for 1000 — it gets registered instead of failed.
There was a problem hiding this comment.
You are right. The watermark assumed ids complete in order; since they do not, it could drop a valid cancellation. Removed the guard — cancellations are now remembered unconditionally, with a regression test for the exact scenario you described.
| pendingLock sync.Mutex | ||
| pendingReqs map[uint64]*request | ||
| // requests cancelled by CloseRequest before their send was registered | ||
| cancelledReqs map[uint64]error |
There was a problem hiding this comment.
Entries are only removed on registration or on connection close. If CloseRequest arrives after the request already completed (or the request is never registered), the entry lingers for the connection's lifetime. If the lastCompletedReqID guard is removed as suggested, please bound this map (e.g. cap and drop oldest) so a long-lived shared connection cannot grow it without limit.
There was a problem hiding this comment.
Covered by the same change: the map is deduped and capped at 1024, dropping the oldest ids first. Ids are never reused, so the cap is never reached in practice.
| // The broker does not always answer in-flight acks when it closes the | ||
| // consumer, so fail the wait instead of waiting for the timeout. | ||
| if closeConsumer != nil { | ||
| if ack := pc.inflightAck.Load(); ack != nil && ack.cnx == cnx { |
There was a problem hiding this comment.
Only the single in-flight ack is covered. Acks already queued in eventsCh (cap 10) when CommandCloseConsumer arrives are not: after the in-flight ack is released, select may pick another ackRequest before connectClosedCh, send it on the old connection/consumer id, and the broker (ServerCnx.handleAck) will not answer -> another full OperationTimeout stall that also blocks reconnection. With the 12 concurrent acking goroutines from #1543 this path is likely to be hit.
Suggest also failing queued acks on that connection: remember the closed connection (or a generation) and have internalAck/internalAckList short-circuit with errAckConsumerClosed when _getConn() is that connection, cleared after a successful reconnect. Otherwise the "reconnect in milliseconds" guarantee only holds for a single in-flight ack.
There was a problem hiding this comment.
Agreed, the in-flight ack was only half the window. The closed connection is now remembered and every waiting ack path (plain, list, transactional) fails fast with errAckConsumerClosed until a successful re-subscribe clears it. The check sits right after the in-flight ack is registered, so a close landing in between is still caught by ConnectionClosed. Fire-and-forget acks are untouched since they cannot stall.
| return m.cnx, nil | ||
| } | ||
|
|
||
| type closeRecordingConnection struct { |
There was a problem hiding this comment.
It is suggested that e2e testing still needs to be supplemented.
There was a problem hiding this comment.
Added TestAckWithResponseSurvivesBrokerClosingConsumer: 12 workers acking with response across a topic unload. Fails on master (only ~4400/10000 consumed within 25s), passes with the fix.
…conditionally - Remember cancelled requests unconditionally in CloseRequest: responses complete out of order, so a completed higher request id must not suppress the cancellation of an id that was allocated but not yet registered. The map of remembered ids is bounded (1024) since ids are never reused. - Fail every waiting ack path (plain, list, transactional) with errAckConsumerClosed while the connection the broker closed is still in use; the check sits right after the in-flight ack is registered, so a close landing in between is still caught by ConnectionClosed. Cleared on a successful re-subscribe. - Add e2e test unloading a topic while 12 workers ack with response: fails on master, passes with the fix.
58f3c94 to
cba308c
Compare
…t.go The file opened with a stray '//' line before the ASF license header, so the hawkeye header check reported a missing header and the lint job failed. Drop the line so the header matches the rest of the repo.
|
Fixed the lint failure — stray |
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved close-state and cancellation handling issues can block ACKs until OperationTimeout, while the test’s deadline failure path leaks CPU-spinning workers.
4 open issues
2 resolved since last review
Review effort: Balanced
| case <-deadline: | ||
| t.Fatalf("consumed only %d/%d messages within 25s", | ||
| consumed.Load(), numMessages) |
There was a problem hiding this comment.
Fixed in 15181e8 — the timeout sets a flag now; workers are stopped and joined before the Fatalf.
| pc.name = res.Response.ConsumerStatsResponse.GetConsumerName() | ||
| } | ||
|
|
||
| pc.clearClosedAckConns() |
There was a problem hiding this comment.
Fixed in 15181e8 — only the old connection's marker is cleared now, so a close recorded on the new one survives.
| first = false | ||
| } | ||
| } | ||
| delete(c.cancelledReqs, oldest) |
There was a problem hiding this comment.
Leaving this one. Eviction needs 1024 cancellations of not-yet-registered requests on one connection; if it ever fires, the evicted ack waits one OperationTimeout, which is the pre-fix behavior.
- grabConn cleared the whole closedAckConns map after a successful re-subscribe, erasing a marker the connection reader may have just recorded for a CLOSE_CONSUMER that arrived between the SUBSCRIBE response and grabConn resuming. Clear only the replaced connections so acks queued behind the reconnect still fail fast. - The e2e test's Fatalf skipped close(stop) and wg.Wait(), leaving 12 workers spinning on the closed consumer for the rest of the suite. Signal the timeout, stop and join the workers, then fail.

Fixes #1543
Motivation
The events loop handles ack requests with
AckWithResponseby callingRequestOnCnxsynchronously: it blocks until the ack response arrives orOperationTimeoutexpires. When the broker closes the consumer (e.g. ontopic unload), it does not always answer in-flight acks, so the loop stays
blocked for a full
OperationTimeout(30s by default). The closenotification sits in
connectClosedChduring that time, which delaysreconnection and blocks every other event queued behind the wait.
As analyzed in #1543, this reproduces reliably under load: the reporter
measured 30s acks and 51 duplicate deliveries on topic unload.
Modifications
partitionConsumerwhileinternalAck/internalAckListwaits for theresponse, and clear it when the wait returns.
Connection.CloseRequest(requestID, err), which fails a pendingrequest without waiting for its response. If the send is still queued,
the cancellation is remembered and applied when the run loop registers
the request, so it cannot be lost to the registration race.
partitionConsumer.ConnectionClosednow receives the closing connectionand fails the in-flight ack sent on that connection with
errAckConsumerClosed, which unblocks the events loop immediately.Comparing against the connection the ack was sent on ensures a close
notification for an old incarnation cannot fail a newer request. The
cancel only triggers for broker-initiated closes; a TCP-level close
keeps failing pending requests through
failPendingRequestsas before.ConsumerHandler.ConnectionClosedgains aConnectionparameter(internal interface, single implementation).
Verifying this change
Added deterministic tests in
pulsar/internalcovering: a cancelledregistered request completes exactly once and a late response for the
same id is ignored; a cancellation that arrives before registration fails
the request at registration time;
failPendingRequestsdrops rememberedcancellations; and
CloseRequestracing the response path completes therequest exactly once (100 trials under
-race).If the close notification arrives before the request starts being tracked
(a window of two adjacent statements on the events loop), the request
falls back to the existing timeout behavior.
Passed locally:
go test -race -count=100 ./pulsar/internal -run 'TestCloseRequest|TestFailPendingRequests'go test ./pulsar/internalVerified against a local standalone broker (Pulsar 4.0.3): 10k messages,
each acked with
AckWithResponse, topic unloaded mid-consumption. Onmaster (e1ed0ef) the in-flight ack times out after 30.001s
(
request timed out) and the events loop stalls; with this change thein-flight ack fails in 1.9ms (
consumer closed before ack response was received) and the consumer reconnects and consumes the backlog (10003messages, including 3 redeliveries).
Broker integration tests have not been run; hitting the in-flight window
depends on timing, so the regression coverage is the deterministic test
matrix above. The normal receipt path is already covered by
TestAckResponseNotBlocked, whose behavior this change does not alter.Does this pull request potentially affect one of the following parts:
Documentation