Skip to content

[Issue 1543][consumer] Cancel in-flight ack requests when the broker closes the consumer - #1562

Open
EdwardShen125 wants to merge 5 commits into
apache:masterfrom
EdwardShen125:issue-1543-cancel-inflight-ack-on-close
Open

EdwardShen125 wants to merge 5 commits into
apache:masterfrom
EdwardShen125:issue-1543-cancel-inflight-ack-on-close

Conversation

@EdwardShen125

Copy link
Copy Markdown

Fixes #1543

Motivation

The events loop handles ack requests with AckWithResponse by calling
RequestOnCnx synchronously: it blocks until the ack response arrives or
OperationTimeout expires. When the broker closes the consumer (e.g. on
topic unload), it does not always answer in-flight acks, so the loop stays
blocked for a full OperationTimeout (30s by default). The close
notification sits in connectClosedCh during that time, which delays
reconnection and blocks every other event queued behind the wait.

As analyzed in #1543, this reproduces reliably under load: the reporter
measured 30s acks and 51 duplicate deliveries on topic unload.

Modifications

  • Track the in-flight ack-with-response (request id + connection) in
    partitionConsumer while internalAck/internalAckList waits for the
    response, and clear it when the wait returns.
  • Add Connection.CloseRequest(requestID, err), which fails a pending
    request without waiting for its response. If the send is still queued,
    the cancellation is remembered and applied when the run loop registers
    the request, so it cannot be lost to the registration race.
  • partitionConsumer.ConnectionClosed now receives the closing connection
    and fails the in-flight ack sent on that connection with
    errAckConsumerClosed, which unblocks the events loop immediately.
    Comparing against the connection the ack was sent on ensures a close
    notification for an old incarnation cannot fail a newer request. The
    cancel only triggers for broker-initiated closes; a TCP-level close
    keeps failing pending requests through failPendingRequests as before.
  • ConsumerHandler.ConnectionClosed gains a Connection parameter
    (internal interface, single implementation).

Verifying this change

  • Make sure that the change passes the CI checks.

Added deterministic tests in pulsar/internal covering: a cancelled
registered request completes exactly once and a late response for the
same id is ignored; a cancellation that arrives before registration fails
the request at registration time; failPendingRequests drops remembered
cancellations; and CloseRequest racing the response path completes the
request exactly once (100 trials under -race).

If the close notification arrives before the request starts being tracked
(a window of two adjacent statements on the events loop), the request
falls back to the existing timeout behavior.

Passed locally:

  • go test -race -count=100 ./pulsar/internal -run 'TestCloseRequest|TestFailPendingRequests'
  • go test ./pulsar/internal

Verified against a local standalone broker (Pulsar 4.0.3): 10k messages,
each acked with AckWithResponse, topic unloaded mid-consumption. On
master (e1ed0ef) the in-flight ack times out after 30.001s
(request timed out) and the events loop stalls; with this change the
in-flight ack fails in 1.9ms (consumer closed before ack response was received) and the consumer reconnects and consumes the backlog (10003
messages, including 3 redeliveries).

Broker integration tests have not been run; hitting the in-flight window
depends on timing, so the regression coverage is the deterministic test
matrix above. The normal receipt path is already covered by
TestAckResponseNotBlocked, whose behavior this change does not alter.

Does this pull request potentially affect one of the following parts:

  • Dependencies: no
  • The public API: no
  • The schema: no
  • The default values of configurations: no
  • The wire protocol: no

Documentation

  • Does this pull request introduce a new feature: no
  • Documentation changes are not required for this internal concurrency fix.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Cancellation races can still stall reconnection or retain completed-request entries, while missing consumer-level tests leave both ACK paths and broker-close handling insufficiently covered.

3 open issues
What changed in this PR

This PR addresses #1543 by adding cancellation of in-flight ACK receipt waits when the broker closes a consumer, aiming to let the consumer event loop reconnect without waiting for OperationTimeout.

Changes:

  • Track the active ACK request and its connection for individual and list acknowledgements.
  • Add connection-level request cancellation and pass the closing connection to consumer callbacks.
  • Add cancellation tests and update connection test doubles.
File Description
pulsar/​producer_test.go Forward request cancellation through the connection mock.
pulsar/​internal/​connection.go Add cancellation bookkeeping and connection-aware close callbacks.
pulsar/​internal/​connection_close_request_test.go Test cancellation, cleanup, and response races.
pulsar/​consumer_partition.go Track in-flight ACKs and cancel them on matching broker-close notifications.
pulsar/​consumer_multitopic_test.go Add cancellation support to the dummy connection interface.

Review effort: Balanced

Comment on lines +1825 to +1827
if closeConsumer != nil {
if ack := pc.inflightAck.Load(); ack != nil && ack.cnx == cnx {
cnx.CloseRequest(ack.requestID, errAckConsumerClosed)
Comment thread pulsar/internal/connection.go Resolved
Comment thread pulsar/internal/connection.go Resolved
@EdwardShen125
EdwardShen125 force-pushed the issue-1543-cancel-inflight-ack-on-close branch from a5beb8a to 17a1e46 Compare October 28, 2026 10:17
…the consumer

The events loop waits synchronously for the response of acks sent with
AckWithResponse. When the broker closes the consumer without answering
in-flight acks (e.g. on topic unload), the loop stays blocked until the
request times out, delaying reconnection by a full OperationTimeout.

Fail the in-flight ack from ConnectionClosed instead, which runs on the
connection read goroutine and is therefore not blocked by the events loop.

Fixes apache#1543
@EdwardShen125
EdwardShen125 force-pushed the issue-1543-cancel-inflight-ack-on-close branch from 17a1e46 to 3d6726b Compare October 29, 2026 02:03
@EdwardShen125

Copy link
Copy Markdown
Author

@crossoverJie Addressed and force-pushed.

Comment thread pulsar/internal/connection.go Outdated
req.callback(nil, err)
return
}
if requestID > c.lastCompletedReqID {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The lastCompletedReqID guard can silently drop a valid cancellation. Request IDs are allocated monotonically per client, but on a shared connection responses complete out of order. An ACK's id can be allocated and then block in SendRequest because incomingRequestsCh (cap 10) is full; meanwhile a higher-id request is enqueued and completed, pushing lastCompletedReqID above the ACK's id. If CommandCloseConsumer arrives before that ACK is registered, CloseRequest sees requestID <= lastCompletedReqID, does not remember the cancellation, and the ACK is then registered/sent and waits the full OperationTimeout — exactly the stall this PR is meant to remove.

Since request ids are never reused, I'd drop the guard and remember the cancellation unconditionally (bounding the map if needed). Repro: complete id 1001, call CloseRequest(1000) while 1000 is not yet registered, then internalSendRequest for 1000 — it gets registered instead of failed.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right. The watermark assumed ids complete in order; since they do not, it could drop a valid cancellation. Removed the guard — cancellations are now remembered unconditionally, with a regression test for the exact scenario you described.

pendingLock sync.Mutex
pendingReqs map[uint64]*request
// requests cancelled by CloseRequest before their send was registered
cancelledReqs map[uint64]error

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Entries are only removed on registration or on connection close. If CloseRequest arrives after the request already completed (or the request is never registered), the entry lingers for the connection's lifetime. If the lastCompletedReqID guard is removed as suggested, please bound this map (e.g. cap and drop oldest) so a long-lived shared connection cannot grow it without limit.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Covered by the same change: the map is deduped and capped at 1024, dropping the oldest ids first. Ids are never reused, so the cap is never reached in practice.

// The broker does not always answer in-flight acks when it closes the
// consumer, so fail the wait instead of waiting for the timeout.
if closeConsumer != nil {
if ack := pc.inflightAck.Load(); ack != nil && ack.cnx == cnx {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only the single in-flight ack is covered. Acks already queued in eventsCh (cap 10) when CommandCloseConsumer arrives are not: after the in-flight ack is released, select may pick another ackRequest before connectClosedCh, send it on the old connection/consumer id, and the broker (ServerCnx.handleAck) will not answer -> another full OperationTimeout stall that also blocks reconnection. With the 12 concurrent acking goroutines from #1543 this path is likely to be hit.

Suggest also failing queued acks on that connection: remember the closed connection (or a generation) and have internalAck/internalAckList short-circuit with errAckConsumerClosed when _getConn() is that connection, cleared after a successful reconnect. Otherwise the "reconnect in milliseconds" guarantee only holds for a single in-flight ack.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, the in-flight ack was only half the window. The closed connection is now remembered and every waiting ack path (plain, list, transactional) fails fast with errAckConsumerClosed until a successful re-subscribe clears it. The check sits right after the in-flight ack is registered, so a close landing in between is still caught by ConnectionClosed. Fire-and-forget acks are untouched since they cannot stall.

return m.cnx, nil
}

type closeRecordingConnection struct {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is suggested that e2e testing still needs to be supplemented.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added TestAckWithResponseSurvivesBrokerClosingConsumer: 12 workers acking with response across a topic unload. Fails on master (only ~4400/10000 consumed within 25s), passes with the fix.

…conditionally

- Remember cancelled requests unconditionally in CloseRequest: responses
  complete out of order, so a completed higher request id must not
  suppress the cancellation of an id that was allocated but not yet
  registered. The map of remembered ids is bounded (1024) since ids are
  never reused.
- Fail every waiting ack path (plain, list, transactional) with
  errAckConsumerClosed while the connection the broker closed is still
  in use; the check sits right after the in-flight ack is registered, so
  a close landing in between is still caught by ConnectionClosed.
  Cleared on a successful re-subscribe.
- Add e2e test unloading a topic while 12 workers ack with response:
  fails on master, passes with the fix.
@EdwardShen125
EdwardShen125 force-pushed the issue-1543-cancel-inflight-ack-on-close branch from 58f3c94 to cba308c Compare October 30, 2026 17:29
…t.go

The file opened with a stray '//' line before the ASF license header,
so the hawkeye header check reported a missing header and the lint job
failed. Drop the line so the header matches the rest of the repo.
@EdwardShen125

EdwardShen125 commented Nov 2, 2026 •

Copy link
Copy Markdown
Author

Fixed the lint failure — stray // above the license header.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved close-state and cancellation handling issues can block ACKs until OperationTimeout, while the test’s deadline failure path leaks CPU-spinning workers.

4 open issues
2 resolved since last review

Review effort: Balanced

Comment on lines +113 to +115
case <-deadline:
t.Fatalf("consumed only %d/%d messages within 25s",
consumed.Load(), numMessages)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 15181e8 — the timeout sets a flag now; workers are stopped and joined before the Fatalf.

Comment thread pulsar/consumer_partition.go Outdated
pc.name = res.Response.ConsumerStatsResponse.GetConsumerName()
}

pc.clearClosedAckConns()

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 15181e8 — only the old connection's marker is cleared now, so a close recorded on the new one survives.

first = false
}
}
delete(c.cancelledReqs, oldest)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this one. Eviction needs 1024 cancellations of not-yet-registered requests on one connection; if it ever fires, the evicted ack waits one OperationTimeout, which is the pre-fix behavior.

- grabConn cleared the whole closedAckConns map after a successful
  re-subscribe, erasing a marker the connection reader may have just
  recorded for a CLOSE_CONSUMER that arrived between the SUBSCRIBE
  response and grabConn resuming. Clear only the replaced connections
  so acks queued behind the reconnect still fail fast.
- The e2e test's Fatalf skipped close(stop) and wg.Wait(), leaving 12
  workers spinning on the closed consumer for the rest of the suite.
  Signal the timeout, stop and join the workers, then fail.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] In-flight ACK receipt blocks consumer reconnection until OperationTimeout after topic unload

3 participants