Repository navigation
[improve][client] Make Consumer.resume() eagerly top up the broker's permit window #25978
Description
Search before reporting
- I searched in the issues and found nothing similar.
Motivation
I was adding pause/resume to the Go client (apache/pulsar-client-go#1507), and while doing that I took a closer look at how Java's Consumer.resume() tells the broker to start sending messages again. I found a small thing worth improving, and it's the same shape as what I fixed on the Go side. When you resume, Java calls increaseAvailablePermits(cnx(), 0):
public void resume() {
if (paused) {
paused = false;
increaseAvailablePermits(cnx(), 0);
}
}But that method only actually sends permits to the broker once the owed count reaches half the receiver queue:
while (available >= getCurrentReceiverQueueSize() / 2 && !paused) {
...
sendFlowPermitsToBroker(currentCnx, available);
...
}That "wait until half the queue" rule is good for normal running - you don't want to send a flow command for every single message. The thing is, it's also applied on resume. So if you resume at a moment when fewer than half a queue of permits are owed, resume sends nothing right then, and the broker's window only fills back up later as you keep consuming.
One correction to how I first described this: it is not a stuck/stall bug. As long as the permit invariant holds - brokerInFlight + queuedAtClient + availablePermits == currentReceiverQueueSize - the consumer always recovers on its own, and in Java the invariant does hold (duplicates, skipped batch entries, and intermediate chunks all give their permit back). I couldn't reproduce a stuck consumer, so I'm dropping that claim. What's left is a nice-to-have: let resume top the broker's window straight back up to the full queue size instead of waiting for the threshold. Thanks @lhotari for pointing out the wrong framing in my first version.
Solution
Give resume its own little flush that skips the half-queue rule and just sends whatever is owed. Because it only ever sends what's owed, it can never send too much:
@Override
public void resume() {
if (paused) {
paused = false;
flushAvailablePermitsToBroker(cnx());
}
}
private void flushAvailablePermitsToBroker(ClientCnx currentCnx) {
int available = AVAILABLE_PERMITS_UPDATER.get(this);
while (available > 0 && !paused) {
if (AVAILABLE_PERMITS_UPDATER.compareAndSet(this, available, 0)) {
sendFlowPermitsToBroker(currentCnx, available);
break;
} else {
available = AVAILABLE_PERMITS_UPDATER.get(this);
}
}
}A few notes:
- It uses the same compare-and-swap the existing code already uses, so there's no race and it won't send twice.
- The
available > 0check means it sends nothing when nothing is owed - including when an auto-scale-down has leftavailablePermitsat zero or below. - If the consumer happens to be disconnected,
sendFlowPermitsToBrokerdoes nothing and the reconnect path grants a fresh batch anyway, so that's safe. MultiTopicsConsumerImpl.resume()just calls resume on each child, so partitioned and multi-topic consumers are covered too.
This is the same fix I already used in the Go client (apache/pulsar-client-go#1507).
On the auto-scaled receiver queue (autoScaledReceiverQueueSizeEnabled(true)): I don't think it needs any special handling. The queue only grows and shrinks on the consume path (expectMoreIncomingMessages / reduceCurrentReceiverQueueSize), never through resume(). The flush sends exactly the owed permits, which by the invariant is always at most the current queue size, so it can't over-grant no matter what size scaling has it at. And the scaled-down case (where permits can go to zero or negative) is handled by the available > 0 guard.
On tests: the current testPauseAndResume drains the whole queue, so the owed count ends up back at a full queue size and the threshold path already flushes - which is exactly why the difference never shows today. A proper test needs to leave fewer permits owed than half the queue while the broker is at zero, then resume and check that a flow command actually goes out with exactly the owed count. The cleanest way is a ConsumerImplTest unit test with a mocked connection, plus an autoScaledReceiverQueueSizeEnabled(true) variant (scaled up: flush equals what's owed, nothing extra; scaled down with availablePermits <= 0: nothing sent).
Alternatives
No response
Anything else?
No response
Are you willing to submit a PR?
- I'm willing to submit a PR!
Activity
lhotari commentedon Jun 9, 2026
I don't think that it's completely wrong and I don't see directly why the existing behavior would be causing the consumer to get stuck. However I do agree that it would make sense to send flow permits for all available permits after resuming the consumer.
When pause is called, the available permits should eventually get in the state where the consumer has
available permits >= receiver queue size / 2. There must be other underlying bugs if the current logic doesn't resume the consumer.It seems that there are existing bugs if this happens. The permits handling should be only about what has been sent or received on the wire. If the broker sends duplicates, it shouldn't matter. Permit leaks would also impact other use cases, not just pause/resume. That's why finding and fixing the root cause is important.
In permit accounting the invariant is
brokerInFlight + queuedAtClient + availablePermits == receiverQueueSize. A consumer is only truly stuck when the broker has 0 permits, the client queue is empty, andavailable permits < receiverQueueSize / 2. If the invariant holds, broker-idle + empty-queue forcesavailable == receiverQueueSize, so resume would flush. A real stall therefore requires that this invariant is broken.PavelZeger commentedon Jun 9, 2026
You're right @lhotari , and I'll fix my framing.
The examples I gave don't actually leak in Java - duplicates, skipped batch entries, and intermediate
chunks all return their permit. So with the invariant holding, today's
resume()would already flush. And I can't reproduce an actual stuck consumer, so I'll drop that claim.I also can't really justify it on over-granting: Java's r
esume()already callsincreaseAvailablePermits(cnx, 0), which sends only what's owed, not a full batch - so it can't over-grant today either. The only real difference my change can make is when owed < receiverQueueSize/2 on resume: current code waits, mine flushes right away. Without a leak that's not a stall, just a small thing - it tops the broker's window back to full the moment you resume instead of letting it refill as messages get consumed.So I'm happy either way:
flushAvailablePermitsToBrokerchange plus aConsumerImplTestunit test, framed as "resume tops up the window" rather than a bug fix.WDYT?
lhotari commentedon Jun 29, 2026
I think that this can be justified. It would be necessary to ensure that the solution is also correct for the "autoscaled receiver queue" feature
.autoScaledReceiverQueueSizeEnabled(true). I'm not sure if that requires any special handling.lhotari commentedon Jun 29, 2026
Please update the issue title and description
[-][Improve] `Consumer.resume()` can leave the consumer stuck and not deliver any messages[/-][+][improve][client] Make Consumer.resume() eagerly top up the broker's permit window [/+]PavelZeger commentedon Jun 30, 2026
Thanks for the review! I've changed the title and the description.
[-][improve][client] Make Consumer.resume() eagerly top up the broker's permit window [/-][+][improve][client] Make `Consumer.resume()` eagerly top up the broker's permit window [/+]PavelZeger commentedon Oct 16, 2026
Hi @lhotari @merlimat
I'd like to understand why this PR was closed in favor of @Asthenia0412 PR.
The timeline:
Consumer.resume()eagerly top up the broker's permit window #25978 describing the problem:resume()only flushes permits once the owed count reaches half the receiver queue, so it doesn't top up the broker's window right away. The fix was aflushAvailablePermitsToBrokerthat sends whatever is owed. It's the same approach I'd already used in the Go client ([feat][pip][consumer] Add Pause/Resume/Paused to Consumer pulsar-client-go#1507).autoScaledReceiverQueueSizeEnabled(true). I updated the title and description, and the issue covers the auto-scaled case and the test plan (ConsumerImplTestwith a mocked connection, scaled-up and scaled-down variants).Consumer.resume()eagerly top up the broker's permit window #25978 or this PR. It was approved and merged, and this one was closed.The PR template asks every contributor to confirm they searched for existing issues and PRs before submitting. A search for
resumepermits would have turned up #25978, which already had a committer's review and an open PR. That overlap wasn't caught by the author or in review, and the PR that came later, with no link to the prior discussion, was the one merged.I also asked several times for a review of this PR and got no response. As an outside contributor, it's hard to tell whether the problem is my contributions or how review attention is allocated. So I'd appreciate direct answers to two questions:
I'd like to keep contributing to Pulsar, but I need to know the work will get looked at.
Thanks,
Pavel
lhotari commentedon Oct 16, 2026
@PavelZeger I'm sorry that it happened. There wasn't a link from the issue to this PR. The link gets added when you add
Fixes #xxxxxorCloses #xxxxxin the PR description.lhotari commentedon Oct 16, 2026
A mistake on my side was that I didn't assign this issue to you since you had added a check box to "I'm willing to submit a PR!". I'm sorry about that.
merlimat commentedon Oct 16, 2026
@PavelZeger Sorry about this.
One small note: this was an issue and not a PR. I couldn't find another PR tied to this issue. This was part of the misunderstanding and why #26919 was tied to this issue.
Denovo1998 commentedon Oct 17, 2026
I think even robots should check before they work.
However, I think we can also "defend" at the same time, check at the beginning of the review whether there is a relevant issue or a PR that solves the same problem. Such tasks used to be quite laborious before AI; I think now they can still be checked very quickly.
Now AI is developing too fast, and it's easy to generate various problems.