Skip to content

backport: bitcoin#31740, bitcoin#31945 (libmultiprocess EventLoop shutdown race fixes) - #8221

Open
PastaPastaPasta wants to merge 2 commits into
dashpay:developfrom
PastaPastaPasta:bp/libmultiprocess-31740
Open

PastaPastaPasta wants to merge 2 commits into
dashpay:developfrom
PastaPastaPasta:bp/libmultiprocess-31740

Conversation

@PastaPastaPasta

@PastaPastaPasta PastaPastaPasta commented Nov 21, 2026 •

Copy link
Copy Markdown
Member

Issue being fixed or feature implemented

ipc_tests fails intermittently in the linux64_tsan unit-test run with a ThreadSanitizer data race inside libmultiprocess: close in mp::EventLoop::loop() races with write in mp::EventLoop::removeClient() on the event loop's wakeup socket. It has failed on two unrelated PRs so far, with identical stacks:

Both started after the CMake cut-over began running the unit tests via ctest in the TSan build job. The bug itself is in the libmultiprocess version Dash pins (abe254b, from bitcoin#31105).

Mechanism: removeClient() drops the client count, unlocks, then writes to the wakeup fd. It wakes the loop even while async work is still queued. When two such writes land, the loop can wake on the first, see the exit condition, leave, and close() the fd while another thread is still about to write() to it. TSan reports the unsynchronized close/write; without TSan this can show up as the write(m_post_fd, ...): Broken pipe crash of bitcoin#31151. A related race in the same code makes EventLoop::post() relock m_mutex after loop() has returned and the EventLoop has been destroyed (libmultiprocess#154, bitcoin#31921). In Dash's IpcTest(), disconnect_client() hits that through loop.sync(), and it shows up as a hang or a TSan "use of an invalid mutex" report.

Upstream fixed both in libmultiprocess and pulled the fixes into Bitcoin Core with two version bumps, which this PR backports instead of patching libmultiprocess locally.

What was done?

Dash adaptations, all recorded in the commit messages:

How Has This Been Tested?

Local TSan build (-DSANITIZERS=thread, clang 19, libstdc++, depends with MULTIPROCESS=1 CC=clang-19 CXX=clang++-19, matching the CI multiprocess depends used by linux64_tsan). test_dash --run_test=ipc_tests was run in a loop, 48 at a time, on a heavily loaded x86_64 host:

build iterations pass close/write race (CI signature) invalid-mutex in post() hung (>7 min)
develop (f62ee01) 1500 1482 5 3 10
this PR 4000 4000 0 0 0

The races reproduced locally have the same stacks as the CI failures. For the hangs, gdb showed the main thread blocked in pthread_mutex_lock inside EventLoop::post(), called from disconnect_client() after the loop thread had exited.

Also built dash-node and test_dash with the final branch (no warnings), ran ipc_tests, and ran lint-whitespace, lint-includes, lint-include-guards and lint-files.

Breaking Changes

None. libmultiprocess is only used by the experimental multiprocess build (MULTIPROCESS=1) and is not part of release builds.

Checklist:

  • I have performed a self-review of my own code
  • I have made corresponding changes to the documentation
  • I have assigned this pull request to a milestone (if you have permission to set one)

🤖 Generated with Claude Code

@PastaPastaPasta PastaPastaPasta added this to the 24.1 milestone Nov 21, 2026
fanquake and others added 2 commits November 21, 2026 17:49
…onverting to subtree

4e0aa18 test: Add test for IPC serialization bug (Ryan Ofsky)
2221c88 depends: Update libmultiprocess library before converting to subtree (Ryan Ofsky)

Pull request description:

  This should be the final update to the libmultiprocess package via the depends system. It brings in the libmultiprocess cmake changes from bitcoin-core/libmultiprocess#136 needed to support building as subtree. After this, followup PR bitcoin#31741 will add libmultiprocess as a git subtree and depends will just use the git subtree instead of hardcoding its own version hash.

  Since there have been libmultiprocess API changes since the last update, this commit also updates bitcoin code to be compatible with them.

  This update has the following new changes since previous update bitcoin#31105:

  bitcoin-core/libmultiprocess#121 ProxyClientBase: avoid static_cast to partially constructed object
  bitcoin-core/libmultiprocess#120 proxy-types.h: add static_assert to detect int/enum size mismatch
  bitcoin-core/libmultiprocess#127 ProxyClientBase: avoid static_cast to partially destructed object
  bitcoin-core/libmultiprocess#129 Fix "disconnected: write(m_post_fd, &buffer, 1): Broken pipe" EventLoop shutdown races.
  bitcoin-core/libmultiprocess#130 refactor: Add CleanupRun function to dedup clean list code
  bitcoin-core/libmultiprocess#131 doc: fix startAsyncThread comment
  bitcoin-core/libmultiprocess#133 Fix debian "libatomic not found" error in downstream builds
  bitcoin-core/libmultiprocess#94 c++ 20 cleanups
  bitcoin-core/libmultiprocess#135 refactor: proxy-types.h API cleanup
  bitcoin-core/libmultiprocess#136 cmake: Support being included with add_subdirectory
  bitcoin-core/libmultiprocess#137 doc: Fix broken markdown links

ACKs for top commit:
  Sjors:
    ACK 4e0aa18
  vasild:
    ACK 4e0aa18
  TheCharlatan:
    ACK 4e0aa18

Tree-SHA512: 6d81cdf7f44762c7f476212295f6224054fd0a61315bb54786bc7758a2b33e5a2fce925c71e36f7bda320049aa14e7218a458ceb03dacbb869632c466c4789b0
(cherry picked from commit ad2f932)

Dash adaptations:
- native_libmultiprocess keeps the bitcoin-core/libmultiprocess download
  path from bitcoin#31982; the archive hash is the same as upstream's.
- Dash's common-types.h never had the std::chrono CustomBuildField and
  CustomReadField overloads or the generic Data CustomBuildField that
  upstream removes here. They came with the parts of bitcoin#30510 Dash
  did not take, so only the new mp/type-*.h includes apply.
- Dash's FooInterface lacks passTransaction, passVectorChar and
  passBlockState (also from bitcoin#30510), so passScript takes capnp
  field ordinal 3 and its check follows passUniValue in IpcTest().
  ipc_test.cpp includes util/strencodings.h for HexStr(), which upstream
  got through ipc_test.h's validation.h and util/fs.h includes that
  Dash's test does not have.
- init-types.h includes mp/proxy-types.h and the mp/type-*.h headers
  init.capnp needs (context, decay, interface, threadmap). Upstream's init-types.h gets
  them through mining.capnp.proxy-types.h and common-types.h, but Dash
  has no Mining interface, so without them dash-node fails to compile
  against the new library.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…I failures

01f7715 depends: Update libmultiprocess library to fix CI failure (Ryan Ofsky)

Pull request description:

  Bump libmultiprocess library to include bugfix bitcoin-core/libmultiprocess#159 which should fix intermittent CI failures reported in bitcoin#31921

  This change is bumping the libmultiprocess version instead of cherry picking the bugfix. It could cherry-pick the bugfix instead, but there are reasons to prefer bumping the version:

  - Bugfix might interact with earlier PRs, and the latest version is better tested with testing done in many CI configurations in [bitcoin#30975](bitcoin#30975) and [bitcoin#31802](bitcoin#31802)

  - Even though we are in feature freeze for a release, the MULTIPROCESS=1 option is currently not enabled for release, so this PR only affect CI builds and local builds, not the release build.

  This update brings in the following changes:

  bitcoin-core/libmultiprocess#140 build: don't clobber user/superproject c++ version
  bitcoin-core/libmultiprocess#142 build: add option for external mpgen binary
  bitcoin-core/libmultiprocess#143 cleanup: initialize vars in the EventLoop constructor in the correct order
  bitcoin-core/libmultiprocess#146 cmake: Suppress compiler warnings from capnproto headers
  bitcoin-core/libmultiprocess#147 cmake: EXTERNAL_MPGEN cleanups
  bitcoin-core/libmultiprocess#148 util: fix -Wpessimizing-move warning
  bitcoin-core/libmultiprocess#145 CTest: Module must be included at the top level
  bitcoin-core/libmultiprocess#149 Avoid `-Wundef` compiler warnings
  bitcoin-core/libmultiprocess#152 refactor: Fix compiler and clang-tidy warnings
  bitcoin-core/libmultiprocess#155 scripted-diff: s/Libmultiprocess_EXTERNAL_MPGEN/MPGEN_EXECUTABLE/g
  bitcoin-core/libmultiprocess#156 refactor: Remove locale-dependent function calls
  bitcoin-core/libmultiprocess#157 refactor: Avoid using std::format
  bitcoin-core/libmultiprocess#159 bugfix: Do not lock EventLoop::mutex after EventLoop is done
  bitcoin-core/libmultiprocess#161 cmake: Avoid including CTest if not top level project
  bitcoin-core/libmultiprocess#164 Bump minimum required cmake to 3.12

  ---

  This PR is part of the [process separation project](bitcoin#28722).

ACKs for top commit:
  fanquake:
    ACK 01f7715

Tree-SHA512: a6a795e4d4e13e9d35c9346f3c83d5da817f1452bdc4a9412aeadc4c652ad6e5047f4c77756570594a70ec9095cc786772a0469e306dc19bb5a508fd515c37f4
(cherry picked from commit ba0a439)

Dash adaptations:
- native_libmultiprocess keeps the bitcoin-core/libmultiprocess download
  path from bitcoin#31982; the archive hash is the same as upstream's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@PastaPastaPasta
PastaPastaPasta force-pushed the bp/libmultiprocess-31740 branch from 3943351 to b082971 Compare November 21, 2026 17:49
@coderabbitai

coderabbitai Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Explain Change →

Important

Review skipped

Review was skipped as selected files did not have any reviewable changes.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 9d0590e6-5713-4101-919b-ef7955feb19c
Commits

Reviewing files that changed from the base of the PR and between 3943351 and b082971.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review

No actionable comments were generated in the recent review.

Walkthrough

The changes update the native libmultiprocess revision and add multiprocess type headers to the Cap’n Proto integration. The echo schema uses includeTypes, and CapnpProtocol::addCleanup stores callbacks in cleanup_fns. The IPC test interface adds passScript, with a CScript implementation that returns its argument. The test checks that a script containing OP_11 retains its hexadecimal encoding after passing through the interface.

Estimated code review effort: 2 (Simple) | ~10 minutes

Sequence Diagram(s)

sequenceDiagram
  participant IpcTest
  participant FooInterface
  participant FooImplementation
  IpcTest->>FooInterface: passScript(Data)
  FooInterface->>FooImplementation: passScript(CScript)
  FooImplementation-->>FooInterface: return CScript
  FooInterface-->>IpcTest: return Data
Loading

Suggested reviewers: kwvg

Merge Risk: Minimal · previous review 39433

This backport updates the experimental multiprocess IPC library to fix shutdown races and adds a script pass-through test. Release builds are unaffected. No code defects were found. The only failing check is the PR description's @mention, which should be removed before merge.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the two backported upstream Bitcoin Core changes (bitcoin#31740 and bitcoin#31945) and the libmultiprocess EventLoop shutdown race fixes. The changeset bumps libmultiprocess to 1954f7f…
Description check Passed The description explains the intermittent ipc_tests TSan data race in libmultiprocess, the two upstream fixes, the Dash-specific adaptations (passScript ordinal, includes, depends download path), and …
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the two backported upstream Bitcoin Core changes (bitcoin#31740 and bitcoin#31945) and the libmultiprocess EventLoop shutdown race fixes. The changeset bumps libmultiprocess to 1954f7f and 07c917f, adds the EventLoop race fixes, and includes the related API and include updates, so the title reflects the main change. It is somewhat terse in that it names backport numbers rather than the behaviour, but it is specific and accurate.

Full details: Description check

Explanation

The description explains the intermittent ipc_tests TSan data race in libmultiprocess, the two upstream fixes, the Dash-specific adaptations (passScript ordinal, includes, depends download path), and the local testing results. It directly relates to the changeset files.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
Fix failing CI checks

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 6035cd45-981a-4166-a0e5-dd881c92063f
Commits

Reviewing files that changed from the base of the PR and between f62ee01 and 3943351.

Files selected for processing: 9

  • depends/packages/native_libmultiprocess.mk
  • src/ipc/capnp/common-types.h
  • src/ipc/capnp/echo-types.h
  • src/ipc/capnp/echo.capnp
  • src/ipc/capnp/init-types.h
  • src/ipc/capnp/protocol.cpp
  • src/test/ipc_test.capnp
  • src/test/ipc_test.cpp
  • src/test/ipc_test.h

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.

@thepastaclaw

thepastaclaw commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Final review complete — no blockers (commit b082971)

@thepastaclaw thepastaclaw left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Final validation — Phase 1 + Phase 2

Updated after discussion: 1 finding(s) of this review were settled in the inline threads; no blocking finding remains.

  • ✅ Withdrawn: 🔴 Missing follow-up: bitcoin#33241 fixes mpgen hangs with older system Cap’n Proto

At b082971, both backports carry the intended EventLoop shutdown fixes, and the multiprocess, TSan, and ASan checks passed. One blocking upstream follow-up is missing: building the newly pinned libmultiprocess revision against older system Cap’n Proto can hang during IPC schema generation.

🔴 1 0 blocking

Review provenance

Source: reviewer 1: glm-5.3-flash (agent: phase1-reviewer, role: general); reviewer 2: gpt-6.2-sol (agent: phase2-reviewer, role: general); reviewer 3: gpt-6.2-sol (agent: phase2-reviewer, role: backport-reviewer); reviewer 4: gpt-6.2-sol (agent: phase2-reviewer, role: dash-core-commit-history); final verifier: gpt-6.2-sol (agent: sol-verifier, role: final-verifier)

  • Triage: low by gpt-6.2-sol (effort low) — The diff is a contained dependency bump with small IPC compatibility updates and a regression test; the shutdown race fixes are in the pinned library, not in Dash’s changed code.
  • Phase 1 reviewers: glm-5.3-flash — general (completed, effort high); agent phase1-reviewer
  • Phase 1 model: glm-5.3-flash — zai quota: 5h 100% left, weekly 33% left; passed over gemini-3.8-flash-high (antigravity quota lookup failed)
  • Fresh verifier: gpt-6.2-sol — final-verifier; agent sol-verifier
  • Phase 2 reviewers: gpt-6.2-sol — general (completed, effort medium); agent phase2-reviewer, gpt-6.2-sol — backport-reviewer (completed, effort medium); agent phase2-reviewer, gpt-6.2-sol — dash-core-commit-history (completed, effort medium); agent phase2-reviewer
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review (PastaClaw). How to work through them:
- Verify each finding against the code; fix only what is real, as narrowly as the finding.
- If a finding is wrong, decline it on its thread in one or two sentences; the bot answers once and stops.
- To defer one, link a tracking issue on its thread (or reply asking the bot to file one).
- Suggestions don't block the merge, but most are worth fixing in this PR; judge each on its merits.
- Keep the PR's scope; don't grow it to satisfy hardening suggestions.
- Carried items were raised in an earlier round: read the thread first, and don't re-fix what you already fixed, declined or deferred there. Disputed and deferred items are already handled and are not listed.

In `depends/packages/native_libmultiprocess.mk`:
- [BLOCKING → WITHDRAWN in discussion; do not fix] depends/packages/native_libmultiprocess.mk:2: Missing follow-up: bitcoin#33241 fixes mpgen hangs with older system Cap’n Proto
  The new pin matches bitcoin#31945, but unlike the base pin it builds standalone `mpgen` as C++20. Its `src/mp/gen.cpp:146` still compares a `kj::StringPtr` with a string literal before generating any output. With Cap’n Proto 0.8.x and affected GCC versions, that comparison recurses; bitcoin#33176 reproduced the resulting build hang. The `WITH_MULTIPROCESS=ON` build documented in `doc/multiprocess.md:30` invokes the installed generator, so it hits the same bug if that generator was built from `1954f7f` against the affected system dependency. The passing depends jobs use Cap’n Proto 1.3.0. Carry the generator fix from libmultiprocess commit `c4cb758e`, which Bitcoin Core incorporated in bitcoin#33241, and identify the fixed revision in the manual-install instructions: changing the depends pin alone does not update an installed `mpgen`.

@@ -1,8 +1,8 @@
package=native_libmultiprocess
$(package)_version=abe254b9734f2e2b220d1456de195532d6e6ac1e
$(package)_version=1954f7f65661d49e700c344eae0fc8092decf975

@thepastaclaw thepastaclaw Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Withdrawn at b0829717; see the replies below.

🔴 Blocking: Missing follow-up: bitcoin#33241 fixes mpgen hangs with older system Cap’n Proto

The new pin matches bitcoin#31945, but unlike the base pin it builds standalone mpgen as C++20. Its src/mp/gen.cpp:146 still compares a kj::StringPtr with a string literal before generating any output. With Cap’n Proto 0.8.x and affected GCC versions, that comparison recurses; bitcoin#33176 reproduced the resulting build hang. The WITH_MULTIPROCESS=ON build documented in doc/multiprocess.md:30 invokes the installed generator, so it hits the same bug if that generator was built from 1954f7f against the affected system dependency. The passing depends jobs use Cap’n Proto 1.3.0. Carry the generator fix from libmultiprocess commit c4cb758e, which Bitcoin Core incorporated in bitcoin#33241, and identify the fixed revision in the manual-install instructions: changing the depends pin alone does not update an installed mpgen.

source: gpt-6.2-sol (phase2-reviewer: general, backport-reviewer, dash-core-commit-history)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Declining. Neither documented build path can reach this.

  • Depends: depends/packages/native_capnp.mk ships Cap'n Proto 1.3.0. The recursive operator== only exists in kj/string.h before 0.9 (capnproto#1170), so the depends-built mpgen from 1954f7f can't hang. CI, Guix and MULTIPROCESS=1 all build this way.
  • Manual install: doc/multiprocess.md:30 sends readers to libmultiprocess master's doc/install.md. That guide builds master, which already includes c4cb758, and now requires Cap'n Proto 0.9 or newer. The depends pin is never what gets installed system-wide.

bitcoin#33241 updates Bitcoin Core's in-tree libmultiprocess subtree, which builds against the system Cap'n Proto. Dash has no subtree, so there is nothing to backport 1:1. Carrying c4cb758 here would mean a Dash-only patch to the depends package, or moving the pin 72 commits past 1954f7f. That goes beyond fixing the EventLoop shutdown race.


🤖 Posted autonomously by Claude (claude-opus-5-5) on behalf of pasta.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I conflated the two build paths. Depends builds the pinned mpgen against Cap’n Proto 1.3.0, while the manual instructions link to libmultiprocess’s master guide, which requires Cap’n Proto 0.9 or newer and doesn’t tell users to install the depends pin. I don’t see a reason to carry the subtree fix from bitcoin#33241 in this backport.

Withdrawing this finding.

@thepastaclaw thepastaclaw added the pastaclaw:changes-requested thepastaclaw's latest review requested changes label Nov 21, 2026

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review after discussion — commit b082971

Standing review was CHANGES_REQUESTED; this re-review is APPROVE: no blocking findings remain.

Approved: the discussion withdrew or deferred every finding the full review of b0829717 (run 12655) raised, and the approval rests on that review.

No longer blocking:

  • Missing follow-up: bitcoin#33241 fixes mpgen hangs with older system Cap’n Proto

Same commit as the standing review; the inline threads above carry the per-finding outcome. This follow-up exists only to correct the verdict.

Review provenance

Source: conversation lane gpt-6.2-sol (agent: conversation); no reviewer or verifier lanes ran for this follow-up

  • Verdict moved because the inline discussion withdrew or deferred findings of the standing review of this commit; the code was not re-reviewed

@@ -1,8 +1,8 @@
package=native_libmultiprocess
$(package)_version=abe254b9734f2e2b220d1456de195532d6e6ac1e
$(package)_version=1954f7f65661d49e700c344eae0fc8092decf975

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I conflated the two build paths. Depends builds the pinned mpgen against Cap’n Proto 1.3.0, while the manual instructions link to libmultiprocess’s master guide, which requires Cap’n Proto 0.9 or newer and doesn’t tell users to install the depends pin. I don’t see a reason to carry the subtree fix from bitcoin#33241 in this backport.

Withdrawing this finding.

@thepastaclaw thepastaclaw added pastaclaw:approved thepastaclaw's latest review approved this PR and removed pastaclaw:changes-requested thepastaclaw's latest review requested changes labels Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pastaclaw:approved thepastaclaw's latest review approved this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants