You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
ipc_tests fails intermittently in the linux64_tsan unit-test run with a ThreadSanitizer data race inside libmultiprocess: close in mp::EventLoop::loop() races with write in mp::EventLoop::removeClient() on the event loop's wakeup socket. It has failed on two unrelated PRs so far, with identical stacks:
Both started after the CMake cut-over began running the unit tests via ctest in the TSan build job. The bug itself is in the libmultiprocess version Dash pins (abe254b, from bitcoin#31105).
Mechanism: removeClient() drops the client count, unlocks, then writes to the wakeup fd. It wakes the loop even while async work is still queued. When two such writes land, the loop can wake on the first, see the exit condition, leave, and close() the fd while another thread is still about to write() to it. TSan reports the unsynchronized close/write; without TSan this can show up as the write(m_post_fd, ...): Broken pipe crash of bitcoin#31151. A related race in the same code makes EventLoop::post() relock m_mutex after loop() has returned and the EventLoop has been destroyed (libmultiprocess#154, bitcoin#31921). In Dash's IpcTest(), disconnect_client() hits that through loop.sync(), and it shows up as a hang or a TSan "use of an invalid mutex" report.
Upstream fixed both in libmultiprocess and pulled the fixes into Bitcoin Core with two version bumps, which this PR backports instead of patching libmultiprocess locally.
What was done?
depends: Update libmultiprocess library before converting to subtree bitcoin/bitcoin#31740 bumps libmultiprocess to 07c917f, which includes libmultiprocess#129 ("Fix write(m_post_fd, ...): Broken pipe EventLoop shutdown races"). It also carries the Bitcoin-side API updates: cleanup → cleanup_fns, the explicit mp/type-*.h includes after the proxy-types.h split, and the new echo-types.h with echo.capnp switched to includeTypes. It also adds the upstream passScript IPC regression test.
Dash's FooInterface lacks passTransaction/passVectorChar/passBlockState (multiprocess: Add IPC wrapper for Mining interface bitcoin/bitcoin#30510), so passScript uses capnp field ordinal 3. ipc_test.cpp includes util/strencodings.h for HexStr(), which upstream gets through includes Dash's test doesn't have.
init-types.h includes mp/proxy-types.h and the mp/type-*.h headers that init.capnp needs. Upstream gets them indirectly through mining.capnp.proxy-types.h, and Dash has no Mining interface, so without them dash-node doesn't compile against the new library.
How Has This Been Tested?
Local TSan build (-DSANITIZERS=thread, clang 19, libstdc++, depends with MULTIPROCESS=1 CC=clang-19 CXX=clang++-19, matching the CI multiprocess depends used by linux64_tsan). test_dash --run_test=ipc_tests was run in a loop, 48 at a time, on a heavily loaded x86_64 host:
The races reproduced locally have the same stacks as the CI failures. For the hangs, gdb showed the main thread blocked in pthread_mutex_lock inside EventLoop::post(), called from disconnect_client() after the loop thread had exited.
Also built dash-node and test_dash with the final branch (no warnings), ran ipc_tests, and ran lint-whitespace, lint-includes, lint-include-guards and lint-files.
Breaking Changes
None. libmultiprocess is only used by the experimental multiprocess build (MULTIPROCESS=1) and is not part of release builds.
Checklist:
I have performed a self-review of my own code
I have made corresponding changes to the documentation
I have assigned this pull request to a milestone (if you have permission to set one)
…onverting to subtree
4e0aa18 test: Add test for IPC serialization bug (Ryan Ofsky)
2221c88 depends: Update libmultiprocess library before converting to subtree (Ryan Ofsky)
Pull request description:
This should be the final update to the libmultiprocess package via the depends system. It brings in the libmultiprocess cmake changes from bitcoin-core/libmultiprocess#136 needed to support building as subtree. After this, followup PR bitcoin#31741 will add libmultiprocess as a git subtree and depends will just use the git subtree instead of hardcoding its own version hash.
Since there have been libmultiprocess API changes since the last update, this commit also updates bitcoin code to be compatible with them.
This update has the following new changes since previous update bitcoin#31105:
bitcoin-core/libmultiprocess#121 ProxyClientBase: avoid static_cast to partially constructed object
bitcoin-core/libmultiprocess#120 proxy-types.h: add static_assert to detect int/enum size mismatch
bitcoin-core/libmultiprocess#127 ProxyClientBase: avoid static_cast to partially destructed object
bitcoin-core/libmultiprocess#129 Fix "disconnected: write(m_post_fd, &buffer, 1): Broken pipe" EventLoop shutdown races.
bitcoin-core/libmultiprocess#130 refactor: Add CleanupRun function to dedup clean list code
bitcoin-core/libmultiprocess#131 doc: fix startAsyncThread comment
bitcoin-core/libmultiprocess#133 Fix debian "libatomic not found" error in downstream builds
bitcoin-core/libmultiprocess#94 c++ 20 cleanups
bitcoin-core/libmultiprocess#135 refactor: proxy-types.h API cleanup
bitcoin-core/libmultiprocess#136 cmake: Support being included with add_subdirectory
bitcoin-core/libmultiprocess#137 doc: Fix broken markdown links
ACKs for top commit:
Sjors:
ACK 4e0aa18
vasild:
ACK 4e0aa18
TheCharlatan:
ACK 4e0aa18
Tree-SHA512: 6d81cdf7f44762c7f476212295f6224054fd0a61315bb54786bc7758a2b33e5a2fce925c71e36f7bda320049aa14e7218a458ceb03dacbb869632c466c4789b0
(cherry picked from commit ad2f932)
Dash adaptations:
- native_libmultiprocess keeps the bitcoin-core/libmultiprocess download
path from bitcoin#31982; the archive hash is the same as upstream's.
- Dash's common-types.h never had the std::chrono CustomBuildField and
CustomReadField overloads or the generic Data CustomBuildField that
upstream removes here. They came with the parts of bitcoin#30510 Dash
did not take, so only the new mp/type-*.h includes apply.
- Dash's FooInterface lacks passTransaction, passVectorChar and
passBlockState (also from bitcoin#30510), so passScript takes capnp
field ordinal 3 and its check follows passUniValue in IpcTest().
ipc_test.cpp includes util/strencodings.h for HexStr(), which upstream
got through ipc_test.h's validation.h and util/fs.h includes that
Dash's test does not have.
- init-types.h includes mp/proxy-types.h and the mp/type-*.h headers
init.capnp needs (context, decay, interface, threadmap). Upstream's init-types.h gets
them through mining.capnp.proxy-types.h and common-types.h, but Dash
has no Mining interface, so without them dash-node fails to compile
against the new library.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…I failures
01f7715 depends: Update libmultiprocess library to fix CI failure (Ryan Ofsky)
Pull request description:
Bump libmultiprocess library to include bugfix bitcoin-core/libmultiprocess#159 which should fix intermittent CI failures reported in bitcoin#31921
This change is bumping the libmultiprocess version instead of cherry picking the bugfix. It could cherry-pick the bugfix instead, but there are reasons to prefer bumping the version:
- Bugfix might interact with earlier PRs, and the latest version is better tested with testing done in many CI configurations in [bitcoin#30975](bitcoin#30975) and [bitcoin#31802](bitcoin#31802)
- Even though we are in feature freeze for a release, the MULTIPROCESS=1 option is currently not enabled for release, so this PR only affect CI builds and local builds, not the release build.
This update brings in the following changes:
bitcoin-core/libmultiprocess#140 build: don't clobber user/superproject c++ version
bitcoin-core/libmultiprocess#142 build: add option for external mpgen binary
bitcoin-core/libmultiprocess#143 cleanup: initialize vars in the EventLoop constructor in the correct order
bitcoin-core/libmultiprocess#146 cmake: Suppress compiler warnings from capnproto headers
bitcoin-core/libmultiprocess#147 cmake: EXTERNAL_MPGEN cleanups
bitcoin-core/libmultiprocess#148 util: fix -Wpessimizing-move warning
bitcoin-core/libmultiprocess#145 CTest: Module must be included at the top level
bitcoin-core/libmultiprocess#149 Avoid `-Wundef` compiler warnings
bitcoin-core/libmultiprocess#152 refactor: Fix compiler and clang-tidy warnings
bitcoin-core/libmultiprocess#155 scripted-diff: s/Libmultiprocess_EXTERNAL_MPGEN/MPGEN_EXECUTABLE/g
bitcoin-core/libmultiprocess#156 refactor: Remove locale-dependent function calls
bitcoin-core/libmultiprocess#157 refactor: Avoid using std::format
bitcoin-core/libmultiprocess#159 bugfix: Do not lock EventLoop::mutex after EventLoop is done
bitcoin-core/libmultiprocess#161 cmake: Avoid including CTest if not top level project
bitcoin-core/libmultiprocess#164 Bump minimum required cmake to 3.12
---
This PR is part of the [process separation project](bitcoin#28722).
ACKs for top commit:
fanquake:
ACK 01f7715
Tree-SHA512: a6a795e4d4e13e9d35c9346f3c83d5da817f1452bdc4a9412aeadc4c652ad6e5047f4c77756570594a70ec9095cc786772a0469e306dc19bb5a508fd515c37f4
(cherry picked from commit ba0a439)
Dash adaptations:
- native_libmultiprocess keeps the bitcoin-core/libmultiprocess download
path from bitcoin#31982; the archive hash is the same as upstream's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review was skipped as selected files did not have any reviewable changes.
Run configuration
Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: Repository UI
Review profile: CHILL
Plan: Advanced
Run ID: 9d0590e6-5713-4101-919b-ef7955feb19c
Commits
Reviewing files that changed from the base of the PR and between 3943351 and b082971.
You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.
Use the checkbox below for a quick retry:
Trigger review
No actionable comments were generated in the recent review.
Walkthrough
The changes update the native libmultiprocess revision and add multiprocess type headers to the Cap’n Proto integration. The echo schema uses includeTypes, and CapnpProtocol::addCleanup stores callbacks in cleanup_fns. The IPC test interface adds passScript, with a CScript implementation that returns its argument. The test checks that a script containing OP_11 retains its hexadecimal encoding after passing through the interface.
This backport updates the experimental multiprocess IPC library to fix shutdown races and adds a script pass-through test. Release builds are unaffected. No code defects were found. The only failing check is the PR description's @mention, which should be removed before merge.
Pre-merge checks: 4
Check name
Status
Explanation
Title check
The title names the two backported upstream Bitcoin Core changes (bitcoin#31740 and bitcoin#31945) and the libmultiprocess EventLoop shutdown race fixes. The changeset bumps libmultiprocess to 1954f7f…
Description check
The description explains the intermittent ipc_tests TSan data race in libmultiprocess, the two upstream fixes, the Dash-specific adaptations (passScript ordinal, includes, depends download path), and …
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Full details: Title check
Explanation
The title names the two backported upstream Bitcoin Core changes (bitcoin#31740 and bitcoin#31945) and the libmultiprocess EventLoop shutdown race fixes. The changeset bumps libmultiprocess to 1954f7f and 07c917f, adds the EventLoop race fixes, and includes the related API and include updates, so the title reflects the main change. It is somewhat terse in that it names backport numbers rather than the behaviour, but it is specific and accurate.
Full details: Description check
Explanation
The description explains the intermittent ipc_tests TSan data race in libmultiprocess, the two upstream fixes, the Dash-specific adaptations (passScript ordinal, includes, depends download path), and the local testing results. It directly relates to the changeset files.
Finishing Touches
Generate unit tests (beta)
Create a new PR
Fix failing CI checks
Create a new PR
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
File: No configuration file from this repository applied
Commit: Unavailable
Layers: Repository UI
Review profile: CHILL
Plan: Advanced
Run ID: 6035cd45-981a-4166-a0e5-dd881c92063f
Commits
Reviewing files that changed from the base of the PR and between f62ee01 and 3943351.
Files selected for processing: 9
depends/packages/native_libmultiprocess.mk
src/ipc/capnp/common-types.h
src/ipc/capnp/echo-types.h
src/ipc/capnp/echo.capnp
src/ipc/capnp/init-types.h
src/ipc/capnp/protocol.cpp
src/test/ipc_test.capnp
src/test/ipc_test.cpp
src/test/ipc_test.h
Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.
The reason will be displayed to describe this comment to others. Learn more.
Final validation — Phase 1 + Phase 2
Updated after discussion: 1 finding(s) of this review were settled in the inline threads; no blocking finding remains.
✅ Withdrawn: 🔴 Missing follow-up: bitcoin#33241 fixes mpgen hangs with older system Cap’n Proto
At b082971, both backports carry the intended EventLoop shutdown fixes, and the multiprocess, TSan, and ASan checks passed. One blocking upstream follow-up is missing: building the newly pinned libmultiprocess revision against older system Cap’n Proto can hang during IPC schema generation.
Triage: low by gpt-6.2-sol (effort low) — The diff is a contained dependency bump with small IPC compatibility updates and a regression test; the shutdown race fixes are in the pinned library, not in Dash’s changed code.
These findings are from an automated code review (PastaClaw). How to work through them:
- Verify each finding against the code; fix only what is real, as narrowly as the finding.
- If a finding is wrong, decline it on its thread in one or two sentences; the bot answers once and stops.
- To defer one, link a tracking issue on its thread (or reply asking the bot to file one).
- Suggestions don't block the merge, but most are worth fixing in this PR; judge each on its merits.
- Keep the PR's scope; don't grow it to satisfy hardening suggestions.
- Carried items were raised in an earlier round: read the thread first, and don't re-fix what you already fixed, declined or deferred there. Disputed and deferred items are already handled and are not listed.
In `depends/packages/native_libmultiprocess.mk`:
- [BLOCKING → WITHDRAWN in discussion; do not fix] depends/packages/native_libmultiprocess.mk:2: Missing follow-up: bitcoin#33241 fixes mpgen hangs with older system Cap’n Proto
The new pin matches bitcoin#31945, but unlike the base pin it builds standalone `mpgen` as C++20. Its `src/mp/gen.cpp:146` still compares a `kj::StringPtr` with a string literal before generating any output. With Cap’n Proto 0.8.x and affected GCC versions, that comparison recurses; bitcoin#33176 reproduced the resulting build hang. The `WITH_MULTIPROCESS=ON` build documented in `doc/multiprocess.md:30` invokes the installed generator, so it hits the same bug if that generator was built from `1954f7f` against the affected system dependency. The passing depends jobs use Cap’n Proto 1.3.0. Carry the generator fix from libmultiprocess commit `c4cb758e`, which Bitcoin Core incorporated in bitcoin#33241, and identify the fixed revision in the manual-install instructions: changing the depends pin alone does not update an installed `mpgen`.
The reason will be displayed to describe this comment to others. Learn more.
✅ Withdrawn at b0829717; see the replies below.
🔴 Blocking: Missing follow-up: bitcoin#33241 fixes mpgen hangs with older system Cap’n Proto
The new pin matches bitcoin#31945, but unlike the base pin it builds standalone mpgen as C++20. Its src/mp/gen.cpp:146 still compares a kj::StringPtr with a string literal before generating any output. With Cap’n Proto 0.8.x and affected GCC versions, that comparison recurses; bitcoin#33176 reproduced the resulting build hang. The WITH_MULTIPROCESS=ON build documented in doc/multiprocess.md:30 invokes the installed generator, so it hits the same bug if that generator was built from 1954f7f against the affected system dependency. The passing depends jobs use Cap’n Proto 1.3.0. Carry the generator fix from libmultiprocess commit c4cb758e, which Bitcoin Core incorporated in bitcoin#33241, and identify the fixed revision in the manual-install instructions: changing the depends pin alone does not update an installed mpgen.
source: gpt-6.2-sol (phase2-reviewer: general, backport-reviewer, dash-core-commit-history)
The reason will be displayed to describe this comment to others. Learn more.
Declining. Neither documented build path can reach this.
Depends:depends/packages/native_capnp.mk ships Cap'n Proto 1.3.0. The recursive operator== only exists in kj/string.h before 0.9 (capnproto#1170), so the depends-built mpgen from 1954f7f can't hang. CI, Guix and MULTIPROCESS=1 all build this way.
Manual install:doc/multiprocess.md:30 sends readers to libmultiprocess master's doc/install.md. That guide builds master, which already includes c4cb758, and now requires Cap'n Proto 0.9 or newer. The depends pin is never what gets installed system-wide.
bitcoin#33241 updates Bitcoin Core's in-tree libmultiprocess subtree, which builds against the system Cap'n Proto. Dash has no subtree, so there is nothing to backport 1:1. Carrying c4cb758 here would mean a Dash-only patch to the depends package, or moving the pin 72 commits past 1954f7f. That goes beyond fixing the EventLoop shutdown race.
🤖 Posted autonomously by Claude (claude-opus-5-5) on behalf of pasta.
The reason will be displayed to describe this comment to others. Learn more.
I conflated the two build paths. Depends builds the pinned mpgen against Cap’n Proto 1.3.0, while the manual instructions link to libmultiprocess’s master guide, which requires Cap’n Proto 0.9 or newer and doesn’t tell users to install the depends pin. I don’t see a reason to carry the subtree fix from bitcoin#33241 in this backport.
The reason will be displayed to describe this comment to others. Learn more.
I conflated the two build paths. Depends builds the pinned mpgen against Cap’n Proto 1.3.0, while the manual instructions link to libmultiprocess’s master guide, which requires Cap’n Proto 0.9 or newer and doesn’t tell users to install the depends pin. I don’t see a reason to carry the subtree fix from bitcoin#33241 in this backport.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue being fixed or feature implemented
ipc_testsfails intermittently in thelinux64_tsanunit-test run with a ThreadSanitizer data race inside libmultiprocess:closeinmp::EventLoop::loop()races withwriteinmp::EventLoop::removeClient()on the event loop's wakeup socket. It has failed on two unrelated PRs so far, with identical stacks:Both started after the CMake cut-over began running the unit tests via ctest in the TSan build job. The bug itself is in the libmultiprocess version Dash pins (abe254b, from bitcoin#31105).
Mechanism:
removeClient()drops the client count, unlocks, then writes to the wakeup fd. It wakes the loop even while async work is still queued. When two such writes land, the loop can wake on the first, see the exit condition, leave, andclose()the fd while another thread is still about towrite()to it. TSan reports the unsynchronized close/write; without TSan this can show up as thewrite(m_post_fd, ...): Broken pipecrash of bitcoin#31151. A related race in the same code makesEventLoop::post()relockm_mutexafterloop()has returned and theEventLoophas been destroyed (libmultiprocess#154, bitcoin#31921). In Dash'sIpcTest(),disconnect_client()hits that throughloop.sync(), and it shows up as a hang or a TSan "use of an invalid mutex" report.Upstream fixed both in libmultiprocess and pulled the fixes into Bitcoin Core with two version bumps, which this PR backports instead of patching libmultiprocess locally.
What was done?
write(m_post_fd, ...): Broken pipe EventLoop shutdown races"). It also carries the Bitcoin-side API updates:cleanup→cleanup_fns, the explicitmp/type-*.hincludes after theproxy-types.hsplit, and the newecho-types.hwithecho.capnpswitched toincludeTypes. It also adds the upstreampassScriptIPC regression test.Dash adaptations, all recorded in the commit messages:
native_libmultiprocess.mkkeeps thebitcoin-core/libmultiprocessdownload path from scripted-diff: rename libmultiprocess repository bitcoin/bitcoin#31982. Both archive hashes are unchanged from upstream and were verified against that URL.common-types.hnever had thestd::chronooverloads or the generic DataCustomBuildFieldthat depends: Update libmultiprocess library before converting to subtree bitcoin/bitcoin#31740 removes (they came with the parts of multiprocess: Add IPC wrapper for Mining interface bitcoin/bitcoin#30510 Dash did not take), so only the new includes apply.FooInterfacelackspassTransaction/passVectorChar/passBlockState(multiprocess: Add IPC wrapper for Mining interface bitcoin/bitcoin#30510), sopassScriptuses capnp field ordinal 3.ipc_test.cppincludesutil/strencodings.hforHexStr(), which upstream gets through includes Dash's test doesn't have.init-types.hincludesmp/proxy-types.hand themp/type-*.hheaders thatinit.capnpneeds. Upstream gets them indirectly throughmining.capnp.proxy-types.h, and Dash has no Mining interface, so without themdash-nodedoesn't compile against the new library.How Has This Been Tested?
Local TSan build (
-DSANITIZERS=thread, clang 19, libstdc++, depends withMULTIPROCESS=1 CC=clang-19 CXX=clang++-19, matching the CI multiprocess depends used bylinux64_tsan).test_dash --run_test=ipc_testswas run in a loop, 48 at a time, on a heavily loaded x86_64 host:post()The races reproduced locally have the same stacks as the CI failures. For the hangs, gdb showed the main thread blocked in
pthread_mutex_lockinsideEventLoop::post(), called fromdisconnect_client()after the loop thread had exited.Also built
dash-nodeandtest_dashwith the final branch (no warnings), ranipc_tests, and ranlint-whitespace,lint-includes,lint-include-guardsandlint-files.Breaking Changes
None. libmultiprocess is only used by the experimental multiprocess build (
MULTIPROCESS=1) and is not part of release builds.Checklist:
🤖 Generated with Claude Code