Repository navigation
feat: cache the package dependency graph keyed by package.xml content hash - #22
Merged
Merged
Conversation
… hash DependencyGraphAnalyzer.scan() re-parsed every package.xml with catkin_pkg on each invocation, which is a measurable cost on autoware-scale workspaces where `ws build` / `inspect` runs the scan repeatedly. Use the previously-defined but unused .cwm/cache directory to persist the scanned graph as JSON, keyed by a content hash (sha256) of every package.xml. A content hash (not mtime) is used so same-second edits are not missed, and the relative path of each file is folded into the key so additions, removals and renames invalidate the cache too. The entry is stored under .cwm/cache/dag/<fingerprint>.json and written atomically (tmp + os.replace) so concurrent worktrees/agents cannot corrupt it; identical package sets are shared across worktrees since paths are stored relative to src. scan() gains an opt-in cache_dir kwarg and stays backward compatible when it is omitted. Cache misses (absent / stale / corrupt / format-version bump) fall back silently to a full rebuild. No CLI flag or env var is added: invalidation is fully automatic via the content hash. This is an additive, contract-preserving change (minor release).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an on-disk cache for the package dependency graph (release R7 of the
review-and-proposals roadmap, minor / v6.4.0).
DependencyGraphAnalyzer.scan()re-parsed everypackage.xmlwith catkin_pkgon each invocation, which is a measurable cost on autoware-scale workspaces
where
ws build/inspectrun the scan repeatedly. The cache lets a secondscan of an unchanged tree skip the XML parse entirely.
How it works
package.xmlundersrc/— eachfile contributes its relative path and the sha256 of its content, so
edits (even same-second), additions, removals, and renames all invalidate
the cache. mtime is intentionally not used.
.cwm/cache/dag/<fingerprint>.json(the previously-defined-but-unused
cache_path). Written atomically(tmp +
os.replace) so concurrent worktrees/agents can't corrupt it.Package paths are stored relative to
src, so worktrees with identicalpackage sets share the entry.
scan()gains an opt-incache_dirkwarg and stays backwardcompatible when omitted.
compute_changesetpassesconfig.dag_cache_dir.(absent / stale / corrupt / format-version bump) falls back silently to a
full rebuild.
Changes
src/cwm/core/dag_cache.py—compute_fingerprint,load_cached_graph,store_cached_graph,CACHE_FORMAT_VERSIONdependency_graph.py— sharediter_package_xmls()walk (single source of truth for_rebuild_from_src+ fingerprint),scan(cache_dir=...)load-or-rebuild,to_dict/from_dictconfig.py—dag_cache_dirpropertychangeset.py— passcache_dirintoscantest_dag_cache.py(new) +TestDGACache; sharedmake_package_xml/write_packagehelpers inconftest.pyTesting
uv run pytest -q→ 228 passedparse_package(monkeypatched to fail),rebuild on edit/add/remove, format-version mismatch, corrupt-cache fallback,
cross-worktree sharing with correct path restoration.