Skip to content

feat: cache the package dependency graph keyed by package.xml content hash - #22

Merged
HansRobo merged 1 commit into
mainfrom
feat/dag-cache
Jun 3, 2026
Merged

HansRobo merged 1 commit into
mainfrom
feat/dag-cache

Conversation

@HansRobo

@HansRobo HansRobo commented Jun 3, 2026

Copy link
Copy Markdown
Owner

Summary

Adds an on-disk cache for the package dependency graph (release R7 of the
review-and-proposals roadmap, minor / v6.4.0).

DependencyGraphAnalyzer.scan() re-parsed every package.xml with catkin_pkg
on each invocation, which is a measurable cost on autoware-scale workspaces
where ws build / inspect run the scan repeatedly. The cache lets a second
scan of an unchanged tree skip the XML parse entirely.

How it works

  • Key: a sha256 fingerprint of every package.xml under src/ — each
    file contributes its relative path and the sha256 of its content, so
    edits (even same-second), additions, removals, and renames all invalidate
    the cache. mtime is intentionally not used.
  • Storage: serialised as JSON under .cwm/cache/dag/<fingerprint>.json
    (the previously-defined-but-unused cache_path). Written atomically
    (tmp + os.replace) so concurrent worktrees/agents can't corrupt it.
    Package paths are stored relative to src, so worktrees with identical
    package sets share the entry.
  • API: scan() gains an opt-in cache_dir kwarg and stays backward
    compatible when omitted. compute_changeset passes config.dag_cache_dir.
  • Invalidation is fully automatic — no CLI flag or env var. A cache miss
    (absent / stale / corrupt / format-version bump) falls back silently to a
    full rebuild.

Changes

  • new src/cwm/core/dag_cache.py — compute_fingerprint, load_cached_graph, store_cached_graph, CACHE_FORMAT_VERSION
  • dependency_graph.py — shared iter_package_xmls() walk (single source of truth for _rebuild_from_src + fingerprint), scan(cache_dir=...) load-or-rebuild, to_dict/from_dict
  • config.py — dag_cache_dir property
  • changeset.py — pass cache_dir into scan
  • tests — test_dag_cache.py (new) + TestDGACache; shared make_package_xml/write_package helpers in conftest.py

Testing

  • uv run pytest -q → 228 passed
  • Key coverage: cache hit skips parse_package (monkeypatched to fail),
    rebuild on edit/add/remove, format-version mismatch, corrupt-cache fallback,
    cross-worktree sharing with correct path restoration.

… hash

DependencyGraphAnalyzer.scan() re-parsed every package.xml with catkin_pkg
on each invocation, which is a measurable cost on autoware-scale workspaces
where `ws build` / `inspect` runs the scan repeatedly.

Use the previously-defined but unused .cwm/cache directory to persist the
scanned graph as JSON, keyed by a content hash (sha256) of every package.xml.
A content hash (not mtime) is used so same-second edits are not missed, and
the relative path of each file is folded into the key so additions, removals
and renames invalidate the cache too. The entry is stored under
.cwm/cache/dag/<fingerprint>.json and written atomically (tmp + os.replace)
so concurrent worktrees/agents cannot corrupt it; identical package sets are
shared across worktrees since paths are stored relative to src.

scan() gains an opt-in cache_dir kwarg and stays backward compatible when it
is omitted. Cache misses (absent / stale / corrupt / format-version bump) fall
back silently to a full rebuild. No CLI flag or env var is added: invalidation
is fully automatic via the content hash.

This is an additive, contract-preserving change (minor release).
@HansRobo HansRobo added the release:minor Bump minor version on merge label Jun 3, 2026
@HansRobo
HansRobo merged commit 35003eb into main Jun 3, 2026
11 of 12 checks passed
@HansRobo
HansRobo deleted the feat/dag-cache branch June 3, 2026 05:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release:minor Bump minor version on merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant