Skip to content

fix(semantic): keep live-context summaries recall-sized - #7697

Open
lester076 wants to merge 3 commits into
volcengine:mainfrom
lester076:fix/live-context-summary-code-leak
Open

lester076 wants to merge 3 commits into
volcengine:mainfrom
lester076:fix/live-context-summary-code-leak

Conversation

@lester076

Copy link
Copy Markdown

Fixes #7692

Problem

A live-context node summary inlined the complete mounted-argument code into the abstract: the live_context_summary prompt demanded "Preserve every mounted argument name and value", so a node whose args.code carries ~2 KB of JS produced a 2,536-character / 979-token summary. The recall assembler's default per-entry cap is 320 tokens, and resources entries have no fallback tier below abstract — so every automatic recall silently degraded this node to <memory detail="uri" />: path only, no content. The agent gets a bare URI it cannot act on without a blind read, which executes the bound MCP tool (~13 s, 4–5 upstream calls in the reporter's setup). Raising max_tokens only makes the 2.5 KB code land in context, which is not what a recall abstract is for.

What changed

  1. live_context_summary.yaml: the prompt now receives the mounted argument names and their total serialized size, and explicitly instructs the model to name the arguments without copying their values (values can carry kilobytes of code or data that belongs to the node file, not to a recall abstract). The full values remain available through the node file and the Tool definition at tool_uri, which the summary already points to.
  2. semantic_processor.py: the file-level summary gate now enforces semantic.abstract_max_chars (256 by default) via the existing _truncate_generated_text. This limit previously applied only to directory overviews and media parsers, so any oversized file summary could enter storage and only be dropped at injection time — the second gap the issue notes (same family as [Bug] Inconsistent abstract size limits: _enforce_size_limits caps at 256 chars but _truncate_abstract_bytes caps at 50,000 bytes #3005).

Verification

New tests/storage/test_live_context_summary.py:

  • the rendered prompt contains Mounted argument names: code and Mounted argument size:, and neither the mounted value nor any code fragment appears in the prompt (failed on main);
  • a generated summary over abstract_max_chars is truncated at the semantic-processing gate (failed on main);
  • a summary within budget passes through byte-identical (passes on main, still passes).

tests/storage/test_semantic_processor_language.py: one fixture (_create_mock_config) now mirrors the real config's semantic.abstract_max_chars, which the new gate reads; suite result returns to its main baseline (94 passed, the pre-existing timezone failure).

@r266-tech

Copy link
Copy Markdown
Contributor

Two findings on exact head c05678baafb71dfd102aced8f14789e1c1d3a345:

  1. The new file-summary gate does not enforce semantic.abstract_max_chars when the first sentence is long. I exercised _generate_single_file_summary() → actual live-context YAML parsing → actual prompt rendering → _generate_text_summary(), with the real default limit of 256 and an offline VLM returning these strings:

    VLM response Returned summary length on this head
    "word " * 200 252
    "word " * 200 + "." 1001
    "中" * 600 + "。" 601
    "Short. " + "中" * 600 + "。" 6

    _truncate_generated_text() deliberately returns the complete first sentence when no sentence ends within the limit (return text[:first_sentence_end].strip()). Reusing that helper therefore leaves the advertised file-level upper bound unenforced for common punctuated output. The 601-character CJK summary remains large enough to lose to the recall cap. These long-first-sentence outputs are unchanged from main@00c8e7b1; the new length gate only fixes some shapes. Please make the new file-summary boundary enforce its declared maximum and cover a long first sentence, while considering the existing directory helper's sentence-preserving behavior separately.

  2. An existing code-summary fallback test newly fails.

    python -m pytest tests/storage/test_live_context_summary.py \
      tests/storage/test_semantic_processor_code_routing.py --no-cov -q
    # 1 failed, 11 passed
    

    The failure is test_missing_skeleton_calls_llm_fallback: its _config() fixture does not set semantic.abstract_max_chars, so the new gate compares a MagicMock with 0 and raises TypeError. All nine code-routing tests pass on main@00c8e7b1; the PR's three new live-context tests pass on this head. This is a fixture update needed for the new config read, not evidence that real configured deployments raise this error.

The prompt change does remove the mounted argument value and retain the argument name in all five paired cases I ran, including real metadata parsing rather than stubbing _parse_live_context_metadata.

AI-only verification on macOS arm64/Python 3.14.10. Only filesystem lookup and VLM responses were substituted; no real model, embedding, native-storage operation, or deployed server was exercised. Seven dependency warnings were present in both test runs.

@lester076

Copy link
Copy Markdown
Author

Both findings fixed on 25af9fc8:

  1. File-summary gate now enforces the declared maximum. _truncate_generated_text gained a keyword-only hard_cap; when set (only by the new file-summary boundary), a long first sentence is cut inside the limit instead of being returned whole. Sentence-preserving behavior for the existing directory-overview callers is unchanged — _enforce_size_limits still calls the helper without hard_cap, and a new test pins that (a 601-char single CJK sentence still passes through whole in directory mode, and is cut to ≤256 in file mode). The reviewer's three response shapes ("word " * 200 + ".", "中" * 600 + ".", "Short. " + "中" * 600 + ".") are now parametrized cases asserting len(summary) <= 256; all three returned over-limit lengths on the previous head and pass now.

  2. _config() in tests/storage/test_semantic_processor_code_routing.py now sets semantic.abstract_max_chars = 256; test_missing_skeleton_calls_llm_fallback passes again.

Full affected suites: test_live_context_summary.py (8 tests) + test_semantic_processor_code_routing.py (9 tests) → 16 passed; test_semantic_processor_language.py → back to its main baseline (94 passed, the pre-existing timezone failure). ruff check + ruff format clean.

@r266-tech

Copy link
Copy Markdown
Contributor

@lester076 Confirmed both fixes on 25af9fc80319c32cd3d06e07a5438abc297faad4.

I reran the same production file-summary path (real YAML metadata parsing and prompt rendering, offline VLM responses) against c05678ba and this head in the same environment:

Response Old summary length New summary length
Long English first sentence 1001 254
Long CJK first sentence 601 256
Short sentence followed by a long CJK sentence 6 6

All five probe cases now stay within the configured 256 characters. The short summary is unchanged; mounted argument values stay out of the prompt while their names remain present.

The two focused modules now pass 16/16 (previous head: 11 passed, 1 failed). The existing test_semantic_processor_l0_l1.py also passes 16/16, including the directory contract that preserves a whole first sentence beyond its limit. Ruff check, format check, and the revision's git diff --check pass. This resolves both findings in my earlier comment.

AI-only verification on macOS arm64 / Python 3.14.10, using the same pinned dependencies and published 0.5.3 native component as the earlier review. Filesystem lookup and VLM responses were substituted; no real model, embedding, native-storage operation, deployed server, or full-suite run. Seven dependency warnings remain in each pytest run.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

[Bug]: live context 节点摘要把 mounted arguments 的完整代码内联,默认召回预算下只返回裸 URI

2 participants