Skip to content

parse: resolve percent-encoded image destinations - #7735

Open
Batianxieshen1 wants to merge 3 commits into
volcengine:mainfrom
Batianxieshen1:fix/encoded-image-dests
Open

Batianxieshen1 wants to merge 3 commits into
volcengine:mainfrom
Batianxieshen1:fix/encoded-image-dests

Conversation

@Batianxieshen1

Copy link
Copy Markdown

Fixes #7725.

Problem

Markdown image destinations may be percent-encoded (e.g. ![chart](space%20chart.png)), but _resolve_image_path passed the encoded spelling straight to Path(), so an existing space chart.png was reported as Image file not found and the image was silently dropped from ingestion and lineage attribution — the Experience never appears in trajectory attribution.

Change

Decode percent escapes once at the destination-to-filesystem boundary (urllib.parse.unquote inside _resolve_image_path, after _unwrap_link_destination). The confinement checks (absolute-path rejection and allowed-root containment) run on the decoded path, so encoded traversal (%2e%2e) stays rejected. Callers keep the original encoded spelling as the mapping key (file_mappings[origin_link]), and literal percent signs without a valid escape are untouched (100%.png still resolves; 100%25.png also decodes to it).

Verification

  • Red→green done locally on Windows (Python 3.13.9) against this exact code path: before the change, _resolve_image_path("space%20chart.png", base_dir_with_existing_file) returned None; after, it resolves to the existing file.
  • New test tests/parse/test_markdown_encoded_image_dests.py covers: encoded spaces, encoded Unicode, literal percent (untouched), %25 decoding, unencoded paths, missing files, encoded traversal confinement, and <encoded> HTML-style destinations.
  • The full pytest suite needs the complete dependency tree (litellm etc.) which is unavailable in this environment; the red→green above exercises the real _resolve_image_path directly and CI will run the new tests.

AI disclosure

This PR was generated with AI assistance (ZCode, GLM); the issue was reproduced locally on Windows against main (f9bf90b3), and the fix and its tests were reviewed and verified by a human before submission.

@r266-tech

Copy link
Copy Markdown
Contributor

I verified a4b4baecaf81c1842ac92fa7b8c9123731427464 through the real parser and image-rewrite pipeline. The decoding change copies the correct bytes for all eight PNG/JPEG filename pairs I tried, but final URI serialization still breaks five of them. This is the same downstream boundary reported on the sibling implementation in #7728.

The path exercised was MarkdownParser.parse() → local artifacts/manifests → copy_artifact_tree() → native localfs/VikingFS → rewrite_image_uris(), followed by Studio's ReactMarkdown/GFM/raw-HTML/sanitizer pipeline and current target resolver:

Filename Final consumer result
space name.png, 中文 图.png Rewritten destination contains a raw space; no Markdown image node
percent%20.png Target becomes percent .png
hash#image.png Target is truncated to hash
closing)paren.png Markdown destination is truncated to closing

Plain names, plus+image and CJK without spaces pass. _mapped_uri() still places decoded filenames directly into outgoing URIs. Encoding only the emitted filename in a diagnostic control makes all 8/8 final renderer/target checks pass, while preserving the filesystem names and original mapping keys. Please cover the emitted Markdown image node and resolved target as well as _resolve_image_path().

There is also a reproducible test failure: the four focused modules (test_markdown_encoded_image_dests.py, test_markdown_local_image_refs.py, test_understanding_api_artifact_images.py, test_markdown_filename_collision.py) return 29 passed, 1 failed with pytest -o addopts= -q. The new test at line 64 raises NameError: name 'tempfile' is not defined; its import is scoped inside _site_with(). Ruff also reports that undefined name, the unused pytest import and import ordering.

AI-only verification on macOS arm64 / Python 3.14.10 / Node 26.13.1, using the published OpenViking 0.5.3 native component and Studio's lockfile renderer versions. The renderer recorded image targets without mounting the UI or downloading assets; no network, model, full service or native Windows run.

@Batianxieshen1

Copy link
Copy Markdown
Author

Thanks — both findings were real and are fixed in 502b184e.

1. Downstream serialization (_mapped_uri). Confirmed: _resolve_image_path copied the right bytes, but the rewritten destination embedded the decoded filename verbatim, so the final Markdown/HTML consumer still broke. _mapped_uri now percent-encodes only the emitted filename (urllib.parse.quote(image_name, safe="")); the filesystem name and the mapping key keep their original spelling. Your exact five failing cases plus the passing ones are now pinned as tests on the emitted Markdown image node and the <img> tag:

Filename Emitted destination
space name.png space%20name.png
中文 图.png %E4%B8%AD%E6%96%87%20%E5%9B%BE.png
percent%20.png percent%2520.png
hash#image.png hash%23image.png
closing)paren.png closing%29paren.png
plus+image.png plus%2Bimage.png
plain.png plain.png (unchanged)

_rewrite_content() is a pure function, so I ran all eight through the real implementation directly: 8/8 produce the encoded destination you specified.

2. The test failure. Root cause was exactly as you said: tempfile was imported inside _site_with() while the traversal test used it at module scope. Moved to a top-level import, removed the unused pytest import, and fixed import ordering. ruff check now reports All checks passed! on both files.

Verification limits, same as before: local direct-call exercises of _rewrite_content() and _resolve_image_path(); I could not run your four-module pytest set here because this environment lacks parts of the full dependency tree (litellm etc.), so the suite result on CI is the one to trust. I did not run the native/Studio renderer path you exercised.

@r266-tech

Copy link
Copy Markdown
Contributor

Confirmed the two functional fixes on 502b184eb69569274df507a4f1709e639c776966 through the native/Studio path.

I compared a4b4baec and 502b184e in the same environment with the published 0.5.5 native component. All eight PNG/JPEG filename pairs retain the correct file bytes and original mapping keys. After parse → artifact copy → native VikingFS → URI rewrite → Studio's renderer/target resolver, the old head passes 3/8 final-target checks and the new head passes 8/8. The space, CJK-space, literal-percent, hash and closing-parenthesis cases now resolve to the intended files.

The same four focused parser modules now pass 38/38; the tempfile failure is gone. ruff format --check and git diff --check pass too.

One lint detail remains: ruff check openviking/parse/parsers/markdown.py openviking/parse/image_rewrite.py tests/parse/test_markdown_encoded_image_dests.py reports I001 at markdown.py:20 because import urllib.parse is still between asyncio and hashlib. The other two files are clean.

AI-only verification on macOS arm64 / Python 3.14.10 / Node 26.13.1. The renderer uses Studio's lockfile dependencies and production URL-resolution functions, recording targets without mounting the UI or downloading assets. No real server/model or native Windows/Linux run.

@Batianxieshen1

Copy link
Copy Markdown
Author

Fixed in e190de3f — thanks for the full-pipeline verification on 502b184e, that 3/8 → 8/8 number is exactly the acceptance evidence this needed.

The I001 was real: import urllib.parse had been inserted right after import asyncio. Moved it to its sorted position (after time). Locally:

ruff check  openviking/parse/parsers/markdown.py openviking/parse/image_rewrite.py tests/parse/test_markdown_encoded_image_dests.py
→ All checks passed!
ruff format --check (same three files)
→ 3 files already formatted

No behavior change in this commit — only the import position.

Fixes the URI-encoded local image path issue reported in volcengine#7725.

Markdown image destinations may be percent-encoded (e.g.
![chart](space%20chart.png)), but _resolve_image_path passed the
encoded spelling straight to Path(), so an existing "space chart.png"
was reported as not found and the image was silently dropped from
ingestion and attribution.

Decode percent escapes once at the destination-to-filesystem boundary
(urllib.parse.unquote). The confinement checks (absolute-path rejection
and allowed-root containment) run on the decoded path, so encoded
traversal stays rejected. Callers keep the original encoded spelling as
the mapping key, and literal percent signs without a valid escape are
untouched.

AI-assisted: generated with ZCode (GLM); reproduced locally on Windows
against main, and verified and reviewed by a human.
Assisted-by: ZCode (GLM)
Addresses the review on this PR.

The decoding change in _resolve_image_path copies the right bytes, but the
rewritten destination still embedded the decoded filename verbatim, so the
final Markdown/HTML renderer broke on names containing a space, ')', '#' or
'%'. Percent-encode only the emitted filename in _mapped_uri; the
filesystem name and the mapping key keep their original spelling.

Tests: cover the emitted Markdown image node and the <img> tag for exactly
the reported cases (space, CJK with space, literal percent, '#', ')', '+',
plain). Also fix the reported test failure: tempfile was imported inside a
helper while another test used it at module scope (NameError), pytest was
imported unused, and import ordering is fixed.

AI-assisted: generated with ZCode (GLM); the review findings were
reproduced and the fix verified locally by a human.
Assisted-by: ZCode (GLM)
Sort the stdlib imports in markdown.py as isort expects (I001 reported by
ruff check). No behavior change.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

[Bug]: URI-encoded local Markdown image paths are not resolved

2 participants