Skip to content

refactor(compile): separate configurations, scheduled runs, and snapshots - #7737

Open
fujiajie666 wants to merge 53 commits into
mainfrom
refactor/compile_cron_version
Open

fujiajie666 wants to merge 53 commits into
mainfrom
refactor/compile_cron_version

Conversation

@fujiajie666

Copy link
Copy Markdown
Collaborator

Description

Separate persistent Compile configurations from executions and scheduled triggers. Configurations are selected by caller-supplied, creator-scoped compile_name; accepted executions keep independent task IDs, frozen inputs, snapshots and incremental baselines.

Human Involvement

  • A human participated in the implementation or review loop
  • This PR was generated entirely by AI agents without human participation in the loop

Related Issue

None linked.

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation update
  • Performance improvement
  • Test update

Changes Made

  • Add configuration creation, named execution, scoped configuration listing and run/output APIs; align CLI, Python/Go/TypeScript SDKs and the bundled runtime.
  • Persist schedule slots and execution identity separately from configuration ownership; preserve progress through cancellation, rollback, rejected submissions and recovery.
  • Queue target tagging independently, guard overlapping targets and serialize writes without waiting for indexing.
  • Use native Git snapshot history and commit metadata, support guarded single-file restores and empty initial snapshots, and make content-write snapshots opt-in.
  • Update bilingual API guides, the wiki Compile skill and regression coverage.

Compatibility

Configuration queries use compile_name instead of to; run endpoints replace operation endpoints. Snapshot version responses use Git commit IDs and before_snapshot pagination instead of a separate UUID/version-number index. Content writes require use_snapshot=true to request snapshots. Direct full-input Compile remains supported with deprecation headers; saved configuration and execution are separate API calls.

Testing

  • git diff --check origin/main...HEAD passed.
  • .venv/bin/python -m pytest -q -o addopts='' tests/service/test_compile_configuration_lifecycle.py tests/service/test_compile_scheduler_contract.py tests/server/test_compile_configurations.py: 121 passed, 1 failed (8 deprecation warnings).
  • Failure: test_slot_recovery_keeps_accepted_owner_after_executor_change raises ExternalTaskError: External task provider is not registered: compile in ExternalTaskService._provider before reaching the expected queue-retry error. This failure remains unresolved in the published branch.
  • Full Python, Rust, Go and TypeScript suites were not run.

Checklist

  • Added regression tests for configuration lifecycle, scheduling, authorization and snapshot behavior
  • Updated corresponding documentation and SDKs
  • Full cross-language test suite passes locally

Additional Notes

The working tree is clean. This PR publishes the existing branch commits. A merge-tree check against the fetched main completed without conflicts.

…_version

# Conflicts:
#	tests/server/test_mcp_endpoint.py
#	tests/test_task_tracker.py
Use compile_name across configuration storage, scheduling, CLI, SDKs, Studio and documentation. Enforce account-wide uniqueness and preserve case-sensitive names on local filesystems. Simplify redundant tests while retaining validation, isolation and recovery coverage.
…ion check

Hide the internal input signature and configuration owner from public compile task metadata.
Require read permission for snapshot commits and restore authenticated executor roles and groups for compile lifecycle operations without ACL bypass. Preserve scheduled execution identity and update snapshot documentation.

Validation: 17 focused tests passed; Ruff and diff checks passed.
Use args.owner_user_id for configuration creation and updates without forwarding it to Runtime. Preserve the saved update schedule when updates are disabled or the schedule is omitted.

Validation: 110 related tests and the focused scheduler regression passed; Ruff and diff checks passed. Test changes add five lines and remove four.
Preserve the compile configuration lifecycle while adapting committed-output reporting and retry checkpoints. Align compile target creation with the current tree-lock API and cover recovery and cancellation behavior.
Remove persisted conflict counts and finalization searches while retaining rollback reindex scheduling. List configurations by account and owner without filtering output access, and update tests and documentation.
…access

Keep snapshot output reads subject to diff permissions and clarify the CLI guidance for inline inputs.
…refactor/compile_cron_version

# Conflicts:
#	crates/ragfs/src/lock/manager.rs
#	tests/server/test_bot_proxy_auth.py
#	tests/server/test_compile_studio.py
Keep snapshot-planned source selection and align simple Compile with the current request schema.

@qin-ctx qin-ctx left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

配置、执行和原生快照分离的方向有价值,值得保留。目前确认了三处项目自带前端的功能回归,以及历史清理新增的大量重复读取,需要在合并前解决。另有一处运行历史分页的非阻塞存储开销建议,具体见行内评论。

content: str
mode: str = "replace"
wait: bool = False
use_snapshot: bool = False

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Bug] (blocking)

用户在项目自带的文件浏览器中编辑文本并保存,现在会收到 HTTP 400,文件没有写入。frontend/packages/react/src/modules/file-browser/fs-file-editing.ts 每次保存仍发送 edit_source: 'manual',这里已删除该字段,同时保留了 extra='forbid'。

用相同请求对比真实 HTTP 路由,并替换下游写入服务:base 会进入 fs.write 并返回 200,本 PR 则在参数校验时返回 body.edit_source: Extra inputs are not permitted,没有进入写入流程。普通 Markdown、JSON、文本编辑和 Wiki 文件都会受影响。合并前需要同步迁移项目自带的保存请求。

)
if key in operation["meta"]
},
"configs": page,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Bug] (blocking)

用户打开 Studio 的 Compile 定时配置页时,即使接口请求成功,页面也会报错:schedule-list.tsx 仍执行 page.targets.map(...),而这里已改为返回 configs。共享文件浏览器中的 readCompileTargets() 也会拒绝这个响应,导致 Wiki 面板加载失败。

这些前端还在按 to 查询 /compile/config,通过携带完整输入的 /compile 请求创建定时计划。用真实 HTTP 路由和下游服务替身对比,相同请求在 base 被接受,在本 PR 返回 400。文档已描述 Studio 按名称管理配置,但 web-studio 和共享前端没有同步迁移,需要在合并前完成。

page = history[:limit]
return {
"uri": uri,
"versions": page,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Bug] (blocking)

版本接口改为返回原生 Git commit 后,Wiki 文件历史对话框无法再读取历史正文。frontend/packages/react/src/modules/file-browser/preview-native.ts 中的 readPreviewHistory() 仍按旧结构读取 version_id、version_number、snapshot_id、snapshot_path 和 created_at。收到新的 commit 记录后,版本名称变成 undefined,版本 ID、快照和文件路径也都缺失;随后 readHistoricalText() 发出的 /snapshot/show 请求缺少必填的 target_ref,会被参数校验拒绝。

已分别使用 base 和新接口的响应结构运行实际前端适配函数,确认只有旧结构能生成有效的历史正文请求。现有回退只处理 HTTP 404,版本接口成功返回新结构时不会触发。合并前需要将这个消费者一并迁移。

bypass_acl=True,
)
continue
tasks = await get_task_tracker()._store.list_durable(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Design] (blocking)

每清理一条过期运行历史,这里都会重新读取整个账号的任务文件,再按 Task ID 找其中一条。PersistentTaskStore.list_durable() 不使用任务缓存,而且读取全部任务类型的文件后才筛选,因此读取量随“过期历史数量 × 账号任务文件数量”增长。

用实际存储实现和内存 AGFS 替身计数:账号保留 1000 个其他任务,清理 100 条过期历史,base 读取 201 次,本 PR 读取 100101 次,其中 10 万次是重复读取任务文件。在未启用 CacheFS 的 S3 后端,这些读取会转化为重复对象 GET;每批最多处理 256 条历史并不能限制这部分开销。

每条历史已经保存执行者和 Task ID,本 PR 也提供了不依赖缓存的 get_durable_task()。读取最新任务状态不需要为每条历史扫描整个账号,合并前需要消除这处读放大。

for entry in entries:
name = str(entry.get("name") or "")
if name.endswith(".json"):
run = await self.read_json(f"{directory}/{name}")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Design] (non-blocking)

查询一页运行历史,现在会先读取全部保留的运行 JSON,再应用游标和数量限制。用 1000 条历史计数,存储层取 21 条记录(20 条结果加 1 条用于判断下一页)时,base 的操作历史读取 21 份文件,这里读取 1000 份;每次翻页还会重新全读。读取的也不只是排序字段,还包括完整的执行输入和结果。

七天保留期限制了时间,但没有限制运行次数。建议让排序和分页先确定本页需要读取的记录,减少对象存储读取;是否处理及具体采用哪种实现,由作者决定。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

2 participants