diff --git a/CHANGELOG.md b/CHANGELOG.md index 9a72fb3..0333bc2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,7 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and - Separate companion navigation for text chat, voice chat, and game companion. The game surface remains explicitly unavailable until Windows.Graphics.Capture has validated real DirectX/Vulkan frames. - Screen-awareness implementation framework for scoped display consent, WGC capture, local privacy/quality gates, multimodal AstrBot validation, and low-interruption response control. +- WGC S0.1 target-binding preflight for authorized windows and attached displays. It returns metadata only; no capture session, pixels, disk write, or upload is performed. - Debug-level tracing of AstrBot SSE frame shapes: frame type, which identity fields are present, and the key names inside `data`. No payload text is recorded. Enable with `RUST_LOG`. - Q10-Q15 chat experience increment: lightweight Markdown rendering, message actions, generation controls, session actions, connection controls, and responsive chat layout. - AstrBot OpenAPI adapter foundation: SSE mapper, dual-slot runtime skeleton, credential enrollment IPC, transport-aware profiles (ADR-0005). diff --git a/docs/decisions/pending-decisions.md b/docs/decisions/pending-decisions.md index 4c2f904..6e652a3 100644 --- a/docs/decisions/pending-decisions.md +++ b/docs/decisions/pending-decisions.md @@ -114,10 +114,10 @@ - Date noted: 2026-07-27 - Status: direction selected; implementation remains blocked on the required full-desktop Spike and acceptance gates. Planning artifact: docs/reports/companion-ui-and-screen-awareness-plan-2026-07-27.md. -- Decision update (2026-07-27): the product owner selected option B, the scoped full-desktop direction. This approves discovery and design only; it does not authorize capture code before the required Spike passes. +- Decision update (2026-07-27): the product owner selected option B with three selectable scoped ranges: a named window, one display, or all current displays. This approves discovery and design only; it does not authorize capture code before the required Spike passes. - Conflict: the requested product direction asks for randomly timed screen captures after a settings toggle, while ADR-0008 only permits a single user-authorized foreground window and requires a visible indicator, a stop control, Companion-only delivery, no on-disk raw frames, and Rust-owned policy enforcement. - Proposed safe interpretation: “random” means random trigger time, not an unbounded random capture target. V1 may schedule captures only for a live, explicitly authorized foreground-window session. The interval is a hard minimum between successful captures and the configured probability is evaluated only after that interval has elapsed. -- Chosen scope direction: add a full-desktop capability after Spike approval. The eventual consent UI must require the user to select either one monitor or an explicit all-monitors scope; the scheduler is bound to that stable scope and may never choose a different monitor by itself. -- Mandatory gates before implementation: separate scope-specific consent, revised threat model, Windows.Graphics.Capture Spike results, lock-screen/full-screen/protected-content/sensitive-app handling, visible indicator and stop-path validation, Companion-only delivery, and real AstrBot visual acceptance. +- Chosen scope direction: add a full-desktop capability after Spike approval. The eventual consent UI must require the user to select one named window, one monitor, or an explicit all-monitors scope. The scheduler is bound to that stable scope and may never choose a different window or monitor by itself. Window identity changes, display changes, and scope changes revoke the active authorization. +- Mandatory gates before implementation: separate scope-specific consent, revised threat model, Windows.Graphics.Capture Spike results for all three ranges, window-identity/display-change handling, lock-screen/full-screen/protected-content/sensitive-app handling, visible indicator and stop-path validation, Companion-only delivery, and real AstrBot visual acceptance. - Non-negotiable exclusions: no hidden capture, browser-history collection, input logging, process injection, or automatic delivery to Operations. - Safe interim rule: preserve the current capture policy unchanged until the full-desktop Spike has passed and the product owner accepts its evidence. C1–C3 page work may be planned independently; no scheduler, persistence migration, or capture-scope expansion is implemented in this planning phase. diff --git a/docs/implementation-queue.md b/docs/implementation-queue.md index 6b0e2da..b2c015e 100644 --- a/docs/implementation-queue.md +++ b/docs/implementation-queue.md @@ -234,6 +234,15 @@ The following are deliberately not marked complete: HTTPS certificate pinning, s 验收:框架明确“随机仅指触发时间”、Companion-only、无原始内容持久化、真实视觉验收及 S0-S5 的逐阶段停止条件;本轮无 `src/`、`src-tauri/` 或数据库改动。 + +### S0.1:WGC 目标绑定预检(2026-07-27,完成;不读取像素) + +- 新增 Rust-owned WGC 预检:仅验证 Windows 是否支持 WGC,以及已授权窗口和当前已连接显示器能否绑定为 `GraphicsCaptureItem`;返回目标类型和尺寸元数据。 +- 预检通过 `CapturePolicy::inspect` 复用既有窗口授权,不读取窗口文本/像素、不消耗读取预算、不创建 capture session,也不触发编码、持久化、网络或 AstrBot 上传。 +- 仅为 S0 证据,不代表游戏、全屏、锁屏、受保护内容或真实画面捕获可用;S0.2 必须完成 D3D11 真实帧、黑帧/透明帧检测和 Windows 人工证据。 + +验收:隔离目标目录执行 `cargo check -p astrbot-desktop --lib` 通过;`cargo test -p astrbot-desktop --lib` 为 117 passed。 + ### C1:聊天、语音聊天、游戏陪玩三页面壳 - 在现有“聊天”之后增加“语音聊天”“游戏陪玩”两个独立导航入口。 @@ -246,7 +255,7 @@ The following are deliberately not marked complete: HTTPS certificate pinning, s - 在“诊断与设置”增加屏幕感知分组:总开关、授权状态、最短触发间隔、概率、活跃时段、停止和脱敏回执。 - 默认关闭;随机只表示触发时机,绝不随机选择未授权的窗口、显示器或应用。 -- 方案 B 已选:C2–C6 仍受 PD-0014 约束,必须先通过 WGC、显示器范围专属授权、锁屏/受保护内容和敏感应用 Spike,才可扩展当前单窗口范围。 +- 方案 B 已选:C2–C6 仍受 PD-0014 约束,必须先通过 WGC、窗口/显示器范围专属授权、锁屏/受保护内容和敏感应用 Spike,才可扩展当前单窗口范围。 验收:未授权、关闭、锁屏、授权过期、目标变化和断连均不能上传画面;活跃会话始终存在可见指示器与停止按钮。 @@ -260,9 +269,9 @@ The following are deliberately not marked complete: HTTPS certificate pinning, s ### C4:全桌面与游戏陪玩 Windows.Graphics.Capture Spike(方案 B) -- 先验证 WGC 对单显示器、全部显示器和 DirectX/Vulkan 游戏窗口的非黑帧捕获;记录 CPU、GPU、内存、显示器拔插、最小化、窗口切换、锁屏和受保护内容行为。 -- 设计独立的范围专属授权:用户必须明确选定单显示器或全部显示器;随机调度只能复用该范围,不能自行扩张或改选目标。 +- 先验证 WGC 对指定窗口、单显示器、全部显示器和 DirectX/Vulkan 游戏窗口的非黑帧捕获;记录 CPU、GPU、内存、窗口进程变化、显示器拔插、最小化、窗口切换、锁屏和受保护内容行为。 +- 设计独立的范围专属授权:用户必须明确选定指定窗口、单显示器或全部显示器;随机调度只能复用该范围,不能自行扩张、跟随或改选目标。 - Spike 通过前,游戏陪玩和全桌面感知只显示真实阻塞说明,不能发送 GDI 黑图或声称“正在观察”。 - 禁止输入模拟、进程注入和游戏内存读取;自动画面只送 Companion,始终显示指示器和停止入口。 -验收:单显示器和全部显示器均通过 WGC 真实画面、资源预算、敏感/受保护内容降级与可撤销授权测试后,才进入正式全桌面调度或游戏陪玩会话开发。 +验收:指定窗口、单显示器和全部显示器均通过 WGC 真实画面、资源预算、敏感/受保护内容降级与可撤销授权测试后,才进入正式全桌面调度或游戏陪玩会话开发。 diff --git a/docs/progress.md b/docs/progress.md index 97549c7..36e1ca3 100644 --- a/docs/progress.md +++ b/docs/progress.md @@ -730,7 +730,7 @@ Playwright 5 通过,`npm run build` 通过。25 处刻意回归全部被对应 - 已完成:根据当前 UI 截图定义“聊天、语音聊天、游戏陪玩”三个独立导航入口;屏幕感知配置归入“诊断与设置”。 - 已完成:审阅 menglimi/astrbot_plugin_screen_companion 的行为与配置维度,并写入本项目适配规划:docs/reports/companion-ui-and-screen-awareness-plan-2026-07-27.md。 - 已完成:将“随机定时全屏捕获”与 ADR-0008 的单窗口显式授权边界冲突记录为 PD-0014。 -- 已决定:产品所有者于 2026-07-27 选择方案 B(全桌面方向);它要求用户选定单显示器或全部显示器,并先完成 WGC、范围授权、锁屏/受保护内容、敏感应用与真实 AstrBot 图像验收。 +- 已决定:产品所有者于 2026-07-27 选择方案 B(全桌面方向);它要求用户选定指定窗口、指定显示器或全部显示器,并先完成 WGC、范围授权、锁屏/受保护内容、敏感应用与真实 AstrBot 图像验收。 - 规划建立时:C1–C8 的任何代码、数据库迁移、协议变更、调度器或 WGC Spike 均未开始;C1 的后续完成记录见本节下方。 - 阻塞:方案 B 的必要 Spike 尚未开始;游戏陪玩还受 F-80(WGC)阻塞。 - 测试状态:本次仅文档规划,未新增或重跑测试。 @@ -750,3 +750,24 @@ Playwright 5 通过,`npm run build` 通过。25 处刻意回归全部被对应 - **当前阻塞**:S0 WGC Spike 尚未执行;因此全桌面捕获、硬件加速游戏观察、定时自动观察和“Bot 已理解画面”都未实现。 - **下一步**:先执行 S0,并以非黑帧、保护内容/锁屏降级、资源预算、可撤销授权和可见停止入口的实测证据决定是否进入 S1。 - **验证**:本轮为文档评估,无功能代码、数据库、构建产物或协议改动,未运行构建门槛。 + + +## 2026-07-27 · 范围选择补充(未开发) + +- **产品决定**:屏幕感知范围必须在设置中由用户选择:指定窗口、指定显示器或全部显示器;三档均为独立授权,不存在自动跟随前台窗口或自动纳入新显示器。 +- **授权规则**:窗口关闭/最小化/进程身份变化,显示器拔插/标识变化,或用户改变范围时,立即停止旧会话并要求新范围重新确认。 +- **调度规则**:概率和定时器只复用已授权的稳定范围,不能改变、扩大或替换目标。 +- **后续**:S0 WGC Spike 的验收范围扩展为三档;只有每档真实帧、隐私降级、资源预算和停止路径均有证据,才开发该档自动观察。 + + +## 2026-07-27: WGC S0.1 target-binding preflight + +Added the first WGC Spike increment for the three approved capture scopes. Rust can now report WGC support and bind an already-authorized window or each currently attached display as a `GraphicsCaptureItem`, returning only type and dimensions. The window route goes through `CapturePolicy::inspect`, so it keeps the existing switch/session/identity validation and does not consume a reading budget. + +This increment does **not** create a `GraphicsCaptureSession`, read a frame, encode data, persist content, schedule a trigger, or upload to AstrBot. It is target-binding evidence only. S0.2 remains mandatory for D3D11 frame readback, black/transparent-frame rejection and manual game/lock/protected-content evidence. + +### Test status + +- Isolated Rust check: passed (`cargo check -p astrbot-desktop --lib`). +- Isolated Rust library tests: 117 passed, 0 failed. +- Frontend gates remain to be re-run before this increment is committed. diff --git a/docs/reports/companion-ui-and-screen-awareness-plan-2026-07-27.md b/docs/reports/companion-ui-and-screen-awareness-plan-2026-07-27.md index 42779c9..380bc20 100644 --- a/docs/reports/companion-ui-and-screen-awareness-plan-2026-07-27.md +++ b/docs/reports/companion-ui-and-screen-awareness-plan-2026-07-27.md @@ -104,7 +104,7 @@ Disabled -> SettingsConfigured -> AwaitingExplicitAuthorization -> ActiveAuthori | 字段 | 默认 / 范围 | 规则 | | ---------------------------- | ------------------------------- | ----------------------------------------------------------------------------------- | | enabled | false | 默认关闭;关闭立即撤销所有会话 | -| scope | selected_display / all_displays | 方案 B 已选;先完成 WGC、范围专属授权与隐私 Spike。调度器只能使用用户固定选定的范围 | +| scope | selected_window / selected_display / all_displays | 方案 B 已选;先完成 WGC、范围专属授权与隐私 Spike。调度器只能使用用户固定选定的范围 | | min_trigger_interval_minutes | 默认 20;5–240 | 成功发送后重新计时;这是硬下限 | | trigger_probability_percent | 默认 15;0–100 | 只在到达硬下限后抽样;0 表示永不自动触发 | | active_time_range | 默认全天 | 可选 HH:MM-HH:MM;范围外只允许手动一次性发送 | @@ -140,7 +140,7 @@ Disabled -> SettingsConfigured -> AwaitingExplicitAuthorization -> ActiveAuthori ### 4.1 阻塞性 Spike 1. **WGC 游戏捕获 Spike**:验证 WGC 能否在 DirectX/Vulkan 硬件加速窗口获取非黑帧,并测量 1 fps 下 CPU、GPU、内存、窗口切换、最小化和受保护内容行为。 -2. **全屏范围 Spike(方案 B 已选)**:以 Windows.Graphics.Capture 验证单显示器与“全部显示器”范围;定义范围专属授权文案、显示器切换、敏感应用/受保护内容处理、锁屏策略和 Windows 权限表现;不通过则不实现全屏。 +2. **全屏范围 Spike(方案 B 已选)**:以 Windows.Graphics.Capture 验证指定窗口、单显示器与“全部显示器”范围;定义窗口失效、显示器切换、范围专属授权文案、敏感应用/受保护内容处理、锁屏策略和 Windows 权限表现;不通过则不实现对应范围。 3. **真实 AstrBot 多模态验收**:确认服务端实际接收图片附件段、目标模型看见画面、失败时返回何种 SSE/错误。 4. **主动消息预算 Spike**:验证概率、冷却、手动聊天后静默期与通知权限,避免把“陪伴”实现成打扰。 @@ -165,14 +165,14 @@ Disabled -> SettingsConfigured -> AwaitingExplicitAuthorization -> ActiveAuthori | C4 | 屏幕感知设置页与 schema/迁移设计 | 方案 B 的 WGC / 隐私 Spike 通过 | 默认关闭、显示器范围专属授权、参数校验、停止路径和设置恢复测试完成 | | C5 | Rust 调度器与脱敏观察回执 | C4 | 概率、硬下限、活跃时段、断连/锁屏/授权撤销测试完成 | | C6 | 画面上传与 Companion 主动回应 | C5 + 真实 AstrBot 探针 | 服务端视觉确认、冷却策略、无原始画面落盘 | -| C7 | WGC 全桌面与游戏捕获、性能/隐私 Spike | C3 | 单显示器/全显示器范围、非黑帧、资源预算、全屏/最小化安全降级验收通过 | +| C7 | WGC 全桌面与游戏捕获、性能/隐私 Spike | C3 | 指定窗口/单显示器/全显示器范围、非黑帧、资源预算、全屏/最小化安全降级验收通过 | | C8 | 游戏陪玩正式会话 | C6 + C7 | 仅授权范围、可停止、无输入控制、真实游戏验收通过 | -**当前状态**:C1 已完成:左侧导航、语音聊天页和游戏陪玩页的前端壳已落地,仍共享既有连接/会话运行时,未增加本机权限。C2–C8 尚未开始;不得把本文件视为已交付的自动屏幕感知功能。 +**当前状态**:C1 已完成:左侧导航、语音聊天页和游戏陪玩页的前端壳已落地,仍共享既有连接/会话运行时,未增加本机权限。S0.1 已完成 WGC 目标绑定预检,但尚未创建捕获会话或读取像素;C2–C8 仍未开始。不得把本文件视为已交付的自动屏幕感知功能。 ## 6. 产品决定与仍需决定的事项 -- **已决定(2026-07-27)**:采用 PD-0014 的方案 B,全桌面方向。全桌面不是隐式默认权限:启用时必须由用户明确选定单显示器或全部显示器;随机只影响触发时间,绝不改变已选范围。 +- **已决定(2026-07-27)**:采用 PD-0014 的方案 B,全桌面方向。全桌面与窗口感知都不是隐式默认权限:启用时必须由用户明确选定指定窗口、单显示器或全部显示器;随机只影响触发时间,绝不改变已选范围。 - **仍需决定**:“低打扰”默认值是否采用本文建议的 20 分钟 / 15% / 30 分钟回复冷却? - **仍需决定**:游戏陪玩在 WGC 完成前只显示准备页,还是完全隐藏入口?本文建议显示准备页与真实阻塞原因。 - **实施门槛**:方案 B 在 WGC、范围授权、锁屏/受保护内容、敏感应用、可见指示/停止和真实 AstrBot 图像验收均通过前,不得进入捕获、调度或上传实现。 diff --git a/docs/reports/screen-awareness-understanding-framework-2026-07-27.md b/docs/reports/screen-awareness-understanding-framework-2026-07-27.md index b42c6e3..197823c 100644 --- a/docs/reports/screen-awareness-understanding-framework-2026-07-27.md +++ b/docs/reports/screen-awareness-understanding-framework-2026-07-27.md @@ -2,7 +2,7 @@ - 日期:2026-07-27 - 状态:评估完成;仅定义实施框架,**未开发捕获、调度、数据库或网络代码**。 -- 范围:用户可在“诊断与设置”中授权单显示器或全部显示器;在随机时机将授权范围的有限上下文发送给 **Companion**,让已配置视觉能力的 AstrBot 理解用户当前在做什么。 +- 范围:用户可在“诊断与设置”中选择并授权指定窗口、指定显示器或全部显示器;在随机时机将固定授权范围的有限上下文发送给 **Companion**,让已配置视觉能力的 AstrBot 理解用户当前在做什么。 - 关联决策:`PD-0014`(方案 B)、`ADR-0008`。 - 参考:已审阅 `menglimi/astrbot_plugin_screen_companion` 的提交 `325ec215342c0a0fa7cf639afe6e4a19d4fbb6fd`。仅借鉴交互与调度模式;该项目为 AGPL-3.0,禁止复制代码、资源、提示词或 WebUI。 @@ -32,7 +32,7 @@ | 层级 | 当前已存在 | 方案 B 目标 | 实施前置条件 | | -------- | ------------------------ | -------------------------------------- | ---------------------------------------- | -| 授权范围 | 短时、单前台窗口会话 | 单显示器或全部显示器的独立、可撤销授权 | WGC + 范围专属同意流程 Spike | +| 授权范围 | 短时、单前台窗口会话 | 指定窗口、指定显示器或全部显示器的独立、可撤销授权 | WGC + 三档范围专属同意流程 Spike | | 捕获 | GDI `PrintWindow` 截窗口 | WGC 采集显示器/游戏真实帧 | 非黑帧、受保护内容、锁屏和显示器变化验证 | | 内容 | 手动发送窗口文字或图片 | 受控自动观察 + 手动“立即看一眼” | 本地策略、质量检查与调度器 | | 上下文 | 自由文本提示或单张图 | 图片 + 脱敏活动上下文信封 + 触发原因 | 统一多模态消息构造器 | @@ -56,12 +56,18 @@ flowchart LR J --> K[用户可见回执、状态和一键停止] ``` -### 3.1 采集层:只得到授权范围的一帧 +### 3.1 采集层:只得到用户选定范围的一帧 -- 提供两种互斥范围:`selected_display`(用户选定的一台显示器)和 `all_displays`(用户明确确认全部显示器)。不提供“自动选择显示器”。 -- 总开关默认关闭;每种范围各自需要单独同意。修改范围、显示器拔插、锁屏、系统会话切换或授权撤销立即暂停。 +范围是设置页的一级单选项;三项互斥,均默认关闭,均需各自明确同意: + +- `selected_window`(指定窗口):用户通过原生窗口选择器选择一个可见顶层窗口,或选择“授权当前前台窗口”。授权令牌绑定窗口句柄、进程身份和有效期;窗口关闭、最小化、进程变化或句柄复用即停止,不自动跟随新的前台窗口。 +- `selected_display`(指定显示器):用户从显示器名称/预览中选择一台显示器;授权令牌绑定该显示器稳定标识。显示器拔插、标识变化或范围修改即停止。 +- `all_displays`(全部显示器):用户在清楚列出当前显示器数量和名称后,单独确认全部范围;新增或移除显示器不会静默纳入/排除,必须停止并重新确认。 + +- 选择器只用于让用户看到并选择目标;调度器只能复用已保存的授权令牌,绝不自行枚举、改选或跟随窗口/显示器。 +- 修改任何范围时先停止旧会话、清除待执行任务和短期去重状态,再显示新范围的专属同意文案;不能把窗口授权升级为显示器授权。 - WGC 输出只进入内存;达到尺寸、像素、编码、上传大小预算前不构造网络请求。 -- 游戏陪玩仅在 WGC 对 DirectX/Vulkan 实测为非黑帧后复用该能力;否则固定显示“不支持自动观察”,保留手动窗口读取作为另一条能力。 +- 游戏陪玩可使用已选游戏窗口或已选显示器范围,但仅在 WGC 对 DirectX/Vulkan 实测为非黑帧后启用;否则固定显示“不支持自动观察”。 ### 3.2 本地过滤与质量门:先决定“不发” @@ -116,7 +122,7 @@ flowchart LR ### 3.6 可见性、停止与审计 -- 捕获范围生效时,外壳常驻橙色指示器显示“正在读取:单显示器/全部显示器”,并提供直接停止入口。 +- 捕获范围生效时,外壳常驻橙色指示器显示“正在读取:指定窗口/指定显示器/全部显示器”,并提供直接停止入口。 - 设置页显示最后一次**结果**而非原始内容:时间、手动/自动、结果码、范围类型、触发原因、下次最早评估时间。 - 停止需同步取消等待中的定时器、当前捕获、上传和后续主动回应;取消竞态必须有测试。 - 本地只持久化用户设置、授权状态摘要、最近观察时间、脱敏结果码和短期不可逆去重指纹;原始帧、原文摘要及完整模型输出不得进入 SQLite、日志、崩溃报告、测试快照或剪贴板。 @@ -131,14 +137,14 @@ flowchart LR ### S0:WGC 与安全边界 Spike(阻塞后续) -- 验证单显示器、全部显示器、全屏 DirectX/Vulkan 游戏、最小化/切换、显示器拔插、锁屏、受保护内容和敏感应用下的 WGC 行为。 +- 验证指定窗口、单显示器、全部显示器、全屏 DirectX/Vulkan 游戏、最小化/切换、显示器拔插、锁屏、受保护内容和敏感应用下的 WGC 行为。 - 测量每种范围的 CPU、GPU、峰值内存、编码耗时、帧尺寸和失败类型;定义明确预算。 - 验收:真实游戏非黑帧;拒绝/降级路径可解释;指示器和停止在每个失败/成功路径均有效;结果写入 Spike 报告。任一关键项失败则不进入 S1。 ### S1:范围授权、持久化最小设置与状态页 -- 实现单显示器/全部显示器专属授权、总开关、活跃时段、最短间隔、概率、互动风格与增强文字上下文开关。 -- 验收:没有范围授权不可开启;修改范围必须重新授权;所有配置按 Profile 隔离;默认值安全且无原始内容落盘。 +- 实现指定窗口/指定显示器/全部显示器的专属授权、总开关、活跃时段、最短间隔、概率、互动风格与增强文字上下文开关。 +- 验收:没有范围授权不可开启;窗口句柄/进程变化、显示器变化或修改范围均必须停止并重新授权;所有配置按 Profile 隔离;默认值安全且无原始内容落盘。 ### S2:Rust 调度与本地策略门 @@ -165,7 +171,7 @@ flowchart LR - 在 S0 之前,不能宣称全桌面、游戏陪玩自动观察或“Bot 正在看屏幕”。 - 在 S3 之前,不能宣称 Bot 理解了图片;附件可上传不等于视觉模型已接收和理解。 - 在 WGC 的受保护内容策略明确前,不能以“尽量捕获”为目标绕过系统限制。 -- 在用户明确选择范围前,不能因概率或“陪伴模式”自动扩展到其他显示器、窗口或应用。 +- 在用户明确选择范围前,不能因概率或“陪伴模式”自动扩展到其他窗口、显示器或应用。 ## 7. 需要产品确认但不阻塞 S0 的细节 diff --git a/docs/reports/wgc-s0-target-binding-2026-07-27.md b/docs/reports/wgc-s0-target-binding-2026-07-27.md new file mode 100644 index 0000000..c37138c --- /dev/null +++ b/docs/reports/wgc-s0-target-binding-2026-07-27.md @@ -0,0 +1,28 @@ +# WGC S0.1 Target-Binding Spike — 2026-07-27 + +## Purpose + +Establish the narrowest possible evidence that the Windows client can bind explicit capture targets through Windows.Graphics.Capture (WGC), without starting capture or processing screen content. This is an S0 sub-step, not a delivered screen-awareness feature. + +## Implemented boundary + +- `wgc_spike_support_status` reports only whether WGC is supported and the current attached-display count. +- `wgc_spike_probe_authorized_window` reuses the existing `CapturePolicy::inspect` gate and creates a WGC item only for an already authorized window. +- `wgc_spike_probe_displays` creates WGC items for the displays attached at call time. +- Each successful probe returns target kind plus width and height. Invalid or non-positive dimensions are rejected. + +No command creates a `GraphicsCaptureSession`, retrieves a frame, reads text, consumes capture budget, writes raw content, encodes an image, schedules an observation, or calls AstrBot. + +## Validation + +- `cargo fmt --all`: passed. +- With `CARGO_TARGET_DIR=src-tauri/.tmp/wgc-check`, `cargo check -p astrbot-desktop --lib`: passed. +- With the same isolated target, `cargo test -p astrbot-desktop --lib`: 117 passed, 0 failed. + +The isolated target is required in this workspace because the default Cargo target resolves generated Tauri permissions through the mapped UNC path and can fail with Windows access denied (`os error 5`). + +## Non-evidence and next gate + +S0.1 provides no proof that WGC returns useful pixels. It does not validate DirectX/Vulkan games, minimized targets, display topology changes, lock screens, protected content, sensitive-application filtering, quality thresholds, or AstrBot visual understanding. + +S0.2 must create a bounded D3D11 WGC frame path, copy one frame to a staging texture, convert BGRA to RGBA, detect black/transparent/empty results, and record manual Windows evidence before any screen frame can progress to a local privacy or network path. diff --git a/docs/错题本.md b/docs/错题本.md index ccc478d..2b3f714 100644 --- a/docs/错题本.md +++ b/docs/错题本.md @@ -59,3 +59,35 @@ - 修复或规避:诊断页 helper 改为等待目标标题,调用方显式等待;仅该启动集成测试预算设为十秒,不调整全局测试配置或产品代码。 - 预防规则:涉及懒加载页面切换的集成测试必须等待可见的页面就绪信号;单条跨异步边界测试应声明与其真实链路匹配的局部预算。 - 验证证据:修复后运行完整 Vitest、ESLint、TypeScript 和生产 Web 构建;产品逻辑无改动。 + + +## 2026-07-27 — 不能把“当前前台窗口”当成长期授权范围 + +- 阶段:屏幕感知三档范围规划 +- 类型:功能行为 / 安全边界 +- 现象:若自动观察始终读取当前前台窗口,用户切换到聊天、浏览器或敏感工具时,实际捕获目标会在没有新授权的情况下变化。 +- 根因:把“识别当前窗口”误用为“自动跟随当前窗口”,没有将用户选择的捕获范围绑定为稳定授权令牌。 +- 修复或规避:将范围明确为指定窗口、指定显示器、全部显示器三档;窗口身份或显示器拓扑变化时撤销会话,调度器只复用已授权范围。 +- 预防规则:任何自动观察不得根据前台变化自行更换目标;范围变更必须先停止、再显示专属同意、再重新授权。 +- 验证证据:PD-0014、screen-awareness-understanding-framework、实施队列 S0/C2/C4 均更新为三档范围验收。 + + +## 2026-07-27 — WGC interop types are not the Win32 aliases + +- 阶段:S0.1 WGC 目标绑定预检 +- 类型:编码 / Windows API 契约 +- 现象:首次编译 WGC 显示器预检时,`BOOL` 从 `Win32::Foundation` 导入失败,`GraphicsCaptureItem::Size` 也不能直接传给内部尺寸检查。 +- 根因:windows-rs 将此回调的 `BOOL` 暴露在 `windows::core`,而 WinRT `SizeInt32` 与内部 `WgcSize` 是不同类型,不能依赖隐式转换。 +- 修复或规避:改用 `windows::core::BOOL`,并对窗口和显示器的 `Size()` 结果显式 `.into()`。 +- 预防规则:新增 WinRT/WGC 调用先以隔离 `cargo check` 验证每个特征门和跨命名空间类型;相同 API 的两个分支必须保持相同的显式转换。 +- 验证证据:`cargo check -p astrbot-desktop --lib` 通过;`cargo test -p astrbot-desktop --lib` 117 passed。 + +## 2026-07-27 — 隔离 Cargo 目标的冷启动不能按短超时判失败 + +- 阶段:S0.1 WGC 目标绑定预检 +- 类型:测试 / 环境 +- 现象:首次在新的 `src-tauri/.tmp/wgc-check` 目标目录运行 Rust 测试超过两分钟而被调用超时。 +- 根因:隔离目标目录需要重新编译完整 Tauri 依赖图;这不是测试失败,且默认目标目录仍受 UNC 生成权限文件的访问拒绝影响。 +- 修复或规避:保留项目内隔离目标缓存并以足够时限重跑;最终 117 个库测试全部通过。 +- 预防规则:默认目标触发 UNC `os error 5` 时,使用已验证的项目内 `CARGO_TARGET_DIR`;首次冷启动须看退出码和最终测试结果,不能把工具超时当作断言失败。 +- 验证证据:隔离 `cargo check` 和后续 `cargo test -p astrbot-desktop --lib` 均成功。 diff --git a/src-tauri/Cargo.toml b/src-tauri/Cargo.toml index 8ee15d8..8d41b09 100644 --- a/src-tauri/Cargo.toml +++ b/src-tauri/Cargo.toml @@ -44,7 +44,9 @@ zeroize.workspace = true [target.'cfg(windows)'.dependencies] windows = { workspace = true, features = [ "Win32_Foundation", + "Graphics_Capture", "Win32_Graphics_Gdi", + "Win32_System_WinRT_Graphics_Capture", "Win32_NetworkManagement_IpHelper", "Win32_Networking_WinSock", "Win32_Security_Credentials", diff --git a/src-tauri/src/capture_policy.rs b/src-tauri/src/capture_policy.rs index 00d3159..9b46368 100644 --- a/src-tauri/src/capture_policy.rs +++ b/src-tauri/src/capture_policy.rs @@ -317,6 +317,31 @@ impl CapturePolicy { }) } + /// Validates an authorized window without reading its text or pixels. + /// + /// WGC target binding uses this narrow path during S0. It deliberately does + /// not consume a read budget because it does not create a capture session or + /// expose screen content. + pub fn inspect( + &self, + token: Uuid, + body: impl FnOnce(&WindowSelection) -> T, + ) -> Result { + let guard = self.state.lock().expect("capture policy lock poisoned"); + if !guard.enabled { + return Err(CapturePolicyError::Disabled); + } + let session = guard + .sessions + .get(&token) + .ok_or(CapturePolicyError::UnknownSession)?; + session + .selection + .validate() + .map_err(CapturePolicyError::Selection)?; + Ok(body(&session.selection)) + } + /// Checks a read is allowed, and runs it against the authorized window. /// /// The read is a closure so the handle never leaves this module: every gate @@ -396,6 +421,25 @@ mod tests { let _ = &policy; } + #[test] + fn an_inspection_is_refused_while_the_switch_is_off() { + let policy = CapturePolicy::default(); + let error = policy + .inspect(Uuid::new_v4(), |_| ()) + .expect_err("must be refused"); + assert_eq!(error, CapturePolicyError::Disabled); + } + + #[test] + fn an_inspection_is_refused_when_no_session_names_the_window() { + let policy = CapturePolicy::default(); + policy.set_enabled(true); + let error = policy + .inspect(Uuid::new_v4(), |_| ()) + .expect_err("must be refused"); + assert_eq!(error, CapturePolicyError::UnknownSession); + } + #[test] fn a_read_is_refused_while_the_switch_is_off() { let policy = CapturePolicy::default(); diff --git a/src-tauri/src/lib.rs b/src-tauri/src/lib.rs index f59c6f5..f5578b9 100644 --- a/src-tauri/src/lib.rs +++ b/src-tauri/src/lib.rs @@ -555,6 +555,27 @@ async fn begin_window_capture_session( .ok_or_else(|| "capture session expired immediately".to_owned()) } +#[tauri::command] +fn wgc_spike_support_status() -> windows::wgc::WgcSupportStatus { + windows::wgc::support_status() +} + +#[tauri::command] +fn wgc_spike_probe_authorized_window( + app: AppHandle, + session_token: uuid::Uuid, +) -> Result { + app.state::() + .inspect(session_token, windows::wgc::probe_selected_window) + .map_err(|error| error.to_string())? + .map_err(|error| error.to_string()) +} + +#[tauri::command] +fn wgc_spike_probe_displays() -> Result, String> { + windows::wgc::probe_display_targets().map_err(|error| error.to_string()) +} + #[tauri::command] fn end_window_capture_session(app: AppHandle, session_token: uuid::Uuid) -> ScreenReadingStatus { let policy = app.state::(); @@ -942,6 +963,9 @@ pub fn run() { pet_window_preferences, inspect_foreground_window, screen_reading_status, + wgc_spike_support_status, + wgc_spike_probe_authorized_window, + wgc_spike_probe_displays, set_screen_reading_enabled, begin_window_capture_session, end_window_capture_session, diff --git a/src-tauri/src/windows/mod.rs b/src-tauri/src/windows/mod.rs index 524ed8a..a1cd7ee 100644 --- a/src-tauri/src/windows/mod.rs +++ b/src-tauri/src/windows/mod.rs @@ -6,3 +6,4 @@ pub mod foreground_window; pub mod ports; pub mod process; pub mod uia; +pub mod wgc; diff --git a/src-tauri/src/windows/wgc.rs b/src-tauri/src/windows/wgc.rs new file mode 100644 index 0000000..1b4c451 --- /dev/null +++ b/src-tauri/src/windows/wgc.rs @@ -0,0 +1,302 @@ +//! Windows.Graphics.Capture target-binding spike. +//! +//! This module intentionally stops before a pixel frame is requested. It proves +//! whether this Windows build can construct WGC capture items for an already +//! authorized window and for the attached displays, without exposing a frame to +//! React, disk, logging, or the network. The next S0 increment must add a +//! bounded D3D11 frame path and black-frame validation before this can become a +//! capture feature. + +use std::fmt; + +use serde::Serialize; + +use super::foreground_window::{ForegroundWindowError, WindowSelection}; + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "snake_case")] +pub enum WgcTargetKind { + Window, + Display, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct WgcTargetProbe { + pub kind: WgcTargetKind, + pub width: u32, + pub height: u32, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct WgcSupportStatus { + pub supported: bool, + pub display_target_count: u16, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum WgcProbeError { + Selection(ForegroundWindowError), + UnsupportedPlatform, + UnsupportedByWindows, + InvalidTargetSize, + Platform(u32), +} + +impl fmt::Display for WgcProbeError { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::Selection(error) => write!(formatter, "the selected window is invalid: {error}"), + Self::UnsupportedPlatform => { + formatter.write_str("Windows.Graphics.Capture is unavailable on this platform") + } + Self::UnsupportedByWindows => formatter + .write_str("Windows.Graphics.Capture is not supported by this Windows session"), + Self::InvalidTargetSize => { + formatter.write_str("Windows.Graphics.Capture returned an invalid target size") + } + Self::Platform(code) => write!( + formatter, + "Windows.Graphics.Capture failed with HRESULT 0x{code:08x}" + ), + } + } +} + +impl std::error::Error for WgcProbeError {} + +/// Returns only capability metadata. Enumerating display targets never creates +/// a capture session or reads pixels. +pub fn support_status() -> WgcSupportStatus { + #[cfg(windows)] + { + let supported = wgc_supported().unwrap_or(false); + let display_target_count = if supported { + display_handles().map_or(0, |displays| { + u16::try_from(displays.len()).unwrap_or(u16::MAX) + }) + } else { + 0 + }; + WgcSupportStatus { + supported, + display_target_count, + } + } + + #[cfg(not(windows))] + { + WgcSupportStatus { + supported: false, + display_target_count: 0, + } + } +} + +/// Proves that the current, policy-authorized window can be represented by WGC. +/// It does not start a capture session or expose pixels. +pub fn probe_selected_window(selection: &WindowSelection) -> Result { + selection.validate().map_err(WgcProbeError::Selection)?; + + #[cfg(windows)] + { + if !wgc_supported()? { + return Err(WgcProbeError::UnsupportedByWindows); + } + let item = capture_item_for_window(selection)?; + target_probe( + WgcTargetKind::Window, + item.Size().map_err(map_windows_error)?.into(), + ) + } + + #[cfg(not(windows))] + { + let _ = selection; + Err(WgcProbeError::UnsupportedPlatform) + } +} + +/// Probes every attached display target without reading any image data. This is +/// S0 evidence that the eventual one-display and all-display scopes can bind to +/// WGC items; it is not an authorization or capture API. +pub fn probe_display_targets() -> Result, WgcProbeError> { + #[cfg(windows)] + { + if !wgc_supported()? { + return Err(WgcProbeError::UnsupportedByWindows); + } + display_handles()? + .into_iter() + .map(|display| { + let item = capture_item_for_monitor(display)?; + target_probe( + WgcTargetKind::Display, + item.Size().map_err(map_windows_error)?.into(), + ) + }) + .collect() + } + + #[cfg(not(windows))] + { + Err(WgcProbeError::UnsupportedPlatform) + } +} + +fn target_probe(kind: WgcTargetKind, size: WgcSize) -> Result { + let width = u32::try_from(size.width).map_err(|_| WgcProbeError::InvalidTargetSize)?; + let height = u32::try_from(size.height).map_err(|_| WgcProbeError::InvalidTargetSize)?; + if width == 0 || height == 0 { + return Err(WgcProbeError::InvalidTargetSize); + } + Ok(WgcTargetProbe { + kind, + width, + height, + }) +} + +#[derive(Clone, Copy)] +struct WgcSize { + width: i32, + height: i32, +} + +#[cfg(windows)] +impl From for WgcSize { + fn from(value: windows::Graphics::SizeInt32) -> Self { + Self { + width: value.Width, + height: value.Height, + } + } +} + +#[cfg(windows)] +fn wgc_supported() -> Result { + windows::Graphics::Capture::GraphicsCaptureSession::IsSupported().map_err(map_windows_error) +} + +#[cfg(windows)] +fn capture_item_for_window( + selection: &WindowSelection, +) -> Result { + use windows::{ + Win32::{Foundation::HWND, System::WinRT::Graphics::Capture::IGraphicsCaptureItemInterop}, + core::factory, + }; + + let interop = + factory::() + .map_err(map_windows_error)?; + unsafe { + interop + .CreateForWindow(HWND(selection.raw_handle() as *mut _)) + .map_err(map_windows_error) + } +} + +#[cfg(windows)] +fn capture_item_for_monitor( + monitor: windows::Win32::Graphics::Gdi::HMONITOR, +) -> Result { + use windows::{ + Win32::System::WinRT::Graphics::Capture::IGraphicsCaptureItemInterop, core::factory, + }; + + let interop = + factory::() + .map_err(map_windows_error)?; + unsafe { interop.CreateForMonitor(monitor).map_err(map_windows_error) } +} + +#[cfg(windows)] +fn display_handles() -> Result, WgcProbeError> { + use windows::{ + Win32::{ + Foundation::{LPARAM, RECT}, + Graphics::Gdi::{EnumDisplayMonitors, HDC, HMONITOR}, + }, + core::BOOL, + }; + + unsafe extern "system" fn collect_display( + monitor: HMONITOR, + _device_context: HDC, + _bounds: *mut RECT, + data: LPARAM, + ) -> BOOL { + let displays = unsafe { &mut *(data.0 as *mut Vec) }; + displays.push(monitor); + BOOL(1) + } + + let mut displays = Vec::new(); + let succeeded = unsafe { + EnumDisplayMonitors( + None, + None, + Some(collect_display), + LPARAM((&mut displays as *mut Vec) as isize), + ) + } + .as_bool(); + if !succeeded { + return Err(last_windows_error()); + } + Ok(displays) +} + +#[cfg(windows)] +fn map_windows_error(error: windows::core::Error) -> WgcProbeError { + WgcProbeError::Platform(error.code().0 as u32) +} + +#[cfg(windows)] +fn last_windows_error() -> WgcProbeError { + let code = unsafe { windows::Win32::Foundation::GetLastError() }.0; + WgcProbeError::Platform(code) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn refuses_non_positive_target_sizes() { + assert_eq!( + target_probe( + WgcTargetKind::Display, + WgcSize { + width: 0, + height: 1080 + } + ), + Err(WgcProbeError::InvalidTargetSize) + ); + assert_eq!( + target_probe( + WgcTargetKind::Window, + WgcSize { + width: -1, + height: 1080 + } + ), + Err(WgcProbeError::InvalidTargetSize) + ); + } + + #[test] + fn serializable_support_status_starts_with_no_pixels() { + let status = WgcSupportStatus { + supported: false, + display_target_count: 0, + }; + assert_eq!( + serde_json::to_string(&status).expect("serialize"), + "{\"supported\":false,\"displayTargetCount\":0}" + ); + } +}