feat(awareness): probe WGC capture targets
这个提交存在于:
父节点
350c6fd3ad
当前提交
dfc26aa37e
共有 13 个文件被更改,包括 494 次插入 和 24 次删除
|
|
@ -10,6 +10,7 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and
|
||||||
|
|
||||||
- Separate companion navigation for text chat, voice chat, and game companion. The game surface remains explicitly unavailable until Windows.Graphics.Capture has validated real DirectX/Vulkan frames.
|
- Separate companion navigation for text chat, voice chat, and game companion. The game surface remains explicitly unavailable until Windows.Graphics.Capture has validated real DirectX/Vulkan frames.
|
||||||
- Screen-awareness implementation framework for scoped display consent, WGC capture, local privacy/quality gates, multimodal AstrBot validation, and low-interruption response control.
|
- Screen-awareness implementation framework for scoped display consent, WGC capture, local privacy/quality gates, multimodal AstrBot validation, and low-interruption response control.
|
||||||
|
- WGC S0.1 target-binding preflight for authorized windows and attached displays. It returns metadata only; no capture session, pixels, disk write, or upload is performed.
|
||||||
- Debug-level tracing of AstrBot SSE frame shapes: frame type, which identity fields are present, and the key names inside `data`. No payload text is recorded. Enable with `RUST_LOG`.
|
- Debug-level tracing of AstrBot SSE frame shapes: frame type, which identity fields are present, and the key names inside `data`. No payload text is recorded. Enable with `RUST_LOG`.
|
||||||
- Q10-Q15 chat experience increment: lightweight Markdown rendering, message actions, generation controls, session actions, connection controls, and responsive chat layout.
|
- Q10-Q15 chat experience increment: lightweight Markdown rendering, message actions, generation controls, session actions, connection controls, and responsive chat layout.
|
||||||
- AstrBot OpenAPI adapter foundation: SSE mapper, dual-slot runtime skeleton, credential enrollment IPC, transport-aware profiles (ADR-0005).
|
- AstrBot OpenAPI adapter foundation: SSE mapper, dual-slot runtime skeleton, credential enrollment IPC, transport-aware profiles (ADR-0005).
|
||||||
|
|
|
||||||
|
|
@ -114,10 +114,10 @@
|
||||||
|
|
||||||
- Date noted: 2026-07-27
|
- Date noted: 2026-07-27
|
||||||
- Status: direction selected; implementation remains blocked on the required full-desktop Spike and acceptance gates. Planning artifact: docs/reports/companion-ui-and-screen-awareness-plan-2026-07-27.md.
|
- Status: direction selected; implementation remains blocked on the required full-desktop Spike and acceptance gates. Planning artifact: docs/reports/companion-ui-and-screen-awareness-plan-2026-07-27.md.
|
||||||
- Decision update (2026-07-27): the product owner selected option B, the scoped full-desktop direction. This approves discovery and design only; it does not authorize capture code before the required Spike passes.
|
- Decision update (2026-07-27): the product owner selected option B with three selectable scoped ranges: a named window, one display, or all current displays. This approves discovery and design only; it does not authorize capture code before the required Spike passes.
|
||||||
- Conflict: the requested product direction asks for randomly timed screen captures after a settings toggle, while ADR-0008 only permits a single user-authorized foreground window and requires a visible indicator, a stop control, Companion-only delivery, no on-disk raw frames, and Rust-owned policy enforcement.
|
- Conflict: the requested product direction asks for randomly timed screen captures after a settings toggle, while ADR-0008 only permits a single user-authorized foreground window and requires a visible indicator, a stop control, Companion-only delivery, no on-disk raw frames, and Rust-owned policy enforcement.
|
||||||
- Proposed safe interpretation: “random” means random trigger time, not an unbounded random capture target. V1 may schedule captures only for a live, explicitly authorized foreground-window session. The interval is a hard minimum between successful captures and the configured probability is evaluated only after that interval has elapsed.
|
- Proposed safe interpretation: “random” means random trigger time, not an unbounded random capture target. V1 may schedule captures only for a live, explicitly authorized foreground-window session. The interval is a hard minimum between successful captures and the configured probability is evaluated only after that interval has elapsed.
|
||||||
- Chosen scope direction: add a full-desktop capability after Spike approval. The eventual consent UI must require the user to select either one monitor or an explicit all-monitors scope; the scheduler is bound to that stable scope and may never choose a different monitor by itself.
|
- Chosen scope direction: add a full-desktop capability after Spike approval. The eventual consent UI must require the user to select one named window, one monitor, or an explicit all-monitors scope. The scheduler is bound to that stable scope and may never choose a different window or monitor by itself. Window identity changes, display changes, and scope changes revoke the active authorization.
|
||||||
- Mandatory gates before implementation: separate scope-specific consent, revised threat model, Windows.Graphics.Capture Spike results, lock-screen/full-screen/protected-content/sensitive-app handling, visible indicator and stop-path validation, Companion-only delivery, and real AstrBot visual acceptance.
|
- Mandatory gates before implementation: separate scope-specific consent, revised threat model, Windows.Graphics.Capture Spike results for all three ranges, window-identity/display-change handling, lock-screen/full-screen/protected-content/sensitive-app handling, visible indicator and stop-path validation, Companion-only delivery, and real AstrBot visual acceptance.
|
||||||
- Non-negotiable exclusions: no hidden capture, browser-history collection, input logging, process injection, or automatic delivery to Operations.
|
- Non-negotiable exclusions: no hidden capture, browser-history collection, input logging, process injection, or automatic delivery to Operations.
|
||||||
- Safe interim rule: preserve the current capture policy unchanged until the full-desktop Spike has passed and the product owner accepts its evidence. C1–C3 page work may be planned independently; no scheduler, persistence migration, or capture-scope expansion is implemented in this planning phase.
|
- Safe interim rule: preserve the current capture policy unchanged until the full-desktop Spike has passed and the product owner accepts its evidence. C1–C3 page work may be planned independently; no scheduler, persistence migration, or capture-scope expansion is implemented in this planning phase.
|
||||||
|
|
|
||||||
|
|
@ -234,6 +234,15 @@ The following are deliberately not marked complete: HTTPS certificate pinning, s
|
||||||
|
|
||||||
验收:框架明确“随机仅指触发时间”、Companion-only、无原始内容持久化、真实视觉验收及 S0-S5 的逐阶段停止条件;本轮无 `src/`、`src-tauri/` 或数据库改动。
|
验收:框架明确“随机仅指触发时间”、Companion-only、无原始内容持久化、真实视觉验收及 S0-S5 的逐阶段停止条件;本轮无 `src/`、`src-tauri/` 或数据库改动。
|
||||||
|
|
||||||
|
|
||||||
|
### S0.1:WGC 目标绑定预检(2026-07-27,完成;不读取像素)
|
||||||
|
|
||||||
|
- 新增 Rust-owned WGC 预检:仅验证 Windows 是否支持 WGC,以及已授权窗口和当前已连接显示器能否绑定为 `GraphicsCaptureItem`;返回目标类型和尺寸元数据。
|
||||||
|
- 预检通过 `CapturePolicy::inspect` 复用既有窗口授权,不读取窗口文本/像素、不消耗读取预算、不创建 capture session,也不触发编码、持久化、网络或 AstrBot 上传。
|
||||||
|
- 仅为 S0 证据,不代表游戏、全屏、锁屏、受保护内容或真实画面捕获可用;S0.2 必须完成 D3D11 真实帧、黑帧/透明帧检测和 Windows 人工证据。
|
||||||
|
|
||||||
|
验收:隔离目标目录执行 `cargo check -p astrbot-desktop --lib` 通过;`cargo test -p astrbot-desktop --lib` 为 117 passed。
|
||||||
|
|
||||||
### C1:聊天、语音聊天、游戏陪玩三页面壳
|
### C1:聊天、语音聊天、游戏陪玩三页面壳
|
||||||
|
|
||||||
- 在现有“聊天”之后增加“语音聊天”“游戏陪玩”两个独立导航入口。
|
- 在现有“聊天”之后增加“语音聊天”“游戏陪玩”两个独立导航入口。
|
||||||
|
|
@ -246,7 +255,7 @@ The following are deliberately not marked complete: HTTPS certificate pinning, s
|
||||||
|
|
||||||
- 在“诊断与设置”增加屏幕感知分组:总开关、授权状态、最短触发间隔、概率、活跃时段、停止和脱敏回执。
|
- 在“诊断与设置”增加屏幕感知分组:总开关、授权状态、最短触发间隔、概率、活跃时段、停止和脱敏回执。
|
||||||
- 默认关闭;随机只表示触发时机,绝不随机选择未授权的窗口、显示器或应用。
|
- 默认关闭;随机只表示触发时机,绝不随机选择未授权的窗口、显示器或应用。
|
||||||
- 方案 B 已选:C2–C6 仍受 PD-0014 约束,必须先通过 WGC、显示器范围专属授权、锁屏/受保护内容和敏感应用 Spike,才可扩展当前单窗口范围。
|
- 方案 B 已选:C2–C6 仍受 PD-0014 约束,必须先通过 WGC、窗口/显示器范围专属授权、锁屏/受保护内容和敏感应用 Spike,才可扩展当前单窗口范围。
|
||||||
|
|
||||||
验收:未授权、关闭、锁屏、授权过期、目标变化和断连均不能上传画面;活跃会话始终存在可见指示器与停止按钮。
|
验收:未授权、关闭、锁屏、授权过期、目标变化和断连均不能上传画面;活跃会话始终存在可见指示器与停止按钮。
|
||||||
|
|
||||||
|
|
@ -260,9 +269,9 @@ The following are deliberately not marked complete: HTTPS certificate pinning, s
|
||||||
|
|
||||||
### C4:全桌面与游戏陪玩 Windows.Graphics.Capture Spike(方案 B)
|
### C4:全桌面与游戏陪玩 Windows.Graphics.Capture Spike(方案 B)
|
||||||
|
|
||||||
- 先验证 WGC 对单显示器、全部显示器和 DirectX/Vulkan 游戏窗口的非黑帧捕获;记录 CPU、GPU、内存、显示器拔插、最小化、窗口切换、锁屏和受保护内容行为。
|
- 先验证 WGC 对指定窗口、单显示器、全部显示器和 DirectX/Vulkan 游戏窗口的非黑帧捕获;记录 CPU、GPU、内存、窗口进程变化、显示器拔插、最小化、窗口切换、锁屏和受保护内容行为。
|
||||||
- 设计独立的范围专属授权:用户必须明确选定单显示器或全部显示器;随机调度只能复用该范围,不能自行扩张或改选目标。
|
- 设计独立的范围专属授权:用户必须明确选定指定窗口、单显示器或全部显示器;随机调度只能复用该范围,不能自行扩张、跟随或改选目标。
|
||||||
- Spike 通过前,游戏陪玩和全桌面感知只显示真实阻塞说明,不能发送 GDI 黑图或声称“正在观察”。
|
- Spike 通过前,游戏陪玩和全桌面感知只显示真实阻塞说明,不能发送 GDI 黑图或声称“正在观察”。
|
||||||
- 禁止输入模拟、进程注入和游戏内存读取;自动画面只送 Companion,始终显示指示器和停止入口。
|
- 禁止输入模拟、进程注入和游戏内存读取;自动画面只送 Companion,始终显示指示器和停止入口。
|
||||||
|
|
||||||
验收:单显示器和全部显示器均通过 WGC 真实画面、资源预算、敏感/受保护内容降级与可撤销授权测试后,才进入正式全桌面调度或游戏陪玩会话开发。
|
验收:指定窗口、单显示器和全部显示器均通过 WGC 真实画面、资源预算、敏感/受保护内容降级与可撤销授权测试后,才进入正式全桌面调度或游戏陪玩会话开发。
|
||||||
|
|
|
||||||
|
|
@ -730,7 +730,7 @@ Playwright 5 通过,`npm run build` 通过。25 处刻意回归全部被对应
|
||||||
- 已完成:根据当前 UI 截图定义“聊天、语音聊天、游戏陪玩”三个独立导航入口;屏幕感知配置归入“诊断与设置”。
|
- 已完成:根据当前 UI 截图定义“聊天、语音聊天、游戏陪玩”三个独立导航入口;屏幕感知配置归入“诊断与设置”。
|
||||||
- 已完成:审阅 menglimi/astrbot_plugin_screen_companion 的行为与配置维度,并写入本项目适配规划:docs/reports/companion-ui-and-screen-awareness-plan-2026-07-27.md。
|
- 已完成:审阅 menglimi/astrbot_plugin_screen_companion 的行为与配置维度,并写入本项目适配规划:docs/reports/companion-ui-and-screen-awareness-plan-2026-07-27.md。
|
||||||
- 已完成:将“随机定时全屏捕获”与 ADR-0008 的单窗口显式授权边界冲突记录为 PD-0014。
|
- 已完成:将“随机定时全屏捕获”与 ADR-0008 的单窗口显式授权边界冲突记录为 PD-0014。
|
||||||
- 已决定:产品所有者于 2026-07-27 选择方案 B(全桌面方向);它要求用户选定单显示器或全部显示器,并先完成 WGC、范围授权、锁屏/受保护内容、敏感应用与真实 AstrBot 图像验收。
|
- 已决定:产品所有者于 2026-07-27 选择方案 B(全桌面方向);它要求用户选定指定窗口、指定显示器或全部显示器,并先完成 WGC、范围授权、锁屏/受保护内容、敏感应用与真实 AstrBot 图像验收。
|
||||||
- 规划建立时:C1–C8 的任何代码、数据库迁移、协议变更、调度器或 WGC Spike 均未开始;C1 的后续完成记录见本节下方。
|
- 规划建立时:C1–C8 的任何代码、数据库迁移、协议变更、调度器或 WGC Spike 均未开始;C1 的后续完成记录见本节下方。
|
||||||
- 阻塞:方案 B 的必要 Spike 尚未开始;游戏陪玩还受 F-80(WGC)阻塞。
|
- 阻塞:方案 B 的必要 Spike 尚未开始;游戏陪玩还受 F-80(WGC)阻塞。
|
||||||
- 测试状态:本次仅文档规划,未新增或重跑测试。
|
- 测试状态:本次仅文档规划,未新增或重跑测试。
|
||||||
|
|
@ -750,3 +750,24 @@ Playwright 5 通过,`npm run build` 通过。25 处刻意回归全部被对应
|
||||||
- **当前阻塞**:S0 WGC Spike 尚未执行;因此全桌面捕获、硬件加速游戏观察、定时自动观察和“Bot 已理解画面”都未实现。
|
- **当前阻塞**:S0 WGC Spike 尚未执行;因此全桌面捕获、硬件加速游戏观察、定时自动观察和“Bot 已理解画面”都未实现。
|
||||||
- **下一步**:先执行 S0,并以非黑帧、保护内容/锁屏降级、资源预算、可撤销授权和可见停止入口的实测证据决定是否进入 S1。
|
- **下一步**:先执行 S0,并以非黑帧、保护内容/锁屏降级、资源预算、可撤销授权和可见停止入口的实测证据决定是否进入 S1。
|
||||||
- **验证**:本轮为文档评估,无功能代码、数据库、构建产物或协议改动,未运行构建门槛。
|
- **验证**:本轮为文档评估,无功能代码、数据库、构建产物或协议改动,未运行构建门槛。
|
||||||
|
|
||||||
|
|
||||||
|
## 2026-07-27 · 范围选择补充(未开发)
|
||||||
|
|
||||||
|
- **产品决定**:屏幕感知范围必须在设置中由用户选择:指定窗口、指定显示器或全部显示器;三档均为独立授权,不存在自动跟随前台窗口或自动纳入新显示器。
|
||||||
|
- **授权规则**:窗口关闭/最小化/进程身份变化,显示器拔插/标识变化,或用户改变范围时,立即停止旧会话并要求新范围重新确认。
|
||||||
|
- **调度规则**:概率和定时器只复用已授权的稳定范围,不能改变、扩大或替换目标。
|
||||||
|
- **后续**:S0 WGC Spike 的验收范围扩展为三档;只有每档真实帧、隐私降级、资源预算和停止路径均有证据,才开发该档自动观察。
|
||||||
|
|
||||||
|
|
||||||
|
## 2026-07-27: WGC S0.1 target-binding preflight
|
||||||
|
|
||||||
|
Added the first WGC Spike increment for the three approved capture scopes. Rust can now report WGC support and bind an already-authorized window or each currently attached display as a `GraphicsCaptureItem`, returning only type and dimensions. The window route goes through `CapturePolicy::inspect`, so it keeps the existing switch/session/identity validation and does not consume a reading budget.
|
||||||
|
|
||||||
|
This increment does **not** create a `GraphicsCaptureSession`, read a frame, encode data, persist content, schedule a trigger, or upload to AstrBot. It is target-binding evidence only. S0.2 remains mandatory for D3D11 frame readback, black/transparent-frame rejection and manual game/lock/protected-content evidence.
|
||||||
|
|
||||||
|
### Test status
|
||||||
|
|
||||||
|
- Isolated Rust check: passed (`cargo check -p astrbot-desktop --lib`).
|
||||||
|
- Isolated Rust library tests: 117 passed, 0 failed.
|
||||||
|
- Frontend gates remain to be re-run before this increment is committed.
|
||||||
|
|
|
||||||
|
|
@ -104,7 +104,7 @@ Disabled -> SettingsConfigured -> AwaitingExplicitAuthorization -> ActiveAuthori
|
||||||
| 字段 | 默认 / 范围 | 规则 |
|
| 字段 | 默认 / 范围 | 规则 |
|
||||||
| ---------------------------- | ------------------------------- | ----------------------------------------------------------------------------------- |
|
| ---------------------------- | ------------------------------- | ----------------------------------------------------------------------------------- |
|
||||||
| enabled | false | 默认关闭;关闭立即撤销所有会话 |
|
| enabled | false | 默认关闭;关闭立即撤销所有会话 |
|
||||||
| scope | selected_display / all_displays | 方案 B 已选;先完成 WGC、范围专属授权与隐私 Spike。调度器只能使用用户固定选定的范围 |
|
| scope | selected_window / selected_display / all_displays | 方案 B 已选;先完成 WGC、范围专属授权与隐私 Spike。调度器只能使用用户固定选定的范围 |
|
||||||
| min_trigger_interval_minutes | 默认 20;5–240 | 成功发送后重新计时;这是硬下限 |
|
| min_trigger_interval_minutes | 默认 20;5–240 | 成功发送后重新计时;这是硬下限 |
|
||||||
| trigger_probability_percent | 默认 15;0–100 | 只在到达硬下限后抽样;0 表示永不自动触发 |
|
| trigger_probability_percent | 默认 15;0–100 | 只在到达硬下限后抽样;0 表示永不自动触发 |
|
||||||
| active_time_range | 默认全天 | 可选 HH:MM-HH:MM;范围外只允许手动一次性发送 |
|
| active_time_range | 默认全天 | 可选 HH:MM-HH:MM;范围外只允许手动一次性发送 |
|
||||||
|
|
@ -140,7 +140,7 @@ Disabled -> SettingsConfigured -> AwaitingExplicitAuthorization -> ActiveAuthori
|
||||||
### 4.1 阻塞性 Spike
|
### 4.1 阻塞性 Spike
|
||||||
|
|
||||||
1. **WGC 游戏捕获 Spike**:验证 WGC 能否在 DirectX/Vulkan 硬件加速窗口获取非黑帧,并测量 1 fps 下 CPU、GPU、内存、窗口切换、最小化和受保护内容行为。
|
1. **WGC 游戏捕获 Spike**:验证 WGC 能否在 DirectX/Vulkan 硬件加速窗口获取非黑帧,并测量 1 fps 下 CPU、GPU、内存、窗口切换、最小化和受保护内容行为。
|
||||||
2. **全屏范围 Spike(方案 B 已选)**:以 Windows.Graphics.Capture 验证单显示器与“全部显示器”范围;定义范围专属授权文案、显示器切换、敏感应用/受保护内容处理、锁屏策略和 Windows 权限表现;不通过则不实现全屏。
|
2. **全屏范围 Spike(方案 B 已选)**:以 Windows.Graphics.Capture 验证指定窗口、单显示器与“全部显示器”范围;定义窗口失效、显示器切换、范围专属授权文案、敏感应用/受保护内容处理、锁屏策略和 Windows 权限表现;不通过则不实现对应范围。
|
||||||
3. **真实 AstrBot 多模态验收**:确认服务端实际接收图片附件段、目标模型看见画面、失败时返回何种 SSE/错误。
|
3. **真实 AstrBot 多模态验收**:确认服务端实际接收图片附件段、目标模型看见画面、失败时返回何种 SSE/错误。
|
||||||
4. **主动消息预算 Spike**:验证概率、冷却、手动聊天后静默期与通知权限,避免把“陪伴”实现成打扰。
|
4. **主动消息预算 Spike**:验证概率、冷却、手动聊天后静默期与通知权限,避免把“陪伴”实现成打扰。
|
||||||
|
|
||||||
|
|
@ -165,14 +165,14 @@ Disabled -> SettingsConfigured -> AwaitingExplicitAuthorization -> ActiveAuthori
|
||||||
| C4 | 屏幕感知设置页与 schema/迁移设计 | 方案 B 的 WGC / 隐私 Spike 通过 | 默认关闭、显示器范围专属授权、参数校验、停止路径和设置恢复测试完成 |
|
| C4 | 屏幕感知设置页与 schema/迁移设计 | 方案 B 的 WGC / 隐私 Spike 通过 | 默认关闭、显示器范围专属授权、参数校验、停止路径和设置恢复测试完成 |
|
||||||
| C5 | Rust 调度器与脱敏观察回执 | C4 | 概率、硬下限、活跃时段、断连/锁屏/授权撤销测试完成 |
|
| C5 | Rust 调度器与脱敏观察回执 | C4 | 概率、硬下限、活跃时段、断连/锁屏/授权撤销测试完成 |
|
||||||
| C6 | 画面上传与 Companion 主动回应 | C5 + 真实 AstrBot 探针 | 服务端视觉确认、冷却策略、无原始画面落盘 |
|
| C6 | 画面上传与 Companion 主动回应 | C5 + 真实 AstrBot 探针 | 服务端视觉确认、冷却策略、无原始画面落盘 |
|
||||||
| C7 | WGC 全桌面与游戏捕获、性能/隐私 Spike | C3 | 单显示器/全显示器范围、非黑帧、资源预算、全屏/最小化安全降级验收通过 |
|
| C7 | WGC 全桌面与游戏捕获、性能/隐私 Spike | C3 | 指定窗口/单显示器/全显示器范围、非黑帧、资源预算、全屏/最小化安全降级验收通过 |
|
||||||
| C8 | 游戏陪玩正式会话 | C6 + C7 | 仅授权范围、可停止、无输入控制、真实游戏验收通过 |
|
| C8 | 游戏陪玩正式会话 | C6 + C7 | 仅授权范围、可停止、无输入控制、真实游戏验收通过 |
|
||||||
|
|
||||||
**当前状态**:C1 已完成:左侧导航、语音聊天页和游戏陪玩页的前端壳已落地,仍共享既有连接/会话运行时,未增加本机权限。C2–C8 尚未开始;不得把本文件视为已交付的自动屏幕感知功能。
|
**当前状态**:C1 已完成:左侧导航、语音聊天页和游戏陪玩页的前端壳已落地,仍共享既有连接/会话运行时,未增加本机权限。S0.1 已完成 WGC 目标绑定预检,但尚未创建捕获会话或读取像素;C2–C8 仍未开始。不得把本文件视为已交付的自动屏幕感知功能。
|
||||||
|
|
||||||
## 6. 产品决定与仍需决定的事项
|
## 6. 产品决定与仍需决定的事项
|
||||||
|
|
||||||
- **已决定(2026-07-27)**:采用 PD-0014 的方案 B,全桌面方向。全桌面不是隐式默认权限:启用时必须由用户明确选定单显示器或全部显示器;随机只影响触发时间,绝不改变已选范围。
|
- **已决定(2026-07-27)**:采用 PD-0014 的方案 B,全桌面方向。全桌面与窗口感知都不是隐式默认权限:启用时必须由用户明确选定指定窗口、单显示器或全部显示器;随机只影响触发时间,绝不改变已选范围。
|
||||||
- **仍需决定**:“低打扰”默认值是否采用本文建议的 20 分钟 / 15% / 30 分钟回复冷却?
|
- **仍需决定**:“低打扰”默认值是否采用本文建议的 20 分钟 / 15% / 30 分钟回复冷却?
|
||||||
- **仍需决定**:游戏陪玩在 WGC 完成前只显示准备页,还是完全隐藏入口?本文建议显示准备页与真实阻塞原因。
|
- **仍需决定**:游戏陪玩在 WGC 完成前只显示准备页,还是完全隐藏入口?本文建议显示准备页与真实阻塞原因。
|
||||||
- **实施门槛**:方案 B 在 WGC、范围授权、锁屏/受保护内容、敏感应用、可见指示/停止和真实 AstrBot 图像验收均通过前,不得进入捕获、调度或上传实现。
|
- **实施门槛**:方案 B 在 WGC、范围授权、锁屏/受保护内容、敏感应用、可见指示/停止和真实 AstrBot 图像验收均通过前,不得进入捕获、调度或上传实现。
|
||||||
|
|
|
||||||
|
|
@ -2,7 +2,7 @@
|
||||||
|
|
||||||
- 日期:2026-07-27
|
- 日期:2026-07-27
|
||||||
- 状态:评估完成;仅定义实施框架,**未开发捕获、调度、数据库或网络代码**。
|
- 状态:评估完成;仅定义实施框架,**未开发捕获、调度、数据库或网络代码**。
|
||||||
- 范围:用户可在“诊断与设置”中授权单显示器或全部显示器;在随机时机将授权范围的有限上下文发送给 **Companion**,让已配置视觉能力的 AstrBot 理解用户当前在做什么。
|
- 范围:用户可在“诊断与设置”中选择并授权指定窗口、指定显示器或全部显示器;在随机时机将固定授权范围的有限上下文发送给 **Companion**,让已配置视觉能力的 AstrBot 理解用户当前在做什么。
|
||||||
- 关联决策:`PD-0014`(方案 B)、`ADR-0008`。
|
- 关联决策:`PD-0014`(方案 B)、`ADR-0008`。
|
||||||
- 参考:已审阅 `menglimi/astrbot_plugin_screen_companion` 的提交 `325ec215342c0a0fa7cf639afe6e4a19d4fbb6fd`。仅借鉴交互与调度模式;该项目为 AGPL-3.0,禁止复制代码、资源、提示词或 WebUI。
|
- 参考:已审阅 `menglimi/astrbot_plugin_screen_companion` 的提交 `325ec215342c0a0fa7cf639afe6e4a19d4fbb6fd`。仅借鉴交互与调度模式;该项目为 AGPL-3.0,禁止复制代码、资源、提示词或 WebUI。
|
||||||
|
|
||||||
|
|
@ -32,7 +32,7 @@
|
||||||
|
|
||||||
| 层级 | 当前已存在 | 方案 B 目标 | 实施前置条件 |
|
| 层级 | 当前已存在 | 方案 B 目标 | 实施前置条件 |
|
||||||
| -------- | ------------------------ | -------------------------------------- | ---------------------------------------- |
|
| -------- | ------------------------ | -------------------------------------- | ---------------------------------------- |
|
||||||
| 授权范围 | 短时、单前台窗口会话 | 单显示器或全部显示器的独立、可撤销授权 | WGC + 范围专属同意流程 Spike |
|
| 授权范围 | 短时、单前台窗口会话 | 指定窗口、指定显示器或全部显示器的独立、可撤销授权 | WGC + 三档范围专属同意流程 Spike |
|
||||||
| 捕获 | GDI `PrintWindow` 截窗口 | WGC 采集显示器/游戏真实帧 | 非黑帧、受保护内容、锁屏和显示器变化验证 |
|
| 捕获 | GDI `PrintWindow` 截窗口 | WGC 采集显示器/游戏真实帧 | 非黑帧、受保护内容、锁屏和显示器变化验证 |
|
||||||
| 内容 | 手动发送窗口文字或图片 | 受控自动观察 + 手动“立即看一眼” | 本地策略、质量检查与调度器 |
|
| 内容 | 手动发送窗口文字或图片 | 受控自动观察 + 手动“立即看一眼” | 本地策略、质量检查与调度器 |
|
||||||
| 上下文 | 自由文本提示或单张图 | 图片 + 脱敏活动上下文信封 + 触发原因 | 统一多模态消息构造器 |
|
| 上下文 | 自由文本提示或单张图 | 图片 + 脱敏活动上下文信封 + 触发原因 | 统一多模态消息构造器 |
|
||||||
|
|
@ -56,12 +56,18 @@ flowchart LR
|
||||||
J --> K[用户可见回执、状态和一键停止]
|
J --> K[用户可见回执、状态和一键停止]
|
||||||
```
|
```
|
||||||
|
|
||||||
### 3.1 采集层:只得到授权范围的一帧
|
### 3.1 采集层:只得到用户选定范围的一帧
|
||||||
|
|
||||||
- 提供两种互斥范围:`selected_display`(用户选定的一台显示器)和 `all_displays`(用户明确确认全部显示器)。不提供“自动选择显示器”。
|
范围是设置页的一级单选项;三项互斥,均默认关闭,均需各自明确同意:
|
||||||
- 总开关默认关闭;每种范围各自需要单独同意。修改范围、显示器拔插、锁屏、系统会话切换或授权撤销立即暂停。
|
|
||||||
|
- `selected_window`(指定窗口):用户通过原生窗口选择器选择一个可见顶层窗口,或选择“授权当前前台窗口”。授权令牌绑定窗口句柄、进程身份和有效期;窗口关闭、最小化、进程变化或句柄复用即停止,不自动跟随新的前台窗口。
|
||||||
|
- `selected_display`(指定显示器):用户从显示器名称/预览中选择一台显示器;授权令牌绑定该显示器稳定标识。显示器拔插、标识变化或范围修改即停止。
|
||||||
|
- `all_displays`(全部显示器):用户在清楚列出当前显示器数量和名称后,单独确认全部范围;新增或移除显示器不会静默纳入/排除,必须停止并重新确认。
|
||||||
|
|
||||||
|
- 选择器只用于让用户看到并选择目标;调度器只能复用已保存的授权令牌,绝不自行枚举、改选或跟随窗口/显示器。
|
||||||
|
- 修改任何范围时先停止旧会话、清除待执行任务和短期去重状态,再显示新范围的专属同意文案;不能把窗口授权升级为显示器授权。
|
||||||
- WGC 输出只进入内存;达到尺寸、像素、编码、上传大小预算前不构造网络请求。
|
- WGC 输出只进入内存;达到尺寸、像素、编码、上传大小预算前不构造网络请求。
|
||||||
- 游戏陪玩仅在 WGC 对 DirectX/Vulkan 实测为非黑帧后复用该能力;否则固定显示“不支持自动观察”,保留手动窗口读取作为另一条能力。
|
- 游戏陪玩可使用已选游戏窗口或已选显示器范围,但仅在 WGC 对 DirectX/Vulkan 实测为非黑帧后启用;否则固定显示“不支持自动观察”。
|
||||||
|
|
||||||
### 3.2 本地过滤与质量门:先决定“不发”
|
### 3.2 本地过滤与质量门:先决定“不发”
|
||||||
|
|
||||||
|
|
@ -116,7 +122,7 @@ flowchart LR
|
||||||
|
|
||||||
### 3.6 可见性、停止与审计
|
### 3.6 可见性、停止与审计
|
||||||
|
|
||||||
- 捕获范围生效时,外壳常驻橙色指示器显示“正在读取:单显示器/全部显示器”,并提供直接停止入口。
|
- 捕获范围生效时,外壳常驻橙色指示器显示“正在读取:指定窗口/指定显示器/全部显示器”,并提供直接停止入口。
|
||||||
- 设置页显示最后一次**结果**而非原始内容:时间、手动/自动、结果码、范围类型、触发原因、下次最早评估时间。
|
- 设置页显示最后一次**结果**而非原始内容:时间、手动/自动、结果码、范围类型、触发原因、下次最早评估时间。
|
||||||
- 停止需同步取消等待中的定时器、当前捕获、上传和后续主动回应;取消竞态必须有测试。
|
- 停止需同步取消等待中的定时器、当前捕获、上传和后续主动回应;取消竞态必须有测试。
|
||||||
- 本地只持久化用户设置、授权状态摘要、最近观察时间、脱敏结果码和短期不可逆去重指纹;原始帧、原文摘要及完整模型输出不得进入 SQLite、日志、崩溃报告、测试快照或剪贴板。
|
- 本地只持久化用户设置、授权状态摘要、最近观察时间、脱敏结果码和短期不可逆去重指纹;原始帧、原文摘要及完整模型输出不得进入 SQLite、日志、崩溃报告、测试快照或剪贴板。
|
||||||
|
|
@ -131,14 +137,14 @@ flowchart LR
|
||||||
|
|
||||||
### S0:WGC 与安全边界 Spike(阻塞后续)
|
### S0:WGC 与安全边界 Spike(阻塞后续)
|
||||||
|
|
||||||
- 验证单显示器、全部显示器、全屏 DirectX/Vulkan 游戏、最小化/切换、显示器拔插、锁屏、受保护内容和敏感应用下的 WGC 行为。
|
- 验证指定窗口、单显示器、全部显示器、全屏 DirectX/Vulkan 游戏、最小化/切换、显示器拔插、锁屏、受保护内容和敏感应用下的 WGC 行为。
|
||||||
- 测量每种范围的 CPU、GPU、峰值内存、编码耗时、帧尺寸和失败类型;定义明确预算。
|
- 测量每种范围的 CPU、GPU、峰值内存、编码耗时、帧尺寸和失败类型;定义明确预算。
|
||||||
- 验收:真实游戏非黑帧;拒绝/降级路径可解释;指示器和停止在每个失败/成功路径均有效;结果写入 Spike 报告。任一关键项失败则不进入 S1。
|
- 验收:真实游戏非黑帧;拒绝/降级路径可解释;指示器和停止在每个失败/成功路径均有效;结果写入 Spike 报告。任一关键项失败则不进入 S1。
|
||||||
|
|
||||||
### S1:范围授权、持久化最小设置与状态页
|
### S1:范围授权、持久化最小设置与状态页
|
||||||
|
|
||||||
- 实现单显示器/全部显示器专属授权、总开关、活跃时段、最短间隔、概率、互动风格与增强文字上下文开关。
|
- 实现指定窗口/指定显示器/全部显示器的专属授权、总开关、活跃时段、最短间隔、概率、互动风格与增强文字上下文开关。
|
||||||
- 验收:没有范围授权不可开启;修改范围必须重新授权;所有配置按 Profile 隔离;默认值安全且无原始内容落盘。
|
- 验收:没有范围授权不可开启;窗口句柄/进程变化、显示器变化或修改范围均必须停止并重新授权;所有配置按 Profile 隔离;默认值安全且无原始内容落盘。
|
||||||
|
|
||||||
### S2:Rust 调度与本地策略门
|
### S2:Rust 调度与本地策略门
|
||||||
|
|
||||||
|
|
@ -165,7 +171,7 @@ flowchart LR
|
||||||
- 在 S0 之前,不能宣称全桌面、游戏陪玩自动观察或“Bot 正在看屏幕”。
|
- 在 S0 之前,不能宣称全桌面、游戏陪玩自动观察或“Bot 正在看屏幕”。
|
||||||
- 在 S3 之前,不能宣称 Bot 理解了图片;附件可上传不等于视觉模型已接收和理解。
|
- 在 S3 之前,不能宣称 Bot 理解了图片;附件可上传不等于视觉模型已接收和理解。
|
||||||
- 在 WGC 的受保护内容策略明确前,不能以“尽量捕获”为目标绕过系统限制。
|
- 在 WGC 的受保护内容策略明确前,不能以“尽量捕获”为目标绕过系统限制。
|
||||||
- 在用户明确选择范围前,不能因概率或“陪伴模式”自动扩展到其他显示器、窗口或应用。
|
- 在用户明确选择范围前,不能因概率或“陪伴模式”自动扩展到其他窗口、显示器或应用。
|
||||||
|
|
||||||
## 7. 需要产品确认但不阻塞 S0 的细节
|
## 7. 需要产品确认但不阻塞 S0 的细节
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,28 @@
|
||||||
|
# WGC S0.1 Target-Binding Spike — 2026-07-27
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Establish the narrowest possible evidence that the Windows client can bind explicit capture targets through Windows.Graphics.Capture (WGC), without starting capture or processing screen content. This is an S0 sub-step, not a delivered screen-awareness feature.
|
||||||
|
|
||||||
|
## Implemented boundary
|
||||||
|
|
||||||
|
- `wgc_spike_support_status` reports only whether WGC is supported and the current attached-display count.
|
||||||
|
- `wgc_spike_probe_authorized_window` reuses the existing `CapturePolicy::inspect` gate and creates a WGC item only for an already authorized window.
|
||||||
|
- `wgc_spike_probe_displays` creates WGC items for the displays attached at call time.
|
||||||
|
- Each successful probe returns target kind plus width and height. Invalid or non-positive dimensions are rejected.
|
||||||
|
|
||||||
|
No command creates a `GraphicsCaptureSession`, retrieves a frame, reads text, consumes capture budget, writes raw content, encodes an image, schedules an observation, or calls AstrBot.
|
||||||
|
|
||||||
|
## Validation
|
||||||
|
|
||||||
|
- `cargo fmt --all`: passed.
|
||||||
|
- With `CARGO_TARGET_DIR=src-tauri/.tmp/wgc-check`, `cargo check -p astrbot-desktop --lib`: passed.
|
||||||
|
- With the same isolated target, `cargo test -p astrbot-desktop --lib`: 117 passed, 0 failed.
|
||||||
|
|
||||||
|
The isolated target is required in this workspace because the default Cargo target resolves generated Tauri permissions through the mapped UNC path and can fail with Windows access denied (`os error 5`).
|
||||||
|
|
||||||
|
## Non-evidence and next gate
|
||||||
|
|
||||||
|
S0.1 provides no proof that WGC returns useful pixels. It does not validate DirectX/Vulkan games, minimized targets, display topology changes, lock screens, protected content, sensitive-application filtering, quality thresholds, or AstrBot visual understanding.
|
||||||
|
|
||||||
|
S0.2 must create a bounded D3D11 WGC frame path, copy one frame to a staging texture, convert BGRA to RGBA, detect black/transparent/empty results, and record manual Windows evidence before any screen frame can progress to a local privacy or network path.
|
||||||
32
docs/错题本.md
32
docs/错题本.md
|
|
@ -59,3 +59,35 @@
|
||||||
- 修复或规避:诊断页 helper 改为等待目标标题,调用方显式等待;仅该启动集成测试预算设为十秒,不调整全局测试配置或产品代码。
|
- 修复或规避:诊断页 helper 改为等待目标标题,调用方显式等待;仅该启动集成测试预算设为十秒,不调整全局测试配置或产品代码。
|
||||||
- 预防规则:涉及懒加载页面切换的集成测试必须等待可见的页面就绪信号;单条跨异步边界测试应声明与其真实链路匹配的局部预算。
|
- 预防规则:涉及懒加载页面切换的集成测试必须等待可见的页面就绪信号;单条跨异步边界测试应声明与其真实链路匹配的局部预算。
|
||||||
- 验证证据:修复后运行完整 Vitest、ESLint、TypeScript 和生产 Web 构建;产品逻辑无改动。
|
- 验证证据:修复后运行完整 Vitest、ESLint、TypeScript 和生产 Web 构建;产品逻辑无改动。
|
||||||
|
|
||||||
|
|
||||||
|
## 2026-07-27 — 不能把“当前前台窗口”当成长期授权范围
|
||||||
|
|
||||||
|
- 阶段:屏幕感知三档范围规划
|
||||||
|
- 类型:功能行为 / 安全边界
|
||||||
|
- 现象:若自动观察始终读取当前前台窗口,用户切换到聊天、浏览器或敏感工具时,实际捕获目标会在没有新授权的情况下变化。
|
||||||
|
- 根因:把“识别当前窗口”误用为“自动跟随当前窗口”,没有将用户选择的捕获范围绑定为稳定授权令牌。
|
||||||
|
- 修复或规避:将范围明确为指定窗口、指定显示器、全部显示器三档;窗口身份或显示器拓扑变化时撤销会话,调度器只复用已授权范围。
|
||||||
|
- 预防规则:任何自动观察不得根据前台变化自行更换目标;范围变更必须先停止、再显示专属同意、再重新授权。
|
||||||
|
- 验证证据:PD-0014、screen-awareness-understanding-framework、实施队列 S0/C2/C4 均更新为三档范围验收。
|
||||||
|
|
||||||
|
|
||||||
|
## 2026-07-27 — WGC interop types are not the Win32 aliases
|
||||||
|
|
||||||
|
- 阶段:S0.1 WGC 目标绑定预检
|
||||||
|
- 类型:编码 / Windows API 契约
|
||||||
|
- 现象:首次编译 WGC 显示器预检时,`BOOL` 从 `Win32::Foundation` 导入失败,`GraphicsCaptureItem::Size` 也不能直接传给内部尺寸检查。
|
||||||
|
- 根因:windows-rs 将此回调的 `BOOL` 暴露在 `windows::core`,而 WinRT `SizeInt32` 与内部 `WgcSize` 是不同类型,不能依赖隐式转换。
|
||||||
|
- 修复或规避:改用 `windows::core::BOOL`,并对窗口和显示器的 `Size()` 结果显式 `.into()`。
|
||||||
|
- 预防规则:新增 WinRT/WGC 调用先以隔离 `cargo check` 验证每个特征门和跨命名空间类型;相同 API 的两个分支必须保持相同的显式转换。
|
||||||
|
- 验证证据:`cargo check -p astrbot-desktop --lib` 通过;`cargo test -p astrbot-desktop --lib` 117 passed。
|
||||||
|
|
||||||
|
## 2026-07-27 — 隔离 Cargo 目标的冷启动不能按短超时判失败
|
||||||
|
|
||||||
|
- 阶段:S0.1 WGC 目标绑定预检
|
||||||
|
- 类型:测试 / 环境
|
||||||
|
- 现象:首次在新的 `src-tauri/.tmp/wgc-check` 目标目录运行 Rust 测试超过两分钟而被调用超时。
|
||||||
|
- 根因:隔离目标目录需要重新编译完整 Tauri 依赖图;这不是测试失败,且默认目标目录仍受 UNC 生成权限文件的访问拒绝影响。
|
||||||
|
- 修复或规避:保留项目内隔离目标缓存并以足够时限重跑;最终 117 个库测试全部通过。
|
||||||
|
- 预防规则:默认目标触发 UNC `os error 5` 时,使用已验证的项目内 `CARGO_TARGET_DIR`;首次冷启动须看退出码和最终测试结果,不能把工具超时当作断言失败。
|
||||||
|
- 验证证据:隔离 `cargo check` 和后续 `cargo test -p astrbot-desktop --lib` 均成功。
|
||||||
|
|
|
||||||
|
|
@ -44,7 +44,9 @@ zeroize.workspace = true
|
||||||
[target.'cfg(windows)'.dependencies]
|
[target.'cfg(windows)'.dependencies]
|
||||||
windows = { workspace = true, features = [
|
windows = { workspace = true, features = [
|
||||||
"Win32_Foundation",
|
"Win32_Foundation",
|
||||||
|
"Graphics_Capture",
|
||||||
"Win32_Graphics_Gdi",
|
"Win32_Graphics_Gdi",
|
||||||
|
"Win32_System_WinRT_Graphics_Capture",
|
||||||
"Win32_NetworkManagement_IpHelper",
|
"Win32_NetworkManagement_IpHelper",
|
||||||
"Win32_Networking_WinSock",
|
"Win32_Networking_WinSock",
|
||||||
"Win32_Security_Credentials",
|
"Win32_Security_Credentials",
|
||||||
|
|
|
||||||
|
|
@ -317,6 +317,31 @@ impl CapturePolicy {
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Validates an authorized window without reading its text or pixels.
|
||||||
|
///
|
||||||
|
/// WGC target binding uses this narrow path during S0. It deliberately does
|
||||||
|
/// not consume a read budget because it does not create a capture session or
|
||||||
|
/// expose screen content.
|
||||||
|
pub fn inspect<T>(
|
||||||
|
&self,
|
||||||
|
token: Uuid,
|
||||||
|
body: impl FnOnce(&WindowSelection) -> T,
|
||||||
|
) -> Result<T, CapturePolicyError> {
|
||||||
|
let guard = self.state.lock().expect("capture policy lock poisoned");
|
||||||
|
if !guard.enabled {
|
||||||
|
return Err(CapturePolicyError::Disabled);
|
||||||
|
}
|
||||||
|
let session = guard
|
||||||
|
.sessions
|
||||||
|
.get(&token)
|
||||||
|
.ok_or(CapturePolicyError::UnknownSession)?;
|
||||||
|
session
|
||||||
|
.selection
|
||||||
|
.validate()
|
||||||
|
.map_err(CapturePolicyError::Selection)?;
|
||||||
|
Ok(body(&session.selection))
|
||||||
|
}
|
||||||
|
|
||||||
/// Checks a read is allowed, and runs it against the authorized window.
|
/// Checks a read is allowed, and runs it against the authorized window.
|
||||||
///
|
///
|
||||||
/// The read is a closure so the handle never leaves this module: every gate
|
/// The read is a closure so the handle never leaves this module: every gate
|
||||||
|
|
@ -396,6 +421,25 @@ mod tests {
|
||||||
let _ = &policy;
|
let _ = &policy;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn an_inspection_is_refused_while_the_switch_is_off() {
|
||||||
|
let policy = CapturePolicy::default();
|
||||||
|
let error = policy
|
||||||
|
.inspect(Uuid::new_v4(), |_| ())
|
||||||
|
.expect_err("must be refused");
|
||||||
|
assert_eq!(error, CapturePolicyError::Disabled);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn an_inspection_is_refused_when_no_session_names_the_window() {
|
||||||
|
let policy = CapturePolicy::default();
|
||||||
|
policy.set_enabled(true);
|
||||||
|
let error = policy
|
||||||
|
.inspect(Uuid::new_v4(), |_| ())
|
||||||
|
.expect_err("must be refused");
|
||||||
|
assert_eq!(error, CapturePolicyError::UnknownSession);
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn a_read_is_refused_while_the_switch_is_off() {
|
fn a_read_is_refused_while_the_switch_is_off() {
|
||||||
let policy = CapturePolicy::default();
|
let policy = CapturePolicy::default();
|
||||||
|
|
|
||||||
|
|
@ -555,6 +555,27 @@ async fn begin_window_capture_session(
|
||||||
.ok_or_else(|| "capture session expired immediately".to_owned())
|
.ok_or_else(|| "capture session expired immediately".to_owned())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[tauri::command]
|
||||||
|
fn wgc_spike_support_status() -> windows::wgc::WgcSupportStatus {
|
||||||
|
windows::wgc::support_status()
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tauri::command]
|
||||||
|
fn wgc_spike_probe_authorized_window(
|
||||||
|
app: AppHandle,
|
||||||
|
session_token: uuid::Uuid,
|
||||||
|
) -> Result<windows::wgc::WgcTargetProbe, String> {
|
||||||
|
app.state::<capture_policy::CapturePolicy>()
|
||||||
|
.inspect(session_token, windows::wgc::probe_selected_window)
|
||||||
|
.map_err(|error| error.to_string())?
|
||||||
|
.map_err(|error| error.to_string())
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tauri::command]
|
||||||
|
fn wgc_spike_probe_displays() -> Result<Vec<windows::wgc::WgcTargetProbe>, String> {
|
||||||
|
windows::wgc::probe_display_targets().map_err(|error| error.to_string())
|
||||||
|
}
|
||||||
|
|
||||||
#[tauri::command]
|
#[tauri::command]
|
||||||
fn end_window_capture_session(app: AppHandle, session_token: uuid::Uuid) -> ScreenReadingStatus {
|
fn end_window_capture_session(app: AppHandle, session_token: uuid::Uuid) -> ScreenReadingStatus {
|
||||||
let policy = app.state::<capture_policy::CapturePolicy>();
|
let policy = app.state::<capture_policy::CapturePolicy>();
|
||||||
|
|
@ -942,6 +963,9 @@ pub fn run() {
|
||||||
pet_window_preferences,
|
pet_window_preferences,
|
||||||
inspect_foreground_window,
|
inspect_foreground_window,
|
||||||
screen_reading_status,
|
screen_reading_status,
|
||||||
|
wgc_spike_support_status,
|
||||||
|
wgc_spike_probe_authorized_window,
|
||||||
|
wgc_spike_probe_displays,
|
||||||
set_screen_reading_enabled,
|
set_screen_reading_enabled,
|
||||||
begin_window_capture_session,
|
begin_window_capture_session,
|
||||||
end_window_capture_session,
|
end_window_capture_session,
|
||||||
|
|
|
||||||
|
|
@ -6,3 +6,4 @@ pub mod foreground_window;
|
||||||
pub mod ports;
|
pub mod ports;
|
||||||
pub mod process;
|
pub mod process;
|
||||||
pub mod uia;
|
pub mod uia;
|
||||||
|
pub mod wgc;
|
||||||
|
|
|
||||||
302
src-tauri/src/windows/wgc.rs
普通文件
302
src-tauri/src/windows/wgc.rs
普通文件
|
|
@ -0,0 +1,302 @@
|
||||||
|
//! Windows.Graphics.Capture target-binding spike.
|
||||||
|
//!
|
||||||
|
//! This module intentionally stops before a pixel frame is requested. It proves
|
||||||
|
//! whether this Windows build can construct WGC capture items for an already
|
||||||
|
//! authorized window and for the attached displays, without exposing a frame to
|
||||||
|
//! React, disk, logging, or the network. The next S0 increment must add a
|
||||||
|
//! bounded D3D11 frame path and black-frame validation before this can become a
|
||||||
|
//! capture feature.
|
||||||
|
|
||||||
|
use std::fmt;
|
||||||
|
|
||||||
|
use serde::Serialize;
|
||||||
|
|
||||||
|
use super::foreground_window::{ForegroundWindowError, WindowSelection};
|
||||||
|
|
||||||
|
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)]
|
||||||
|
#[serde(rename_all = "snake_case")]
|
||||||
|
pub enum WgcTargetKind {
|
||||||
|
Window,
|
||||||
|
Display,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
|
||||||
|
#[serde(rename_all = "camelCase")]
|
||||||
|
pub struct WgcTargetProbe {
|
||||||
|
pub kind: WgcTargetKind,
|
||||||
|
pub width: u32,
|
||||||
|
pub height: u32,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
|
||||||
|
#[serde(rename_all = "camelCase")]
|
||||||
|
pub struct WgcSupportStatus {
|
||||||
|
pub supported: bool,
|
||||||
|
pub display_target_count: u16,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||||
|
pub enum WgcProbeError {
|
||||||
|
Selection(ForegroundWindowError),
|
||||||
|
UnsupportedPlatform,
|
||||||
|
UnsupportedByWindows,
|
||||||
|
InvalidTargetSize,
|
||||||
|
Platform(u32),
|
||||||
|
}
|
||||||
|
|
||||||
|
impl fmt::Display for WgcProbeError {
|
||||||
|
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
|
||||||
|
match self {
|
||||||
|
Self::Selection(error) => write!(formatter, "the selected window is invalid: {error}"),
|
||||||
|
Self::UnsupportedPlatform => {
|
||||||
|
formatter.write_str("Windows.Graphics.Capture is unavailable on this platform")
|
||||||
|
}
|
||||||
|
Self::UnsupportedByWindows => formatter
|
||||||
|
.write_str("Windows.Graphics.Capture is not supported by this Windows session"),
|
||||||
|
Self::InvalidTargetSize => {
|
||||||
|
formatter.write_str("Windows.Graphics.Capture returned an invalid target size")
|
||||||
|
}
|
||||||
|
Self::Platform(code) => write!(
|
||||||
|
formatter,
|
||||||
|
"Windows.Graphics.Capture failed with HRESULT 0x{code:08x}"
|
||||||
|
),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::error::Error for WgcProbeError {}
|
||||||
|
|
||||||
|
/// Returns only capability metadata. Enumerating display targets never creates
|
||||||
|
/// a capture session or reads pixels.
|
||||||
|
pub fn support_status() -> WgcSupportStatus {
|
||||||
|
#[cfg(windows)]
|
||||||
|
{
|
||||||
|
let supported = wgc_supported().unwrap_or(false);
|
||||||
|
let display_target_count = if supported {
|
||||||
|
display_handles().map_or(0, |displays| {
|
||||||
|
u16::try_from(displays.len()).unwrap_or(u16::MAX)
|
||||||
|
})
|
||||||
|
} else {
|
||||||
|
0
|
||||||
|
};
|
||||||
|
WgcSupportStatus {
|
||||||
|
supported,
|
||||||
|
display_target_count,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(not(windows))]
|
||||||
|
{
|
||||||
|
WgcSupportStatus {
|
||||||
|
supported: false,
|
||||||
|
display_target_count: 0,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Proves that the current, policy-authorized window can be represented by WGC.
|
||||||
|
/// It does not start a capture session or expose pixels.
|
||||||
|
pub fn probe_selected_window(selection: &WindowSelection) -> Result<WgcTargetProbe, WgcProbeError> {
|
||||||
|
selection.validate().map_err(WgcProbeError::Selection)?;
|
||||||
|
|
||||||
|
#[cfg(windows)]
|
||||||
|
{
|
||||||
|
if !wgc_supported()? {
|
||||||
|
return Err(WgcProbeError::UnsupportedByWindows);
|
||||||
|
}
|
||||||
|
let item = capture_item_for_window(selection)?;
|
||||||
|
target_probe(
|
||||||
|
WgcTargetKind::Window,
|
||||||
|
item.Size().map_err(map_windows_error)?.into(),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(not(windows))]
|
||||||
|
{
|
||||||
|
let _ = selection;
|
||||||
|
Err(WgcProbeError::UnsupportedPlatform)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Probes every attached display target without reading any image data. This is
|
||||||
|
/// S0 evidence that the eventual one-display and all-display scopes can bind to
|
||||||
|
/// WGC items; it is not an authorization or capture API.
|
||||||
|
pub fn probe_display_targets() -> Result<Vec<WgcTargetProbe>, WgcProbeError> {
|
||||||
|
#[cfg(windows)]
|
||||||
|
{
|
||||||
|
if !wgc_supported()? {
|
||||||
|
return Err(WgcProbeError::UnsupportedByWindows);
|
||||||
|
}
|
||||||
|
display_handles()?
|
||||||
|
.into_iter()
|
||||||
|
.map(|display| {
|
||||||
|
let item = capture_item_for_monitor(display)?;
|
||||||
|
target_probe(
|
||||||
|
WgcTargetKind::Display,
|
||||||
|
item.Size().map_err(map_windows_error)?.into(),
|
||||||
|
)
|
||||||
|
})
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(not(windows))]
|
||||||
|
{
|
||||||
|
Err(WgcProbeError::UnsupportedPlatform)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn target_probe(kind: WgcTargetKind, size: WgcSize) -> Result<WgcTargetProbe, WgcProbeError> {
|
||||||
|
let width = u32::try_from(size.width).map_err(|_| WgcProbeError::InvalidTargetSize)?;
|
||||||
|
let height = u32::try_from(size.height).map_err(|_| WgcProbeError::InvalidTargetSize)?;
|
||||||
|
if width == 0 || height == 0 {
|
||||||
|
return Err(WgcProbeError::InvalidTargetSize);
|
||||||
|
}
|
||||||
|
Ok(WgcTargetProbe {
|
||||||
|
kind,
|
||||||
|
width,
|
||||||
|
height,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Clone, Copy)]
|
||||||
|
struct WgcSize {
|
||||||
|
width: i32,
|
||||||
|
height: i32,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(windows)]
|
||||||
|
impl From<windows::Graphics::SizeInt32> for WgcSize {
|
||||||
|
fn from(value: windows::Graphics::SizeInt32) -> Self {
|
||||||
|
Self {
|
||||||
|
width: value.Width,
|
||||||
|
height: value.Height,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(windows)]
|
||||||
|
fn wgc_supported() -> Result<bool, WgcProbeError> {
|
||||||
|
windows::Graphics::Capture::GraphicsCaptureSession::IsSupported().map_err(map_windows_error)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(windows)]
|
||||||
|
fn capture_item_for_window(
|
||||||
|
selection: &WindowSelection,
|
||||||
|
) -> Result<windows::Graphics::Capture::GraphicsCaptureItem, WgcProbeError> {
|
||||||
|
use windows::{
|
||||||
|
Win32::{Foundation::HWND, System::WinRT::Graphics::Capture::IGraphicsCaptureItemInterop},
|
||||||
|
core::factory,
|
||||||
|
};
|
||||||
|
|
||||||
|
let interop =
|
||||||
|
factory::<windows::Graphics::Capture::GraphicsCaptureItem, IGraphicsCaptureItemInterop>()
|
||||||
|
.map_err(map_windows_error)?;
|
||||||
|
unsafe {
|
||||||
|
interop
|
||||||
|
.CreateForWindow(HWND(selection.raw_handle() as *mut _))
|
||||||
|
.map_err(map_windows_error)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(windows)]
|
||||||
|
fn capture_item_for_monitor(
|
||||||
|
monitor: windows::Win32::Graphics::Gdi::HMONITOR,
|
||||||
|
) -> Result<windows::Graphics::Capture::GraphicsCaptureItem, WgcProbeError> {
|
||||||
|
use windows::{
|
||||||
|
Win32::System::WinRT::Graphics::Capture::IGraphicsCaptureItemInterop, core::factory,
|
||||||
|
};
|
||||||
|
|
||||||
|
let interop =
|
||||||
|
factory::<windows::Graphics::Capture::GraphicsCaptureItem, IGraphicsCaptureItemInterop>()
|
||||||
|
.map_err(map_windows_error)?;
|
||||||
|
unsafe { interop.CreateForMonitor(monitor).map_err(map_windows_error) }
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(windows)]
|
||||||
|
fn display_handles() -> Result<Vec<windows::Win32::Graphics::Gdi::HMONITOR>, WgcProbeError> {
|
||||||
|
use windows::{
|
||||||
|
Win32::{
|
||||||
|
Foundation::{LPARAM, RECT},
|
||||||
|
Graphics::Gdi::{EnumDisplayMonitors, HDC, HMONITOR},
|
||||||
|
},
|
||||||
|
core::BOOL,
|
||||||
|
};
|
||||||
|
|
||||||
|
unsafe extern "system" fn collect_display(
|
||||||
|
monitor: HMONITOR,
|
||||||
|
_device_context: HDC,
|
||||||
|
_bounds: *mut RECT,
|
||||||
|
data: LPARAM,
|
||||||
|
) -> BOOL {
|
||||||
|
let displays = unsafe { &mut *(data.0 as *mut Vec<HMONITOR>) };
|
||||||
|
displays.push(monitor);
|
||||||
|
BOOL(1)
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut displays = Vec::new();
|
||||||
|
let succeeded = unsafe {
|
||||||
|
EnumDisplayMonitors(
|
||||||
|
None,
|
||||||
|
None,
|
||||||
|
Some(collect_display),
|
||||||
|
LPARAM((&mut displays as *mut Vec<HMONITOR>) as isize),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
.as_bool();
|
||||||
|
if !succeeded {
|
||||||
|
return Err(last_windows_error());
|
||||||
|
}
|
||||||
|
Ok(displays)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(windows)]
|
||||||
|
fn map_windows_error(error: windows::core::Error) -> WgcProbeError {
|
||||||
|
WgcProbeError::Platform(error.code().0 as u32)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(windows)]
|
||||||
|
fn last_windows_error() -> WgcProbeError {
|
||||||
|
let code = unsafe { windows::Win32::Foundation::GetLastError() }.0;
|
||||||
|
WgcProbeError::Platform(code)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn refuses_non_positive_target_sizes() {
|
||||||
|
assert_eq!(
|
||||||
|
target_probe(
|
||||||
|
WgcTargetKind::Display,
|
||||||
|
WgcSize {
|
||||||
|
width: 0,
|
||||||
|
height: 1080
|
||||||
|
}
|
||||||
|
),
|
||||||
|
Err(WgcProbeError::InvalidTargetSize)
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
target_probe(
|
||||||
|
WgcTargetKind::Window,
|
||||||
|
WgcSize {
|
||||||
|
width: -1,
|
||||||
|
height: 1080
|
||||||
|
}
|
||||||
|
),
|
||||||
|
Err(WgcProbeError::InvalidTargetSize)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn serializable_support_status_starts_with_no_pixels() {
|
||||||
|
let status = WgcSupportStatus {
|
||||||
|
supported: false,
|
||||||
|
display_target_count: 0,
|
||||||
|
};
|
||||||
|
assert_eq!(
|
||||||
|
serde_json::to_string(&status).expect("serialize"),
|
||||||
|
"{\"supported\":false,\"displayTargetCount\":0}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
正在加载…
添加表格
在新工单中引用