AstrBot-Desktop/docs/decisions/ADR-0008-screen-awareness-and-voice-boundary.md
YokiiroBW 7aead6fa48
一些检测仍在等待运行
ci / Portable quality checks (push) Waiting to run
ci / Windows Tauri build (push) Waiting to run
feat(policy): gate screen reading behind consent, a rate limit, and a switch
implementation-queue.md:129 requires an authorization prompt, a frequency
limit, and a privacy switch to exist before screenshots or window
summaries are wired to anything. None of the three existed anywhere in
the repository, so this is them, plus ADR-0008 recording what the
authoritative documents actually permit.

The ADR corrects a mis-reading that would have decided the feature the
wrong way. 01_Architecture_Design.md:250-261 is titled 「不应放在前端的
能力」, so the 「原始截图捕获」 at :259 forbids screenshot capture in
React — it is not a product-level ban, which is how an earlier audit read
it before concluding screen reading was out of scope. The product
affirmatively permits 「用户授权的单次窗口截图」 (:739) and, for phase 7,
「手动和低频截图」 (:1424). What it does exclude is 「实时全双工语音」,
「唤醒词」 and 「高频视频分析」 (:1349-1351).

The module owns all three gates so none of them depends on the frontend
being correct. Reading takes a closure, so the window handle never leaves
the module and there is no accessor that skips the switch, the session,
the expiry check, or the budget. The switch defaults to off and clearing
it drops live sessions, so flipping it mid-capture stops the next read
rather than only preventing new authorizations. `now` is a parameter to
the budget rather than read inside it, which is what makes the interval
testable without sleeping.

Two decisions worth naming. The pixel interval defaults to two seconds
with a 500ms floor no caller can lower — three orders of magnitude away
from a video frame rate, because high-frequency video analysis is out of
scope and what phase 7 asks for is manual, low-frequency capture. And a
session cannot be bound to the operations slot at all, refused at
creation rather than at each read, because :1077 says the operations slot
does not automatically receive game screenshots, window state, or game
events — and a session is exactly that automatic delivery. The slot rule
is a free function so it is asserted directly rather than through a live
window.

Also recorded, not worked around: capture.rs uses PrintWindow over GDI,
which does not capture a hardware-accelerated game surface. Game
companionship needs Windows.Graphics.Capture before it works on the case
it exists for, and a black frame delivered to the model would be worse
than an error.

Rust tests 101 -> 110.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-27 06:22:15 +08:00

8.9 KiB

ADR-0008: Screen awareness, voice capture, and the game companion boundary

Context

Voice chat, window awareness, and game companionship are in scope for v0.3.0-alpha.1, and the Rust modules for reading the foreground window (foreground_window.rs), walking its UI Automation tree (uia.rs), and capturing a selected window (capture.rs) are already written. None of them has a Tauri command, so nothing in the product can reach them, and there is no consent, rate-limit, or privacy concept anywhere in the repository — not in policy.rs, not in runtime_guards.rs, not in the capability files, not in the database schema, not in the UI.

The authoritative documents were read directly for this decision, because an earlier reading of them was wrong in a way that would have changed the outcome:

  • 01_Architecture_Design.md:250-261 is titled 「不应放在前端的能力」. The 「原始截图捕获」 at line 259 is therefore a frontend exclusion — screenshot capture must not live in React — and not a product-level ban. An earlier audit cited it as the latter and concluded that screen reading was out of scope. That was a mis-read.
  • What the product affirmatively permits: 「用户授权的单次窗口截图」 (:739, among the inputs a diagnostic may gather) and, for phase 7, 「按键说话;TTS;游戏窗口绑定;时长事件;手动和低频截图」 (:1418-1424).
  • What it excludes outright: 「实时全双工语音」, 「唤醒词」, and 「高频视频分析」 (:1349-1351). AGENTS.md:32 excludes 「无限制桌面控制」.
  • The voice flow is fixed at :1057-1063 as 按键录音 → 上传音频 → STT → 文本回复 → 可选 TTS, with the microphone bound to the focused instance by default (:1065) and one recording never sent to both slots unless the user explicitly asks (:1071).
  • The slot isolation rule is one sentence at :1077: 「运维 Slot 不自动接收游戏截图、窗口状态或游戏 事件。」
  • implementation-queue.md:129 already states the gate: 「游戏模式接入截图/状态摘要前,先完成授权 提示、频率限制和隐私开关。」

So the policy does not have to be invented. It has to be written down precisely enough to implement, and one thing genuinely is missing from every document: whether the user must be able to see that the application is reading their screen.

Decision

  1. Screen and window reading happens only inside an explicit capture session. A session is started by a user action, names exactly one target window, and carries an expiry. No command reads a window, its UI Automation tree, or its pixels outside a live session. There is no ambient or background reading, and no path that reads the desktop or a monitor rather than a window.

  2. The session is the authorization record, and it lives in Rust. The frontend may ask to start one and may end one; it cannot name a window that no session covers. WindowSelection currently exists only inside inspect_foreground_window and is dropped when that command returns, so a registry in Tauri state is the prerequisite for any of this, not an optimization.

  3. Every read is rate limited by the session, in Rust. The limit is a minimum interval between reads of the same kind, enforced where the read happens rather than in the caller. Frontend throttling does not count: implementation-queue.md:129 requires a frequency limit before screenshots are wired at all, and a limit the renderer can skip is not one. The default interval for pixel capture is deliberately far above a video frame rate, since 「高频视频分析」 is out of scope — this is 「手动和低频截图」.

  4. A privacy switch disables the whole capability, and it wins over any session. When it is off, the commands fail rather than returning stale or partial data, and no session can be started.

  5. The user can see when a session is live, and can end it from there. Nothing in the documents requires this, so it is proposed here: reading someone's screen without a visible sign is the kind of behaviour that is indistinguishable from spyware from the outside, and a kill switch the user has to go looking for is not a kill switch. The indicator is part of the feature, not a later polish item.

  6. Captured frames and window text are never written to disk by this feature. They are held in memory, sent to the bound slot, and dropped. 01_Architecture_Design.md:1487 already says a diagnostic keeps no raw screenshot unless the user saves it; this extends the same rule to the companion path, where there is no save affordance at all.

  7. Screen and window data go to the companion slot only. The operations slot never receives a capture, a window summary, or a game event as a consequence of a session — only as a consequence of the user explicitly sending one. This is :1077 restated as an invariant that must be enforced in Rust and covered by a test that fails if the guard is removed.

  8. Voice follows the documented flow, server side. Press to record, upload the audio, let AstrBot transcribe it, receive text, optionally receive speech back. The client does not run speech recognition of its own: WebView2 provides no SpeechRecognition, and the current ChatPanel.tsx call to it can never have worked. Audio is uploaded through the same /api/v1/files endpoint as any other attachment and referenced as a record segment inside message, which is the contract AstrBot actually reads.

  9. No always-listening capability. No wake word, no voice activity detection that starts a recording on its own, no continuous duplex stream. Recording starts and stops on a user action. This is :1349-1350.

  10. Speech-to-text requires a provider the user configures, and this project does not choose one for them. Every AstrBot STT provider needs either an API key or a self-hosted model. Enabling one sends the user's voice to whichever service they picked, so the credential and the choice are theirs. The client's job is to make the path work once they have configured it, to report clearly when they have not, and never to enrol a provider on their behalf.

Rationale

  • The gate in implementation-queue.md:129 is a precondition, not a checklist item to close later. Wiring the capture commands first and adding consent afterwards would mean shipping a build in which any renderer bug can read the user's screen.
  • Putting the session, the rate limit, and the switch in Rust follows the boundary 01_Architecture_Design.md:250-261 already draws. It also means the guarantee does not depend on the frontend being correct, which is where every silent failure in this codebase so far has lived.
  • A single-window target rather than a monitor keeps the feature inside 「用户授权的单次窗口截图」 and makes the authorization meaningful: the user is agreeing to one specific window, not to their desktop.
  • Binding to the companion slot by default and forbidding automatic delivery to operations keeps the two conversations from leaking into each other, which is the whole point of the dual-slot design.
  • Leaving the STT provider to the user is not caution for its own sake. It is the one part of this feature that sends their voice somewhere; picking a default would be making that decision for them.

Consequences

  • Add a capture_policy module in Rust owning the switch, the session registry, and the rate limit, with the session expiry and the interval as pure functions so they can be tested without Windows.
  • Register commands for the UI Automation summary and the window capture, each taking a session identifier and failing when no live session covers it. WindowSelection::raw_handle becomes reachable through the registry rather than being minted and dropped inside one command.
  • Add the capability entries for the new commands, and a test that reads the shipped capability files — a missing entry fails silently at runtime, which is how F-03 survived.
  • Surface the live session and its stop control in the UI, and the privacy switch in settings.
  • capture.rs uses PrintWindow over GDI. That does not capture a hardware-accelerated game surface, so game companionship needs a different capture path (Windows.Graphics.Capture) before it can work on the case it exists for. Recorded rather than worked around: a black frame delivered to the model is worse than an error.
  • Voice input needs microphone permission from WebView2. Tauri registers a PermissionRequested handler only when clipboard is enabled, which this application does not enable, so the default behaviour decides the outcome and has to be established on a real build rather than assumed.
  • Do not implement mouse or keyboard control, process injection, or reading a game's memory. Detect the foreground window through the existing bounded inspection only, per implementation-queue.md:118.