implementation-queue.md:129 requires an authorization prompt, a frequency limit, and a privacy switch to exist before screenshots or window summaries are wired to anything. None of the three existed anywhere in the repository, so this is them, plus ADR-0008 recording what the authoritative documents actually permit. The ADR corrects a mis-reading that would have decided the feature the wrong way. 01_Architecture_Design.md:250-261 is titled 「不应放在前端的 能力」, so the 「原始截图捕获」 at :259 forbids screenshot capture in React — it is not a product-level ban, which is how an earlier audit read it before concluding screen reading was out of scope. The product affirmatively permits 「用户授权的单次窗口截图」 (:739) and, for phase 7, 「手动和低频截图」 (:1424). What it does exclude is 「实时全双工语音」, 「唤醒词」 and 「高频视频分析」 (:1349-1351). The module owns all three gates so none of them depends on the frontend being correct. Reading takes a closure, so the window handle never leaves the module and there is no accessor that skips the switch, the session, the expiry check, or the budget. The switch defaults to off and clearing it drops live sessions, so flipping it mid-capture stops the next read rather than only preventing new authorizations. `now` is a parameter to the budget rather than read inside it, which is what makes the interval testable without sleeping. Two decisions worth naming. The pixel interval defaults to two seconds with a 500ms floor no caller can lower — three orders of magnitude away from a video frame rate, because high-frequency video analysis is out of scope and what phase 7 asks for is manual, low-frequency capture. And a session cannot be bound to the operations slot at all, refused at creation rather than at each read, because :1077 says the operations slot does not automatically receive game screenshots, window state, or game events — and a session is exactly that automatic delivery. The slot rule is a free function so it is asserted directly rather than through a live window. Also recorded, not worked around: capture.rs uses PrintWindow over GDI, which does not capture a hardware-accelerated game surface. Game companionship needs Windows.Graphics.Capture before it works on the case it exists for, and a black frame delivered to the model would be worse than an error. Rust tests 101 -> 110. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
8.9 KiB
ADR-0008: Screen awareness, voice capture, and the game companion boundary
Context
Voice chat, window awareness, and game companionship are in scope for v0.3.0-alpha.1, and the Rust
modules for reading the foreground window (foreground_window.rs), walking its UI Automation tree
(uia.rs), and capturing a selected window (capture.rs) are already written. None of them has a
Tauri command, so nothing in the product can reach them, and there is no consent, rate-limit, or
privacy concept anywhere in the repository — not in policy.rs, not in runtime_guards.rs, not in
the capability files, not in the database schema, not in the UI.
The authoritative documents were read directly for this decision, because an earlier reading of them was wrong in a way that would have changed the outcome:
01_Architecture_Design.md:250-261is titled 「不应放在前端的能力」. The 「原始截图捕获」 at line 259 is therefore a frontend exclusion — screenshot capture must not live in React — and not a product-level ban. An earlier audit cited it as the latter and concluded that screen reading was out of scope. That was a mis-read.- What the product affirmatively permits: 「用户授权的单次窗口截图」 (
:739, among the inputs a diagnostic may gather) and, for phase 7, 「按键说话;TTS;游戏窗口绑定;时长事件;手动和低频截图」 (:1418-1424). - What it excludes outright: 「实时全双工语音」, 「唤醒词」, and 「高频视频分析」 (
:1349-1351).AGENTS.md:32excludes 「无限制桌面控制」. - The voice flow is fixed at
:1057-1063as 按键录音 → 上传音频 → STT → 文本回复 → 可选 TTS, with the microphone bound to the focused instance by default (:1065) and one recording never sent to both slots unless the user explicitly asks (:1071). - The slot isolation rule is one sentence at
:1077: 「运维 Slot 不自动接收游戏截图、窗口状态或游戏 事件。」 implementation-queue.md:129already states the gate: 「游戏模式接入截图/状态摘要前,先完成授权 提示、频率限制和隐私开关。」
So the policy does not have to be invented. It has to be written down precisely enough to implement, and one thing genuinely is missing from every document: whether the user must be able to see that the application is reading their screen.
Decision
-
Screen and window reading happens only inside an explicit capture session. A session is started by a user action, names exactly one target window, and carries an expiry. No command reads a window, its UI Automation tree, or its pixels outside a live session. There is no ambient or background reading, and no path that reads the desktop or a monitor rather than a window.
-
The session is the authorization record, and it lives in Rust. The frontend may ask to start one and may end one; it cannot name a window that no session covers.
WindowSelectioncurrently exists only insideinspect_foreground_windowand is dropped when that command returns, so a registry in Tauri state is the prerequisite for any of this, not an optimization. -
Every read is rate limited by the session, in Rust. The limit is a minimum interval between reads of the same kind, enforced where the read happens rather than in the caller. Frontend throttling does not count:
implementation-queue.md:129requires a frequency limit before screenshots are wired at all, and a limit the renderer can skip is not one. The default interval for pixel capture is deliberately far above a video frame rate, since 「高频视频分析」 is out of scope — this is 「手动和低频截图」. -
A privacy switch disables the whole capability, and it wins over any session. When it is off, the commands fail rather than returning stale or partial data, and no session can be started.
-
The user can see when a session is live, and can end it from there. Nothing in the documents requires this, so it is proposed here: reading someone's screen without a visible sign is the kind of behaviour that is indistinguishable from spyware from the outside, and a kill switch the user has to go looking for is not a kill switch. The indicator is part of the feature, not a later polish item.
-
Captured frames and window text are never written to disk by this feature. They are held in memory, sent to the bound slot, and dropped.
01_Architecture_Design.md:1487already says a diagnostic keeps no raw screenshot unless the user saves it; this extends the same rule to the companion path, where there is no save affordance at all. -
Screen and window data go to the companion slot only. The operations slot never receives a capture, a window summary, or a game event as a consequence of a session — only as a consequence of the user explicitly sending one. This is
:1077restated as an invariant that must be enforced in Rust and covered by a test that fails if the guard is removed. -
Voice follows the documented flow, server side. Press to record, upload the audio, let AstrBot transcribe it, receive text, optionally receive speech back. The client does not run speech recognition of its own:
WebView2provides noSpeechRecognition, and the currentChatPanel.tsxcall to it can never have worked. Audio is uploaded through the same/api/v1/filesendpoint as any other attachment and referenced as arecordsegment insidemessage, which is the contract AstrBot actually reads. -
No always-listening capability. No wake word, no voice activity detection that starts a recording on its own, no continuous duplex stream. Recording starts and stops on a user action. This is
:1349-1350. -
Speech-to-text requires a provider the user configures, and this project does not choose one for them. Every AstrBot STT provider needs either an API key or a self-hosted model. Enabling one sends the user's voice to whichever service they picked, so the credential and the choice are theirs. The client's job is to make the path work once they have configured it, to report clearly when they have not, and never to enrol a provider on their behalf.
Rationale
- The gate in
implementation-queue.md:129is a precondition, not a checklist item to close later. Wiring the capture commands first and adding consent afterwards would mean shipping a build in which any renderer bug can read the user's screen. - Putting the session, the rate limit, and the switch in Rust follows the boundary
01_Architecture_Design.md:250-261already draws. It also means the guarantee does not depend on the frontend being correct, which is where every silent failure in this codebase so far has lived. - A single-window target rather than a monitor keeps the feature inside 「用户授权的单次窗口截图」 and makes the authorization meaningful: the user is agreeing to one specific window, not to their desktop.
- Binding to the companion slot by default and forbidding automatic delivery to operations keeps the two conversations from leaking into each other, which is the whole point of the dual-slot design.
- Leaving the STT provider to the user is not caution for its own sake. It is the one part of this feature that sends their voice somewhere; picking a default would be making that decision for them.
Consequences
- Add a
capture_policymodule in Rust owning the switch, the session registry, and the rate limit, with the session expiry and the interval as pure functions so they can be tested without Windows. - Register commands for the UI Automation summary and the window capture, each taking a session
identifier and failing when no live session covers it.
WindowSelection::raw_handlebecomes reachable through the registry rather than being minted and dropped inside one command. - Add the capability entries for the new commands, and a test that reads the shipped capability files — a missing entry fails silently at runtime, which is how F-03 survived.
- Surface the live session and its stop control in the UI, and the privacy switch in settings.
capture.rsusesPrintWindowover GDI. That does not capture a hardware-accelerated game surface, so game companionship needs a different capture path (Windows.Graphics.Capture) before it can work on the case it exists for. Recorded rather than worked around: a black frame delivered to the model is worse than an error.- Voice input needs microphone permission from WebView2. Tauri registers a
PermissionRequestedhandler only whenclipboardis enabled, which this application does not enable, so the default behaviour decides the outcome and has to be established on a real build rather than assumed. - Do not implement mouse or keyboard control, process injection, or reading a game's memory. Detect
the foreground window through the existing bounded inspection only, per
implementation-queue.md:118.