128 行
8.9 KiB
Markdown
128 行
8.9 KiB
Markdown
|
|
# ADR-0008: Screen awareness, voice capture, and the game companion boundary
|
||
|
|
|
||
|
|
## Context
|
||
|
|
|
||
|
|
Voice chat, window awareness, and game companionship are in scope for `v0.3.0-alpha.1`, and the Rust
|
||
|
|
modules for reading the foreground window (`foreground_window.rs`), walking its UI Automation tree
|
||
|
|
(`uia.rs`), and capturing a selected window (`capture.rs`) are already written. None of them has a
|
||
|
|
Tauri command, so nothing in the product can reach them, and there is no consent, rate-limit, or
|
||
|
|
privacy concept anywhere in the repository — not in `policy.rs`, not in `runtime_guards.rs`, not in
|
||
|
|
the capability files, not in the database schema, not in the UI.
|
||
|
|
|
||
|
|
The authoritative documents were read directly for this decision, because an earlier reading of them
|
||
|
|
was wrong in a way that would have changed the outcome:
|
||
|
|
|
||
|
|
- `01_Architecture_Design.md:250-261` is titled 「不应放在前端的能力」. The 「原始截图捕获」 at
|
||
|
|
line 259 is therefore a **frontend** exclusion — screenshot capture must not live in React — and
|
||
|
|
**not** a product-level ban. An earlier audit cited it as the latter and concluded that screen
|
||
|
|
reading was out of scope. That was a mis-read.
|
||
|
|
- What the product affirmatively permits: 「用户授权的单次窗口截图」 (`:739`, among the inputs a
|
||
|
|
diagnostic may gather) and, for phase 7, 「按键说话;TTS;游戏窗口绑定;时长事件;手动和低频截图」
|
||
|
|
(`:1418-1424`).
|
||
|
|
- What it excludes outright: 「实时全双工语音」, 「唤醒词」, and 「高频视频分析」 (`:1349-1351`).
|
||
|
|
`AGENTS.md:32` excludes 「无限制桌面控制」.
|
||
|
|
- The voice flow is fixed at `:1057-1063` as 按键录音 → 上传音频 → STT → 文本回复 → 可选 TTS, with
|
||
|
|
the microphone bound to the focused instance by default (`:1065`) and one recording never sent to
|
||
|
|
both slots unless the user explicitly asks (`:1071`).
|
||
|
|
- The slot isolation rule is one sentence at `:1077`: 「运维 Slot 不自动接收游戏截图、窗口状态或游戏
|
||
|
|
事件。」
|
||
|
|
- `implementation-queue.md:129` already states the gate: 「游戏模式接入截图/状态摘要前,先完成授权
|
||
|
|
提示、频率限制和隐私开关。」
|
||
|
|
|
||
|
|
So the policy does not have to be invented. It has to be written down precisely enough to implement,
|
||
|
|
and one thing genuinely is missing from every document: whether the user must be able to see that
|
||
|
|
the application is reading their screen.
|
||
|
|
|
||
|
|
## Decision
|
||
|
|
|
||
|
|
1. **Screen and window reading happens only inside an explicit capture session.** A session is
|
||
|
|
started by a user action, names exactly one target window, and carries an expiry. No command reads
|
||
|
|
a window, its UI Automation tree, or its pixels outside a live session. There is no ambient or
|
||
|
|
background reading, and no path that reads the desktop or a monitor rather than a window.
|
||
|
|
|
||
|
|
2. **The session is the authorization record, and it lives in Rust.** The frontend may ask to start
|
||
|
|
one and may end one; it cannot name a window that no session covers. `WindowSelection` currently
|
||
|
|
exists only inside `inspect_foreground_window` and is dropped when that command returns, so a
|
||
|
|
registry in Tauri state is the prerequisite for any of this, not an optimization.
|
||
|
|
|
||
|
|
3. **Every read is rate limited by the session, in Rust.** The limit is a minimum interval between
|
||
|
|
reads of the same kind, enforced where the read happens rather than in the caller. Frontend
|
||
|
|
throttling does not count: `implementation-queue.md:129` requires a frequency limit before
|
||
|
|
screenshots are wired at all, and a limit the renderer can skip is not one. The default interval
|
||
|
|
for pixel capture is deliberately far above a video frame rate, since 「高频视频分析」 is out of
|
||
|
|
scope — this is 「手动和低频截图」.
|
||
|
|
|
||
|
|
4. **A privacy switch disables the whole capability, and it wins over any session.** When it is off,
|
||
|
|
the commands fail rather than returning stale or partial data, and no session can be started.
|
||
|
|
|
||
|
|
5. **The user can see when a session is live, and can end it from there.** Nothing in the documents
|
||
|
|
requires this, so it is proposed here: reading someone's screen without a visible sign is the kind
|
||
|
|
of behaviour that is indistinguishable from spyware from the outside, and a kill switch the user
|
||
|
|
has to go looking for is not a kill switch. The indicator is part of the feature, not a later
|
||
|
|
polish item.
|
||
|
|
|
||
|
|
6. **Captured frames and window text are never written to disk by this feature.** They are held in
|
||
|
|
memory, sent to the bound slot, and dropped. `01_Architecture_Design.md:1487` already says a
|
||
|
|
diagnostic keeps no raw screenshot unless the user saves it; this extends the same rule to the
|
||
|
|
companion path, where there is no save affordance at all.
|
||
|
|
|
||
|
|
7. **Screen and window data go to the companion slot only.** The operations slot never receives a
|
||
|
|
capture, a window summary, or a game event as a consequence of a session — only as a consequence
|
||
|
|
of the user explicitly sending one. This is `:1077` restated as an invariant that must be
|
||
|
|
enforced in Rust and covered by a test that fails if the guard is removed.
|
||
|
|
|
||
|
|
8. **Voice follows the documented flow, server side.** Press to record, upload the audio, let AstrBot
|
||
|
|
transcribe it, receive text, optionally receive speech back. The client does not run speech
|
||
|
|
recognition of its own: `WebView2` provides no `SpeechRecognition`, and the current
|
||
|
|
`ChatPanel.tsx` call to it can never have worked. Audio is uploaded through the same
|
||
|
|
`/api/v1/files` endpoint as any other attachment and referenced as a `record` segment inside
|
||
|
|
`message`, which is the contract AstrBot actually reads.
|
||
|
|
|
||
|
|
9. **No always-listening capability.** No wake word, no voice activity detection that starts a
|
||
|
|
recording on its own, no continuous duplex stream. Recording starts and stops on a user action.
|
||
|
|
This is `:1349-1350`.
|
||
|
|
|
||
|
|
10. **Speech-to-text requires a provider the user configures, and this project does not choose one
|
||
|
|
for them.** Every AstrBot STT provider needs either an API key or a self-hosted model. Enabling
|
||
|
|
one sends the user's voice to whichever service they picked, so the credential and the choice are
|
||
|
|
theirs. The client's job is to make the path work once they have configured it, to report clearly
|
||
|
|
when they have not, and never to enrol a provider on their behalf.
|
||
|
|
|
||
|
|
## Rationale
|
||
|
|
|
||
|
|
- The gate in `implementation-queue.md:129` is a precondition, not a checklist item to close later.
|
||
|
|
Wiring the capture commands first and adding consent afterwards would mean shipping a build in
|
||
|
|
which any renderer bug can read the user's screen.
|
||
|
|
- Putting the session, the rate limit, and the switch in Rust follows the boundary
|
||
|
|
`01_Architecture_Design.md:250-261` already draws. It also means the guarantee does not depend on
|
||
|
|
the frontend being correct, which is where every silent failure in this codebase so far has lived.
|
||
|
|
- A single-window target rather than a monitor keeps the feature inside 「用户授权的单次窗口截图」
|
||
|
|
and makes the authorization meaningful: the user is agreeing to one specific window, not to their
|
||
|
|
desktop.
|
||
|
|
- Binding to the companion slot by default and forbidding automatic delivery to operations keeps the
|
||
|
|
two conversations from leaking into each other, which is the whole point of the dual-slot design.
|
||
|
|
- Leaving the STT provider to the user is not caution for its own sake. It is the one part of this
|
||
|
|
feature that sends their voice somewhere; picking a default would be making that decision for
|
||
|
|
them.
|
||
|
|
|
||
|
|
## Consequences
|
||
|
|
|
||
|
|
- Add a `capture_policy` module in Rust owning the switch, the session registry, and the rate limit,
|
||
|
|
with the session expiry and the interval as pure functions so they can be tested without Windows.
|
||
|
|
- Register commands for the UI Automation summary and the window capture, each taking a session
|
||
|
|
identifier and failing when no live session covers it. `WindowSelection::raw_handle` becomes
|
||
|
|
reachable through the registry rather than being minted and dropped inside one command.
|
||
|
|
- Add the capability entries for the new commands, and a test that reads the shipped capability files
|
||
|
|
— a missing entry fails silently at runtime, which is how F-03 survived.
|
||
|
|
- Surface the live session and its stop control in the UI, and the privacy switch in settings.
|
||
|
|
- `capture.rs` uses `PrintWindow` over GDI. That does not capture a hardware-accelerated game
|
||
|
|
surface, so game companionship needs a different capture path (`Windows.Graphics.Capture`) before
|
||
|
|
it can work on the case it exists for. Recorded rather than worked around: a black frame delivered
|
||
|
|
to the model is worse than an error.
|
||
|
|
- Voice input needs microphone permission from WebView2. Tauri registers a `PermissionRequested`
|
||
|
|
handler only when `clipboard` is enabled, which this application does not enable, so the default
|
||
|
|
behaviour decides the outcome and has to be established on a real build rather than assumed.
|
||
|
|
- Do not implement mouse or keyboard control, process injection, or reading a game's memory. Detect
|
||
|
|
the foreground window through the existing bounded inspection only, per
|
||
|
|
`implementation-queue.md:118`.
|