diff --git a/docs/decisions/ADR-0008-screen-awareness-and-voice-boundary.md b/docs/decisions/ADR-0008-screen-awareness-and-voice-boundary.md new file mode 100644 index 0000000..6d8e462 --- /dev/null +++ b/docs/decisions/ADR-0008-screen-awareness-and-voice-boundary.md @@ -0,0 +1,127 @@ +# ADR-0008: Screen awareness, voice capture, and the game companion boundary + +## Context + +Voice chat, window awareness, and game companionship are in scope for `v0.3.0-alpha.1`, and the Rust +modules for reading the foreground window (`foreground_window.rs`), walking its UI Automation tree +(`uia.rs`), and capturing a selected window (`capture.rs`) are already written. None of them has a +Tauri command, so nothing in the product can reach them, and there is no consent, rate-limit, or +privacy concept anywhere in the repository — not in `policy.rs`, not in `runtime_guards.rs`, not in +the capability files, not in the database schema, not in the UI. + +The authoritative documents were read directly for this decision, because an earlier reading of them +was wrong in a way that would have changed the outcome: + +- `01_Architecture_Design.md:250-261` is titled 「不应放在前端的能力」. The 「原始截图捕获」 at + line 259 is therefore a **frontend** exclusion — screenshot capture must not live in React — and + **not** a product-level ban. An earlier audit cited it as the latter and concluded that screen + reading was out of scope. That was a mis-read. +- What the product affirmatively permits: 「用户授权的单次窗口截图」 (`:739`, among the inputs a + diagnostic may gather) and, for phase 7, 「按键说话;TTS;游戏窗口绑定;时长事件;手动和低频截图」 + (`:1418-1424`). +- What it excludes outright: 「实时全双工语音」, 「唤醒词」, and 「高频视频分析」 (`:1349-1351`). + `AGENTS.md:32` excludes 「无限制桌面控制」. +- The voice flow is fixed at `:1057-1063` as 按键录音 → 上传音频 → STT → 文本回复 → 可选 TTS, with + the microphone bound to the focused instance by default (`:1065`) and one recording never sent to + both slots unless the user explicitly asks (`:1071`). +- The slot isolation rule is one sentence at `:1077`: 「运维 Slot 不自动接收游戏截图、窗口状态或游戏 + 事件。」 +- `implementation-queue.md:129` already states the gate: 「游戏模式接入截图/状态摘要前,先完成授权 + 提示、频率限制和隐私开关。」 + +So the policy does not have to be invented. It has to be written down precisely enough to implement, +and one thing genuinely is missing from every document: whether the user must be able to see that +the application is reading their screen. + +## Decision + +1. **Screen and window reading happens only inside an explicit capture session.** A session is + started by a user action, names exactly one target window, and carries an expiry. No command reads + a window, its UI Automation tree, or its pixels outside a live session. There is no ambient or + background reading, and no path that reads the desktop or a monitor rather than a window. + +2. **The session is the authorization record, and it lives in Rust.** The frontend may ask to start + one and may end one; it cannot name a window that no session covers. `WindowSelection` currently + exists only inside `inspect_foreground_window` and is dropped when that command returns, so a + registry in Tauri state is the prerequisite for any of this, not an optimization. + +3. **Every read is rate limited by the session, in Rust.** The limit is a minimum interval between + reads of the same kind, enforced where the read happens rather than in the caller. Frontend + throttling does not count: `implementation-queue.md:129` requires a frequency limit before + screenshots are wired at all, and a limit the renderer can skip is not one. The default interval + for pixel capture is deliberately far above a video frame rate, since 「高频视频分析」 is out of + scope — this is 「手动和低频截图」. + +4. **A privacy switch disables the whole capability, and it wins over any session.** When it is off, + the commands fail rather than returning stale or partial data, and no session can be started. + +5. **The user can see when a session is live, and can end it from there.** Nothing in the documents + requires this, so it is proposed here: reading someone's screen without a visible sign is the kind + of behaviour that is indistinguishable from spyware from the outside, and a kill switch the user + has to go looking for is not a kill switch. The indicator is part of the feature, not a later + polish item. + +6. **Captured frames and window text are never written to disk by this feature.** They are held in + memory, sent to the bound slot, and dropped. `01_Architecture_Design.md:1487` already says a + diagnostic keeps no raw screenshot unless the user saves it; this extends the same rule to the + companion path, where there is no save affordance at all. + +7. **Screen and window data go to the companion slot only.** The operations slot never receives a + capture, a window summary, or a game event as a consequence of a session — only as a consequence + of the user explicitly sending one. This is `:1077` restated as an invariant that must be + enforced in Rust and covered by a test that fails if the guard is removed. + +8. **Voice follows the documented flow, server side.** Press to record, upload the audio, let AstrBot + transcribe it, receive text, optionally receive speech back. The client does not run speech + recognition of its own: `WebView2` provides no `SpeechRecognition`, and the current + `ChatPanel.tsx` call to it can never have worked. Audio is uploaded through the same + `/api/v1/files` endpoint as any other attachment and referenced as a `record` segment inside + `message`, which is the contract AstrBot actually reads. + +9. **No always-listening capability.** No wake word, no voice activity detection that starts a + recording on its own, no continuous duplex stream. Recording starts and stops on a user action. + This is `:1349-1350`. + +10. **Speech-to-text requires a provider the user configures, and this project does not choose one + for them.** Every AstrBot STT provider needs either an API key or a self-hosted model. Enabling + one sends the user's voice to whichever service they picked, so the credential and the choice are + theirs. The client's job is to make the path work once they have configured it, to report clearly + when they have not, and never to enrol a provider on their behalf. + +## Rationale + +- The gate in `implementation-queue.md:129` is a precondition, not a checklist item to close later. + Wiring the capture commands first and adding consent afterwards would mean shipping a build in + which any renderer bug can read the user's screen. +- Putting the session, the rate limit, and the switch in Rust follows the boundary + `01_Architecture_Design.md:250-261` already draws. It also means the guarantee does not depend on + the frontend being correct, which is where every silent failure in this codebase so far has lived. +- A single-window target rather than a monitor keeps the feature inside 「用户授权的单次窗口截图」 + and makes the authorization meaningful: the user is agreeing to one specific window, not to their + desktop. +- Binding to the companion slot by default and forbidding automatic delivery to operations keeps the + two conversations from leaking into each other, which is the whole point of the dual-slot design. +- Leaving the STT provider to the user is not caution for its own sake. It is the one part of this + feature that sends their voice somewhere; picking a default would be making that decision for + them. + +## Consequences + +- Add a `capture_policy` module in Rust owning the switch, the session registry, and the rate limit, + with the session expiry and the interval as pure functions so they can be tested without Windows. +- Register commands for the UI Automation summary and the window capture, each taking a session + identifier and failing when no live session covers it. `WindowSelection::raw_handle` becomes + reachable through the registry rather than being minted and dropped inside one command. +- Add the capability entries for the new commands, and a test that reads the shipped capability files + — a missing entry fails silently at runtime, which is how F-03 survived. +- Surface the live session and its stop control in the UI, and the privacy switch in settings. +- `capture.rs` uses `PrintWindow` over GDI. That does not capture a hardware-accelerated game + surface, so game companionship needs a different capture path (`Windows.Graphics.Capture`) before + it can work on the case it exists for. Recorded rather than worked around: a black frame delivered + to the model is worse than an error. +- Voice input needs microphone permission from WebView2. Tauri registers a `PermissionRequested` + handler only when `clipboard` is enabled, which this application does not enable, so the default + behaviour decides the outcome and has to be established on a real build rather than assumed. +- Do not implement mouse or keyboard control, process injection, or reading a game's memory. Detect + the foreground window through the existing bounded inspection only, per + `implementation-queue.md:118`. diff --git a/src-tauri/src/capture_policy.rs b/src-tauri/src/capture_policy.rs new file mode 100644 index 0000000..3ca7b03 --- /dev/null +++ b/src-tauri/src/capture_policy.rs @@ -0,0 +1,427 @@ +//! Authorization for reading the user's screen, per ADR-0008. +//! +//! `implementation-queue.md:129` requires an authorization prompt, a frequency +//! limit, and a privacy switch to exist *before* screenshots or window summaries +//! are wired to anything. This module is those three, and it is deliberately the +//! only way to reach a window handle: `WindowSelection` used to be minted inside +//! `inspect_foreground_window` and dropped when the command returned, so a +//! second IPC call had no way to name a window the user had authorized — and no +//! way to be told it could not. +//! +//! Everything that decides whether a read is allowed lives here rather than in +//! the caller. A limit the renderer can skip is not a limit. + +use std::{ + collections::HashMap, + fmt, + sync::Mutex, + time::{Duration, Instant}, +}; + +use serde::{Deserialize, Serialize}; +use uuid::Uuid; + +use crate::windows::foreground_window::{ForegroundWindowError, WindowSelection}; + +/// Minimum gap between two pixel captures in one session. +/// +/// Two seconds, not two frames. 「高频视频分析」 is out of scope +/// (`01_Architecture_Design.md:1351`) and what phase 7 asks for is 「手动和低频 +/// 截图」 (`:1424`), so the default sits three orders of magnitude away from a +/// video frame rate. +pub const DEFAULT_PIXEL_INTERVAL: Duration = Duration::from_secs(2); + +/// Floor on the configurable interval, so no caller can turn the limit off. +pub const MINIMUM_PIXEL_INTERVAL: Duration = Duration::from_millis(500); + +/// Window text is cheaper than pixels and changes faster, but still not a stream. +pub const DEFAULT_TEXT_INTERVAL: Duration = Duration::from_millis(750); + +/// Most sessions a user can have open at once. +/// +/// One window per session and no reason to watch several at a time; the bound +/// exists so a renderer bug cannot accumulate authorizations. +pub const MAX_ACTIVE_SESSIONS: usize = 4; + +/// What a session is being asked to read. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub enum ReadKind { + /// The window's pixels. + Pixels, + /// The window's title and UI Automation text. + Text, +} + +impl ReadKind { + fn default_interval(self) -> Duration { + match self { + ReadKind::Pixels => DEFAULT_PIXEL_INTERVAL, + ReadKind::Text => DEFAULT_TEXT_INTERVAL, + } + } +} + +#[derive(Debug, PartialEq, Eq)] +pub enum CapturePolicyError { + /// The privacy switch is off. Checked before anything else, so turning it + /// off ends every session rather than leaving live ones running. + Disabled, + /// No live session carries this token. + UnknownSession, + /// The session's authorization has run out, or the window is gone. + Selection(ForegroundWindowError), + /// This read came too soon after the last one of its kind. + RateLimited { retry_after: Duration }, + /// Screen data may not be bound to the operations slot. + /// + /// `01_Architecture_Design.md:1077`: 「运维 Slot 不自动接收游戏截图、窗口状态 + /// 或游戏事件。」 A session is exactly that automatic delivery, so the slot is + /// refused at the point the session is created rather than at each read. + SlotNotPermitted, + /// Too many sessions are already open. + TooManySessions, +} + +impl fmt::Display for CapturePolicyError { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::Disabled => write!(formatter, "屏幕读取已在设置中关闭"), + Self::UnknownSession => write!(formatter, "没有对应的屏幕读取会话,请重新授权"), + Self::Selection(error) => write!(formatter, "窗口授权已失效:{error}"), + Self::RateLimited { retry_after } => write!( + formatter, + "读取过于频繁,请在 {} 毫秒后重试", + retry_after.as_millis() + ), + Self::SlotNotPermitted => { + write!(formatter, "屏幕内容只能发送给陪伴槽位") + } + Self::TooManySessions => write!( + formatter, + "同时开启的屏幕读取会话不能超过 {MAX_ACTIVE_SESSIONS} 个" + ), + } + } +} + +impl std::error::Error for CapturePolicyError {} + +/// Enforces a minimum gap between reads of one kind. +/// +/// `now` is a parameter rather than read inside, so the limit is testable +/// without sleeping and cannot drift from whatever clock the caller used to +/// decide the read was due. +#[derive(Debug, Clone, Copy)] +pub struct ReadBudget { + interval: Duration, + last: Option, +} + +impl ReadBudget { + pub fn new(interval: Duration) -> Self { + Self { + interval: interval.max(MINIMUM_PIXEL_INTERVAL), + last: None, + } + } + + /// Records a read at `now`, or reports how long the caller must wait. + /// + /// The timestamp is only advanced on success, so a rejected read cannot push + /// the next permitted one further away. + pub fn take(&mut self, now: Instant) -> Result<(), Duration> { + if let Some(last) = self.last { + let elapsed = now.saturating_duration_since(last); + if elapsed < self.interval { + return Err(self.interval - elapsed); + } + } + self.last = Some(now); + Ok(()) + } +} + +/// One window the user has authorized this application to read. +#[derive(Debug)] +struct CaptureSession { + selection: WindowSelection, + slot: desktop_protocol::ActiveSlotKind, + budgets: HashMap, +} + +/// What the UI shows while a session is live, so the user can see it and stop it. +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct CaptureSessionSummary { + pub session_token: Uuid, + pub window_title: String, + pub process_id: u32, + pub slot: desktop_protocol::ActiveSlotKind, + pub expired: bool, +} + +/// Whether screen data may be bound to `slot`. +/// +/// A free function so the rule can be asserted without a live window, and so +/// there is one place to read rather than a condition inlined at each call site. +pub fn slot_permits_capture( + slot: desktop_protocol::ActiveSlotKind, +) -> Result<(), CapturePolicyError> { + match slot { + desktop_protocol::ActiveSlotKind::Companion => Ok(()), + desktop_protocol::ActiveSlotKind::Operations => Err(CapturePolicyError::SlotNotPermitted), + } +} + +#[derive(Debug, Default)] +struct PolicyState { + /// The privacy switch. `false` until the user turns it on: an authorization + /// the user never gave is not an authorization. + enabled: bool, + sessions: HashMap, +} + +/// The Tauri-managed owner of the switch, the sessions, and the rate limits. +#[derive(Debug, Default)] +pub struct CapturePolicy { + state: Mutex, +} + +impl CapturePolicy { + fn with_state(&self, body: impl FnOnce(&mut PolicyState) -> T) -> T { + match self.state.lock() { + Ok(mut guard) => body(&mut guard), + Err(poisoned) => body(&mut poisoned.into_inner()), + } + } + + pub fn is_enabled(&self) -> bool { + self.with_state(|state| state.enabled) + } + + /// Turns the capability on or off. + /// + /// Turning it off drops every live session, so a switch flipped mid-capture + /// stops the next read rather than only preventing new authorizations. + pub fn set_enabled(&self, enabled: bool) { + self.with_state(|state| { + state.enabled = enabled; + if !enabled { + state.sessions.clear(); + } + }); + } + + /// Authorizes reading one window until its selection expires. + pub fn begin_session( + &self, + selection: WindowSelection, + slot: desktop_protocol::ActiveSlotKind, + ) -> Result { + slot_permits_capture(slot)?; + selection + .validate() + .map_err(CapturePolicyError::Selection)?; + self.with_state(|state| { + if !state.enabled { + return Err(CapturePolicyError::Disabled); + } + // Expired sessions are cleared first, so a user who let one lapse is + // not told they have too many. + state + .sessions + .retain(|_, session| !session.selection.is_expired()); + if state.sessions.len() >= MAX_ACTIVE_SESSIONS { + return Err(CapturePolicyError::TooManySessions); + } + let token = Uuid::new_v4(); + state.sessions.insert( + token, + CaptureSession { + selection, + slot, + budgets: HashMap::new(), + }, + ); + Ok(token) + }) + } + + pub fn end_session(&self, token: Uuid) { + self.with_state(|state| { + state.sessions.remove(&token); + }); + } + + /// Every live session, for the indicator the user stops a capture from. + pub fn active_sessions(&self) -> Vec { + self.with_state(|state| { + state + .sessions + .retain(|_, session| !session.selection.is_expired()); + let mut summaries: Vec<_> = state + .sessions + .iter() + .map(|(token, session)| { + let summary = session.selection.summary(); + CaptureSessionSummary { + session_token: *token, + window_title: summary.title, + process_id: summary.process_id, + slot: session.slot, + expired: session.selection.is_expired(), + } + }) + .collect(); + // Stable order so the indicator does not reshuffle between polls. + summaries.sort_by_key(|summary| summary.session_token); + summaries + }) + } + + /// Checks a read is allowed, and runs it against the authorized window. + /// + /// The read is a closure so the handle never leaves this module: every gate + /// — switch, session, expiry, rate limit — is passed before the caller sees + /// a `WindowSelection`, and there is no accessor that skips them. + pub fn read( + &self, + token: Uuid, + kind: ReadKind, + now: Instant, + body: impl FnOnce(&WindowSelection) -> T, + ) -> Result { + self.with_state(|state| { + if !state.enabled { + return Err(CapturePolicyError::Disabled); + } + let session = state + .sessions + .get_mut(&token) + .ok_or(CapturePolicyError::UnknownSession)?; + session + .selection + .validate() + .map_err(CapturePolicyError::Selection)?; + session + .budgets + .entry(kind) + .or_insert_with(|| ReadBudget::new(kind.default_interval())) + .take(now) + .map_err(|retry_after| CapturePolicyError::RateLimited { retry_after })?; + Ok(body(&session.selection)) + }) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn the_capability_is_off_until_the_user_turns_it_on() { + let policy = CapturePolicy::default(); + assert!(!policy.is_enabled()); + assert!(policy.active_sessions().is_empty()); + } + + #[test] + fn turning_the_switch_off_ends_every_live_session() { + let policy = CapturePolicy::default(); + policy.set_enabled(true); + // Sessions are inserted through begin_session, which needs a real window + // handle, so this exercises the switch through the state it owns. + assert!(policy.is_enabled()); + policy.set_enabled(false); + assert!(!policy.is_enabled()); + assert!(policy.active_sessions().is_empty()); + } + + /// The operations slot never receives screen data as a consequence of a + /// session (`01_Architecture_Design.md:1077`). Refused at creation, so no + /// read path has to remember. + #[test] + fn a_session_cannot_be_bound_to_the_operations_slot() { + let policy = CapturePolicy::default(); + policy.set_enabled(true); + // Checked before the window handle is touched — the slot is wrong + // regardless of whether the window is still there — which is also what + // lets the rule be asserted without one. + assert_eq!( + slot_permits_capture(desktop_protocol::ActiveSlotKind::Operations), + Err(CapturePolicyError::SlotNotPermitted) + ); + assert_eq!( + slot_permits_capture(desktop_protocol::ActiveSlotKind::Companion), + Ok(()) + ); + let _ = &policy; + } + + #[test] + fn a_read_is_refused_while_the_switch_is_off() { + let policy = CapturePolicy::default(); + let error = policy + .read(Uuid::new_v4(), ReadKind::Pixels, Instant::now(), |_| ()) + .expect_err("must be refused"); + assert_eq!(error, CapturePolicyError::Disabled); + } + + #[test] + fn a_read_is_refused_when_no_session_names_the_window() { + let policy = CapturePolicy::default(); + policy.set_enabled(true); + let error = policy + .read(Uuid::new_v4(), ReadKind::Pixels, Instant::now(), |_| ()) + .expect_err("must be refused"); + assert_eq!(error, CapturePolicyError::UnknownSession); + } + + #[test] + fn a_budget_refuses_a_second_read_inside_its_interval() { + let start = Instant::now(); + let mut budget = ReadBudget::new(Duration::from_secs(2)); + assert!(budget.take(start).is_ok()); + + let retry_after = budget + .take(start + Duration::from_millis(500)) + .expect_err("must be limited"); + assert_eq!(retry_after, Duration::from_millis(1500)); + + assert!(budget.take(start + Duration::from_secs(2)).is_ok()); + } + + /// A refused read must not push the next permitted one further away, or a + /// caller that polls faster than the limit is locked out for good. + #[test] + fn a_refused_read_does_not_extend_the_wait() { + let start = Instant::now(); + let mut budget = ReadBudget::new(Duration::from_secs(2)); + budget.take(start).expect("first read"); + for offset in [200, 400, 600, 800] { + assert!(budget.take(start + Duration::from_millis(offset)).is_err()); + } + assert!(budget.take(start + Duration::from_secs(2)).is_ok()); + } + + /// The interval cannot be configured below the floor, so no caller can turn + /// the frequency limit off by asking for zero. + #[test] + fn the_interval_cannot_be_set_below_the_floor() { + let start = Instant::now(); + let mut budget = ReadBudget::new(Duration::ZERO); + budget.take(start).expect("first read"); + assert!(budget.take(start).is_err()); + assert!(budget.take(start + MINIMUM_PIXEL_INTERVAL).is_ok()); + } + + /// Pixels are limited far more strictly than a video frame rate, because + /// 「高频视频分析」 is out of scope. + #[test] + fn the_pixel_interval_is_nowhere_near_a_frame_rate() { + assert!(DEFAULT_PIXEL_INTERVAL >= Duration::from_secs(1)); + assert!(MINIMUM_PIXEL_INTERVAL >= Duration::from_millis(500)); + assert!(DEFAULT_TEXT_INTERVAL >= Duration::from_millis(500)); + } +} diff --git a/src-tauri/src/lib.rs b/src-tauri/src/lib.rs index 0761a1d..4865105 100644 --- a/src-tauri/src/lib.rs +++ b/src-tauri/src/lib.rs @@ -1,4 +1,5 @@ pub mod auth; +pub mod capture_policy; pub mod credentials; pub mod database; pub mod diagnostics;