feat(awareness): let the companion read one authorized window
Wires the window-awareness modules to the product for the first time. uia.rs and capture.rs were fully written and had no Tauri command at all; their only caller anywhere was an #[ignore]d smoke test, so nothing in a shipped build could reach them. Six commands, all behind the ADR-0008 policy: the privacy switch, the session list, start and stop, a window frame, and the window's UI Automation text. The read takes a closure so the window handle never leaves the policy module — there is no accessor that skips the switch, the session lookup, the expiry check, or the rate limit. Starting a session has an interaction problem worth naming: reading the foreground window is the only selection mechanism these modules have, and at the moment the user clicks the button, the foreground window is this application. So the button gives them a few bounded seconds to bring the target forward, and a window belonging to this process is refused rather than captured — which is the most likely outcome if the delay elapses while they are still on the settings page. The panel sends either the window's text or its picture to the companion slot. Text first in the UI order, deliberately: it is far less revealing than a screenshot and usually enough to answer "what am I looking at". The indicator in the shell is ADR-0008 item 5, which no project document required. Reading someone's screen with no visible sign is indistinguishable from spyware from the outside, and a stop control the user has to go looking for is not a stop control. It polls rather than waiting to be told, because an authorization can lapse on its own — and when it cannot read the status it renders nothing, claiming neither that a capture is live nor that one is not. The app's own commands need no capability entries: only plugin commands go through the ACL, which the existing commands working under `core:default` alone already demonstrates. Rust tests 110 -> 113, frontend 43 -> 47. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
这个提交存在于:
父节点
7aead6fa48
当前提交
b6a21a70ec
共有 9 个文件被更改,包括 804 次插入 和 0 次删除
|
|
@ -81,6 +81,8 @@ pub enum CapturePolicyError {
|
|||
SlotNotPermitted,
|
||||
/// Too many sessions are already open.
|
||||
TooManySessions,
|
||||
/// The window in front belongs to this application.
|
||||
OwnWindow,
|
||||
}
|
||||
|
||||
impl fmt::Display for CapturePolicyError {
|
||||
|
|
@ -101,6 +103,7 @@ impl fmt::Display for CapturePolicyError {
|
|||
formatter,
|
||||
"同时开启的屏幕读取会话不能超过 {MAX_ACTIVE_SESSIONS} 个"
|
||||
),
|
||||
Self::OwnWindow => write!(formatter, "捕获到的是本应用的窗口,请先切换到要共享的窗口"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -161,6 +164,40 @@ pub struct CaptureSessionSummary {
|
|||
pub expired: bool,
|
||||
}
|
||||
|
||||
/// Longest a single authorization lasts.
|
||||
///
|
||||
/// `WindowSelection` caps its own TTL at five minutes, and this takes that cap
|
||||
/// rather than inventing a shorter one. A companion session that outlives it has
|
||||
/// to be renewed by the user, which is another consent event — the alternative
|
||||
/// is an authorization with no end, which is what the single-window rule exists
|
||||
/// to avoid.
|
||||
pub const SESSION_TTL: Duration = Duration::from_secs(5 * 60);
|
||||
|
||||
/// Longest the user may be given to bring the target window forward.
|
||||
pub const MAX_FOCUS_DELAY: Duration = Duration::from_secs(10);
|
||||
|
||||
/// Clamps a requested focus delay.
|
||||
///
|
||||
/// Starting a session reads whichever window is in front, and at the moment the
|
||||
/// user clicks the button that is this application. So the caller is given a few
|
||||
/// seconds to bring the target forward — bounded, because a long delay turns a
|
||||
/// deliberate action into something the user has forgotten they started.
|
||||
pub fn focus_delay(requested_seconds: u64) -> Duration {
|
||||
Duration::from_secs(requested_seconds).min(MAX_FOCUS_DELAY)
|
||||
}
|
||||
|
||||
/// Whether a window belonging to `process_id` may be captured.
|
||||
///
|
||||
/// Refuses this application's own windows. Capturing ourselves is never what the
|
||||
/// user meant, and it is the most likely outcome of a delay that elapsed while
|
||||
/// they were still looking at the settings page.
|
||||
pub fn foreign_process(process_id: u32, own_process_id: u32) -> Result<(), CapturePolicyError> {
|
||||
if process_id == own_process_id {
|
||||
return Err(CapturePolicyError::OwnWindow);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Whether screen data may be bound to `slot`.
|
||||
///
|
||||
/// A free function so the rule can be asserted without a live window, and so
|
||||
|
|
@ -416,6 +453,33 @@ mod tests {
|
|||
assert!(budget.take(start + MINIMUM_PIXEL_INTERVAL).is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_focus_delay_is_bounded() {
|
||||
assert_eq!(focus_delay(3), Duration::from_secs(3));
|
||||
assert_eq!(focus_delay(0), Duration::ZERO);
|
||||
// A renderer asking for an hour gets ten seconds.
|
||||
assert_eq!(focus_delay(3600), MAX_FOCUS_DELAY);
|
||||
assert_eq!(focus_delay(u64::MAX), MAX_FOCUS_DELAY);
|
||||
}
|
||||
|
||||
/// Capturing our own window is never what the user meant, and it is the most
|
||||
/// likely outcome if the delay elapses while they are still on the settings
|
||||
/// page.
|
||||
#[test]
|
||||
fn this_applications_own_window_is_refused() {
|
||||
assert_eq!(
|
||||
foreign_process(4321, 4321),
|
||||
Err(CapturePolicyError::OwnWindow)
|
||||
);
|
||||
assert_eq!(foreign_process(4322, 4321), Ok(()));
|
||||
}
|
||||
|
||||
/// A session never outlives the window authorization it was built from.
|
||||
#[test]
|
||||
fn the_session_lifetime_matches_the_selection_cap() {
|
||||
assert_eq!(SESSION_TTL, Duration::from_secs(5 * 60));
|
||||
}
|
||||
|
||||
/// Pixels are limited far more strictly than a video frame rate, because
|
||||
/// 「高频视频分析」 is out of scope.
|
||||
#[test]
|
||||
|
|
|
|||
|
|
@ -485,6 +485,162 @@ fn pet_window_preferences(app: AppHandle) -> PetWindowPreferences {
|
|||
app.state::<PetPreferencesState>().snapshot()
|
||||
}
|
||||
|
||||
/// What the settings page and the live-capture indicator read.
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
struct ScreenReadingStatus {
|
||||
enabled: bool,
|
||||
sessions: Vec<capture_policy::CaptureSessionSummary>,
|
||||
pixel_interval_ms: u64,
|
||||
text_interval_ms: u64,
|
||||
session_ttl_seconds: u64,
|
||||
}
|
||||
|
||||
fn screen_reading_snapshot(policy: &capture_policy::CapturePolicy) -> ScreenReadingStatus {
|
||||
ScreenReadingStatus {
|
||||
enabled: policy.is_enabled(),
|
||||
sessions: policy.active_sessions(),
|
||||
pixel_interval_ms: capture_policy::DEFAULT_PIXEL_INTERVAL.as_millis() as u64,
|
||||
text_interval_ms: capture_policy::DEFAULT_TEXT_INTERVAL.as_millis() as u64,
|
||||
session_ttl_seconds: capture_policy::SESSION_TTL.as_secs(),
|
||||
}
|
||||
}
|
||||
|
||||
#[tauri::command]
|
||||
fn screen_reading_status(app: AppHandle) -> ScreenReadingStatus {
|
||||
screen_reading_snapshot(&app.state::<capture_policy::CapturePolicy>())
|
||||
}
|
||||
|
||||
/// The privacy switch ADR-0008 requires. Returns the resulting state so the UI
|
||||
/// reflects what Rust decided rather than what it asked for.
|
||||
#[tauri::command]
|
||||
fn set_screen_reading_enabled(app: AppHandle, enabled: bool) -> ScreenReadingStatus {
|
||||
let policy = app.state::<capture_policy::CapturePolicy>();
|
||||
policy.set_enabled(enabled);
|
||||
screen_reading_snapshot(&policy)
|
||||
}
|
||||
|
||||
/// Authorizes reading one window, after giving the user time to bring it forward.
|
||||
///
|
||||
/// Reading the foreground window is the only selection mechanism these modules
|
||||
/// have, and at the moment the user clicks, the foreground window is this
|
||||
/// application. So the delay is the interaction: "switch to the window you want
|
||||
/// to share". A window belonging to this process is refused rather than
|
||||
/// captured.
|
||||
#[tauri::command]
|
||||
async fn begin_window_capture_session(
|
||||
app: AppHandle,
|
||||
slot: desktop_protocol::ActiveSlotKind,
|
||||
focus_delay_seconds: u64,
|
||||
) -> Result<capture_policy::CaptureSessionSummary, String> {
|
||||
// Checked before the wait, so a caller that names the wrong slot is told
|
||||
// immediately instead of after ten seconds.
|
||||
capture_policy::slot_permits_capture(slot).map_err(|error| error.to_string())?;
|
||||
tokio::time::sleep(capture_policy::focus_delay(focus_delay_seconds)).await;
|
||||
|
||||
let selection =
|
||||
windows::foreground_window::inspect_foreground_window(capture_policy::SESSION_TTL)
|
||||
.map_err(|error| error.to_string())?;
|
||||
capture_policy::foreign_process(selection.summary().process_id, std::process::id())
|
||||
.map_err(|error| error.to_string())?;
|
||||
|
||||
let policy = app.state::<capture_policy::CapturePolicy>();
|
||||
let token = policy
|
||||
.begin_session(selection, slot)
|
||||
.map_err(|error| error.to_string())?;
|
||||
policy
|
||||
.active_sessions()
|
||||
.into_iter()
|
||||
.find(|session| session.session_token == token)
|
||||
.ok_or_else(|| "capture session expired immediately".to_owned())
|
||||
}
|
||||
|
||||
#[tauri::command]
|
||||
fn end_window_capture_session(app: AppHandle, session_token: uuid::Uuid) -> ScreenReadingStatus {
|
||||
let policy = app.state::<capture_policy::CapturePolicy>();
|
||||
policy.end_session(session_token);
|
||||
screen_reading_snapshot(&policy)
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
struct WindowFrame {
|
||||
mime_type: &'static str,
|
||||
width: u32,
|
||||
height: u32,
|
||||
bytes: Vec<u8>,
|
||||
}
|
||||
|
||||
/// One frame of the authorized window.
|
||||
///
|
||||
/// Held in memory and handed to the caller; nothing writes it to disk, per
|
||||
/// ADR-0008 and `01_Architecture_Design.md:1487`.
|
||||
#[tauri::command]
|
||||
fn capture_window_frame(app: AppHandle, session_token: uuid::Uuid) -> Result<WindowFrame, String> {
|
||||
app.state::<capture_policy::CapturePolicy>()
|
||||
.read(
|
||||
session_token,
|
||||
capture_policy::ReadKind::Pixels,
|
||||
std::time::Instant::now(),
|
||||
|selection| {
|
||||
windows::capture::capture_selected_window(
|
||||
selection,
|
||||
windows::capture::CaptureLimits::default(),
|
||||
)
|
||||
},
|
||||
)
|
||||
.map_err(|error| error.to_string())?
|
||||
.map(|capture| WindowFrame {
|
||||
mime_type: "image/png",
|
||||
width: capture.width,
|
||||
height: capture.height,
|
||||
bytes: capture.png_bytes,
|
||||
})
|
||||
.map_err(|error| error.to_string())
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
struct WindowTextSummary {
|
||||
text: String,
|
||||
visited_nodes: u16,
|
||||
password_nodes_skipped: u16,
|
||||
foreign_process_nodes_skipped: u16,
|
||||
truncated: bool,
|
||||
}
|
||||
|
||||
/// The authorized window's UI Automation text.
|
||||
///
|
||||
/// Cheaper and far less revealing than pixels, and enough for "what are you
|
||||
/// doing" without sending a picture of the screen.
|
||||
#[tauri::command]
|
||||
fn read_window_text(
|
||||
app: AppHandle,
|
||||
session_token: uuid::Uuid,
|
||||
) -> Result<WindowTextSummary, String> {
|
||||
app.state::<capture_policy::CapturePolicy>()
|
||||
.read(
|
||||
session_token,
|
||||
capture_policy::ReadKind::Text,
|
||||
std::time::Instant::now(),
|
||||
|selection| {
|
||||
windows::uia::read_standard_dialog_text(
|
||||
selection,
|
||||
windows::uia::UiaReadLimits::default(),
|
||||
)
|
||||
},
|
||||
)
|
||||
.map_err(|error| error.to_string())?
|
||||
.map(|snapshot| WindowTextSummary {
|
||||
text: snapshot.text,
|
||||
visited_nodes: snapshot.visited_nodes,
|
||||
password_nodes_skipped: snapshot.password_nodes_skipped,
|
||||
foreign_process_nodes_skipped: snapshot.foreign_process_nodes_skipped,
|
||||
truncated: snapshot.truncated,
|
||||
})
|
||||
.map_err(|error| error.to_string())
|
||||
}
|
||||
|
||||
#[tauri::command]
|
||||
fn inspect_foreground_window() -> Result<windows::foreground_window::ForegroundWindowSummary, String>
|
||||
{
|
||||
|
|
@ -677,6 +833,7 @@ pub fn run() {
|
|||
app.manage(SessionMarker(session_marker));
|
||||
app.manage(RuntimeGuards::new());
|
||||
app.manage(PetPreferencesState::default());
|
||||
app.manage(capture_policy::CapturePolicy::default());
|
||||
app.manage(DualSlotRuntime::new());
|
||||
|
||||
let startup_handle = app.handle().clone();
|
||||
|
|
@ -784,6 +941,12 @@ pub fn run() {
|
|||
set_pet_window_preferences,
|
||||
pet_window_preferences,
|
||||
inspect_foreground_window,
|
||||
screen_reading_status,
|
||||
set_screen_reading_enabled,
|
||||
begin_window_capture_session,
|
||||
end_window_capture_session,
|
||||
capture_window_frame,
|
||||
read_window_text,
|
||||
inspect_process,
|
||||
check_network_target,
|
||||
connection_policy_status,
|
||||
|
|
|
|||
|
|
@ -16,6 +16,7 @@ import {
|
|||
petWindowPreferencesSchema,
|
||||
type PetPositionMode,
|
||||
} from "./features/pet/types";
|
||||
import { ScreenAwarenessIndicator } from "./features/awareness/ScreenAwarenessIndicator";
|
||||
import { ChatPanel } from "./features/chat/ChatPanel";
|
||||
import { StatusBubble } from "./features/chat/StatusBubble";
|
||||
const ProfileManager = lazy(() =>
|
||||
|
|
@ -209,6 +210,7 @@ export function App() {
|
|||
}
|
||||
>
|
||||
{page === "chat" ? null : <PageHeader page={page} />}
|
||||
<ScreenAwarenessIndicator />
|
||||
{error ? (
|
||||
<p className="error-banner" role="alert">
|
||||
IPC 暂不可用:{error}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,77 @@
|
|||
import { render, screen, waitFor } from "@testing-library/react";
|
||||
import { invoke } from "@tauri-apps/api/core";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
import { ScreenAwarenessIndicator } from "./ScreenAwarenessIndicator";
|
||||
|
||||
vi.mock("@tauri-apps/api/core", () => ({ invoke: vi.fn() }));
|
||||
|
||||
const status = (sessions: unknown[]) => ({
|
||||
enabled: true,
|
||||
sessions,
|
||||
pixelIntervalMs: 2000,
|
||||
textIntervalMs: 750,
|
||||
sessionTtlSeconds: 300,
|
||||
});
|
||||
|
||||
const session = (title: string) => ({
|
||||
sessionToken: "6f1e0d3a-4b5c-4d6e-8f90-1a2b3c4d5e6f",
|
||||
windowTitle: title,
|
||||
processId: 4242,
|
||||
slot: "companion",
|
||||
expired: false,
|
||||
});
|
||||
|
||||
describe("the screen awareness indicator", () => {
|
||||
beforeEach(() => {
|
||||
vi.mocked(invoke).mockReset();
|
||||
});
|
||||
afterEach(() => {
|
||||
vi.useRealTimers();
|
||||
});
|
||||
|
||||
/// The indicator is a privacy signal, so a silent failure must not be able to
|
||||
/// hide a live capture — but neither may it claim one that is not happening.
|
||||
it("shows nothing while no window is authorized", async () => {
|
||||
vi.mocked(invoke).mockResolvedValue(status([]));
|
||||
const { container } = render(<ScreenAwarenessIndicator />);
|
||||
|
||||
await waitFor(() => expect(invoke).toHaveBeenCalled());
|
||||
expect(container.firstChild).toBeNull();
|
||||
});
|
||||
|
||||
it("names the authorized window and offers a way to stop", async () => {
|
||||
vi.mocked(invoke).mockResolvedValue(status([session("Notepad")]));
|
||||
render(<ScreenAwarenessIndicator />);
|
||||
|
||||
await waitFor(() =>
|
||||
expect(screen.getByText(/Notepad/)).toBeInTheDocument(),
|
||||
);
|
||||
expect(screen.getByRole("button", { name: "停止" })).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it("stops by turning the privacy switch off, which ends every session", async () => {
|
||||
vi.mocked(invoke).mockResolvedValue(status([session("Notepad")]));
|
||||
render(<ScreenAwarenessIndicator />);
|
||||
const stop = await waitFor(() =>
|
||||
screen.getByRole("button", { name: "停止" }),
|
||||
);
|
||||
|
||||
vi.mocked(invoke).mockClear();
|
||||
stop.click();
|
||||
|
||||
await waitFor(() =>
|
||||
expect(invoke).toHaveBeenCalledWith("set_screen_reading_enabled", {
|
||||
enabled: false,
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
it("claims nothing when the status cannot be read", async () => {
|
||||
vi.mocked(invoke).mockRejectedValue(new Error("unavailable"));
|
||||
const { container } = render(<ScreenAwarenessIndicator />);
|
||||
|
||||
await waitFor(() => expect(invoke).toHaveBeenCalled());
|
||||
expect(container.firstChild).toBeNull();
|
||||
});
|
||||
});
|
||||
|
|
@ -0,0 +1,78 @@
|
|||
import { invoke } from "@tauri-apps/api/core";
|
||||
import { useCallback, useEffect, useState } from "react";
|
||||
|
||||
import { screenReadingStatusSchema, type CaptureSession } from "./types";
|
||||
|
||||
/// How often the shell re-checks whether a capture is authorized.
|
||||
///
|
||||
/// The authorization can lapse on its own — sessions expire, and turning the
|
||||
/// privacy switch off drops them — so the indicator has to poll rather than wait
|
||||
/// to be told, or it would keep claiming a capture is live after it has stopped.
|
||||
const POLL_INTERVAL_MS = 2000;
|
||||
|
||||
/**
|
||||
* Always-visible sign that a window is authorized for reading, with a way to
|
||||
* stop it.
|
||||
*
|
||||
* ADR-0008 item 5: nothing in the project's documents required this, and it is
|
||||
* proposed there for the reason it exists here — reading someone's screen with
|
||||
* no visible sign is indistinguishable from spyware from the outside, and a stop
|
||||
* control the user has to go looking for is not a stop control.
|
||||
*/
|
||||
export function ScreenAwarenessIndicator() {
|
||||
const [sessions, setSessions] = useState<CaptureSession[]>([]);
|
||||
|
||||
const refresh = useCallback(async () => {
|
||||
try {
|
||||
const status = screenReadingStatusSchema.parse(
|
||||
await invoke<unknown>("screen_reading_status"),
|
||||
);
|
||||
setSessions(status.sessions);
|
||||
} catch {
|
||||
// A failure here means the command is unavailable, which is not something
|
||||
// to shout about in the shell. Showing nothing is the honest answer: it
|
||||
// does not claim a capture is live, and it does not claim one is not.
|
||||
setSessions([]);
|
||||
}
|
||||
}, []);
|
||||
|
||||
useEffect(() => {
|
||||
let cancelled = false;
|
||||
const tick = () => {
|
||||
if (!cancelled) void refresh();
|
||||
};
|
||||
tick();
|
||||
const timer = window.setInterval(tick, POLL_INTERVAL_MS);
|
||||
return () => {
|
||||
cancelled = true;
|
||||
window.clearInterval(timer);
|
||||
};
|
||||
}, [refresh]);
|
||||
|
||||
const stopAll = async () => {
|
||||
// Turning the switch off is the one action that ends every session at once,
|
||||
// which is what a user reaching for this actually wants.
|
||||
try {
|
||||
await invoke<unknown>("set_screen_reading_enabled", { enabled: false });
|
||||
} finally {
|
||||
void refresh();
|
||||
}
|
||||
};
|
||||
|
||||
if (!sessions.length) return null;
|
||||
|
||||
return (
|
||||
<div className="awareness-indicator" role="status">
|
||||
<span className="awareness-indicator-dot" aria-hidden="true" />
|
||||
<span>
|
||||
正在允许读取
|
||||
{sessions.length > 1
|
||||
? ` ${sessions.length} 个窗口`
|
||||
: `「${sessions[0].windowTitle || "无标题窗口"}」`}
|
||||
</span>
|
||||
<button type="button" onClick={() => void stopAll()}>
|
||||
停止
|
||||
</button>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
|
@ -0,0 +1,271 @@
|
|||
import { invoke } from "@tauri-apps/api/core";
|
||||
import { useCallback, useEffect, useRef, useState } from "react";
|
||||
|
||||
import { errorMessage } from "../../lib/errorMessage";
|
||||
import { astrbotAttachmentSchema } from "../chat/types";
|
||||
import {
|
||||
screenReadingStatusSchema,
|
||||
windowFrameSchema,
|
||||
windowTextSummarySchema,
|
||||
type ScreenReadingStatus,
|
||||
} from "./types";
|
||||
|
||||
/// The window the user is asked to bring forward before the capture starts.
|
||||
const FOCUS_DELAY_SECONDS = 5;
|
||||
|
||||
/**
|
||||
* Screen and window awareness, per ADR-0008.
|
||||
*
|
||||
* The switch, the session, and the rate limit all live in Rust; this panel only
|
||||
* asks and reports. In particular it never remembers whether it thinks a session
|
||||
* is live — every action re-reads the status Rust returns, so a session that
|
||||
* expired or was dropped by the switch cannot leave the UI claiming otherwise.
|
||||
*/
|
||||
export function ScreenAwarenessPanel() {
|
||||
const [status, setStatus] = useState<ScreenReadingStatus | null>(null);
|
||||
const [error, setError] = useState<string | null>(null);
|
||||
const [notice, setNotice] = useState<string | null>(null);
|
||||
const [busy, setBusy] = useState(false);
|
||||
const [countdown, setCountdown] = useState(0);
|
||||
const countdownRef = useRef<number | null>(null);
|
||||
|
||||
const refresh = useCallback(async () => {
|
||||
try {
|
||||
const value = await invoke<unknown>("screen_reading_status");
|
||||
setStatus(screenReadingStatusSchema.parse(value));
|
||||
setError(null);
|
||||
} catch (reason) {
|
||||
setError(errorMessage(reason));
|
||||
}
|
||||
}, []);
|
||||
|
||||
useEffect(() => {
|
||||
void refresh();
|
||||
}, [refresh]);
|
||||
|
||||
useEffect(
|
||||
() => () => {
|
||||
if (countdownRef.current !== null) {
|
||||
window.clearInterval(countdownRef.current);
|
||||
}
|
||||
},
|
||||
[],
|
||||
);
|
||||
|
||||
const toggle = async (enabled: boolean) => {
|
||||
setBusy(true);
|
||||
try {
|
||||
const value = await invoke<unknown>("set_screen_reading_enabled", {
|
||||
enabled,
|
||||
});
|
||||
setStatus(screenReadingStatusSchema.parse(value));
|
||||
setError(null);
|
||||
setNotice(
|
||||
enabled
|
||||
? "已允许读取窗口。仍需为每个窗口单独授权。"
|
||||
: "已关闭,进行中的会话也一并结束。",
|
||||
);
|
||||
} catch (reason) {
|
||||
setError(errorMessage(reason));
|
||||
} finally {
|
||||
setBusy(false);
|
||||
}
|
||||
};
|
||||
|
||||
const share = async () => {
|
||||
setBusy(true);
|
||||
setError(null);
|
||||
setNotice(null);
|
||||
setCountdown(FOCUS_DELAY_SECONDS);
|
||||
countdownRef.current = window.setInterval(() => {
|
||||
setCountdown((current) => (current > 0 ? current - 1 : 0));
|
||||
}, 1000);
|
||||
try {
|
||||
// Rust waits out the same delay before reading the foreground window, so
|
||||
// this counter is a display of that wait rather than a second one.
|
||||
const value = await invoke<unknown>("begin_window_capture_session", {
|
||||
slot: "companion",
|
||||
focusDelaySeconds: FOCUS_DELAY_SECONDS,
|
||||
});
|
||||
const session =
|
||||
screenReadingStatusSchema.shape.sessions.element.parse(value);
|
||||
setNotice(`已授权读取「${session.windowTitle}」。`);
|
||||
await refresh();
|
||||
} catch (reason) {
|
||||
setError(errorMessage(reason));
|
||||
} finally {
|
||||
if (countdownRef.current !== null) {
|
||||
window.clearInterval(countdownRef.current);
|
||||
countdownRef.current = null;
|
||||
}
|
||||
setCountdown(0);
|
||||
setBusy(false);
|
||||
}
|
||||
};
|
||||
|
||||
const stop = async (sessionToken: string) => {
|
||||
setBusy(true);
|
||||
try {
|
||||
const value = await invoke<unknown>("end_window_capture_session", {
|
||||
sessionToken,
|
||||
});
|
||||
setStatus(screenReadingStatusSchema.parse(value));
|
||||
setNotice("已结束该窗口的授权。");
|
||||
setError(null);
|
||||
} catch (reason) {
|
||||
setError(errorMessage(reason));
|
||||
} finally {
|
||||
setBusy(false);
|
||||
}
|
||||
};
|
||||
|
||||
/// Sends what the window says, as text. Cheaper and far less revealing than a
|
||||
/// picture, and enough for "what am I looking at".
|
||||
const sendText = async (sessionToken: string) => {
|
||||
setBusy(true);
|
||||
setError(null);
|
||||
try {
|
||||
const summary = windowTextSummarySchema.parse(
|
||||
await invoke<unknown>("read_window_text", { sessionToken }),
|
||||
);
|
||||
if (!summary.text.trim()) {
|
||||
setNotice("这个窗口没有可读的文本。");
|
||||
return;
|
||||
}
|
||||
await invoke<unknown>("send_connection_slot_message", {
|
||||
request: {
|
||||
slot: "companion",
|
||||
message: `我正在看的窗口内容:\n\n${summary.text}`,
|
||||
attachments: [],
|
||||
},
|
||||
});
|
||||
setNotice(
|
||||
`已发送窗口文本(${summary.visitedNodes} 个节点${summary.truncated ? ",已截断" : ""})。`,
|
||||
);
|
||||
} catch (reason) {
|
||||
setError(errorMessage(reason));
|
||||
} finally {
|
||||
setBusy(false);
|
||||
}
|
||||
};
|
||||
|
||||
const sendFrame = async (sessionToken: string) => {
|
||||
setBusy(true);
|
||||
setError(null);
|
||||
try {
|
||||
const frame = windowFrameSchema.parse(
|
||||
await invoke<unknown>("capture_window_frame", { sessionToken }),
|
||||
);
|
||||
const attachment = astrbotAttachmentSchema.parse(
|
||||
await invoke<unknown>("upload_connection_slot_file", {
|
||||
request: {
|
||||
slot: "companion",
|
||||
fileName: `window-${frame.width}x${frame.height}.png`,
|
||||
mimeType: frame.mimeType,
|
||||
bytes: frame.bytes,
|
||||
},
|
||||
}),
|
||||
);
|
||||
await invoke<unknown>("send_connection_slot_message", {
|
||||
request: {
|
||||
slot: "companion",
|
||||
message: "这是我现在的窗口画面。",
|
||||
attachments: [attachment],
|
||||
},
|
||||
});
|
||||
setNotice(`已发送画面(${frame.width}×${frame.height})。`);
|
||||
} catch (reason) {
|
||||
setError(errorMessage(reason));
|
||||
} finally {
|
||||
setBusy(false);
|
||||
}
|
||||
};
|
||||
|
||||
const sessions = status?.sessions ?? [];
|
||||
|
||||
return (
|
||||
<section className="awareness-panel">
|
||||
<div>
|
||||
<span className="section-kicker">Screen awareness</span>
|
||||
<h2>窗口读取</h2>
|
||||
<p>
|
||||
读取一个你指定的窗口,用来告诉陪伴实例你正在做什么。只读取单个窗口,
|
||||
不读取整个桌面;画面与文本不写入磁盘;只发送给陪伴槽位。
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<label className="awareness-switch">
|
||||
<input
|
||||
type="checkbox"
|
||||
checked={status?.enabled ?? false}
|
||||
disabled={busy || !status}
|
||||
onChange={(event) => void toggle(event.target.checked)}
|
||||
/>
|
||||
<span>允许本应用读取窗口</span>
|
||||
</label>
|
||||
|
||||
{status ? (
|
||||
<p className="awareness-limits">
|
||||
画面最快每 {Math.round(status.pixelIntervalMs / 1000)} 秒一次,
|
||||
文本最快每 {status.textIntervalMs} 毫秒一次, 每次授权{" "}
|
||||
{Math.round(status.sessionTtlSeconds / 60)} 分钟后失效。
|
||||
</p>
|
||||
) : null}
|
||||
|
||||
<div className="awareness-actions">
|
||||
<button
|
||||
type="button"
|
||||
disabled={busy || !status?.enabled}
|
||||
onClick={() => void share()}
|
||||
>
|
||||
{countdown > 0 ? `请切换到目标窗口…${countdown}` : "授权一个窗口"}
|
||||
</button>
|
||||
</div>
|
||||
|
||||
{sessions.length ? (
|
||||
<ul className="awareness-sessions">
|
||||
{sessions.map((session) => (
|
||||
<li key={session.sessionToken}>
|
||||
<strong>{session.windowTitle || "(无标题窗口)"}</strong>
|
||||
<span className="awareness-session-meta">
|
||||
PID {session.processId}
|
||||
</span>
|
||||
<div className="awareness-session-actions">
|
||||
<button
|
||||
type="button"
|
||||
disabled={busy}
|
||||
onClick={() => void sendText(session.sessionToken)}
|
||||
>
|
||||
发送窗口文本
|
||||
</button>
|
||||
<button
|
||||
type="button"
|
||||
disabled={busy}
|
||||
onClick={() => void sendFrame(session.sessionToken)}
|
||||
>
|
||||
发送窗口画面
|
||||
</button>
|
||||
<button
|
||||
type="button"
|
||||
disabled={busy}
|
||||
onClick={() => void stop(session.sessionToken)}
|
||||
>
|
||||
结束授权
|
||||
</button>
|
||||
</div>
|
||||
</li>
|
||||
))}
|
||||
</ul>
|
||||
) : (
|
||||
<p className="awareness-empty">当前没有已授权的窗口。</p>
|
||||
)}
|
||||
|
||||
{notice ? <p className="awareness-notice">{notice}</p> : null}
|
||||
{error ? (
|
||||
<p className="error-banner" role="alert">
|
||||
{error}
|
||||
</p>
|
||||
) : null}
|
||||
</section>
|
||||
);
|
||||
}
|
||||
55
src/features/awareness/types.ts
普通文件
55
src/features/awareness/types.ts
普通文件
|
|
@ -0,0 +1,55 @@
|
|||
import { z } from "zod";
|
||||
|
||||
import { activeSlotKindSchema } from "../../types/connection";
|
||||
|
||||
/// One window the user has authorized this application to read.
|
||||
export const captureSessionSchema = z
|
||||
.object({
|
||||
sessionToken: z.uuid(),
|
||||
windowTitle: z.string(),
|
||||
processId: z.number().int().nonnegative(),
|
||||
slot: activeSlotKindSchema,
|
||||
expired: z.boolean(),
|
||||
})
|
||||
.strict();
|
||||
|
||||
/**
|
||||
* The privacy switch, the live sessions, and the limits Rust is enforcing.
|
||||
*
|
||||
* The intervals are reported rather than assumed by the UI: the limit lives in
|
||||
* Rust precisely so the renderer cannot be the thing that decides it, and a
|
||||
* countdown built on a hard-coded copy would drift from the real one.
|
||||
*/
|
||||
export const screenReadingStatusSchema = z
|
||||
.object({
|
||||
enabled: z.boolean(),
|
||||
sessions: z.array(captureSessionSchema),
|
||||
pixelIntervalMs: z.number().int().positive(),
|
||||
textIntervalMs: z.number().int().positive(),
|
||||
sessionTtlSeconds: z.number().int().positive(),
|
||||
})
|
||||
.strict();
|
||||
|
||||
export const windowFrameSchema = z
|
||||
.object({
|
||||
mimeType: z.string().min(1),
|
||||
width: z.number().int().positive(),
|
||||
height: z.number().int().positive(),
|
||||
bytes: z.array(z.number().int()),
|
||||
})
|
||||
.strict();
|
||||
|
||||
export const windowTextSummarySchema = z
|
||||
.object({
|
||||
text: z.string(),
|
||||
visitedNodes: z.number().int().nonnegative(),
|
||||
passwordNodesSkipped: z.number().int().nonnegative(),
|
||||
foreignProcessNodesSkipped: z.number().int().nonnegative(),
|
||||
truncated: z.boolean(),
|
||||
})
|
||||
.strict();
|
||||
|
||||
export type CaptureSession = z.infer<typeof captureSessionSchema>;
|
||||
export type ScreenReadingStatus = z.infer<typeof screenReadingStatusSchema>;
|
||||
export type WindowFrame = z.infer<typeof windowFrameSchema>;
|
||||
export type WindowTextSummary = z.infer<typeof windowTextSummarySchema>;
|
||||
|
|
@ -2,6 +2,7 @@
|
|||
import { useState } from "react";
|
||||
|
||||
import { errorMessage } from "../lib/errorMessage";
|
||||
import { ScreenAwarenessPanel } from "../features/awareness/ScreenAwarenessPanel";
|
||||
import type { BootstrapStatus } from "../types/bootstrap";
|
||||
|
||||
export type SettingsChatSlot = "companion" | "operations";
|
||||
|
|
@ -225,6 +226,7 @@ export function DiagnosticsPage({
|
|||
value={String(status.databaseSchemaVersion)}
|
||||
/>
|
||||
</section>
|
||||
<ScreenAwarenessPanel />
|
||||
<DiagnosticsTools />
|
||||
<section className="boundaries">
|
||||
<span className="section-kicker">Security boundary</span>
|
||||
|
|
|
|||
|
|
@ -3508,3 +3508,95 @@ button:disabled {
|
|||
background: var(--ui-surface-soft);
|
||||
font-size: 0.78rem;
|
||||
}
|
||||
|
||||
/* Screen awareness (ADR-0008). Deliberately not styled to disappear: the
|
||||
indicator is a privacy signal, so it is the one banner that must be noticed. */
|
||||
.awareness-panel {
|
||||
display: grid;
|
||||
gap: 0.75rem;
|
||||
padding: 1rem;
|
||||
border: 1px solid var(--ui-line);
|
||||
border-radius: var(--ui-radius-sm);
|
||||
background: var(--ui-surface-soft);
|
||||
}
|
||||
|
||||
.awareness-switch {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 0.5rem;
|
||||
}
|
||||
|
||||
.awareness-limits,
|
||||
.awareness-empty {
|
||||
margin: 0;
|
||||
color: var(--ui-ink-soft, #9aa8bd);
|
||||
font-size: 0.85rem;
|
||||
}
|
||||
|
||||
.awareness-notice {
|
||||
margin: 0;
|
||||
font-size: 0.85rem;
|
||||
}
|
||||
|
||||
.awareness-sessions {
|
||||
display: grid;
|
||||
gap: 0.5rem;
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
list-style: none;
|
||||
}
|
||||
|
||||
.awareness-sessions li {
|
||||
display: grid;
|
||||
gap: 0.35rem;
|
||||
padding: 0.6rem;
|
||||
border: 1px solid var(--ui-line);
|
||||
border-radius: var(--ui-radius-sm);
|
||||
}
|
||||
|
||||
.awareness-session-meta {
|
||||
color: var(--ui-ink-soft, #9aa8bd);
|
||||
font-size: 0.8rem;
|
||||
}
|
||||
|
||||
.awareness-session-actions {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 0.4rem;
|
||||
}
|
||||
|
||||
.awareness-indicator {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 0.5rem;
|
||||
padding: 0.45rem 0.7rem;
|
||||
border: 1px solid #d9a15a;
|
||||
border-radius: var(--ui-radius-sm);
|
||||
background: rgba(217, 161, 90, 0.12);
|
||||
color: #e6bd85;
|
||||
font-size: 0.85rem;
|
||||
}
|
||||
|
||||
.awareness-indicator-dot {
|
||||
width: 0.55rem;
|
||||
height: 0.55rem;
|
||||
border-radius: 50%;
|
||||
background: #e0733b;
|
||||
}
|
||||
|
||||
/* A steady dot rather than a pulse for anyone who asked not to see motion. */
|
||||
@media (prefers-reduced-motion: no-preference) {
|
||||
.awareness-indicator-dot {
|
||||
animation: awareness-pulse 1.6s ease-in-out infinite;
|
||||
}
|
||||
}
|
||||
|
||||
@keyframes awareness-pulse {
|
||||
0%,
|
||||
100% {
|
||||
opacity: 1;
|
||||
}
|
||||
50% {
|
||||
opacity: 0.35;
|
||||
}
|
||||
}
|
||||
|
|
|
|||
正在加载…
添加表格
在新工单中引用