feat(awareness): add manual WGC acceptance action
一些检测仍在等待运行
ci / Portable quality checks (push) Waiting to run
ci / Windows Tauri build (push) Waiting to run

这个提交存在于:
YokiiroBW 2026-07-27 15:26:55 +08:00
父节点 48599db6d4
当前提交 6634cfe91c
共有 9 个文件被更改,包括 191 次插入 和 0 次删除

查看文件

@ -12,6 +12,7 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and
- Screen-awareness implementation framework for scoped display consent, WGC capture, local privacy/quality gates, multimodal AstrBot validation, and low-interruption response control. - Screen-awareness implementation framework for scoped display consent, WGC capture, local privacy/quality gates, multimodal AstrBot validation, and low-interruption response control.
- WGC S0.1 target-binding preflight for authorized windows and attached displays. It returns metadata only; no capture session, pixels, disk write, or upload is performed. - WGC S0.1 target-binding preflight for authorized windows and attached displays. It returns metadata only; no capture session, pixels, disk write, or upload is performed.
- Bounded WGC S0.2 diagnostic readback for an already authorized window: one in-memory D3D11 frame, local blank-frame classification, and discarded PNG conversion metadata only. - Bounded WGC S0.2 diagnostic readback for an already authorized window: one in-memory D3D11 frame, local blank-frame classification, and discarded PNG conversion metadata only.
- Per-session manual WGC acceptance action that displays only local frame-quality metadata and is structurally unable to upload or send an image.
- Debug-level tracing of AstrBot SSE frame shapes: frame type, which identity fields are present, and the key names inside `data`. No payload text is recorded. Enable with `RUST_LOG`. - Debug-level tracing of AstrBot SSE frame shapes: frame type, which identity fields are present, and the key names inside `data`. No payload text is recorded. Enable with `RUST_LOG`.
- Q10-Q15 chat experience increment: lightweight Markdown rendering, message actions, generation controls, session actions, connection controls, and responsive chat layout. - Q10-Q15 chat experience increment: lightweight Markdown rendering, message actions, generation controls, session actions, connection controls, and responsive chat layout.
- AstrBot OpenAPI adapter foundation: SSE mapper, dual-slot runtime skeleton, credential enrollment IPC, transport-aware profiles (ADR-0005). - AstrBot OpenAPI adapter foundation: SSE mapper, dual-slot runtime skeleton, credential enrollment IPC, transport-aware profiles (ADR-0005).

查看文件

@ -252,6 +252,15 @@ The following are deliberately not marked complete: HTTPS certificate pinning, s
验收:隔离 Rust check、121 个库测试和 Clippy 均通过。仍缺少实际窗口/游戏、最小化、锁屏、受保护内容、显示器变化的人工证据,故 S0 整体未完成。 验收:隔离 Rust check、121 个库测试和 Clippy 均通过。仍缺少实际窗口/游戏、最小化、锁屏、受保护内容、显示器变化的人工证据,故 S0 整体未完成。
### S0.3:手动 WGC 验收入口(2026-07-27,完成;等待真实场景证据)
- 在“窗口读取”已授权会话卡片内增加 `WGC 本地帧验收(不发送)`;用户必须先经现有 5 秒前台窗口授权,按钮不创建新授权。
- 结果只显示尺寸、内存中验证的 PNG 大小和 `usable` / `rejected_blank` 状态;不显示缩略图、不返回图像字节、不上传、不发送消息。
- 任何失败继续显示 Rust 的可行动错误;黑帧/透明帧明确表示不支持,不能据此开启游戏陪玩。
验收:前端 typecheck、61 项 Vitest、生产 Vite 构建及本次文件 Prettier 检查通过;Rust S0.2 检查仍为 121 passed / Clippy 通过。人工场景结果仍是 S0 的阻塞项。
### C1:聊天、语音聊天、游戏陪玩三页面壳 ### C1:聊天、语音聊天、游戏陪玩三页面壳
- 在现有“聊天”之后增加“语音聊天”“游戏陪玩”两个独立导航入口。 - 在现有“聊天”之后增加“语音聊天”“游戏陪玩”两个独立导航入口。

查看文件

@ -784,3 +784,17 @@ This is not a user-facing capture feature and has no scheduler, settings UI, dis
- Isolated Rust check: passed. - Isolated Rust check: passed.
- Isolated Rust library tests: 121 passed, 0 failed. - Isolated Rust library tests: 121 passed, 0 failed.
- Isolated Rust Clippy with warnings denied: passed. - Isolated Rust Clippy with warnings denied: passed.
## 2026-07-27: WGC S0.3 manual acceptance entry
Placed a `WGC 本地帧验收(不发送)` action beside each existing authorized-window session. It reuses the short-lived Rust authorization and returns only width, height, temporary PNG byte count, and usable/blank quality. No preview, image byte array, upload, send request, scheduler, or display-scope expansion was added.
A component regression proves the action invokes only `wgc_spike_probe_authorized_window_frame`, never the upload or send commands. This makes manual Windows evidence possible without weakening the no-network S0 boundary.
### Test status
- Frontend typecheck: passed.
- Vitest: 61 passed.
- Production Vite build: passed.
- Rust S0.2 regression remains 121 passed with Clippy clean.

查看文件

@ -39,3 +39,12 @@ Implemented a bounded one-frame diagnostic path for an already authorized window
Validation: isolated `cargo check -p astrbot-desktop --lib`, `cargo test -p astrbot-desktop --lib` (121 passed), and `cargo clippy -p astrbot-desktop --lib -- -D warnings` all passed. Validation: isolated `cargo check -p astrbot-desktop --lib`, `cargo test -p astrbot-desktop --lib` (121 passed), and `cargo clippy -p astrbot-desktop --lib -- -D warnings` all passed.
Manual evidence is still pending. No active UI invokes the diagnostic command yet, and no game, minimized window, lock screen, protected content, or multi-display run has been claimed as validated. Those observations remain blocking S0 acceptance. Manual evidence is still pending. No active UI invokes the diagnostic command yet, and no game, minimized window, lock screen, protected content, or multi-display run has been claimed as validated. Those observations remain blocking S0 acceptance.
## S0.3 manual acceptance entry
The existing authorized-window card in Diagnostics now exposes `WGC 本地帧验收(不发送)`. It is intentionally a per-session manual action rather than a setting or scheduled feature. The UI receives only the probe metadata and renders one of two explicit outcomes:
- `usable`: the local frame was nonblank, with dimensions and the temporary in-memory PNG byte count.
- `rejected_blank`: the local frame was black or transparent, so no game-support conclusion and no outgoing action is permitted.
The component regression asserts that this action invokes only `wgc_spike_probe_authorized_window_frame`; it cannot call either the AstrBot upload or message-send commands. Frontend validation: TypeScript check, 61 Vitest assertions, and production build passed.

查看文件

@ -102,3 +102,14 @@
- 修复或规避:注册 `FrameArrived` 事件并只等待 2 秒,到期返回明确的 `FrameTimedOut`;每次退出移除事件并关闭 frame/session/pool,避免留存捕获资源。 - 修复或规避:注册 `FrameArrived` 事件并只等待 2 秒,到期返回明确的 `FrameTimedOut`;每次退出移除事件并关闭 frame/session/pool,避免留存捕获资源。
- 预防规则:所有主动捕获路径必须有帧、尺寸、内存和生命周期上限;超时是可解释拒绝,不得重试成后台观察。 - 预防规则:所有主动捕获路径必须有帧、尺寸、内存和生命周期上限;超时是可解释拒绝,不得重试成后台观察。
- 验证证据:Rust 121 个库测试通过,Clippy 通过;真实 Windows 场景证据仍待人工验收。 - 验证证据:Rust 121 个库测试通过,Clippy 通过;真实 Windows 场景证据仍待人工验收。
## 2026-07-27 — Metadata result and global notice can intentionally repeat safety copy
- 阶段:S0.3 手动 WGC 验收入口
- 类型:测试 / 功能行为
- 现象:首版组件回归用唯一文本查询 `未保存、未发送`,但结果卡和全局通知都故意说明同一隐私保证,测试失败。
- 根因:断言把可访问的重复安全文案误判为重复渲染;两个位置承担不同作用,前者解释帧结果,后者提供操作反馈。
- 修复或规避:改为断言结果卡独有的尺寸元数据,并继续断言没有调用上传或消息发送命令。
- 预防规则:涉及隐私/安全说明时,允许局部结果和全局反馈重复关键保证;测试应定位结构化状态或唯一业务数据,而不是假设文案唯一。
- 验证证据:前端 61 项测试、类型检查和生产构建通过。

查看文件

@ -0,0 +1,64 @@
import { fireEvent, render, screen, waitFor } from "@testing-library/react";
import { invoke } from "@tauri-apps/api/core";
import { beforeEach, describe, expect, it, vi } from "vitest";
import { ScreenAwarenessPanel } from "./ScreenAwarenessPanel";
vi.mock("@tauri-apps/api/core", () => ({ invoke: vi.fn() }));
const session = {
sessionToken: "6f1e0d3a-4b5c-4d6e-8f90-1a2b3c4d5e6f",
windowTitle: "DirectX game",
processId: 4242,
slot: "companion",
expired: false,
};
const status = {
enabled: true,
sessions: [session],
pixelIntervalMs: 2000,
textIntervalMs: 750,
sessionTtlSeconds: 300,
};
describe("the WGC Spike action", () => {
beforeEach(() => {
vi.mocked(invoke).mockReset();
vi.mocked(invoke).mockImplementation(async (command) => {
if (command === "screen_reading_status") return status;
if (command === "wgc_spike_probe_authorized_window_frame") {
return {
width: 1920,
height: 1080,
encodedPngBytes: 42_000,
quality: "usable",
};
}
throw new Error(`Unexpected command: ${command}`);
});
});
it("runs only the metadata diagnostic and never sends the frame", async () => {
render(<ScreenAwarenessPanel />);
const action = await screen.findByRole("button", { name: /WGC/ });
fireEvent.click(action);
await waitFor(() =>
expect(invoke).toHaveBeenCalledWith(
"wgc_spike_probe_authorized_window_frame",
{ sessionToken: session.sessionToken },
),
);
expect(await screen.findByText(/1920 × 1080/)).toBeInTheDocument();
expect(invoke).not.toHaveBeenCalledWith(
"send_connection_slot_message",
expect.anything(),
);
expect(invoke).not.toHaveBeenCalledWith(
"upload_connection_slot_file",
expect.anything(),
);
});
});

查看文件

@ -7,7 +7,9 @@ import {
screenReadingStatusSchema, screenReadingStatusSchema,
windowFrameSchema, windowFrameSchema,
windowTextSummarySchema, windowTextSummarySchema,
wgcFrameProbeSchema,
type ScreenReadingStatus, type ScreenReadingStatus,
type WgcFrameProbe,
} from "./types"; } from "./types";
/// The window the user is asked to bring forward before the capture starts. /// The window the user is asked to bring forward before the capture starts.
@ -27,6 +29,10 @@ export function ScreenAwarenessPanel() {
const [notice, setNotice] = useState<string | null>(null); const [notice, setNotice] = useState<string | null>(null);
const [busy, setBusy] = useState(false); const [busy, setBusy] = useState(false);
const [countdown, setCountdown] = useState(0); const [countdown, setCountdown] = useState(0);
const [wgcProbe, setWgcProbe] = useState<{
sessionToken: string;
frame: WgcFrameProbe;
} | null>(null);
const countdownRef = useRef<number | null>(null); const countdownRef = useRef<number | null>(null);
const refresh = useCallback(async () => { const refresh = useCallback(async () => {
@ -149,6 +155,32 @@ export function ScreenAwarenessPanel() {
} }
}; };
/// Runs the Spike-only frame path. It returns metadata, not a preview: the
/// raw WGC frame and its temporary PNG are destroyed inside Rust before this
/// function receives a result.
const testWgcFrame = async (sessionToken: string) => {
setBusy(true);
setError(null);
setWgcProbe(null);
try {
const frame = wgcFrameProbeSchema.parse(
await invoke<unknown>("wgc_spike_probe_authorized_window_frame", {
sessionToken,
}),
);
setWgcProbe({ sessionToken, frame });
setNotice(
frame.quality === "usable"
? "WGC 本地验收得到非空画面;图像未保存、未发送。"
: "WGC 返回黑帧或透明帧,已拒绝;不会保存或发送。",
);
} catch (reason) {
setError(errorMessage(reason));
} finally {
setBusy(false);
}
};
const sendFrame = async (sessionToken: string) => { const sendFrame = async (sessionToken: string) => {
setBusy(true); setBusy(true);
setError(null); setError(null);
@ -238,6 +270,13 @@ export function ScreenAwarenessPanel() {
> >
发送窗口文本 发送窗口文本
</button> </button>
<button
type="button"
disabled={busy}
onClick={() => void testWgcFrame(session.sessionToken)}
>
WGC 本地帧验收(不发送)
</button>
<button <button
type="button" type="button"
disabled={busy} disabled={busy}
@ -253,6 +292,17 @@ export function ScreenAwarenessPanel() {
结束授权 结束授权
</button> </button>
</div> </div>
{wgcProbe?.sessionToken === session.sessionToken ? (
<p
className="awareness-spike-result"
data-quality={wgcProbe.frame.quality}
role="status"
>
{wgcProbe.frame.quality === "usable"
? `WGC 已捕获非空帧:${wgcProbe.frame.width} × ${wgcProbe.frame.height},仅在内存中验证了 ${Math.round(wgcProbe.frame.encodedPngBytes / 1024)} KB PNG。未保存、未发送。`
: `WGC 返回黑帧或透明帧(${wgcProbe.frame.width} × ${wgcProbe.frame.height}),已拒绝。游戏陪玩不会据此启用。`}
</p>
) : null}
</li> </li>
))} ))}
</ul> </ul>

查看文件

@ -39,6 +39,15 @@ export const windowFrameSchema = z
}) })
.strict(); .strict();
export const wgcFrameProbeSchema = z
.object({
width: z.number().int().positive(),
height: z.number().int().positive(),
encodedPngBytes: z.number().int().nonnegative(),
quality: z.enum(["usable", "rejected_blank"]),
})
.strict();
export const windowTextSummarySchema = z export const windowTextSummarySchema = z
.object({ .object({
text: z.string(), text: z.string(),
@ -52,4 +61,5 @@ export const windowTextSummarySchema = z
export type CaptureSession = z.infer<typeof captureSessionSchema>; export type CaptureSession = z.infer<typeof captureSessionSchema>;
export type ScreenReadingStatus = z.infer<typeof screenReadingStatusSchema>; export type ScreenReadingStatus = z.infer<typeof screenReadingStatusSchema>;
export type WindowFrame = z.infer<typeof windowFrameSchema>; export type WindowFrame = z.infer<typeof windowFrameSchema>;
export type WgcFrameProbe = z.infer<typeof wgcFrameProbeSchema>;
export type WindowTextSummary = z.infer<typeof windowTextSummarySchema>; export type WindowTextSummary = z.infer<typeof windowTextSummarySchema>;

查看文件

@ -3565,6 +3565,29 @@ button:disabled {
gap: 0.4rem; gap: 0.4rem;
} }
.awareness-spike-result {
margin: 0;
padding: 0.55rem 0.65rem;
border: 1px solid var(--ui-line);
border-radius: var(--ui-radius-sm);
background: color-mix(in srgb, var(--ui-surface) 72%, #d9a15a);
color: var(--ui-ink-soft, #9aa8bd);
font-size: 0.82rem;
line-height: 1.45;
}
.awareness-spike-result[data-quality="usable"] {
border-color: rgba(96, 181, 130, 0.55);
background: rgba(96, 181, 130, 0.1);
color: #b6e2c5;
}
.awareness-spike-result[data-quality="rejected_blank"] {
border-color: rgba(217, 161, 90, 0.55);
background: rgba(217, 161, 90, 0.1);
color: #e6bd85;
}
.awareness-indicator { .awareness-indicator {
display: flex; display: flex;
align-items: center; align-items: center;