AstrBot-Desktop/docs/spikes/P00-03-ui-automation-and-window-capture.md

65 行
3.3 KiB
Markdown

# P00-03: UI Automation and Selected-Window Capture
- Status: Passed for a controlled non-elevated standard dialog; elevated/custom-drawn manual coverage remains
- Date recorded: 2026-07-18
- MVP gate: Required for bounded error-window diagnostics
## Objective
Verify read-only extraction from a standard Windows dialog and a user-selected, single-window capture fallback without running the application as administrator or broadening access to the desktop.
## Environment
- Host: Windows 11 x64
- Target class: standard non-elevated Windows dialog
- Rust toolchain: rustc/cargo 1.97.1
- MSVC C++ build tools: verified by the Windows test and Tauri build
- Elevated target testing: negative boundary only
## Minimal validation implementation
The Spike is expected to:
- Identify the current foreground window.
- Return bounded title, process, and UI Automation text fields.
- Report unsupported, inaccessible, closed, or elevated targets safely.
- Capture only the window explicitly selected by the user.
- Keep image bytes in Rust until a display-safe preview is produced.
- Delete transient capture data after the diagnostic flow unless the user explicitly saves it.
## Required execution
1. Open a standard non-elevated dialog with known text.
2. Read its title and UI Automation text.
3. Verify an unsupported or custom-drawn window produces a structured fallback result.
4. Select one window and capture only its client/window bounds.
5. Verify a closed or invalid handle does not capture another window.
6. Attempt an elevated target and confirm a safe access-denied result without elevation.
7. Confirm transient image data is removed after the test.
## Actual execution result
The ignored Windows smoke test opened a test-owned standard MessageBox containing fixed, non-sensitive text. It selected only that foreground HWND, verified title and process identity, read the expected title/body through UI Automation, captured the same selected window through PrintWindow, and validated an in-memory PNG without writing image bytes to disk.
The test then closed the controlled dialog and verified that the expired HWND identity returned `WindowChanged` and could not capture a replacement window. The test passed in 0.12 seconds.
An elevated target was not created because that would require a UAC/elevation action outside this bounded Spike. The implementation maps access denied and element-unavailable HRESULTs to structured errors, but a real elevated/custom-drawn target remains an explicit manual boundary test.
## Log summary
Command: `cargo test -p astrbot-desktop --test windows_dialog_smoke -- --ignored --nocapture`. Result: 1 passed. Output contained test status only; no extracted text or pixel bytes were logged or persisted.
## Known limitations
- UI Automation does not expose all custom-rendered content.
- Protected, elevated, DRM, or hardware-accelerated windows may reject inspection or capture.
- DPI scaling and multi-monitor coordinates require explicit tests.
## Security impact
Window content can contain personal or secret data. Capture must be user-selected, bounded to one target, previewable, and excluded from durable logs.
## Recommendation
Keep UI Automation as the preferred text path and selected-window capture as an explicit fallback. Do not add continuous desktop capture, global hooks, input injection, or privilege escalation.