AstrBot-Desktop/docs/spikes/P00-03-ui-automation-and-window-capture.md
YokiiroBW 353a439b41
一些检测仍在等待运行
ci / Portable quality checks (push) Waiting to run
ci / Windows Tauri build (push) Waiting to run
docs(p00): record spike results and unresolved boundaries
2026-07-18 21:00:49 +08:00

3.3 KiB

P00-03: UI Automation and Selected-Window Capture

  • Status: Passed for a controlled non-elevated standard dialog; elevated/custom-drawn manual coverage remains
  • Date recorded: 2026-07-18
  • MVP gate: Required for bounded error-window diagnostics

Objective

Verify read-only extraction from a standard Windows dialog and a user-selected, single-window capture fallback without running the application as administrator or broadening access to the desktop.

Environment

  • Host: Windows 11 x64
  • Target class: standard non-elevated Windows dialog
  • Rust toolchain: rustc/cargo 1.97.1
  • MSVC C++ build tools: verified by the Windows test and Tauri build
  • Elevated target testing: negative boundary only

Minimal validation implementation

The Spike is expected to:

  • Identify the current foreground window.
  • Return bounded title, process, and UI Automation text fields.
  • Report unsupported, inaccessible, closed, or elevated targets safely.
  • Capture only the window explicitly selected by the user.
  • Keep image bytes in Rust until a display-safe preview is produced.
  • Delete transient capture data after the diagnostic flow unless the user explicitly saves it.

Required execution

  1. Open a standard non-elevated dialog with known text.
  2. Read its title and UI Automation text.
  3. Verify an unsupported or custom-drawn window produces a structured fallback result.
  4. Select one window and capture only its client/window bounds.
  5. Verify a closed or invalid handle does not capture another window.
  6. Attempt an elevated target and confirm a safe access-denied result without elevation.
  7. Confirm transient image data is removed after the test.

Actual execution result

The ignored Windows smoke test opened a test-owned standard MessageBox containing fixed, non-sensitive text. It selected only that foreground HWND, verified title and process identity, read the expected title/body through UI Automation, captured the same selected window through PrintWindow, and validated an in-memory PNG without writing image bytes to disk.

The test then closed the controlled dialog and verified that the expired HWND identity returned WindowChanged and could not capture a replacement window. The test passed in 0.12 seconds.

An elevated target was not created because that would require a UAC/elevation action outside this bounded Spike. The implementation maps access denied and element-unavailable HRESULTs to structured errors, but a real elevated/custom-drawn target remains an explicit manual boundary test.

Log summary

Command: cargo test -p astrbot-desktop --test windows_dialog_smoke -- --ignored --nocapture. Result: 1 passed. Output contained test status only; no extracted text or pixel bytes were logged or persisted.

Known limitations

  • UI Automation does not expose all custom-rendered content.
  • Protected, elevated, DRM, or hardware-accelerated windows may reject inspection or capture.
  • DPI scaling and multi-monitor coordinates require explicit tests.

Security impact

Window content can contain personal or secret data. Capture must be user-selected, bounded to one target, previewable, and excluded from durable logs.

Recommendation

Keep UI Automation as the preferred text path and selected-window capture as an explicit fallback. Do not add continuous desktop capture, global hooks, input injection, or privilege escalation.