charles/ws-rpc-test
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Screenshot capture on test failure (GNOME, focused-window) #18

New issue
Closed
opened 2026-04-11 11:01:11 +00:00 by charles · 0 comments
charles commented 2026-04-11 11:01:11 +00:00
Owner
Copy link

User story

As a test author whose app under test has a GUI on GNOME, I want a screenshot of the app's window captured automatically the moment a test fails, so that I can diagnose visual bugs ("wrong dialog appeared", "window froze", "modal stuck behind another window") without having to reproduce the failure interactively — and without leaking unrelated desktop content into the artifact.

Scope

v0.1 targets GNOME only (both X11 and Wayland sessions). Other desktops (Sway, Hyprland, KDE, generic X11) are explicitly deferred — the user is on GNOME and we'd rather have one well-tested path than five half-working ones.

The captured window is the focused window at the moment of failure, not a PID-resolved one. This works because:

  • GNOME (especially Wayland) does not expose a stable, unprivileged PID → window CLI.
  • In a test run the harness has just spawned the app and is actively driving it via WebSocket — the app's window is normally the focused one. Tests where another window steals focus mid-run are out of scope for v0.1.

This stays within the framework's non-goal of "browser/DOM testing" — it's a passive raster snapshot, not DOM introspection.

Acceptance criteria

Configuration

  • RunConfig.screenshot_on_failure: bool (default false).
  • RunConfig.screenshot_dir: PathBuf (default ./test-screenshots/).
  • CLI flags: --screenshot-on-failure, --screenshot-dir <PATH> wired through #15.
  • Builder shortcut on TestRunner: runner.screenshot_on_failure(dir).

Capture trigger

  • On test failure (any Err outcome that isn't Skip), the runner attempts to capture a screenshot before running after_each so cleanup hooks don't alter what's on screen.
  • Capture has a hard 5-second timeout. If exceeded, kill the subprocess(es) and log a warning; the test result is not affected.
  • Never fail the test or the run because the screenshot pipeline breaks. All errors are logged warnings.

Capture mechanism (GNOME)

The framework uses GNOME-native interfaces only. Both X11 and Wayland sessions are supported through the same code path because GNOME Shell exposes the same D-Bus interface in both.

  • Primary: GNOME Shell D-Bus interface org.gnome.Shell.Screenshot, method ScreenshotWindow: 
    gdbus call --session \
  --dest org.gnome.Shell \
  --object-path /org/gnome/Shell/Screenshot \
  --method org.gnome.Shell.Screenshot.ScreenshotWindow \
  false false false "<output_path>"
    Args: include_frame=false, include_cursor=false, flash=false, filename. Captures the currently focused window. Works on GNOME X11 and Wayland.
  • Fallback: gnome-screenshot --window --file=<output_path> if the D-Bus call fails (older GNOME install, stripped Shell, etc.).
  • If both methods fail or gdbus/gnome-screenshot are not installed, log a one-line warning to stderr (screenshot capture skipped: GNOME Shell screenshot interface unavailable (gdbus rc=N, gnome-screenshot rc=N)) and continue. Do not fall back to full-screen.

Session detection & gating

  • At runner startup (when screenshot_on_failure is enabled), probe once:
    • Is gdbus on PATH?
    • Does gdbus introspect --session --dest org.gnome.Shell --object-path /org/gnome/Shell/Screenshot succeed?
    • If neither D-Bus nor gnome-screenshot is available, log a one-line warning at startup and disable screenshot capture for the run.
  • Detect non-GNOME sessions via XDG_CURRENT_DESKTOP (GNOME, GNOME-Classic, ubuntu:GNOME, etc.). On other desktops, log a one-line warning at startup naming the detected desktop and disable screenshot capture for the run.
  • All capture commands inherit DISPLAY / WAYLAND_DISPLAY / XDG_RUNTIME_DIR / DBUS_SESSION_BUS_ADDRESS from the same env the harness child sees.

Output

  • Filename pattern: {screenshot_dir}/{timestamp}_{test_name_sanitized}.png where timestamp is YYYYMMDD-HHMMSS-mmm (millisecond precision to avoid collisions in fast suites) and test_name_sanitized replaces non-alphanumerics with _.
  • The directory is created if it doesn't exist (fs::create_dir_all).
  • Captured path stored on TestCaseResult.screenshot: Option<PathBuf>. None means capture was attempted but skipped or failed.

Reporter integration

  • Console reporter (#13): when a test fails and a screenshot was captured, append a line below the failure block:
    Screenshot: ./test-screenshots/20260411-103045-123_generation_cancel.png
  • JSON reporter (#14): include "screenshot": "<path>" in the test object when captured (omitted otherwise).
  • JUnit reporter (#14): emit the screenshot path inside the <failure> body and as a <system-out> element so CI dashboards (Jenkins, GitLab) pick it up as an artifact.

Tests

  • Self-test verifies the happy path with a fake gdbus script that touches a PNG when called with the expected args; assert path-storage and reporter integration.
  • Self-test verifies the D-Bus-fails-fallback-succeeds path: fake gdbus returns non-zero, fake gnome-screenshot produces the file, capture still succeeds.
  • Self-test verifies the both-fail path: both fakes return non-zero, run completes, warning is logged, screenshot on the test result is None.
  • Self-test verifies the non-GNOME desktop path: stub XDG_CURRENT_DESKTOP=KDE, confirm capture is disabled at startup with a warning and never attempted.
  • Self-test verifies the timeout: a fake gdbus script that sleep 30s gets killed within ~5 s and the test still completes.

Documentation requirement

The README and the RunConfig.screenshot_on_failure doc comment must clearly state:

  1. GNOME-only for v0.1 (other desktops will be added based on demand).
  2. Captures the focused window — assumes the app under test holds focus at failure time. Tests that switch focus to another app mid-run will capture the wrong window.
  3. Wayland and X11 GNOME sessions are both supported via the same D-Bus interface.

Out of scope (still deferred)

  • Sway, Hyprland, KDE, generic X11 — to be added in later releases if needed.
  • PID → window resolution (GNOME doesn't expose this without an extension; focused-window is the pragmatic substitute).
  • Bringing the app's window to focus before capture.
  • Video recording / animated diagnostics.
  • Screenshot capture on success (would balloon disk usage).
  • Automatic upload to S3/artifact stores. The path is surfaced; uploading is the CI's job.
  • Comparing screenshots against baselines (snapshot testing — listed under spec §11 future extensions).
  • Full-screen capture — explicitly excluded per project decision.

References

## User story As a **test author whose app under test has a GUI on GNOME**, I want a screenshot of the app's window captured automatically the moment a test fails, so that I can diagnose visual bugs ("wrong dialog appeared", "window froze", "modal stuck behind another window") without having to reproduce the failure interactively — and without leaking unrelated desktop content into the artifact. ## Scope **v0.1 targets GNOME only** (both X11 and Wayland sessions). Other desktops (Sway, Hyprland, KDE, generic X11) are explicitly deferred — the user is on GNOME and we'd rather have one well-tested path than five half-working ones. **The captured window is the *focused* window at the moment of failure**, not a PID-resolved one. This works because: - GNOME (especially Wayland) does not expose a stable, unprivileged PID → window CLI. - In a test run the harness has just spawned the app and is actively driving it via WebSocket — the app's window is normally the focused one. Tests where another window steals focus mid-run are out of scope for v0.1. This stays within the framework's non-goal of "browser/DOM testing" — it's a passive raster snapshot, not DOM introspection. ## Acceptance criteria ### Configuration - [ ] `RunConfig.screenshot_on_failure: bool` (default `false`). - [ ] `RunConfig.screenshot_dir: PathBuf` (default `./test-screenshots/`). - [ ] CLI flags: `--screenshot-on-failure`, `--screenshot-dir <PATH>` wired through #15. - [ ] Builder shortcut on `TestRunner`: `runner.screenshot_on_failure(dir)`. ### Capture trigger - [ ] On test failure (any `Err` outcome that isn't `Skip`), the runner attempts to capture a screenshot **before** running `after_each` so cleanup hooks don't alter what's on screen. - [ ] Capture has a hard 5-second timeout. If exceeded, kill the subprocess(es) and log a warning; the test result is not affected. - [ ] **Never fail the test or the run because the screenshot pipeline breaks.** All errors are logged warnings. ### Capture mechanism (GNOME) The framework uses GNOME-native interfaces only. Both X11 and Wayland sessions are supported through the same code path because GNOME Shell exposes the same D-Bus interface in both. - [ ] **Primary**: GNOME Shell D-Bus interface `org.gnome.Shell.Screenshot`, method `ScreenshotWindow`: ``` gdbus call --session \ --dest org.gnome.Shell \ --object-path /org/gnome/Shell/Screenshot \ --method org.gnome.Shell.Screenshot.ScreenshotWindow \ false false false "<output_path>" ``` Args: `include_frame=false`, `include_cursor=false`, `flash=false`, `filename`. Captures the currently focused window. Works on GNOME X11 and Wayland. - [ ] **Fallback**: `gnome-screenshot --window --file=<output_path>` if the D-Bus call fails (older GNOME install, stripped Shell, etc.). - [ ] If both methods fail or `gdbus`/`gnome-screenshot` are not installed, log a one-line warning to stderr (`screenshot capture skipped: GNOME Shell screenshot interface unavailable (gdbus rc=N, gnome-screenshot rc=N)`) and continue. **Do not fall back to full-screen.** ### Session detection & gating - [ ] At runner startup (when `screenshot_on_failure` is enabled), probe once: - Is `gdbus` on `PATH`? - Does `gdbus introspect --session --dest org.gnome.Shell --object-path /org/gnome/Shell/Screenshot` succeed? - If neither D-Bus nor `gnome-screenshot` is available, log a one-line warning at startup and disable screenshot capture for the run. - [ ] Detect non-GNOME sessions via `XDG_CURRENT_DESKTOP` (`GNOME`, `GNOME-Classic`, `ubuntu:GNOME`, etc.). On other desktops, log a one-line warning at startup naming the detected desktop and disable screenshot capture for the run. - [ ] All capture commands inherit `DISPLAY` / `WAYLAND_DISPLAY` / `XDG_RUNTIME_DIR` / `DBUS_SESSION_BUS_ADDRESS` from the same env the harness child sees. ### Output - [ ] Filename pattern: `{screenshot_dir}/{timestamp}_{test_name_sanitized}.png` where `timestamp` is `YYYYMMDD-HHMMSS-mmm` (millisecond precision to avoid collisions in fast suites) and `test_name_sanitized` replaces non-alphanumerics with `_`. - [ ] The directory is created if it doesn't exist (`fs::create_dir_all`). - [ ] Captured path stored on `TestCaseResult.screenshot: Option<PathBuf>`. `None` means capture was attempted but skipped or failed. ### Reporter integration - [ ] **Console reporter (#13)**: when a test fails and a screenshot was captured, append a line below the failure block: ` Screenshot: ./test-screenshots/20260411-103045-123_generation_cancel.png` - [ ] **JSON reporter (#14)**: include `"screenshot": "<path>"` in the test object when captured (omitted otherwise). - [ ] **JUnit reporter (#14)**: emit the screenshot path inside the `<failure>` body and as a `<system-out>` element so CI dashboards (Jenkins, GitLab) pick it up as an artifact. ### Tests - [ ] Self-test verifies the **happy path** with a fake `gdbus` script that `touch`es a PNG when called with the expected args; assert path-storage and reporter integration. - [ ] Self-test verifies the **D-Bus-fails-fallback-succeeds** path: fake `gdbus` returns non-zero, fake `gnome-screenshot` produces the file, capture still succeeds. - [ ] Self-test verifies the **both-fail** path: both fakes return non-zero, run completes, warning is logged, `screenshot` on the test result is `None`. - [ ] Self-test verifies the **non-GNOME desktop** path: stub `XDG_CURRENT_DESKTOP=KDE`, confirm capture is disabled at startup with a warning and never attempted. - [ ] Self-test verifies the **timeout**: a fake `gdbus` script that `sleep 30`s gets killed within ~5 s and the test still completes. ## Documentation requirement The README and the `RunConfig.screenshot_on_failure` doc comment must clearly state: 1. **GNOME-only** for v0.1 (other desktops will be added based on demand). 2. **Captures the focused window** — assumes the app under test holds focus at failure time. Tests that switch focus to another app mid-run will capture the wrong window. 3. **Wayland and X11 GNOME sessions** are both supported via the same D-Bus interface. ## Out of scope (still deferred) - Sway, Hyprland, KDE, generic X11 — to be added in later releases if needed. - PID → window resolution (GNOME doesn't expose this without an extension; focused-window is the pragmatic substitute). - Bringing the app's window to focus before capture. - Video recording / animated diagnostics. - Screenshot capture on success (would balloon disk usage). - Automatic upload to S3/artifact stores. The path is surfaced; uploading is the CI's job. - Comparing screenshots against baselines (snapshot testing — listed under spec §11 future extensions). - Full-screen capture — explicitly excluded per project decision. ## References - Spec §11 (mentions snapshot testing as a future extension; this is the lighter-weight cousin) - Issue #2 — already forwards `DISPLAY` / `XDG_RUNTIME_DIR` and exposes the child PID - Issue #13 — console reporter integration - Issue #14 — JSON / JUnit reporter integration - Issue #15 — CLI flags - GNOME Shell DBus interface: https://gitlab.gnome.org/GNOME/gnome-shell/-/blob/main/data/dbus-interfaces/org.gnome.Shell.Screenshot.xml
charles added this to the v0.1.0 milestone 2026-04-11 11:01:11 +00:00
charles changed title from Optional screenshot capture on test failure for GUI apps under test to Screenshot capture on test failure (app-window-only) for GUI apps under test 2026-04-11 11:05:53 +00:00
charles changed title from Screenshot capture on test failure (app-window-only) for GUI apps under test to Screenshot capture on test failure (app-window only) 2026-04-11 11:08:36 +00:00
charles changed title from Screenshot capture on test failure (app-window only) to Screenshot capture on test failure (GNOME, focused-window) 2026-04-11 11:10:11 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
area:assertions
Assertion macros and helpers

  
area:cli
CLI argument parsing

  
area:client
RpcClient (WebSocket, JSON-RPC, events)

  
area:harness
Process harness (spawn, health, shutdown)

  
area:meta
Project setup, examples, self-tests

  
area:reporting
Console, JSON, JUnit reporters

  
area:runner
Test runner (registration, execution, hooks)

  
type:user-story
User story

No labels
area:assertions
area:cli
area:client
area:harness
area:meta
area:reporting
area:runner
type:user-story
Milestone
Clear milestone
No items
No milestone
v0.1.0
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
charles/ws-rpc-test#18
No description provided.