charles/ws-rpc-test
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(runner): TestRunner registration + sequential execution (#9) #27

Merged
charles merged 1 commit from feat/9-runner-sequential into main 2026-04-11 18:43:13 +00:00
Conversation 2 Commits 1 Files changed 1 +474 -6
charles commented 2026-04-11 12:41:20 +00:00
Owner
Copy link

Closes #9. Stacked on #26.

Summary

Baseline test runner: register closures, start harness, connect client, execute sequentially, report. Hooks (#10), filtering/skip/fail-fast (#11), parallel mode (#12), and the coloured reporter (#13) arrive in later PRs.

API

let mut runner = TestRunner::new(harness);
runner.test("health check", |c| async move {
    let r = c.call("health_check", json!({})).await?;
    assert_ok!(r);
    Ok(())
}).tag("read-only");

let report = runner.run().await?;

Internals

  • TestCase { name, tags, body } where body is stored as Box<dyn Fn(Arc<RpcClient>) -> Pin<Box<dyn Future<Output = TestResult> + Send + 'static>> + Send + Sync>.
  • TestRunner.test(...) returns &mut TestCase so chaining .tag("foo") works.
  • run() starts the harness, connects the client, iterates tests sequentially with tokio::time::timeout(config.timeout, ...), records results, stops the harness.
  • Per-test outcomes:
    • Ok(()) → Passed
    • Err(TestError::Skip(reason)) → Skipped (reason preserved)
    • Any other Err → Failed with the error
    • Timeout elapsed → Failed with TestError::Timeout { event: "test", duration }
  • RunConfig { timeout: 60s default } — filtering / format / parallel fields land in their own tickets.
  • Minimal plain-text console output for now; full colourised reporter is #13.

Checklist (from issue #9)

Registration

  • TestRunner::new(harness) takes ownership
  • runner.test(name, closure) returns &mut TestCase for chaining
  • Closure: Arc<RpcClient> → impl Future<Output = TestResult>, 'static + Send

Run

  • Starts harness → connects client → runs tests → stops harness → returns TestReport
  • Harness / connect failure: prints error, returns early, no tests run
  • Per-test timeout (RunConfig.timeout, default 60 s)

Per-test outcome

  • Ok(()) → passed
  • Err(TestError::Skip(_)) → skipped with reason
  • Anything else → failed with preserved error

Report

  • TestReport { tests, passed, failed, skipped, filtered, duration }
  • TestCaseResult { name, tags, status, duration, error, skip_reason }
  • Per-test timing collected

Test plan (7 new tests)

  • runner_registers_and_exposes_tags
  • runner_executes_passing_test — in-process HTTP + WS echo fixtures, asserts passed == 1
  • runner_reports_failure_from_err — test returns TestError::Assertion, asserts status + error
  • runner_reports_skip_from_skip_variant — test returns TestError::Skip("requires network"), asserts reason
  • runner_applies_per_test_timeout — 100 ms timeout, test sleeps 2 s, asserts TestError::Timeout { event: "test" }
  • runner_returns_err_when_harness_fails_to_start — closed port, 300 ms startup deadline, asserts TestError::Connection
  • runner_runs_multiple_tests_in_order — three tests, asserts ordering
  • just qa green locally — 69/69 unit tests
  • CI green on Forgejo

Notes for the reviewer

  • The in-process fixtures (spawn_health_server, spawn_ws_echo_server) duplicate some code from harness.rs / client.rs tests. That's intentional for now — extracting a shared test_utils module would be its own refactor. All three test modules will converge on the mock server from ticket #16.
  • TestRunner::default() is now a real derived impl (clippy caught the manual one earlier). ProcessHarness::default(), RunConfig::default(), and Vec::default() all derive-compose cleanly.
  • The tag() method on TestCase returns &mut Self so additional chained tags compose naturally.
Closes #9. **Stacked on #26.** ## Summary Baseline test runner: register closures, start harness, connect client, execute sequentially, report. Hooks (#10), filtering/skip/fail-fast (#11), parallel mode (#12), and the coloured reporter (#13) arrive in later PRs. ### API ```rust let mut runner = TestRunner::new(harness); runner.test("health check", |c| async move { let r = c.call("health_check", json!({})).await?; assert_ok!(r); Ok(()) }).tag("read-only"); let report = runner.run().await?; ``` ### Internals - `TestCase { name, tags, body }` where `body` is stored as `Box<dyn Fn(Arc<RpcClient>) -> Pin<Box<dyn Future<Output = TestResult> + Send + 'static>> + Send + Sync>`. - `TestRunner.test(...)` returns `&mut TestCase` so chaining `.tag("foo")` works. - `run()` starts the harness, connects the client, iterates tests sequentially with `tokio::time::timeout(config.timeout, ...)`, records results, stops the harness. - Per-test outcomes: - `Ok(())` → `Passed` - `Err(TestError::Skip(reason))` → `Skipped` (reason preserved) - Any other `Err` → `Failed` with the error - Timeout elapsed → `Failed` with `TestError::Timeout { event: "test", duration }` - `RunConfig { timeout: 60s default }` — filtering / format / parallel fields land in their own tickets. - Minimal plain-text console output for now; full colourised reporter is #13. ## Checklist (from issue #9) ### Registration - [x] `TestRunner::new(harness)` takes ownership - [x] `runner.test(name, closure)` returns `&mut TestCase` for chaining - [x] Closure: `Arc<RpcClient> → impl Future<Output = TestResult>`, `'static + Send` ### Run - [x] Starts harness → connects client → runs tests → stops harness → returns `TestReport` - [x] Harness / connect failure: prints error, returns early, no tests run - [x] Per-test timeout (`RunConfig.timeout`, default 60 s) ### Per-test outcome - [x] `Ok(())` → passed - [x] `Err(TestError::Skip(_))` → skipped with reason - [x] Anything else → failed with preserved error ### Report - [x] `TestReport { tests, passed, failed, skipped, filtered, duration }` - [x] `TestCaseResult { name, tags, status, duration, error, skip_reason }` - [x] Per-test timing collected ## Test plan (7 new tests) - [x] `runner_registers_and_exposes_tags` - [x] `runner_executes_passing_test` — in-process HTTP + WS echo fixtures, asserts `passed == 1` - [x] `runner_reports_failure_from_err` — test returns `TestError::Assertion`, asserts status + error - [x] `runner_reports_skip_from_skip_variant` — test returns `TestError::Skip("requires network")`, asserts reason - [x] `runner_applies_per_test_timeout` — 100 ms timeout, test sleeps 2 s, asserts `TestError::Timeout { event: "test" }` - [x] `runner_returns_err_when_harness_fails_to_start` — closed port, 300 ms startup deadline, asserts `TestError::Connection` - [x] `runner_runs_multiple_tests_in_order` — three tests, asserts ordering - [x] `just qa` green locally — 69/69 unit tests - [ ] CI green on Forgejo ## Notes for the reviewer - The in-process fixtures (`spawn_health_server`, `spawn_ws_echo_server`) duplicate some code from `harness.rs` / `client.rs` tests. That's intentional for now — extracting a shared `test_utils` module would be its own refactor. All three test modules will converge on the mock server from ticket #16. - `TestRunner::default()` is now a real derived impl (clippy caught the manual one earlier). `ProcessHarness::default()`, `RunConfig::default()`, and `Vec::default()` all derive-compose cleanly. - The `tag()` method on `TestCase` returns `&mut Self` so additional chained tags compose naturally. 
Implements ticket #9. The runner now:

- Accepts test closures via runner.test(name, |client| async { ... })
  returning &mut TestCase for chaining (TestCase.tag("backend")).
- The closure signature is
    F: Fn(Arc<RpcClient>) -> impl Future<Output = TestResult> + Send + 'static
  stored as Box<dyn Fn(...) -> Pin<Box<dyn Future<...>>>>.
- runner.run() starts the harness, connects an Arc<RpcClient> to its
  ws_url, runs each registered test sequentially with a per-test
  timeout (RunConfig.timeout, default 60 s), stops the harness, and
  returns a TestReport.
- Per-test outcomes:
    Ok(())                         → TestStatus::Passed
    Err(TestError::Skip(reason))   → TestStatus::Skipped (reason preserved)
    Err(other)                     → TestStatus::Failed (error preserved)
    timeout                        → TestStatus::Failed with
                                     TestError::Timeout { event: "test", duration }
- Harness start failure or RpcClient connect failure short-circuits
  with an early Err and no tests run. The harness is stopped cleanly
  in both the happy and error paths.
- Minimal console output for now: one line per test with PASS/FAIL/SKIP,
  timing, and an indented error message. The full coloured Reporter
  lands in #13.

Types
- TestCase { name, tags, body }
- RunConfig { timeout }
- TestStatus { Passed, Failed, Skipped }
- TestCaseResult { name, tags, status, duration, error, skip_reason }
- TestReport { tests, passed, failed, skipped, filtered, duration }
- TestReport::record() increments counters and pushes the result.

Tests (7 in src/runner.rs)
- Small in-process fixtures: spawn_health_server (raw HTTP 200) and
  spawn_ws_echo_server (tokio-tungstenite echo responder).
- runner_registers_and_exposes_tags
- runner_executes_passing_test (against the echo fixture)
- runner_reports_failure_from_err
- runner_reports_skip_from_skip_variant
- runner_applies_per_test_timeout (100 ms deadline, test sleeps 2 s)
- runner_returns_err_when_harness_fails_to_start (closed port, 300 ms
  startup deadline)
- runner_runs_multiple_tests_in_order

just qa green: 69/69 unit tests, fmt and clippy -D warnings clean.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
charles reviewed 2026-04-11 17:53:33 +00:00
charles left a comment
Copy link

Review — TestRunner registration + exécution séquentielle (#9)

Design général

  • BoxedTestFn = Box<dyn Fn(...)> — le corps de test est Fn (pas FnOnce), ce qui le rend réutilisable. Les tests pourraient théoriquement être re-run. C'est un choix délibéré ou une conséquence de l'API de la PR#12 (parallel) qui drain les tests ? À confirmer.
  • test() retourne &mut TestCase pour permettre le chaînage .tag(...) — API ergonomique.

run() — inline println!

La sortie console est gérée directement dans run() avec des println!. Elle sera remplacée par ConsoleReporter en PR#13. Pas de problème puisque c'est une PR intermédiaire.

run_one — gestion du TestError::Skip

Ok(Err(TestError::Skip(reason))) => TestCaseResult { status: Skipped, ... }

Élégant — l'utilisateur peut retourner Err(TestError::Skip(...)) depuis son corps de test pour un skip runtime. Compatible avec skip_if qui arrive en #11.

Tests

  • runner_applies_per_test_timeout avec un sleep de 2s dans le corps — test direct de l'enforcement du timeout.
  • runner_returns_err_when_harness_fails_to_start — vérifie que le client n'essaie pas de se connecter si le harness échoue.
  • Bon ordre de couverture bottom-up.

Aucun bloquant.

## Review — TestRunner registration + exécution séquentielle (#9) ### Design général - `BoxedTestFn = Box<dyn Fn(...)>` — le corps de test est `Fn` (pas `FnOnce`), ce qui le rend réutilisable. Les tests pourraient théoriquement être re-run. C'est un choix délibéré ou une conséquence de l'API de la PR#12 (parallel) qui drain les tests ? À confirmer. - `test()` retourne `&mut TestCase` pour permettre le chaînage `.tag(...)` — API ergonomique. ### `run()` — inline `println!` La sortie console est gérée directement dans `run()` avec des `println!`. Elle sera remplacée par `ConsoleReporter` en PR#13. Pas de problème puisque c'est une PR intermédiaire. ### `run_one` — gestion du `TestError::Skip` ```rust Ok(Err(TestError::Skip(reason))) => TestCaseResult { status: Skipped, ... } ``` Élégant — l'utilisateur peut retourner `Err(TestError::Skip(...))` depuis son corps de test pour un skip runtime. Compatible avec `skip_if` qui arrive en #11. ### Tests - `runner_applies_per_test_timeout` avec un sleep de 2s dans le corps — test direct de l'enforcement du timeout. - `runner_returns_err_when_harness_fails_to_start` — vérifie que le client n'essaie pas de se connecter si le harness échoue. - Bon ordre de couverture bottom-up. Aucun bloquant.
charles reviewed 2026-04-11 17:53:45 +00:00
charles left a comment
Copy link

✅ Pas de bloquant — runner séquentiel propre, gestion du Skip élégante via TestError::Skip.

✅ Pas de bloquant — runner séquentiel propre, gestion du Skip élégante via `TestError::Skip`.
charles changed target branch from feat/8-assertion-macros to main 2026-04-11 18:43:05 +00:00
charles deleted branch feat/9-runner-sequential 2026-04-11 18:43:13 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
area:assertions
Assertion macros and helpers

  
area:cli
CLI argument parsing

  
area:client
RpcClient (WebSocket, JSON-RPC, events)

  
area:harness
Process harness (spawn, health, shutdown)

  
area:meta
Project setup, examples, self-tests

  
area:reporting
Console, JSON, JUnit reporters

  
area:runner
Test runner (registration, execution, hooks)

  
type:user-story
User story

No labels
area:assertions
area:cli
area:client
area:harness
area:meta
area:reporting
area:runner
type:user-story
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
charles/ws-rpc-test!27
No description provided.