charles/ws-rpc-test
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(runner): name/tag filter, skip_if, fail-fast (#11) #29

Merged
charles merged 2 commits from feat/11-filter-skip-failfast into main 2026-04-11 18:43:29 +00:00
Conversation 2 Commits 2 Files changed 1 +312 -10
charles commented 2026-04-11 12:47:05 +00:00
Owner
Copy link

Closes #11. Stacked on #28.

Summary

Adds everything the runner needs to run a subset of the suite and bail early on failure.

Filtering

  • runner.filter("gallery*") — glob-like pattern over test names. Supports exact, prefix*, *suffix, *contains*, prefix*suffix, and *. Implemented inline with starts_with / ends_with / contains — no regex dep.
  • runner.filter_tag("backend") — exact tag match.
  • Name + tag is AND, not OR (per issue).
  • Filtered-out tests go to report.filtered and produce no output line.

Skip

  • Runtime: already supported (Err(TestError::Skip("reason")) since #9) — re-exercised by new tests.
  • Declarative: runner.test(...).skip_if(|| cond_at_runtime, "reason"). Condition is a Fn() -> bool + Send + Sync + 'static evaluated at run-time (not registration) so it can read env / fs state. When true, hooks and body are skipped, result is Skipped(reason).

Fail-fast

  • runner.fail_fast(true) — after the first Failed test, remaining tests are reported as Skipped with skip_reason = "fail-fast". after_each still runs for the failing test; after_all still runs at the end.

Precedence inside the run loop

  1. filter (name + tag) — filtered-out never reach the hook layer
  2. fail-fast short-circuit — recorded as Skipped("fail-fast")
  3. skip_if — recorded as Skipped(reason)
  4. before_each → body → after_each

Checklist (from issue #11)

Filtering

  • RunConfig.filter: Option<String> glob
  • RunConfig.tag: Option<String> exact
  • --filter + --tag is AND (wiring to CLI happens in #15)
  • Filtered tests counted in report.filtered, produce no output
  • Filter logic unit-tested without spinning up the harness

Skip

  • Runtime via Err(TestError::Skip(...))
  • Declarative via .skip_if(|| cond, "reason") evaluated at run time

Fail-fast

  • After first failure, remainder reported as Skipped("fail-fast") (not silently dropped)
  • after_each / after_all still run per the existing hook contract

Test plan (11 new tests)

  • glob_match_{exact,prefix,suffix,surround,middle,star_all} — 6 pure-function tests, no fixture needed
  • filter_by_name_skips_non_matches_counted_as_filtered
  • filter_by_tag_and_name_uses_and_semantics — 3 tests; only one matches both filters
  • skip_if_true_skips_test_at_runtime — asserts reason preserved
  • skip_if_false_runs_test_normally
  • fail_fast_skips_remaining_tests — 4 tests, second fails, last two recorded as Skipped("fail-fast")
  • just qa green locally — 85/85 unit tests
  • CI green on Forgejo

Notes for the reviewer

  • The glob matcher doesn't support multi-star patterns beyond the five shapes listed. Anything more exotic should trigger a second look at whether we really need regex support in v0.1 — deferred for now, keeps the dep list small.
  • TestCase.skip_condition is stored as Option<(Box<dyn Fn() -> bool + Send + Sync>, String)>. The Fn-pointer trick means .skip_if closures are 'static + Send + Sync; that matches the rest of the runner's closure constraints.
  • RunConfig::Default::default() is now a derive, with a separate with_defaults() helper that sets the 60 s timeout. Calling TestRunner::new() always uses with_defaults(); derive-default is only for callers who want the zero-field-config shape.
  • skip_if evaluating before fail_fast would be the other reasonable ordering. I put fail-fast first because it represents "the whole run is done" whereas skip_if is per-test — the user probably wants fail-fast to win in all cases.
Closes #11. **Stacked on #28.** ## Summary Adds everything the runner needs to run a subset of the suite and bail early on failure. ### Filtering - `runner.filter("gallery*")` — glob-like pattern over test names. Supports `exact`, `prefix*`, `*suffix`, `*contains*`, `prefix*suffix`, and `*`. Implemented inline with `starts_with` / `ends_with` / `contains` — **no regex dep**. - `runner.filter_tag("backend")` — exact tag match. - Name + tag is **AND**, not OR (per issue). - Filtered-out tests go to `report.filtered` and produce no output line. ### Skip - **Runtime**: already supported (`Err(TestError::Skip("reason"))` since #9) — re-exercised by new tests. - **Declarative**: `runner.test(...).skip_if(|| cond_at_runtime, "reason")`. Condition is a `Fn() -> bool + Send + Sync + 'static` evaluated at run-time (not registration) so it can read env / fs state. When true, hooks and body are skipped, result is `Skipped(reason)`. ### Fail-fast - `runner.fail_fast(true)` — after the first `Failed` test, remaining tests are reported as `Skipped` with `skip_reason = "fail-fast"`. `after_each` still runs for the failing test; `after_all` still runs at the end. ### Precedence inside the run loop 1. filter (name + tag) — filtered-out never reach the hook layer 2. fail-fast short-circuit — recorded as Skipped("fail-fast") 3. `skip_if` — recorded as Skipped(reason) 4. `before_each` → body → `after_each` ## Checklist (from issue #11) ### Filtering - [x] `RunConfig.filter: Option<String>` glob - [x] `RunConfig.tag: Option<String>` exact - [x] `--filter` + `--tag` is AND (wiring to CLI happens in #15) - [x] Filtered tests counted in `report.filtered`, produce no output - [x] Filter logic unit-tested without spinning up the harness ### Skip - [x] Runtime via `Err(TestError::Skip(...))` - [x] Declarative via `.skip_if(|| cond, "reason")` evaluated at run time ### Fail-fast - [x] After first failure, remainder reported as `Skipped("fail-fast")` (not silently dropped) - [x] `after_each` / `after_all` still run per the existing hook contract ## Test plan (11 new tests) - [x] `glob_match_{exact,prefix,suffix,surround,middle,star_all}` — 6 pure-function tests, no fixture needed - [x] `filter_by_name_skips_non_matches_counted_as_filtered` - [x] `filter_by_tag_and_name_uses_and_semantics` — 3 tests; only one matches both filters - [x] `skip_if_true_skips_test_at_runtime` — asserts reason preserved - [x] `skip_if_false_runs_test_normally` - [x] `fail_fast_skips_remaining_tests` — 4 tests, second fails, last two recorded as `Skipped("fail-fast")` - [x] `just qa` green locally — 85/85 unit tests - [ ] CI green on Forgejo ## Notes for the reviewer - The glob matcher doesn't support multi-star patterns beyond the five shapes listed. Anything more exotic should trigger a second look at whether we really need regex support in v0.1 — deferred for now, keeps the dep list small. - `TestCase.skip_condition` is stored as `Option<(Box<dyn Fn() -> bool + Send + Sync>, String)>`. The Fn-pointer trick means `.skip_if` closures are `'static` + `Send + Sync`; that matches the rest of the runner's closure constraints. - `RunConfig::Default::default()` is now a derive, with a separate `with_defaults()` helper that sets the 60 s timeout. Calling `TestRunner::new()` always uses `with_defaults()`; derive-default is only for callers who want the zero-field-config shape. - `skip_if` evaluating before `fail_fast` would be the other reasonable ordering. I put fail-fast first because it represents "the whole run is done" whereas `skip_if` is per-test — the user probably wants fail-fast to win in all cases. 
Implements ticket #11 on top of #10's hook lifecycle.

Filtering (RunConfig.filter / RunConfig.tag)
- runner.filter("gallery*") sets a glob-like pattern over test names.
  Supported shapes: exact, "prefix*", "*suffix", "*contains*",
  "prefix*suffix", and "*". Implemented inline (no regex dep) via
  split_*/starts_with/ends_with.
- runner.filter_tag("backend") sets an exact tag match.
- Combining filter + tag is AND (per issue acceptance).
- Filtered-out tests are counted in TestReport.filtered and produce
  no output line.
- 6 pure-function unit tests cover the glob matcher.

Runtime skip (TestError::Skip)
- Already supported since #9; re-exercised here by tests.

Declarative skip (.skip_if)
- TestCase now has an optional (cond, reason) pair. cond: Fn() -> bool
  is evaluated at run-time (not registration) so it can observe env
  state. When true, the test is recorded as Skipped with the reason;
  hooks and body are NOT executed.

Fail-fast (RunConfig.fail_fast)
- runner.fail_fast(true) causes remaining tests (after the first
  Failed result) to be reported as Skipped with reason "fail-fast"
  rather than silently dropped.

Precedence inside the run loop
  1. filter (name + tag) — filtered-out never reach the hook layer
  2. fail-fast short-circuit
  3. skip_if
  4. before_each → body → after_each

Tests (11 new in src/runner.rs)
- glob_match_exact / _prefix / _suffix / _surround / _middle / _star_all
- filter_by_name_skips_non_matches_counted_as_filtered
- filter_by_tag_and_name_uses_and_semantics
- skip_if_true_skips_test_at_runtime (asserts reason preserved)
- skip_if_false_runs_test_normally
- fail_fast_skips_remaining_tests (4 tests, fail-fast kicks in after
  the second, remaining two recorded as Skipped("fail-fast"))

just qa green: 85/85 unit tests, fmt and clippy -D warnings clean.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
charles reviewed 2026-04-11 17:54:28 +00:00
charles left a comment
Copy link

Review — filter, skip_if, fail-fast (#11)

🐛 Bug : RunConfig::default() a un timeout de 0s

La PR a remplacé l'impl Default for RunConfig manuel (qui retournait timeout: DEFAULT_TEST_TIMEOUT) par #[derive(Default)]. Or Duration::default() == Duration::ZERO, donc :

RunConfig::default().timeout == Duration::ZERO  // chaque test timeout immédiatement !

TestRunner::new() est correct (il appelle RunConfig::with_defaults()), mais TestRunner::default() — qui est #[derive(Default)] — crée un RunConfig::default() avec un timeout nul. Tout utilisateur qui écrit :

let mut runner = TestRunner::default();
runner.test("foo", |_| async { Ok(()) });
runner.run().await  // → chaque test fail avec Timeout { event: "test", duration: 0s }

obtiendra des échecs silencieux immédiats.

Fix suggéré : supprimer #[derive(Default)] sur RunConfig et réimplémenter manuellement pour déléguer à with_defaults() :

impl Default for RunConfig {
    fn default() -> Self { Self::with_defaults() }
}

Ça rend with_defaults() redondant (on peut le garder comme alias ou le supprimer).

glob_match — bien

Implémentation O(n), sans dépendance, supporte les patterns courants documentés. La limitation multi-* est explicite.

skip_if — évaluation lazy

La condition est évaluée à l'exécution (pas à l'enregistrement) — correct et documenté.

RunConfig::name_matches + tag_matches

Séparation propre des deux filtres. Sémantique AND (les deux doivent passer) — conforme au test filter_by_tag_and_name_uses_and_semantics.

Tests glob

Coverage complète : exact, prefix, suffix, surround, middle, *-all — excellent.

Bloquant : timeout 0s via RunConfig::default().

## Review — filter, skip_if, fail-fast (#11) ### 🐛 Bug : `RunConfig::default()` a un timeout de 0s La PR a remplacé l'`impl Default for RunConfig` manuel (qui retournait `timeout: DEFAULT_TEST_TIMEOUT`) par `#[derive(Default)]`. Or `Duration::default() == Duration::ZERO`, donc : ```rust RunConfig::default().timeout == Duration::ZERO // chaque test timeout immédiatement ! ``` `TestRunner::new()` est correct (il appelle `RunConfig::with_defaults()`), mais `TestRunner::default()` — qui est `#[derive(Default)]` — crée un `RunConfig::default()` avec un timeout nul. Tout utilisateur qui écrit : ```rust let mut runner = TestRunner::default(); runner.test("foo", |_| async { Ok(()) }); runner.run().await // → chaque test fail avec Timeout { event: "test", duration: 0s } ``` obtiendra des échecs silencieux immédiats. **Fix suggéré** : supprimer `#[derive(Default)]` sur `RunConfig` et réimplémenter manuellement pour déléguer à `with_defaults()` : ```rust impl Default for RunConfig { fn default() -> Self { Self::with_defaults() } } ``` Ça rend `with_defaults()` redondant (on peut le garder comme alias ou le supprimer). --- ### `glob_match` — bien Implémentation O(n), sans dépendance, supporte les patterns courants documentés. La limitation multi-`*` est explicite. ### `skip_if` — évaluation lazy La condition est évaluée à l'exécution (pas à l'enregistrement) — correct et documenté. ### `RunConfig::name_matches` + `tag_matches` Séparation propre des deux filtres. Sémantique AND (les deux doivent passer) — conforme au test `filter_by_tag_and_name_uses_and_semantics`. ### Tests glob Coverage complète : exact, prefix, suffix, surround, middle, `*`-all — excellent. **Bloquant : timeout 0s via `RunConfig::default()`.**
charles reviewed 2026-04-11 17:55:24 +00:00
charles left a comment
Copy link

🐛 Bug bloquant : RunConfig::default() retourne maintenant timeout: Duration::ZERO (0 secondes) suite au passage de impl Default manuel à #[derive(Default)]. TestRunner::default() hérite de ce bug — chaque test timeout immédiatement. Fix : réimplémenter Default for RunConfig manuellement en déléguant à with_defaults().

🐛 **Bug bloquant** : `RunConfig::default()` retourne maintenant `timeout: Duration::ZERO` (0 secondes) suite au passage de `impl Default` manuel à `#[derive(Default)]`. `TestRunner::default()` hérite de ce bug — chaque test timeout immédiatement. Fix : réimplémenter `Default for RunConfig` manuellement en déléguant à `with_defaults()`.
#[derive(Default)] gives Duration::ZERO for timeout fields.
Delegate to with_defaults() which sets DEFAULT_TEST_TIMEOUT (60 s).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
charles changed target branch from feat/10-runner-hooks to main 2026-04-11 18:43:21 +00:00
charles deleted branch feat/11-filter-skip-failfast 2026-04-11 18:43:29 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
area:assertions
Assertion macros and helpers

  
area:cli
CLI argument parsing

  
area:client
RpcClient (WebSocket, JSON-RPC, events)

  
area:harness
Process harness (spawn, health, shutdown)

  
area:meta
Project setup, examples, self-tests

  
area:reporting
Console, JSON, JUnit reporters

  
area:runner
Test runner (registration, execution, hooks)

  
type:user-story
User story

No labels
area:assertions
area:cli
area:client
area:harness
area:meta
area:reporting
area:runner
type:user-story
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
charles/ws-rpc-test!29
No description provided.