TestRunner: filtering, skip mechanism, fail-fast

charles commented

2026-04-11 10:55:39 +00:00

Owner

User story

As a test author, I want to run a subset of tests by name or tag, skip tests at runtime or registration, and bail on the first failure when iterating, so that the inner-loop feedback cycle stays fast.

Acceptance criteria

Filtering

RunConfig.filter: Option<String> — glob-like matcher (* wildcard) against test names. Empty/None means no filter.
RunConfig.tag: Option<String> — exact tag match against TestCase.tags.
Combining --filter and --tag is AND, not OR.
Filtered-out tests do not appear in run output but are counted in report.filtered.
Filter logic is unit-tested without spinning up the harness.

Skip

A test can return Err(TestError::Skip("reason".into())) from its body to skip itself at runtime. (Resolves spec review §7.)
Declarative skip at registration: runner.test(...).skip_if(|| condition_bool, "reason"). The condition is evaluated at run time (not registration time).
Skipped tests show up in the report with their reason.

Fail-fast

RunConfig.fail_fast: bool — stops execution after the first failure.
Remaining tests are reported as skipped with reason "fail-fast" (not silently dropped).
after_each for the failing test still runs; after_all still runs.

References

Spec §4.2 (filtering, fail-fast); spec review §7 (skip)

## User story As a **test author**, I want to run a subset of tests by name or tag, skip tests at runtime or registration, and bail on the first failure when iterating, so that the inner-loop feedback cycle stays fast. ## Acceptance criteria ### Filtering - [ ] `RunConfig.filter: Option<String>` — glob-like matcher (`*` wildcard) against test names. Empty/None means no filter. - [ ] `RunConfig.tag: Option<String>` — exact tag match against `TestCase.tags`. - [ ] Combining `--filter` and `--tag` is **AND**, not OR. - [ ] Filtered-out tests do not appear in run output but are counted in `report.filtered`. - [ ] Filter logic is unit-tested without spinning up the harness. ### Skip - [ ] A test can return `Err(TestError::Skip("reason".into()))` from its body to skip itself at runtime. *(Resolves spec review §7.)* - [ ] Declarative skip at registration: `runner.test(...).skip_if(|| condition_bool, "reason")`. The condition is evaluated at run time (not registration time). - [ ] Skipped tests show up in the report with their reason. ### Fail-fast - [ ] `RunConfig.fail_fast: bool` — stops execution after the first failure. - [ ] Remaining tests are reported as `skipped` with reason `"fail-fast"` (not silently dropped). - [ ] `after_each` for the failing test still runs; `after_all` still runs. ## References - Spec §4.2 (filtering, fail-fast); spec review §7 (skip)

charles added this to the v0.1.0 milestone

2026-04-11 10:55:39 +00:00

charles added the

area:runner

type:user-story

labels

2026-04-11 10:55:39 +00:00

charles referenced this issue

2026-04-11 11:58:50 +00:00

feat: bootstrap crate skeleton, deps, and TestError (#1) #19

charles referenced this issue

2026-04-11 12:41:20 +00:00