Flows YAML — engine + ops (loader, expression, registry, executor, ops) #1075

Closed
opened 2026-05-10 15:42:51 +00:00 by claude-desktop · 0 comments
Collaborator

User story

As a maintainer, I want a complete YAML flow engine — file loader, expression language, op registry, linear executor, internal triggers, and the full set of operations — so that the new dispatch path can run any of the 9 default flows end-to-end behind a feature flag.

Resolves audit findings §1.1.1 (flow + legacy collision), §1.1.2 (injection bundle bloat), §1.1.3 (DAG model unused), §1.1.4 (string-based filters), §1.1.6 (no async hooks).

Acceptance criteria

Loader + file watcher

  • apps/server/src/domain/flows-yaml/loader.ts parses every *.yml under flows/defaults/ and flows/custom/ at boot; custom shadows defaults by name.
  • Parse failures logged with path + YAML error; bad file skipped, last-good kept active.
  • chokidar (or bun fs.watch) with 200ms debounce; emits flow.changed { name } on add/change/unlink.
  • POST /flows/reload forces full re-scan, returns { loaded, errors[] }.

Expression language

  • Hand-rolled recursive-descent parser in flows-yaml/expr/parser.ts.
  • Operators: &&, ||, !, ==, !=, in, matches, <, <=, >, >=. Literals: string, number, bool, null. Member access. Function call.
  • Evaluator with short-circuit, typed errors with line+column.
  • Built-in functions: has_label, has_any_label, has_all_labels, author_is, repo_matches, comment_matches, event_action.
  • Built-in variables: event.*, steps.<id>.outputs.<key>, env.<KEY>.
  • ${{ expr }} interpolation in with: arg values; whole-string returns native type, embedded stringifies.

Op registry

  • Operation type: name, argsSchema (Zod), outputsSchema (Zod), run(ctx, args), deps (capability keys).
  • One file per op under flows-yaml/ops/<name>.ts; index aggregates.
  • buildOpContext(op, env) returns only declared capabilities; tests substitute fakes per op.
  • Output validated against outputsSchema before storing under steps.<id>.outputs.

Linear executor + parallel block

  • flows-yaml/executor.ts runs steps in order: evaluate if: → resolve interpolations → run op → record flow_node_runs row.
  • Step error fails flow unless continue-on-error: true; if: falseskipped, flow continues.
  • steps[].parallel: [...] runs concurrently with Promise.all; outputs visible via steps.<parent>.outputs.<sub>.….
  • concurrency.group (expression-evaluated) gates with mutex; cancel-in-progress: true aborts in-flight run.
  • Multi-flow match sorted by priority desc then name asc.

Audit

  • flow_runs opens on start, closes on terminal status (completed/failed/cancelled).
  • flow_node_runs rows hold step status, duration, output, error.
  • Drizzle migration renames flow_node_runs.node_*step_*.
  • New column flow_runs.internal_trigger_source records parent flow/task id.

Internal triggers

  • Worker fires task.completed / task.failed / task.timed_out on terminal task state.
  • Executor fires flow.completed / flow.failed.
  • Forge handler fires pr.merged after Forgejo confirms merge (distinct from pull_request.closed).
  • YAML on: { task: [completed], pr: [merged], flow: [failed] } consumes them.
  • Chain-depth counter incremented per re-entry; depth > 4 → log + drop.

Ops to author (one file per op)

  • Forge-event: dispatch, dispatch_breakdown, resolve_agent, dedup_check, dedup_record, dedup_cancel_stale, guard_reviewer_dispatch, guard_author_dispatch, detect_change_request, set_label, remove_label, comment.
  • Imperative-handler ports: cancel_tasks (replaces handleIssueUnassigned at webhook.ts:296), pr_dependency_markers (replaces handlePrDependencyMarkers at webhook.ts:325), propagate_dependencies (replaces handleStackedRebaseCascade at webhook.ts:333 and the legacy handleIssueClosed callback).
  • No op imports the legacy defaultArgInjections bundle.
  • Each op has a .test.ts against a fake ctx; recorded fixture replays produce identical observable side effects to the legacy path.

Tests

  • Loader: parse minimal, parse with overlay, parse with bad YAML, custom overrides default; SSE event fires on file change.
  • Expression: parser round-trip, evaluator short-circuit, type errors, every builtin against synthetic event.
  • Op registry: missing-capability ctx panics with clear message.
  • Executor: sequential order, mid-failure halt, continue-on-error, parallel concurrency, concurrency mutex.
  • Internal triggers: emit on terminal state, depth cap fires drop on 5th hop, on: task.completed flow runs after task.

Out of scope

  • Default YAML files + JSON Schema + REST CRUD (covered by Defaults+API ticket).
  • Editor UI (covered by Editor UI ticket).
  • Shadow-mode + cutover + legacy deletion (covered by Migration ticket).

References

  • Spec: docs/specs/flows-yaml.md §3, §5.2, §6, §7, §8, §9, §10.
  • Replaces: apps/server/src/domain/flows/{flow-dispatch,executor,registry,validation}.ts, inline imperative calls at apps/server/src/http/webhook.ts:296–341, defaultArgInjections at flow-dispatch.ts:113–218, matchesFilters at flow-dispatch.ts:356–407.
## User story As a maintainer, I want a complete YAML flow engine — file loader, expression language, op registry, linear executor, internal triggers, and the full set of operations — so that the new dispatch path can run any of the 9 default flows end-to-end behind a feature flag. Resolves audit findings §1.1.1 (flow + legacy collision), §1.1.2 (injection bundle bloat), §1.1.3 (DAG model unused), §1.1.4 (string-based filters), §1.1.6 (no async hooks). ## Acceptance criteria ### Loader + file watcher - [ ] `apps/server/src/domain/flows-yaml/loader.ts` parses every `*.yml` under `flows/defaults/` and `flows/custom/` at boot; `custom` shadows `defaults` by name. - [ ] Parse failures logged with path + YAML error; bad file skipped, last-good kept active. - [ ] `chokidar` (or bun fs.watch) with 200ms debounce; emits `flow.changed { name }` on add/change/unlink. - [ ] `POST /flows/reload` forces full re-scan, returns `{ loaded, errors[] }`. ### Expression language - [ ] Hand-rolled recursive-descent parser in `flows-yaml/expr/parser.ts`. - [ ] Operators: `&&`, `||`, `!`, `==`, `!=`, `in`, `matches`, `<`, `<=`, `>`, `>=`. Literals: string, number, bool, null. Member access. Function call. - [ ] Evaluator with short-circuit, typed errors with line+column. - [ ] Built-in functions: `has_label`, `has_any_label`, `has_all_labels`, `author_is`, `repo_matches`, `comment_matches`, `event_action`. - [ ] Built-in variables: `event.*`, `steps.<id>.outputs.<key>`, `env.<KEY>`. - [ ] `${{ expr }}` interpolation in `with:` arg values; whole-string returns native type, embedded stringifies. ### Op registry - [ ] `Operation` type: `name`, `argsSchema` (Zod), `outputsSchema` (Zod), `run(ctx, args)`, `deps` (capability keys). - [ ] One file per op under `flows-yaml/ops/<name>.ts`; index aggregates. - [ ] `buildOpContext(op, env)` returns only declared capabilities; tests substitute fakes per op. - [ ] Output validated against `outputsSchema` before storing under `steps.<id>.outputs`. ### Linear executor + parallel block - [ ] `flows-yaml/executor.ts` runs steps in order: evaluate `if:` → resolve interpolations → run op → record `flow_node_runs` row. - [ ] Step error fails flow unless `continue-on-error: true`; `if: false` → `skipped`, flow continues. - [ ] `steps[].parallel: [...]` runs concurrently with `Promise.all`; outputs visible via `steps.<parent>.outputs.<sub>.…`. - [ ] `concurrency.group` (expression-evaluated) gates with mutex; `cancel-in-progress: true` aborts in-flight run. - [ ] Multi-flow match sorted by `priority` desc then `name` asc. ### Audit - [ ] `flow_runs` opens on start, closes on terminal status (`completed`/`failed`/`cancelled`). - [ ] `flow_node_runs` rows hold step status, duration, output, error. - [ ] Drizzle migration renames `flow_node_runs.node_*` → `step_*`. - [ ] New column `flow_runs.internal_trigger_source` records parent flow/task id. ### Internal triggers - [ ] Worker fires `task.completed` / `task.failed` / `task.timed_out` on terminal task state. - [ ] Executor fires `flow.completed` / `flow.failed`. - [ ] Forge handler fires `pr.merged` after Forgejo confirms merge (distinct from `pull_request.closed`). - [ ] YAML `on: { task: [completed], pr: [merged], flow: [failed] }` consumes them. - [ ] Chain-depth counter incremented per re-entry; depth > 4 → log + drop. ### Ops to author (one file per op) - [ ] Forge-event: `dispatch`, `dispatch_breakdown`, `resolve_agent`, `dedup_check`, `dedup_record`, `dedup_cancel_stale`, `guard_reviewer_dispatch`, `guard_author_dispatch`, `detect_change_request`, `set_label`, `remove_label`, `comment`. - [ ] Imperative-handler ports: `cancel_tasks` (replaces `handleIssueUnassigned` at `webhook.ts:296`), `pr_dependency_markers` (replaces `handlePrDependencyMarkers` at `webhook.ts:325`), `propagate_dependencies` (replaces `handleStackedRebaseCascade` at `webhook.ts:333` and the legacy `handleIssueClosed` callback). - [ ] No op imports the legacy `defaultArgInjections` bundle. - [ ] Each op has a `.test.ts` against a fake ctx; recorded fixture replays produce identical observable side effects to the legacy path. ### Tests - [ ] Loader: parse minimal, parse with overlay, parse with bad YAML, custom overrides default; SSE event fires on file change. - [ ] Expression: parser round-trip, evaluator short-circuit, type errors, every builtin against synthetic event. - [ ] Op registry: missing-capability ctx panics with clear message. - [ ] Executor: sequential order, mid-failure halt, `continue-on-error`, parallel concurrency, `concurrency` mutex. - [ ] Internal triggers: emit on terminal state, depth cap fires drop on 5th hop, `on: task.completed` flow runs after task. ## Out of scope - Default YAML files + JSON Schema + REST CRUD (covered by Defaults+API ticket). - Editor UI (covered by Editor UI ticket). - Shadow-mode + cutover + legacy deletion (covered by Migration ticket). ## References - Spec: `docs/specs/flows-yaml.md` §3, §5.2, §6, §7, §8, §9, §10. - Replaces: `apps/server/src/domain/flows/{flow-dispatch,executor,registry,validation}.ts`, inline imperative calls at `apps/server/src/http/webhook.ts:296–341`, `defaultArgInjections` at `flow-dispatch.ts:113–218`, `matchesFilters` at `flow-dispatch.ts:356–407`.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
charles/claude-hooks#1075
No description provided.