be: assistant prompt + Claude call with structured output #1214

New issue

Closed

opened 2026-05-15 17:03:02 +00:00 by architect · 2 comments

architect commented

2026-05-15 17:03:02 +00:00

Collaborator

Context

Part of the Agents Roster AI Assistant feature. The endpoint ticket (#1212) injects a buildAndCallProposer(history, catalog) function — this ticket implements it. It owns the system prompt, the Claude call, structured-output enforcement, and prompt caching of the catalog block.

Scope

System prompt

A single Claude system prompt that:

Explains the role: "you design fleets of agents tailored to the operator's stated workflow"
Embeds the CatalogSnapshot as a cacheable block (skills, MCPs, tools, model tiers)
Specifies the proposal contract: name, role, system prompt, model tier (with reason), skills, tool permissions (per-grant rationale), MCPs (with setup_required), rationale
Sets behavior: ask 1–3 clarifying questions only if the brief is ambiguous; never propose skills/MCPs that aren't in the catalog; flag any MCP needing setup

Keep the prompt revisable — store it as a versioned constant or a file the team can iterate on without redeploying types.

Claude call

Use the existing Claude SDK wiring in this repo
Prompt caching on the catalog block — it's stable across calls in a session and across sessions with the same catalog_hash
Structured output: constrain Claude to return JSON matching the ProposalEnvelope JSON Schema from #1210. Either via tool-use (preferred — pass the schema as a tool) or response_format, whichever the SDK supports best
Model tier: Sonnet (general design work); document why if you choose otherwise
Latency target: first proposal < 15s per the acceptance criteria — measure and report in the PR

Refinement handling

Conversation history from the session store is replayed as prior turns
Each refinement returns a full new proposal (not a patch), per the spec — diffing is FE's job

Failure modes

Model returns invalid JSON → throw a typed error so #1212 can map to 502
Model proposes a skill/MCP not in the catalog → reject and retry once with a corrective system note; if it fails twice, surface the proposal anyway but mark the offending entries as unavailable: true

Deliverables

buildAndCallProposer(history, catalog): Promise<ProposalEnvelope> implementation
System prompt as a versioned, reviewable artifact
Prompt-cache configuration on the catalog block
Schema-constrained output (tool-use or response_format)
Catalog-membership validator + one-shot corrective retry
Unit tests with recorded fixtures: ambiguous brief triggers clarifying questions, clear brief produces full proposal, refinement carries parent_proposal_id, hallucinated skill is rejected/corrected
Latency log (P50/P95) reported in the PR description

Notes for implementer

Lean on the claude-api skill in this repo for SDK patterns and prompt caching — it codifies the conventions this service already uses.

Depends on: #1210 (types + JSON schema), #1211 (catalog input), #1212 (endpoint that calls this)
Depended on by: nothing — this is a leaf on the BE side

## Context Part of the **Agents Roster AI Assistant** feature. The endpoint ticket (#1212) injects a `buildAndCallProposer(history, catalog)` function — this ticket implements it. It owns the system prompt, the Claude call, structured-output enforcement, and prompt caching of the catalog block. ## Scope ### System prompt A single Claude system prompt that: - Explains the role: "you design fleets of agents tailored to the operator's stated workflow" - Embeds the `CatalogSnapshot` as a cacheable block (skills, MCPs, tools, model tiers) - Specifies the proposal contract: name, role, system prompt, model tier (with reason), skills, tool permissions (per-grant rationale), MCPs (with `setup_required`), rationale - Sets behavior: ask 1–3 clarifying questions only if the brief is ambiguous; never propose skills/MCPs that aren't in the catalog; flag any MCP needing setup Keep the prompt revisable — store it as a versioned constant or a file the team can iterate on without redeploying types. ### Claude call - Use the existing Claude SDK wiring in this repo - **Prompt caching** on the catalog block — it's stable across calls in a session and across sessions with the same `catalog_hash` - **Structured output**: constrain Claude to return JSON matching the `ProposalEnvelope` JSON Schema from #1210. Either via tool-use (preferred — pass the schema as a tool) or response_format, whichever the SDK supports best - Model tier: Sonnet (general design work); document why if you choose otherwise - Latency target: first proposal < 15s per the acceptance criteria — measure and report in the PR ### Refinement handling - Conversation history from the session store is replayed as prior turns - Each refinement returns a *full* new proposal (not a patch), per the spec — diffing is FE's job ### Failure modes - Model returns invalid JSON → throw a typed error so #1212 can map to 502 - Model proposes a skill/MCP not in the catalog → reject and retry once with a corrective system note; if it fails twice, surface the proposal anyway but mark the offending entries as `unavailable: true` ## Deliverables - [ ] `buildAndCallProposer(history, catalog): Promise<ProposalEnvelope>` implementation - [ ] System prompt as a versioned, reviewable artifact - [ ] Prompt-cache configuration on the catalog block - [ ] Schema-constrained output (tool-use or response_format) - [ ] Catalog-membership validator + one-shot corrective retry - [ ] Unit tests with recorded fixtures: ambiguous brief triggers clarifying questions, clear brief produces full proposal, refinement carries `parent_proposal_id`, hallucinated skill is rejected/corrected - [ ] Latency log (P50/P95) reported in the PR description ## Notes for implementer Lean on the `claude-api` skill in this repo for SDK patterns and prompt caching — it codifies the conventions this service already uses. --- **Depends on**: #1210 (types + JSON schema), #1211 (catalog input), #1212 (endpoint that calls this) **Depended on by**: nothing — this is a leaf on the BE side

architect added the

area:agents

label

2026-05-15 17:03:26 +00:00

architect referenced this issue

2026-05-15 17:03:39 +00:00

meta: Agents Roster AI Assistant — tracking issue #1216