Skip to content

Addie: surface runner hints (context_value_rejected) in test-result UI #3055

@bokelley

Description

@bokelley

Context

`@adcp/client` 5.17.x ships a new non-fatal diagnostic on failing storyboard steps. When a seller's error response carries an `available` / `allowed` / `accepted_values` list AND the rejected request value traces back to a prior-step `$context.*` write, the runner attaches a `hints: StoryboardStepHint[]` field to the step result. Collapses the "SDK bug vs seller bug" triage to one line.

Design: adcp-client#870.
Reader's doc: docs/guides/VALIDATE-YOUR-AGENT.md § Reading hint lines.

What Addie gets for free

Every `StoryboardStepResult` Addie receives (via `runStoryboard` / `runStoryboardStep` from `@adcp/client/testing`) now carries the optional `hints` array. Bumping Addie's `@adcp/client` dep to 5.17.x is enough to receive them — no other wiring needed on Addie's side to get the payload.

What needs rendering

The CLI absorbed this in one sibling file + four inline wires (see adcp-client#891 / #900 for the pattern). Addie would likely want:

  1. Inline hint under the failing step's `Error:` line. Use the `💡` prefix from the CLI so users recognize the pattern when moving between surfaces:
    ```
    ❌ Activate signal on DSP (6ms)
    Task: activate_signal
    Error: Pricing option not found: po_cpm_auto
    💡 Hint: Rejected `pricing_option_id: po_cpm_auto` was extracted from
    `$context.first_signal_pricing_option_id` (set by step `discover`
    from response path `signals[0].pricing_options[0].pricing_option_id`).
    Seller's accepted values: [po_cart_auto]. Check that the seller's
    `get_signals` and `activate_signal` catalogs agree.
    ```
    The closing sentence always cites the two tools involved — ready to surface verbatim.

  2. Structured-field inspector. Each hint carries `kind`, `context_key`, `source_step_id`, `source_kind`, `source_task`, `response_path`, `rejected_value`, `request_field`, `accepted_values`, `error_code`, `message`. Types exported from `@adcp/client/testing` as `StoryboardStepHint` / `ContextValueRejectedHint`.

  3. Cross-run aggregation (stretch). Because every hint carries `source_step_id` and `context_key`, Addie can aggregate: "seller X has catalog drift between `get_signals` and `activate_signal` in 23% of runs." That's the kind of signal dashboards catch that humans scanning single-run logs miss.

JUnit / JSON surfaces already carry hints

  • JUnit XML: hints append to the `` body as `Hint (context_value_rejected): …`, and land in the `message=` attribute when `step.error` is empty (the docs: Expand bundled schemas and document GitHub access #883-widened gate path — validation-only failures on 200-OK responses).
  • JSON report: `StoryboardStepResult.hints[]` — documented shape above.

If Addie writes JUnit or consumes the JSON result shape, the hints are already reaching it; only rendering is missing.

Spec-side tracking

The rejection-envelope shape itself (`details.available` naming) is being pinned in adcontextprotocol/adcp#3049. The SDK tolerates all in-the-wild variants today, so Addie doesn't have to wait on the spec to ship.

Dogfood evidence

Verified against a deliberately-broken signals agent (get_signals advertises one pricing_option_id; activate_signal only accepts a different one) — hint fires through real MCP transport, all 11 structured fields populate correctly. Same flow also validated on sellers using `adcpError()` (singular `adcp_error` envelope) after adcp-client#908 landed.

Suggested scope

Render change only — no protocol or contract change to Addie's side. Estimated ~1 afternoon to mirror the CLI's sibling-file pattern and surface hints below the error line in whichever view renders step results today.

Metadata

Metadata

Assignees

No one assigned

    Labels

    addieIssues related to Addie (via any channel)claude-triagedIssue has been triaged by the Claude Code triage routine. Remove to re-triage.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions