5.13 runner: error_handling / schema_compliance tracks gated behind get_products, leaving signals-only agents with no error/schema coverage

[EvgenyAndroid]

Subproject: adcontextprotocol/adcp-client (routed to adcp by default — maintainer can move if needed)

Summary

In the 5.13 storyboard runner, the error_handling and schema_compliance tracks appear to be gated behind get_products availability. For signals-only agents (supported_protocols: ["signals"], no get_products tool), this means the entire error/schema track is skipped — not just inapplicable scenarios, but all of them.

This is a regression from the 5.6 runner, where error handling and schema compliance ran as protocol-level baselines independent of which domain a given agent supported.

Expected behavior

error_handling and schema_compliance are universal protocol concerns — every AdCP agent, regardless of declared protocols, should be tested against them. These tracks should run unconditionally, or at minimum have signals-specific paths (e.g., testing activate_signal error responses) rather than collapsing entirely when get_products is absent.

Observed behavior

• Signals-only agent with supported_protocols: ["signals"] — error_handling and schema_compliance tracks produce 0 steps / are not run
• Same agent on 5.6 runner — both tracks ran against signals endpoints

Likely cause

Applicability logic in the runner treats get_products as a proxy for "agent supports protocol-level error/schema testing," rather than using supported_protocols to select the appropriate endpoints per track.

Related

• past_start_enforcement gating bug (same pattern — media-buy assumptions baked into universal checks)
• Storyboard signal_id / signal_agent_segment_id field name bug (separate issue)

Impact

Signals-only agents have no compliance coverage for error handling or schema validation in 5.13. A compliant agent could silently regress on error behavior with no runner signal.

[bokelley]

Thanks for filing!

Triage

Classification: Bug (regression)
Bucket(s): compliance suite
Status: drafting-pr
Milestone: (omitted — compliance suite addition, no protocol version bump)

What the experts said:

• ad-tech-protocol-expert: Confirmed. Both error-compliance.yaml and schema-validation.yaml sit under universal/ but carry required_tools: [get_products] at the storyboard root — the 5.13 runner correctly treats this as an applicability gate and emits missing_tool, skipping the storyboard entirely for signals agents. REFERENCE_NOT_FOUND (not SIGNAL_NOT_FOUND) is the canonical error code for an unresolvable signal_agent_segment_id; the VERSION_UNSUPPORTED path via get_signals is spec-sound.
• code-reviewer: All structural checks pass — idempotency keys present on all activate_signal steps, correct model field for signals pricing, sample_request_skip_schema on the intentionally-malformed step.

My take: Root cause is that error-compliance.yaml and schema-validation.yaml were authored for media-buy agents but placed in universal/ — their required_tools: [get_products] gate has always excluded signals agents. The fix adds two new storyboards (error-compliance-signals, schema-validation-signals) scoped to [get_signals, activate_signal] and [get_signals] respectively, restoring error handling and schema compliance coverage for signals agents. The adcp-client runner may also need a complementary fix to use supported_protocols for track applicability rather than required_tools as a proxy (tracked in the PR body).

Drafting PR: #3358 — adds error-compliance-signals.yaml (5 phases: capability discovery, error responses, error structure, version negotiation, transport binding) and schema-validation-signals.yaml (2 phases: capability discovery, schema compliance) as new universal storyboards.

---

Triaged by Claude Code. Session: https://claude.ai/code/session_01J2r8RXmLEsooTH82Q3XyY9

---

Generated by Claude Code