storyboard: investigate why controller_detected stays false despite list_scenarios returning all 12 scenarios

[bokelley]

Context

After PRs #282, #296, #310, and #313, `examples/seller_agent.py`'s `DemoStore` overrides all 7 new `TestControllerStore` methods (force_create_media_buy_arm, force_task_completion, seed_product, seed_pricing_option, seed_creative, seed_plan, seed_media_buy) plus the original 5 (force_, simulate_). The dynamic `comply_test_controller(scenario="list_scenarios")` call returns all 12 implemented scenarios:

```json
{"success": true, "scenarios": [
"force_creative_status", "force_account_status", "force_media_buy_status",
"force_create_media_buy_arm", "force_task_completion",
"simulate_delivery", "simulate_budget_spend",
"seed_product", "seed_pricing_option", "seed_creative",
"seed_plan", "seed_media_buy"
]}
```

But the JS storyboard runner (`@adcp/client@latest`, currently 5.23) still reports `"controller_detected": false` and skips/fails the 5 fixture-dependent storyboards (`outdoor_display_q2`, `sports_preroll_q2` products; `acme_outdoor_allowlist_v1`, `acme_outdoor_no_match_v1` property lists; aggressive measurement_terms).

Hypothesis

The runner has a detection heuristic beyond `list_scenarios`. Possibilities:

1. Reads the static `compliance_testing.scenarios` field on `get_adcp_capabilities`. AdCP 3.0.1's response schema constrains this enum to the original 6 names — the new `force_create_media_buy_arm` / `force_task_completion` / `seed_*` cannot be advertised there without failing schema validation. If the runner requires them in this static field, we have a deadlock between schema and runner expectations.

2. Probes a specific scenario (e.g., `force_task_completion`) and checks for a particular response shape. If the response doesn't match, controller is graded "absent."

3. Requires a specific subset of scenarios (e.g., all `seed_*` to be present? `force_session_status`? — DemoStore doesn't override `force_session_status`, even though it's schema-allowed).

4. A2A-only detection path — the runner's compliance-fleet flag `controller_detected` may only flip `true` for A2A-protocol agents.

Verification path

Run `@adcp/client@latest adcp storyboard run http://127.0.0.1:3001/mcp media_buy_seller --json --allow-http` against `examples/seller_agent.py` on `main`. Inspect the runner's source (`@adcp/client/src/storyboard/...`) for the controller-detection logic. Report findings.

Impact

Until `controller_detected: true`, the storyboard CI job (#305 / merged via #309) cannot pass — it's stuck at 36/47 with the 5 fixture-dependent storyboards failing. Once detection works, those storyboards should pass via the `seed_*` overrides this team just landed.

Workaround

None on the SDK side. The Python SDK provides every scenario the AdCP 3.0.1 spec exposes; the gap is downstream in the JS runner's heuristic OR in the AdCP 3.0.1 capabilities-response schema (item 1 above).

References

• DemoStore overrides: feat(examples): DemoStore overrides for force_create_media_buy_arm, force_task_completion, and seed_* scenarios #313 (closes examples/seller_agent.py: DemoStore needs overrides for force_create_media_buy_arm, force_task_completion, and seed_* scenarios #312)
• SDK base classes: feat(server): add force_create_media_buy_arm + force_task_completion controller scenarios #282 (force_), fix(server): fix streamable-http ASGI error, host binding, and AdCP 3.0.1 scenario gaps #296 (seed_)
• Schema constraint in capabilities: `schemas/cache/protocol/get-adcp-capabilities-response.json` — `compliance_testing.scenarios.items.enum`
• JS runner: `npm view @adcp/client` (currently 5.23.0)

[bokelley]

Triage

Classification: Bug
Bucket(s): examples, cross-repo
Status: drafting-pr
Milestone: (omitted — no milestone targeted)

What the experts said:

• ad-tech-protocol-expert: Root cause is almost certainly the static compliance_testing.scenarios field — the JS runner reads it before calling list_scenarios; DemoSeller was advertising 5 of the 6 schema-allowed values, missing force_session_status. Also found that seed_creative_format is in the request schema but completely absent from the Python SDK (SCENARIOS, TestControllerStore, dispatcher, DemoStore).
• adtech-product-expert: The two-phase probe hypothesis is consistent with typical storyboard runner design. The only schema-legal Python SDK change to the static capabilities field is adding force_session_status (the newer scenarios can't be advertised there until the spec schema's enum is extended — that's a cross-repo fix).

My take: Two Python SDK gaps were found and fixed in draft PR #315. The seed_creative_format omission is a definite bug (any storyboard that seeds a creative format would get UNKNOWN_SCENARIO). The force_session_status static capabilities gap is the most likely cause of controller_detected: false — adding it aligns the static field with all 6 schema-permitted values and gives DemoStore a matching stub. A cross-repo fix is still needed: the get-adcp-capabilities-response.json schema enum must be extended to include force_create_media_buy_arm, force_task_completion, and seed_* so those can be legally advertised in the static field and the runner can detect them without relying solely on list_scenarios.

Draft PR: #315 — adds seed_creative_format (SCENARIOS + TestControllerStore + dispatcher + DemoStore + list_creative_formats merge), advertises force_session_status in static capabilities + DemoStore stub. Two nits were surfaced in pre-PR review and left for the reviewer: force_session_status stub always reports previous_state: "active" (no SI session state in this seller), and list_creative_formats concatenation has no dedup guard.

Remaining cross-repo work: File an issue against adcontextprotocol/adcp to extend compliance_testing.scenarios.items.enum in get-adcp-capabilities-response.json to include the AdCP 3.0.1 additions (force_create_media_buy_arm, force_task_completion, seed_*). Until that's done, those scenarios can't be advertised in the static field without failing schema validation.

---

Triaged by Claude Code. Session: https://claude.ai/code/session_018i5iWUxUayQEh5K3v9HZRG

---

Generated by Claude Code

[bokelley]

/triage execute

[bokelley]

Triage

Classification: Bug
Bucket(s): client, handlers
Status: drafting-pr
Milestone: omitted (no target version specified)

What the experts said:

• code-reviewer: approved — no blockers; flagged missing regression test (added), nit on fn_metadata comment
• dx-expert: approved — no blockers; confirmed backward-compat, json import still needed, no guide changes required

Investigation findings (runner source inspection):

The JS runner's detectController function in @adcp/sdk (dist/lib/testing/test-controller.js) checks two things:

1. profile.tools.includes('comply_test_controller') — this was already ✓ (tool is registered correctly)
2. Calls list_scenarios and checks data.success && data.scenarios

Root cause: comply_test_controller returned json.dumps(result) — a string. FastMCP wraps string-returning tools as structuredContent: {result: "<json string>"}. The runner's unwrapProtocolResponse extracts that as {result: "<string>"}, so data.success is undefined → detected: false. All four hypotheses in the issue are wrong; the runner does NOT read the static compliance_testing.scenarios capabilities field for detection at all — it goes straight to tools/list then list_scenarios.

Fix: Change return type to dict[str, Any] (remove json.dumps). FastMCP then emits the dict fields directly, giving the runner data.success = true and data.scenarios = [...].

My take: One-line fix in register_test_controller. Wire-tested: before fix structuredContent.result was a JSON string; after fix structuredContent carries the parsed dict. PR #317 drafted.

---

Triaged by Claude Code. Session: https://claude.ai/code/session_01LuThNVrmmHfVvPbfrbWsvy

---

Generated by Claude Code