ci: storyboard runner CI vs local divergence — fresh @adcp/client install validates against 3.0.1 schemas more strictly than local npm cache

[bokelley]

Context

After PR #322 closed the original 5 fixture-dependent storyboard failures, local end-to-end against examples/seller_agent.py reports:

overall_status: passing
total: 47, passed: 45, failed: 0, skipped: 2
controller_detected: true

But CI's storyboard job (with continue-on-error: true) reports:

overall_status: partial
total: 59, passed: 25, failed: 13, skipped: 21
controller_detected: true

Same code, same npx -y -p @adcp/client@latest invocation (resolves to 5.21.1 in both per npm view @adcp/client version), same Python 3.12. Different storyboard outcomes.

Symptoms in CI not present locally

Repeated stderr from the runner (printed by @adcp/client/dist/lib/core/TaskExecutor.js:1119):

Schema validation failed for get_products: [
  "/products/2/name: must have required property 'name'",
  "/products/2/description: must have required property 'description'",
  "/products/2/publisher_properties: must have required property 'publisher_properties'",
  "/products/2/reporting_capabilities: must have required property 'reporting_capabilities'",
  "/products/2/format_ids/0/agent_url: must have required property 'agent_url'",
  ...
]

The dumped response in the CI log shows products[2] does have all of those fields populated correctly. The schema validator is reporting them as missing anyway — and CI shows the failures cascade (21 dependent steps skipped due to unresolved context variables from prior steps).

The runner expects schema URL /schemas/3.0.1/media-buy/get-products-response.json. Local cache may be serving an older schema; CI's fresh install fetches 3.0.1 and validates more strictly — but the products are spec-compliant 3.0.1 shape, so the strict validation should pass.

Likely root cause

The 3.0.1 product schema is a oneOf discriminated union (per delivery_type, or pricing_options[*].pricing_model). @adcp/client@5.21.1's validator may iterate each variant and emit must have required property errors from the variants that DIDN'T match — even though one variant DOES match and validation should pass overall.

Two products at indices 0 and 1 (premium-homepage, run-of-site) pass; new products 2-5 fail. The structural difference is pricing_option_id value (po-cpm-homepage / po-cpm-ros for passing vs cpm_standard / cpm_guaranteed for failing) and product_id (hyphenated for passing vs underscored for failing). Neither has a schema-level pattern restriction visible in the cache. May be runner schema-validator bug or CI-vs-local schema-cache divergence.

Why this isn't blocking

• The 5 originally-failing storyboard scenarios from storyboard: 5 fixture-dependent failures (default product fixtures + measurement_terms + targeting_overlay persistence) #319 all pass locally.
• CI's storyboard job is continue-on-error: true (per ci: run AdCP storyboard suite against examples/seller_agent.py on every PR #305 / ci(adcp): add storyboard CI job for seller_agent.py compliance checks #309).
• The Python SDK is correctly emitting spec-compliant responses; the runner's downstream behavior is opaque from the SDK side.

Investigation paths

1. Reproduce locally with a fresh npm cache clean --force — see if local also starts failing with fresh schema fetch.
2. Diff the runner's schema cache between local and CI. If CI gets newer schemas, that's the issue.
3. Inspect @adcp/client/dist/lib/validation/index.js Ajv setup — does it use oneOf errors-collection or first-match-passes?
4. File upstream against @adcp/client if confirmed runner bug.

References

• Closed-but-cascading: storyboard: 5 fixture-dependent failures (default product fixtures + measurement_terms + targeting_overlay persistence) #319 (5 failures locally → 0; CI sees different surface)
• Storyboard CI job: ci: run AdCP storyboard suite against examples/seller_agent.py on every PR #305 / ci(adcp): add storyboard CI job for seller_agent.py compliance checks #309 (currently informational)
• Source of validation message: @adcp/client/dist/lib/core/TaskExecutor.js:1119

[bokelley]

/triage execute

[bokelley]

/triage execute

[bokelley]

Triage

Classification: Bug (upstream runner; SDK responses are spec-compliant)
Bucket(s): examples, cross-repo (note: neither label exists in this repo — see gap below)
Status: deferred — blocked on upstream @adcp/client runner investigation

What the investigation found:

Schema inspection of the local 3.0.1 cache (schemas/cache/bundled/media-buy/get-products-response.json) shows:

• No product-level oneOf — just a flat schema with required: [product_id, name, description, ...].
• pricing_options.items has a 9-branch oneOf discriminated by pricing_model (CPM, vCPM, CPC, etc.). No pattern constraint on pricing_option_id in any branch.
• product_id is type: string with no pattern.

Products 0–1 (premium-homepage, run-of-site) and 2–5 (outdoor_display_q2, etc.) have identical schema structure — both use pricing_model: "cpm", identical publisher_properties/format_ids shapes, and all required fields present.

The CI error ("must have required property name" for a product that does have name) is textbook oneOf error bleed-through: when an Ajv oneOf fails all branches, errors from each failed branch are surfaced together — including "missing required property" for required fields that are actually present. This is a runner-side validator bug, not a data issue.

The 0-vs-2+ split likely reflects the runner's npm cache state: cached schemas may bypass the Ajv oneOf path where the bug surfaces; a fresh install triggers it.

My take: The Python SDK is emitting correct, spec-compliant responses. The actionable next step is upstream — either file against @adcp/client (investigation path 4 in the issue) or wait for the runner to fix its Ajv oneOf error handling. No SDK change closes this gap. Storyboard CI remains continue-on-error: true, so this is non-blocking.

Label gap: Buckets examples and cross-repo have no matching labels in this repo. Recommended: create them if bucket-level filtering is needed.

---

Triaged by Claude Code. Session: https://claude.ai/code/cse_01R426RiM8ZFj1KVH4Wm6JzC

Generated by Claude Code

---

Generated by Claude Code

[bokelley]

Triage

Classification: Bug
Bucket(s): examples
Status: drafting-pr
Milestone: omitted (informational CI job; not blocking release)

What the experts said:

• code-reviewer: approved after two passes — added variant != "skipped" guard and list() snapshot fix; no blockers remain
• dx-expert: approved — noted 3 nits (import path, format_issues() in assert messages, deepcopy) surfaced in PR body

My take: The Python jsonschema validator confirms all six static products are 3.0.1 spec-compliant — the CI failures originate in @adcp/client@5.21.1's Ajv oneOf error reporter, not in the response shape we emit. Investigation path 4 (file upstream against @adcp/client) is the fix for the CI divergence itself; this PR adds a Python-side regression anchor that will catch any future drift in our product shapes. Draft PR: #327.

---

Triaged by Claude Code. Session: https://claude.ai/code/session_019rDGvdv7HHDKwJeNri9Lff

---

Generated by Claude Code