Motivation
Every seller that integrates the AdCP compliance suite wants a CI gate that fails a PR if it regresses storyboard pass-rate. The pattern is simple but non-obvious: run storyboards, parse the output, compare against a rolling floor, fail if below.
Our training-agent workflow at `.github/workflows/training-agent-storyboards.yml` has the shape:
```yaml
-
name: Run storyboards
id: run
run: |
npx tsx server/tests/manual/run-storyboards.ts 2>&1 | tee /tmp/storyboards.log
clean=$(grep -oE 'storyboards: [0-9]+/[0-9]+' /tmp/storyboards.log | tail -1 | grep -oE '^storyboards: [0-9]+' | grep -oE '[0-9]+$')
echo "clean=${clean}" >> "$GITHUB_OUTPUT"
-
name: Enforce non-regression
env:
CLEAN: ${{ steps.run.outputs.clean }}
MIN_CLEAN: ${{ matrix.min_clean_storyboards }}
run: |
if [ "${CLEAN}" -lt "${MIN_CLEAN}" ]; then
echo "::error::Regression: ${CLEAN} clean storyboards is below the ${MIN_CLEAN} floor."
exit 1
fi
```
Plus the matrix of legacy/framework modes, the ratcheting floor comments explaining each bump, and the `paths:` filter. Every seller is going to need ~60 lines of this.
Proposal
Ship a reusable composite action at `adcontextprotocol/adcp-client/.github/actions/storyboard-gate`:
```yaml
- uses: adcontextprotocol/adcp-client/.github/actions/storyboard-gate@v1
with:
agent-url: http://localhost:4444/api/mcp
min-clean: 45
min-passing: 324
mode: json # table | json | junit
```
Assumes the CLI from the companion RFC is installed. Emits `::error::` annotations on regression so the PR comment is structured.
What the action does
- Install `@adcp/client` (or use a pre-installed one via `skip-install`).
- Invoke `npx @adcp/client run-storyboards --format json` (or local equivalent).
- Parse the JSON output for `clean_count` + `passing_count`.
- Compare to `min-clean` / `min-passing` inputs.
- Upload the full run report as a workflow artifact.
- Emit a GH check summary with the ratchet comment (if floor bumped upward, suggest a follow-up PR to bump the input).
Ratcheting helper (optional)
Separate action `adcontextprotocol/adcp-client/.github/actions/storyboard-ratchet` that bumps the floor when the main branch pass-rate exceeds it for N consecutive runs. Reduces human toil on the "we added a fix, now bump the floor" PR.
Why separate from the CLI RFC
The CLI is where the work is; the action is a thin wrapper. Keeping them as separate issues lets the CLI land first and sellers adopt with bespoke workflows in the meantime.
Reference implementation
`adcontextprotocol/adcp`'s training-agent workflow, specifically the `min_clean_storyboards` / `min_passing_steps` matrix entries with their floor-bump comments that document each compliance milestone.
Complementary RFC: `@adcp/client` CLI (separate issue).
Motivation
Every seller that integrates the AdCP compliance suite wants a CI gate that fails a PR if it regresses storyboard pass-rate. The pattern is simple but non-obvious: run storyboards, parse the output, compare against a rolling floor, fail if below.
Our training-agent workflow at `.github/workflows/training-agent-storyboards.yml` has the shape:
```yaml
name: Run storyboards
id: run
run: |
npx tsx server/tests/manual/run-storyboards.ts 2>&1 | tee /tmp/storyboards.log
clean=$(grep -oE 'storyboards: [0-9]+/[0-9]+' /tmp/storyboards.log | tail -1 | grep -oE '^storyboards: [0-9]+' | grep -oE '[0-9]+$')
echo "clean=${clean}" >> "$GITHUB_OUTPUT"
name: Enforce non-regression
env:
CLEAN: ${{ steps.run.outputs.clean }}
MIN_CLEAN: ${{ matrix.min_clean_storyboards }}
run: |
if [ "${CLEAN}" -lt "${MIN_CLEAN}" ]; then
echo "::error::Regression: ${CLEAN} clean storyboards is below the ${MIN_CLEAN} floor."
exit 1
fi
```
Plus the matrix of legacy/framework modes, the ratcheting floor comments explaining each bump, and the `paths:` filter. Every seller is going to need ~60 lines of this.
Proposal
Ship a reusable composite action at `adcontextprotocol/adcp-client/.github/actions/storyboard-gate`:
```yaml
with:
agent-url: http://localhost:4444/api/mcp
min-clean: 45
min-passing: 324
mode: json # table | json | junit
```
Assumes the CLI from the companion RFC is installed. Emits `::error::` annotations on regression so the PR comment is structured.
What the action does
Ratcheting helper (optional)
Separate action `adcontextprotocol/adcp-client/.github/actions/storyboard-ratchet` that bumps the floor when the main branch pass-rate exceeds it for N consecutive runs. Reduces human toil on the "we added a fix, now bump the floor" PR.
Why separate from the CLI RFC
The CLI is where the work is; the action is a thin wrapper. Keeping them as separate issues lets the CLI land first and sellers adopt with bespoke workflows in the meantime.
Reference implementation
`adcontextprotocol/adcp`'s training-agent workflow, specifically the `min_clean_storyboards` / `min_passing_steps` matrix entries with their floor-bump comments that document each compliance milestone.
Complementary RFC: `@adcp/client` CLI (separate issue).