Skip to content

RFC: reusable GitHub Action for storyboard non-regression gating #787

@bokelley

Description

@bokelley

Motivation

Every seller that integrates the AdCP compliance suite wants a CI gate that fails a PR if it regresses storyboard pass-rate. The pattern is simple but non-obvious: run storyboards, parse the output, compare against a rolling floor, fail if below.

Our training-agent workflow at `.github/workflows/training-agent-storyboards.yml` has the shape:

```yaml

  • name: Run storyboards
    id: run
    run: |
    npx tsx server/tests/manual/run-storyboards.ts 2>&1 | tee /tmp/storyboards.log
    clean=$(grep -oE 'storyboards: [0-9]+/[0-9]+' /tmp/storyboards.log | tail -1 | grep -oE '^storyboards: [0-9]+' | grep -oE '[0-9]+$')
    echo "clean=${clean}" >> "$GITHUB_OUTPUT"

  • name: Enforce non-regression
    env:
    CLEAN: ${{ steps.run.outputs.clean }}
    MIN_CLEAN: ${{ matrix.min_clean_storyboards }}
    run: |
    if [ "${CLEAN}" -lt "${MIN_CLEAN}" ]; then
    echo "::error::Regression: ${CLEAN} clean storyboards is below the ${MIN_CLEAN} floor."
    exit 1
    fi
    ```

Plus the matrix of legacy/framework modes, the ratcheting floor comments explaining each bump, and the `paths:` filter. Every seller is going to need ~60 lines of this.

Proposal

Ship a reusable composite action at `adcontextprotocol/adcp-client/.github/actions/storyboard-gate`:

```yaml

  • uses: adcontextprotocol/adcp-client/.github/actions/storyboard-gate@v1
    with:
    agent-url: http://localhost:4444/api/mcp
    min-clean: 45
    min-passing: 324
    mode: json # table | json | junit
    ```

Assumes the CLI from the companion RFC is installed. Emits `::error::` annotations on regression so the PR comment is structured.

What the action does

  1. Install `@adcp/client` (or use a pre-installed one via `skip-install`).
  2. Invoke `npx @adcp/client run-storyboards --format json` (or local equivalent).
  3. Parse the JSON output for `clean_count` + `passing_count`.
  4. Compare to `min-clean` / `min-passing` inputs.
  5. Upload the full run report as a workflow artifact.
  6. Emit a GH check summary with the ratchet comment (if floor bumped upward, suggest a follow-up PR to bump the input).

Ratcheting helper (optional)

Separate action `adcontextprotocol/adcp-client/.github/actions/storyboard-ratchet` that bumps the floor when the main branch pass-rate exceeds it for N consecutive runs. Reduces human toil on the "we added a fix, now bump the floor" PR.

Why separate from the CLI RFC

The CLI is where the work is; the action is a thin wrapper. Keeping them as separate issues lets the CLI land first and sellers adopt with bespoke workflows in the meantime.

Reference implementation

`adcontextprotocol/adcp`'s training-agent workflow, specifically the `min_clean_storyboards` / `min_passing_steps` matrix entries with their floor-bump comments that document each compliance milestone.

Complementary RFC: `@adcp/client` CLI (separate issue).

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions