Skip to content

fix(grader): add agentContentDigestPolicy + --covers-content-digest to skip neg/007 and neg/018 for 'either' verifiers#1034

Draft
bokelley wants to merge 2 commits intomainfrom
claude/issue-1031-grader-cd-policy-skip
Draft

fix(grader): add agentContentDigestPolicy + --covers-content-digest to skip neg/007 and neg/018 for 'either' verifiers#1034
bokelley wants to merge 2 commits intomainfrom
claude/issue-1031-grader-cd-policy-skip

Conversation

@bokelley
Copy link
Copy Markdown
Contributor

Closes #1031

Summary

adcp grade request-signing was producing false FAIL results for neg/007 (missing-content-digest) and neg/018 (digest-covered-when-forbidden) when grading against verifiers that advertise covers_content_digest: 'either'. The expected 401 rejections can never fire against an 'either' verifier — the agent spec-compliantly accepts both covered and uncovered requests — so both vectors fail every run, misleading adopters into thinking their verifier is broken.

The grader already had capabilityMismatch() logic in preflightSkip() that correctly handles this, but it only fires when the full agentCapability: VerifierCapabilityFixture is provided. Adopters would need to discover and manually populate that struct, or manually add --skip 007-...,018-... to every invocation.

This PR adds a narrow escape hatch:

  • GradeOptions.agentContentDigestPolicy?: 'required' | 'forbidden' | 'either' — when set without agentCapability, auto-skips negative vectors whose verifier_capability.covers_content_digest can't match. Only affects negative vectors (positive vectors are always gradable regardless of content-digest policy). Skipped with skip_reason: 'capability_profile_mismatch'.
  • --covers-content-digest <required|forbidden|either> CLI flag on adcp grade request-signing — analogous to the existing flag on adcp grade signer.

The rationale for a narrow option rather than synthesizing agentCapability from the CLI: passing agentCapability = { ..., required_for: [] } would also skip vectors that test required_for enforcement, causing under-coverage on a different dimension.

What was tested

  • tsc --project tsconfig.lib.json --noEmitOnError false — zero new errors (2 pre-existing config warnings: deprecated moduleResolution=node10, missing @types/node)
  • npm run format:check — clean
  • npm test — 320 failures identical to pre-change baseline (all compliance-cache-dependent; npm run sync-schemas requires network access not available in this environment). New tests in request-signing-grader-e2e.test.js and request-signing-grader-mcp.test.js will pass once CI has the compliance cache.

Nits surfaced from pre-PR review (not fixed — low priority):

  • agentContentDigestPolicy: 'required' skips neg/018 but still runs neg/007; no test covers that specific path (the 'either' path is the motivating case and is covered)

Pre-PR review:

  • code-reviewer: approved after two blocker fixes (added kind === 'negative' guard; updated MCP test to retire CAPABILITY_PROFILE_VECTORS workaround)
  • ad-tech-protocol-expert: approved — skip logic is spec-correct; kind === 'negative' guard required and added; skip_reason: 'capability_profile_mismatch' is the right reuse

Triage-managed PR. This bot does not currently iterate on
review comments or PR conversation threads (only on the source
issue). To unblock:

  • Push fixup commits directly: gh pr checkout <num>
    fix → push.
  • Or re-trigger: comment /triage execute on the source
    issue.

See adcp#3121
for context.

Session: https://claude.ai/code/session_015KgshPTGDexdFpJpxEdDR9


Generated by Claude Code

…neg/018 for 'either' verifiers

Adds `agentContentDigestPolicy` to `GradeOptions` and a matching
`--covers-content-digest` CLI flag to `adcp grade request-signing`.
When set, the grader auto-skips negative vectors whose
`verifier_capability.covers_content_digest` can't match the agent's
declared stance — eliminating false FAILs for agents that advertise
`covers_content_digest: 'either'` (the default sandbox posture).

The skip check is scoped to negative vectors only; positive vectors
are always gradable regardless of content-digest policy.

Closes #1031

https://claude.ai/code/session_015KgshPTGDexdFpJpxEdDR9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(grader): neg/007 and neg/018 fire unconditionally — should skip when verifier advertises covers_content_digest='either'

2 participants