Conversation
…neg/018 for 'either' verifiers Adds `agentContentDigestPolicy` to `GradeOptions` and a matching `--covers-content-digest` CLI flag to `adcp grade request-signing`. When set, the grader auto-skips negative vectors whose `verifier_capability.covers_content_digest` can't match the agent's declared stance — eliminating false FAILs for agents that advertise `covers_content_digest: 'either'` (the default sandbox posture). The skip check is scoped to negative vectors only; positive vectors are always gradable regardless of content-digest policy. Closes #1031 https://claude.ai/code/session_015KgshPTGDexdFpJpxEdDR9
Auto-generated by sync-version during pre-push validation. https://claude.ai/code/session_015KgshPTGDexdFpJpxEdDR9
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1031
Summary
adcp grade request-signingwas producing false FAIL results for neg/007 (missing-content-digest) and neg/018 (digest-covered-when-forbidden) when grading against verifiers that advertisecovers_content_digest: 'either'. The expected 401 rejections can never fire against an'either'verifier — the agent spec-compliantly accepts both covered and uncovered requests — so both vectors fail every run, misleading adopters into thinking their verifier is broken.The grader already had
capabilityMismatch()logic inpreflightSkip()that correctly handles this, but it only fires when the fullagentCapability: VerifierCapabilityFixtureis provided. Adopters would need to discover and manually populate that struct, or manually add--skip 007-...,018-...to every invocation.This PR adds a narrow escape hatch:
GradeOptions.agentContentDigestPolicy?: 'required' | 'forbidden' | 'either'— when set withoutagentCapability, auto-skips negative vectors whoseverifier_capability.covers_content_digestcan't match. Only affects negative vectors (positive vectors are always gradable regardless of content-digest policy). Skipped withskip_reason: 'capability_profile_mismatch'.--covers-content-digest <required|forbidden|either>CLI flag onadcp grade request-signing— analogous to the existing flag onadcp grade signer.The rationale for a narrow option rather than synthesizing
agentCapabilityfrom the CLI: passingagentCapability = { ..., required_for: [] }would also skip vectors that testrequired_forenforcement, causing under-coverage on a different dimension.What was tested
tsc --project tsconfig.lib.json --noEmitOnError false— zero new errors (2 pre-existing config warnings: deprecatedmoduleResolution=node10, missing@types/node)npm run format:check— cleannpm test— 320 failures identical to pre-change baseline (all compliance-cache-dependent;npm run sync-schemasrequires network access not available in this environment). New tests inrequest-signing-grader-e2e.test.jsandrequest-signing-grader-mcp.test.jswill pass once CI has the compliance cache.Nits surfaced from pre-PR review (not fixed — low priority):
agentContentDigestPolicy: 'required'skips neg/018 but still runs neg/007; no test covers that specific path (the'either'path is the motivating case and is covered)Pre-PR review:
kind === 'negative'guard; updated MCP test to retireCAPABILITY_PROFILE_VECTORSworkaround)kind === 'negative'guard required and added;skip_reason: 'capability_profile_mismatch'is the right reuseSession: https://claude.ai/code/session_015KgshPTGDexdFpJpxEdDR9
Generated by Claude Code