Skip to content

Introduce kosli evaluate#671

Draft
tooky wants to merge 51 commits intomainfrom
introduce-kosli-evaluate
Draft

Introduce kosli evaluate#671
tooky wants to merge 51 commits intomainfrom
introduce-kosli-evaluate

Conversation

@tooky
Copy link
Contributor

@tooky tooky commented Feb 28, 2026

Why: Customers are duplicating attestation types to work around the lack of evaluation logic — creating separate types per environment just to encode different pass/fail rules. Three independent customers (JB, Deutsche Bank, Norsk Tipping) have hit this, and two more (Blackstone, NatWest) are building toward it. This is the most consistent product gap in our pipeline.

Objective: Add kosli evaluate — a CLI command that applies a Rego policy to trail data and returns a structured pass/fail decision. This separates what you collect (attestation type) from how you judge it (evaluation), and is the first step toward controls as a first-class product concept.

How this was built

This PR is also a demonstration of elephant carpaccio + TDD with Claude Code. The entire feature was built as a conversation — 48 commits, each one a single red-green-refactor step. Slices were kept thin enough to review independently, and the commit list reads bottom-to-top as a narrative of how the feature grew. Later commits came from reviewing the branch against Beck's Rules of Simple Design. The branch is intentionally unrebased so you can follow the progression.

Capabilities

  • kosli evaluate trail <name> — single trail against a Rego policy
  • kosli evaluate trails <name>... — multiple trails in one policy call
  • --output json|table — structured audit output or human-readable (default)
  • --show-input — include policy input in JSON output for debugging
  • --attestations — filter which attestations reach the policy (plain name for trail-level, artifact.name for artifact-level)
  • Exit code 0/1 reflects the policy decision — designed for CI/CD gates

Example

Validate all PRs are approved:

package policy

import rego.v1

default allow = false

violations contains msg if {
    some trail in input.trails
    some pr in trail.compliance_status.attestations_statuses["pull-request"].pull_requests
    count(pr.approvers) == 0
    msg := sprintf("trail '%v': pull-request %v has no approvers", [trail.name, pr.url])
}

allow if {
    count(violations) == 0
}

Kosli Server:

$ kosli evaluate trails \
  --policy pr-approved.rego \
  --org kosli \
  --flow server \
  --attestations pull-request \
  c643b06bf2efaa8f35d4da54c9e34a34a28bd251 \
  bd8254c58d20826df7248772cedf523f715516b6 \
  012cb304aab50bc4a3cc96fba7840ff29ea4d19e \
  a49a603c04b73c58d18909aace2f13e98892089f \
  9373bda52a51550b8ecb2236ed94cb88aa6e3a98
RESULT:  ALLOWED

CyberDojo Dashboard:

$ kosli evaluate trails \
  --policy tmp/pr-approved.rego \
  --org cyber-dojo \
  --flow dashboard-ci \
  9978a1ca82c273a68afaa85fc37dd60d1e394f84 \
  b334d371eb85c9a5c811776de1b65fb80b52d952 \
  5abd63aa1d64af7be5b5900af974dc73ae425bd6 \
  cb3ec71f5ce1103779009abaf4e8f8a3ed97d813
RESULT:      DENIED
VIOLATIONS:  trail '5abd63aa1d64af7be5b5900af974dc73ae425bd6': pull-request https://github.com/cyber-dojo/dashboard/pull/342 has no approvers
             trail '9978a1ca82c273a68afaa85fc37dd60d1e394f84': pull-request https://github.com/cyber-dojo/dashboard/pull/344 has no approvers
             trail 'b334d371eb85c9a5c811776de1b65fb80b52d952': pull-request https://github.com/cyber-dojo/dashboard/pull/343 has no approvers
             trail 'cb3ec71f5ce1103779009abaf4e8f8a3ed97d813': pull-request https://github.com/cyber-dojo/dashboard/pull/341 has no approvers
Error: policy denied: [trail '5abd63aa1d64af7be5b5900af974dc73ae425bd6': pull-request https://github.com/cyber-dojo/dashboard/pull/342 has no approvers trail '9978a1ca82c273a68afaa85fc37dd60d1e394f84': pull-request https://github.com/cyber-dojo/dashboard/pull/344 has no approvers trail 'b334d371eb85c9a5c811776de1b65fb80b52d952': pull-request https://github.com/cyber-dojo/dashboard/pull/343 has no approvers trail 'cb3ec71f5ce1103779009abaf4e8f8a3ed97d813': pull-request https://github.com/cyber-dojo/dashboard/pull/341 has no approvers]

Architecture

  • internal/evaluate/ — OPA Rego engine + trail data transforms (array-to-map, rehydration from detail API, filtering)
  • cmd/kosli/evaluateHelpers.go — shared options, flag registration, fetch+enrich pipeline, output dispatch
  • cmd/kosli/evaluateTrail.go / evaluateTrails.go — thin command wrappers

Also included

tooky and others added 30 commits February 27, 2026 17:12
Slice 1 of kosli evaluate trail - adds the evaluate parent command
and evaluate trail subcommand that fetches a trail from the API and
wraps the response in {"trail": ...} JSON output.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds internal/evaluate with Evaluate() function that validates and
evaluates Rego policies. Validates package name is 'policy', requires
an 'allow' rule, collects violations on deny.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds --policy flag to evaluate trail command. Reads a .rego file,
evaluates it against the trail input using OPA, exits 0 on allow
and 1 on deny. Uses Rego v1 syntax. Policy must use package policy
and declare an allow rule.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…e and violations

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…wed text

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…lations

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…violations

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tests: non-map passthrough, no compliance_status passthrough,
empty array to empty map, single attestation keyed by name.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tests: multiple trail attestations, artifact-level, both levels,
multiple artifacts, entries without attestation_name skipped.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…maps

Wire TransformTrail into evaluateTrail after JSON parse. Tests verify
trail-level and artifact-level maps, plus Rego policy access by name.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…tion

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…vels

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…te existing fields

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
tooky and others added 21 commits February 28, 2026 07:08
…hanged

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…evel attestation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…mixed, no-match)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…trail

Slice 5c complete: FilterAttestations filters trail-level (plain name)
and artifact-level (dot-qualified) attestations before rehydration,
saving API calls for excluded attestations.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Aligns evaluate trail with the codebase convention of --output table|json
(with -o shorthand, defaulting to "table") instead of --format text|json.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds `kosli evaluate trails TRAIL-NAME [TRAIL-NAME...]` command that
fetches multiple trails, wraps them in {"trails": [...]} input shape,
and evaluates against a Rego policy. Supports --policy, --output,
and --show-input flags with the same semantics as evaluate trail.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds rehydration (CollectAttestationIDs + RehydrateTrail) and
--attestations filtering to evaluate trails, matching the same
enrichment pipeline as evaluate trail. Each trail is independently
transformed, filtered, and rehydrated before being collected into
the trails array.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove the no-policy code path (raw JSON dump) since there's no use
case for it without evaluation. Add "policy" to RequireFlags, remove
early-return blocks, update tests accordingly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move fetchAndEnrichTrail, evaluateAndPrintResult, and printEvaluateInput
into evaluateHelpers.go. Both evaluate trail and evaluate trails now
delegate to these shared functions, eliminating duplicated code.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Both evaluateTrailOptions and evaluateTrailsOptions now embed
commonEvaluateOptions instead of duplicating the same five fields.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace manual if/else output format branching with the
output.FormattedPrint dispatcher pattern used by other commands
like getTrail. Adds printEvaluateResultAsJson and
printEvaluateResultAsTable format functions. Removes the manual
--output validation from both run methods, letting FormattedPrint
handle unsupported formats consistently.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…etching

Log HTTP and JSON parse errors at debug level when fetching
attestation details during trail enrichment, instead of silently
continuing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use assert for leaf value checks in TestTransformTrail, consistent
with the other test functions in the file. Keep require only for
the IsType guard that validates the type before subsequent access.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ration

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…sform.go

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ailEnrichment

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fixes SNYK-GOLANG-GOLANGORGXNETHTTP2-15363313 — missing nil check
in http2 causing server panic on malformed frames.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant