feat(ci): add test failure retry support to auto-rerun workflow

[radical]

Description

Adds test failure retry support to the auto-rerun transient CI failures workflow. When a CI job fails due to test execution failures, the workflow now downloads TRX test result artifacts, parses them, and matches individual test failures against configurable retry-safe patterns (e.g., network errors, MCR rate limiting).

The design of the retry patterns configuration was inspired by arcade's test configuration JSON schema.

Retry Patterns Config (eng/test-retry-patterns.json)

The new eng/test-retry-patterns.json file defines which test and job failures are safe to automatically retry. It uses a declarative JSON format with two pattern arrays:

Structure

{
  "version": 1,
  "testFailurePatterns": [ ... ],
  "jobFailurePatterns": [ ... ]
}

Each pattern has:

• output — a substring or { "regex": "..." } matched against test output or job logs
• reason — human-readable explanation shown in PR comments and workflow summaries
• testName (optional, test patterns only) — filter by test name (substring or regex)
• testProject (optional, test patterns only) — filter by test project/assembly name
• jobName (optional, job patterns only) — filter by job name (substring or regex)

Example: Simple substring match

Match any test whose output contains ECONNRESET:

{
  "output": "ECONNRESET",
  "reason": "Transient network connection reset"
}

Example: Regex pattern

Match MCR rate-limiting responses (403 within ~500 chars of the MCR URL):

{
  "output": { "regex": "mcr\\.microsoft\\.com[\\s\\S]{0,500}403 Forbidden" },
  "reason": "MCR registry rate limiting (HTTP 403)"
}

Example: Job-scoped pattern with job name filter

Match the Windows process init failure only on Windows jobs:

{
  "jobName": { "regex": ".*windows.*" },
  "output": "0xC0000142",
  "reason": "Windows process initialization failure (0xC0000142)"
}

Example: Test-scoped pattern with test name filter

Match a specific flaky test by name and output:

{
  "testName": { "regex": ".*MyFlakyTest.*" },
  "output": "Connection refused",
  "reason": "Known transient failure in MyFlakyTest"
}

Adding new patterns

1. Edit eng/test-retry-patterns.json
2. Add a new entry to testFailurePatterns or jobFailurePatterns
3. The config is validated at runtime — unknown properties, missing reason, or invalid regex will produce clear error messages
4. Run the Infrastructure.Tests to verify: dotnet test --project tests/Infrastructure.Tests/Infrastructure.Tests.csproj

Checklist

• Is this feature complete?
  • Yes. Ready to ship.
• Are you including unit tests for the changes and scenario tests if relevant?
  • Yes
• Did you add public API?
  • No
• Does the change make any security assumptions or guarantees?
  • No
• Does the change require an update in our Aspire docs?
  • No

[github-actions]

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 16446

Or

• Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 16446"

[Copilot]

Pull request overview

Adds configurable retry-safe matching for test execution failures in the “auto-rerun transient CI failures” workflow by introducing a versioned retry-patterns JSON config, TRX artifact analysis, and expanded test coverage/documentation.

Changes:

• Introduces eng/test-retry-patterns.json to define retry-safe test/job failure patterns (substring or regex + reasons).
• Extends the workflow to load the config, match job logs for test failures, download/unpack All-TestResults, parse TRX files, and promote test-execution-failure jobs when patterns match.
• Adds extensive Infrastructure.Tests coverage plus updated documentation describing the new matching passes and configuration format.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
tests/Infrastructure.Tests/WorkflowScripts/auto-rerun-transient-ci-failures.harness.js	Expands the Node harness surface area to invoke new JS helper functions from C# tests.
tests/Infrastructure.Tests/WorkflowScripts/AutoRerunTransientCiFailuresTests.cs	Adds many xUnit tests for config validation, matching semantics, TRX parsing, artifact selection, and promotion logic.
eng/test-retry-patterns.json	Adds the initial curated set of retry-safe patterns for test output and job logs.
docs/ci/auto-rerun-transient-ci-failures.md	Documents the new multi-pass analysis and how to configure retry patterns.
.github/workflows/auto-rerun-transient-ci-failures.yml	Loads retry-pattern config, performs job-log + TRX-based analysis, and forwards matched test info to the rerun/comment step.
.github/workflows/auto-rerun-transient-ci-failures.js	Implements config loading/validation and the new pattern matching + TRX parsing/promotion utilities.