Tracking issue for auditing and possibly eliminating difference between PR CI and Auto CI jobs

[jieyouxu]

Currently, rust-lang/rust PR jobs are not a subset of Auto jobs in the strict sense. That is, a defined PR job does not imply it is run with the exact same configuration in Auto CI.

Why might this be a problem?

When there are CI jobs (or job configurations) that are PR-only, and not also run as part of Auto jobs, it's possible to have master end up in a state where all subsequent PR jobs will be red, even in unrelated PRs. See for instance #144183 and discussions in #t-infra > Spellcheck workflow now fails on all PRs (tree bad?).

Potentially ideal CI invariant

Ideally, we may want to enforce the CI invariant that PR CI jobs are a subset of Full CI jobs in the strict-sense, with no carve-outs (other than fail-fast or not).

• See Enforce that PR CI jobs are a subset of Auto CI jobs modulo carve-outs #144244 where a weaker version of the subset relationship with env and continue_on_error carve-outs is enforced. However, env being excluded is quite substantial, because env vars can substantially enforce what gets run.

Kinds of differences

Warning

This listing is non-exhaustive. Please feel free to add more known differences.

• x86_64-gnu-tools: gets DEPLOY_TOOLSTATES_JSON: toolstates-linux.json set only in Auto environment.
  • Auto:

    rust/src/ci/github-actions/jobs.yml

    Lines 440 to 443 in 6781992

    	- name: x86_64-gnu-tools
    	env:
    	DEPLOY_TOOLSTATES_JSON: toolstates-linux.json
    	<<: *job-linux-4c

  • PR:

    rust/src/ci/github-actions/jobs.yml

    Lines 158 to 159 in 6781992

    	- name: x86_64-gnu-tools
    	<<: *job-linux-4c

Implementation history

• Enforce that PR CI jobs are a subset of Auto CI jobs modulo carve-outs #144244

Non-blocking jobs

It was discussed that we should use/implement toolstate or some other mechanism for experimental jobs, and not expose contributors to failing PR CI if said job does break.