User journey: as a template author who maintains templates outside of the abc repo, I want an easy way to test my templates as part of the pull request process.
We could create a single command that would be invoked in anon: pull_request GitHub Action. This command would:
- Make sure all templates have at least one golden test. If not, print a helpful message about using the
new-test command which will be released soon.
- Make sure all golden tests are passing (
verify). If they're failing due to a test data mismatch, suggest running the record command. If they're failing due to a broken spec.yaml, identify the failing spec.yaml and describe the problem with it.
All problems should be reported, not just the first problem that is found.
The command could be abc templates golden-test check. It could have an optional parameter giving the starting directory.
We could also provide a github action that would download the latest build of abc and run this new command.
A good first customer for this would be the github.com/abcxyz/guardian repo. @verbanicm @gjonathanhong
EDIT:
Here's another possibility: expand the scope of this from just golden-tests to everything having to do with abc templates. We could have a command abc templates check-clean that would perform all the above checks, and also look for upgrade operations with incomplete merges (there should be no files with an .abcmerge_... suffix).
User journey: as a template author who maintains templates outside of the abc repo, I want an easy way to test my templates as part of the pull request process.
We could create a single command that would be invoked in an
on: pull_requestGitHub Action. This command would:new-testcommand which will be released soon.verify). If they're failing due to a test data mismatch, suggest running therecordcommand. If they're failing due to a broken spec.yaml, identify the failing spec.yaml and describe the problem with it.All problems should be reported, not just the first problem that is found.
The command could be
abc templates golden-test check. It could have an optional parameter giving the starting directory.We could also provide a github action that would download the latest build of
abcand run this new command.A good first customer for this would be the github.com/abcxyz/guardian repo. @verbanicm @gjonathanhong
EDIT:
Here's another possibility: expand the scope of this from just golden-tests to everything having to do with abc templates. We could have a command
abc templates check-cleanthat would perform all the above checks, and also look for upgrade operations with incomplete merges (there should be no files with an.abcmerge_...suffix).