Add solitary/sociable/integration test levels to agent prompts and quality gate#2545
Merged
hiroshinishio merged 1 commit intomainfrom Apr 17, 2026
Merged
Add solitary/sociable/integration test levels to agent prompts and quality gate#2545hiroshinishio merged 1 commit intomainfrom
hiroshinishio merged 1 commit intomainfrom
Conversation
…ality gate - Define 3 test levels in coding_standards.xml: solitary (mocked), sociable (real collaborators), integration (real external services) - Add integration test quality gate in checklist.py (db_operations_use_real_test_db, api_calls_tested_end_to_end, env_var_guards_for_secrets) - Add env var guard patterns for free services (skip locally, run in CI) and paid APIs (always skip) - Remove duplicated rules from pr.xml that already exist in coding_standards.xml - Update PR title from "Add unit tests" to "Add unit and integration tests" - Prefer "test" over "test:unit" npm script to run all test levels - Update "unit test" references globally in prompts, constants, and test fixtures
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Social Media Post (GitAuto)
Upgrade test generation to produce solitary, sociable, and integration tests
Social Media Post (Wes)
Customer told us they wanted more integration tests from our agent. Checked the PRs. Every single one was pure mocks, even when their repo had test DB helpers sitting right there. The prompt said "use test infrastructure if available" but the model ignored it every time. Why? One buried line loses against the rest of the prompt biasing toward mocks. Prompt alone is not enough. Added a quality gate that programmatically flags missing integration tests.