From 38aac056ddadfde53aa1c35b78940072568d9492 Mon Sep 17 00:00:00 2001 From: Timo van Veenendaal Date: Wed, 22 Apr 2026 13:35:55 -0700 Subject: [PATCH 1/7] Live and recorded tests skill --- .../SKILL.md | 52 +++++++++++++++++++ .../eval.yaml | 29 +++++++++++ .../references/test-modes.md | 31 +++++++++++ .../references/test-resource-cleanup.md | 37 +++++++++++++ .../references/test-resource-deployment.md | 48 +++++++++++++++++ .../tasks/anti-trigger-write-tests.yaml | 15 ++++++ .../tasks/anti-trigger.yaml | 15 ++++++ .../tasks/cleanup-resources.yaml | 18 +++++++ .../tasks/deploy-and-run-live.yaml | 22 ++++++++ .../tasks/record-with-env.yaml | 22 ++++++++ 10 files changed, 289 insertions(+) create mode 100644 .github/skills/azsdk-common-live-and-recorded-tests/SKILL.md create mode 100644 .github/skills/azsdk-common-live-and-recorded-tests/eval.yaml create mode 100644 .github/skills/azsdk-common-live-and-recorded-tests/references/test-modes.md create mode 100644 .github/skills/azsdk-common-live-and-recorded-tests/references/test-resource-cleanup.md create mode 100644 .github/skills/azsdk-common-live-and-recorded-tests/references/test-resource-deployment.md create mode 100644 .github/skills/azsdk-common-live-and-recorded-tests/tasks/anti-trigger-write-tests.yaml create mode 100644 .github/skills/azsdk-common-live-and-recorded-tests/tasks/anti-trigger.yaml create mode 100644 .github/skills/azsdk-common-live-and-recorded-tests/tasks/cleanup-resources.yaml create mode 100644 .github/skills/azsdk-common-live-and-recorded-tests/tasks/deploy-and-run-live.yaml create mode 100644 .github/skills/azsdk-common-live-and-recorded-tests/tasks/record-with-env.yaml diff --git a/.github/skills/azsdk-common-live-and-recorded-tests/SKILL.md b/.github/skills/azsdk-common-live-and-recorded-tests/SKILL.md new file mode 100644 index 000000000000..3fa682c758c8 --- /dev/null +++ b/.github/skills/azsdk-common-live-and-recorded-tests/SKILL.md @@ -0,0 +1,52 @@ +--- +name: azsdk-common-live-and-recorded-tests +license: MIT +metadata: + version: "0.1.0" + distribution: shared +description: "Deploy test resources and run Azure SDK tests in live, record, or playback mode. WHEN: \"run live tests\", \"run recorded tests\", \"deploy test resources\", \"record tests\", \"run tests in record mode\", \"clean up test resources\", \"run tests against live resources\". DO NOT USE FOR: writing new tests, authoring Bicep templates, playback-only test runs without resource deployment. INVOKES: azure-sdk-mcp:azsdk_package_run_tests." +compatibility: + requires: "azure-sdk-mcp server, Azure PowerShell (Az module), local azure-sdk-for-{language} clone" +--- + +# Live and Recorded Tests + +## MCP Tools + +| Tool | Purpose | +|------|---------| +| `azure-sdk-mcp:azsdk_package_run_tests` | Run tests in playback, record, or live mode | + +## Prerequisites + +- azure-sdk-mcp server must be running +- Azure PowerShell module (`Az`) installed +- For live/record modes: an Azure subscription with permissions to create resources + - Use the Azure SDK Test Resources - TME (id: 4d042dc6-fe17-4698-a23f-ec6a8d1e98f4) subscription if not already authenticated and no other subscription is specified. + +## Steps + +1. **Identify service directory** — Determine the service directory for the package under test (e.g., `keyvault`, `storage`). This is required by the test resource deployment script. +2. **Check for existing deployment** — Look for a `.env` file at the service directory level (next to `test-resources.bicep`). If one exists, inform the user that a previous deployment appears to be available and ask whether to **reuse the existing deployment** or **redeploy test resources**. If reusing, skip to step 7. +3. **Verify Azure context** — Run `Get-AzContext` to check for an active Azure PowerShell session. If no context exists, guide the user through `Connect-AzAccount`. Confirm the correct subscription is selected. +4. **Confirm deployment** — Even if no existing `.env` file is found, confirm with the user before proceeding to deploy test resources. Deployment creates Azure resources that may incur costs. +5. **Deploy test resources** — Run `eng/common/TestResources/New-TestResources.ps1` with the service directory and any user-provided parameters. See [deployment parameters](references/test-resource-deployment.md) for details. The script outputs environment variables needed for live/record test runs. +6. **Save environment** — Capture the environment variables output by the deployment script. If the script writes a `.env` file, note its path. Otherwise, collect the environment variables from the script output. +7. **Run tests** — Run `azure-sdk-mcp:azsdk_package_run_tests` with the appropriate test mode (`record`, `live`, or `playback`) and the path to the `.env` file containing test environment variables. When tests run in record mode and all tests pass, the tool automatically pushes recorded test assets to the assets repo. +8. **Clean up** — Ask the user whether to clean up test resources. If yes, run `eng/common/TestResources/Remove-TestResources.ps1`. If no, inform the user that resources remain deployed for subsequent test runs. See [cleanup details](references/test-resource-cleanup.md). + +## Examples + +- "Deploy test resources and run live tests for this package" +- "Run the tests in record mode using the live test deployment specified in .env" +- "Clean up my test resources for keyvault" +- "Run tests in record mode against my existing deployment" +- "Set up live test resources for storage and run all tests" + +## Troubleshooting + +- **No Azure context:** Run `Connect-AzAccount` and select the target subscription with `Set-AzContext -SubscriptionId `. +- **Deployment fails with auth error:** Verify that the signed-in account has Contributor or Owner role on the target subscription. Check `Get-AzContext` output. +- **Deployment fails with resource conflict:** A previous deployment may still exist. Try running `Remove-TestResources.ps1` first, or use a different `BaseName`. +- **Tests fail in record mode:** Check that all environment variables from the deployment are being passed correctly. Verify the `.env` file path is correct. +- **Assets push fails after recording:** Ensure the assets repo is configured and accessible. Check git authentication. diff --git a/.github/skills/azsdk-common-live-and-recorded-tests/eval.yaml b/.github/skills/azsdk-common-live-and-recorded-tests/eval.yaml new file mode 100644 index 000000000000..357b0165c999 --- /dev/null +++ b/.github/skills/azsdk-common-live-and-recorded-tests/eval.yaml @@ -0,0 +1,29 @@ +name: azsdk-common-live-and-recorded-tests-eval +description: Evaluation suite for live and recorded tests skill. +skill: azsdk-common-live-and-recorded-tests +version: '1.0' +config: + model: gpt-4o + executor: mock + trials_per_task: 1 + timeout_seconds: 600 + parallel: false +metrics: + - name: task_completion + weight: 0.7 + threshold: 0.8 + description: Did the skill complete trigger and anti-trigger checks? + - name: efficiency + weight: 0.3 + threshold: 0.7 + description: Did the skill stay within behavior limits? +graders: + - type: text + name: no_fatal_errors + config: + regex_not_match: + - (?i)error occurred + - (?i)failed + - (?i)re-run the tool +tasks: + - "tasks/*.yaml" diff --git a/.github/skills/azsdk-common-live-and-recorded-tests/references/test-modes.md b/.github/skills/azsdk-common-live-and-recorded-tests/references/test-modes.md new file mode 100644 index 000000000000..410cf8e26705 --- /dev/null +++ b/.github/skills/azsdk-common-live-and-recorded-tests/references/test-modes.md @@ -0,0 +1,31 @@ +# Test Modes + +## Overview + +| Mode | Description | Requires Azure Resources | Records Traffic | +|------|-------------|--------------------------|-----------------| +| **Playback** | Runs against previously recorded HTTP interactions | No | No | +| **Record** | Runs against real Azure resources and records HTTP interactions | Yes | Yes | +| **Live** | Runs against real Azure resources without recording | Yes | No | + +## Default Behavior + +When no test mode is specified, **Playback** is the default. Playback mode does not require test resource deployment or environment variables. + +## Record Mode + +- Requires deployed test resources and environment variables passed via a `.env` file. +- When all tests pass in record mode, the tool automatically pushes recorded test assets to the assets repo. +- This is the standard workflow for creating or updating test recordings. + +## Live Mode + +- Requires deployed test resources and environment variables passed via a `.env` file. +- Tests run against real Azure resources without recording HTTP traffic. +- Useful for validating that tests work against real services without updating recordings. + +## Playback Mode + +- Does not require deployed Azure resources. +- Tests replay previously recorded HTTP interactions via the test proxy. +- This is the default mode and the fastest way to run tests locally. diff --git a/.github/skills/azsdk-common-live-and-recorded-tests/references/test-resource-cleanup.md b/.github/skills/azsdk-common-live-and-recorded-tests/references/test-resource-cleanup.md new file mode 100644 index 000000000000..138275a70fec --- /dev/null +++ b/.github/skills/azsdk-common-live-and-recorded-tests/references/test-resource-cleanup.md @@ -0,0 +1,37 @@ +# Test Resource Cleanup + +## Script Location + +`eng/common/TestResources/Remove-TestResources.ps1` + +## Common Parameters + +| Parameter | Required | Description | +|-----------|----------|-------------| +| `ServiceDirectory` | Yes* | Service directory name (e.g., `keyvault`). Required if `ResourceGroupName` not provided. | +| `-ResourceGroupName` | No | Explicit resource group name to delete | +| `-Force` | No | Skip confirmation prompt | + +*Either `ServiceDirectory` or `-ResourceGroupName` must be provided. If the test resources were deployed to the TME tenant, you must use the resource group name, which should include the `SSS3PT_` prefix as output by the test resource creation script. + +## Usage Examples + +### Basic cleanup + +```powershell +eng/common/TestResources/Remove-TestResources.ps1 keyvault -Force +``` + +### Cleanup by resource group name + +```powershell +eng/common/TestResources/Remove-TestResources.ps1 ` + -ResourceGroupName 'rg-mytest' ` + -Force +``` + +## Notes + +- The script also purges purgeable resources (e.g., Key Vault soft-deleted vaults). +- Any `.env` files created by `New-TestResources.ps1` are also removed. +- Always ask the user before cleaning up — they may want to reuse resources for additional test runs. diff --git a/.github/skills/azsdk-common-live-and-recorded-tests/references/test-resource-deployment.md b/.github/skills/azsdk-common-live-and-recorded-tests/references/test-resource-deployment.md new file mode 100644 index 000000000000..843ece1416aa --- /dev/null +++ b/.github/skills/azsdk-common-live-and-recorded-tests/references/test-resource-deployment.md @@ -0,0 +1,48 @@ +# Test Resource Deployment + +## Script Location + +`eng/common/TestResources/New-TestResources.ps1` + +## Common Parameters + +| Parameter | Required | Description | +|-----------|----------|-------------| +| `ServiceDirectory` | Yes | Service directory name under `sdk/` (e.g., `keyvault`, `storage`) | +| `-OutFile` | Yes | Required for a .env file to be output. | +| `-BaseName` | No | Base name for resource group (default: auto-generated) | +| `-Location` | No | Azure region for deployment (default: script default) | +| `-SubscriptionId` | No | Target subscription (uses current Az context if omitted) | +| `-ResourceGroupName` | No | Custom resource group name | + +## Usage Examples + +### Basic deployment (interactive, uses current Az context) + +```powershell +Connect-AzAccount -Subscription 'SUBSCRIPTION_ID' +eng/common/TestResources/New-TestResources.ps1 keyvault -OutFile +``` + +### Deployment with custom parameters + +```powershell +Connect-AzAccount -Subscription 'SUBSCRIPTION_ID' +eng/common/TestResources/New-TestResources.ps1 ` + -BaseName 'azsdk' ` + -ServiceDirectory 'keyvault' ` + -SubscriptionId 'SUBSCRIPTION_ID' ` + -ResourceGroupName 'rg-mytest' ` + -Location 'eastus' + -OutFile +``` + +## Output + +The script outputs environment variables needed for live and record test runs. If `$SupportsTestResourcesDotenv=$true` in the language repo's `LanguageSettings.ps1`, a `.env` file is written next to the `test-resources.bicep` file. + +Environment variables typically include credentials and resource endpoints needed by test frameworks. + +## Additional Parameters + +Some services require extra parameters (e.g., `-AdditionalParameters @{enableHsm=$true}` for Key Vault HSM tests). Check the service's `test-resources.bicep` or `test-resources.json` for required parameters. diff --git a/.github/skills/azsdk-common-live-and-recorded-tests/tasks/anti-trigger-write-tests.yaml b/.github/skills/azsdk-common-live-and-recorded-tests/tasks/anti-trigger-write-tests.yaml new file mode 100644 index 000000000000..d02fe9469685 --- /dev/null +++ b/.github/skills/azsdk-common-live-and-recorded-tests/tasks/anti-trigger-write-tests.yaml @@ -0,0 +1,15 @@ +id: live-tests-negative-002 +name: Anti-Trigger - Write New Tests +description: | + Test that the skill does NOT activate when the user wants to + author new test cases rather than run existing tests. +tags: + - negative-test + - anti-trigger +inputs: + prompt: "Help me write unit tests for my new Azure SDK client" +expected: + output_not_contains: + - "New-TestResources" + - "azsdk_package_run_tests" + - "Connect-AzAccount" diff --git a/.github/skills/azsdk-common-live-and-recorded-tests/tasks/anti-trigger.yaml b/.github/skills/azsdk-common-live-and-recorded-tests/tasks/anti-trigger.yaml new file mode 100644 index 000000000000..a7da18e73ef4 --- /dev/null +++ b/.github/skills/azsdk-common-live-and-recorded-tests/tasks/anti-trigger.yaml @@ -0,0 +1,15 @@ +id: live-tests-negative-001 +name: Anti-Trigger - Playback Only Test Run +description: | + Test that the skill does NOT activate for playback-only test runs + that don't need resource deployment. +tags: + - negative-test + - anti-trigger +inputs: + prompt: "Run the tests for this package in playback mode" +expected: + output_not_contains: + - "New-TestResources" + - "Connect-AzAccount" + - "Get-AzContext" diff --git a/.github/skills/azsdk-common-live-and-recorded-tests/tasks/cleanup-resources.yaml b/.github/skills/azsdk-common-live-and-recorded-tests/tasks/cleanup-resources.yaml new file mode 100644 index 000000000000..90822086f975 --- /dev/null +++ b/.github/skills/azsdk-common-live-and-recorded-tests/tasks/cleanup-resources.yaml @@ -0,0 +1,18 @@ +id: cleanup-resources-001 +name: Clean Up Test Resources +description: | + Test that the skill offers and performs cleanup of test resources + after a test run. +tags: + - basic + - happy-path + - cleanup +inputs: + prompt: "Clean up the test resources I deployed for keyvault" +expected: + output_contains: + - "Remove-TestResources" + outcomes: [] + behavior: + max_tool_calls: 5 + max_response_time_ms: 60000 diff --git a/.github/skills/azsdk-common-live-and-recorded-tests/tasks/deploy-and-run-live.yaml b/.github/skills/azsdk-common-live-and-recorded-tests/tasks/deploy-and-run-live.yaml new file mode 100644 index 000000000000..bf687f176167 --- /dev/null +++ b/.github/skills/azsdk-common-live-and-recorded-tests/tasks/deploy-and-run-live.yaml @@ -0,0 +1,22 @@ +id: live-tests-basic-001 +name: Deploy Test Resources and Run Live Tests +description: | + Test that the skill guides through deploying test resources and + running tests in live mode for a package. +tags: + - basic + - happy-path + - live-mode +inputs: + prompt: "Deploy test resources and run live tests for the keyvault package at ~/repos/azure-sdk-for-python/sdk/keyvault/azure-keyvault-keys" +expected: + output_contains: + - "Get-AzContext" + - "New-TestResources" + - "live" + outcomes: + - type: tool_called + tool_name: azsdk_package_run_tests + behavior: + max_tool_calls: 10 + max_response_time_ms: 60000 diff --git a/.github/skills/azsdk-common-live-and-recorded-tests/tasks/record-with-env.yaml b/.github/skills/azsdk-common-live-and-recorded-tests/tasks/record-with-env.yaml new file mode 100644 index 000000000000..445122602189 --- /dev/null +++ b/.github/skills/azsdk-common-live-and-recorded-tests/tasks/record-with-env.yaml @@ -0,0 +1,22 @@ +id: record-tests-with-env-001 +name: Run Tests in Record Mode with Existing Environment +description: | + Test that the skill runs tests in record mode when a .env file + is already provided, skipping resource deployment. +tags: + - basic + - happy-path + - record-mode +inputs: + prompt: "Run the tests in record mode using the live test deployment specified in sdk/keyvault/azure-keyvault-keys/.env" +expected: + output_contains: + - "record" + output_not_contains: + - "New-TestResources" + outcomes: + - type: tool_called + tool_name: azsdk_package_run_tests + behavior: + max_tool_calls: 5 + max_response_time_ms: 60000 From d450b37f5d09348225588f001ff7e673ffaab041 Mon Sep 17 00:00:00 2001 From: Timo van Veenendaal Date: Thu, 23 Apr 2026 13:28:16 -0700 Subject: [PATCH 2/7] Make it more clear that the MCP tool should be used Co-authored-by: Copilot --- .github/skills/azsdk-common-live-and-recorded-tests/SKILL.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/.github/skills/azsdk-common-live-and-recorded-tests/SKILL.md b/.github/skills/azsdk-common-live-and-recorded-tests/SKILL.md index 3fa682c758c8..32f5376cafee 100644 --- a/.github/skills/azsdk-common-live-and-recorded-tests/SKILL.md +++ b/.github/skills/azsdk-common-live-and-recorded-tests/SKILL.md @@ -17,6 +17,8 @@ compatibility: |------|---------| | `azure-sdk-mcp:azsdk_package_run_tests` | Run tests in playback, record, or live mode | +> **IMPORTANT:** ALWAYS use the `azure-sdk-mcp:azsdk_package_run_tests` MCP tool to run tests. NEVER run test commands directly in the terminal (e.g., `pytest`, `dotnet test`, `mvn test`, `npm test`, `go test`). The MCP tool handles test mode configuration, environment setup, and automatic asset pushing in record mode. + ## Prerequisites - azure-sdk-mcp server must be running @@ -32,7 +34,7 @@ compatibility: 4. **Confirm deployment** — Even if no existing `.env` file is found, confirm with the user before proceeding to deploy test resources. Deployment creates Azure resources that may incur costs. 5. **Deploy test resources** — Run `eng/common/TestResources/New-TestResources.ps1` with the service directory and any user-provided parameters. See [deployment parameters](references/test-resource-deployment.md) for details. The script outputs environment variables needed for live/record test runs. 6. **Save environment** — Capture the environment variables output by the deployment script. If the script writes a `.env` file, note its path. Otherwise, collect the environment variables from the script output. -7. **Run tests** — Run `azure-sdk-mcp:azsdk_package_run_tests` with the appropriate test mode (`record`, `live`, or `playback`) and the path to the `.env` file containing test environment variables. When tests run in record mode and all tests pass, the tool automatically pushes recorded test assets to the assets repo. +7. **Run tests** — Call the `azure-sdk-mcp:azsdk_package_run_tests` MCP tool (do NOT run test commands directly in the terminal). Provide the appropriate test mode (`record`, `live`, or `playback`) and the path to the `.env` file containing test environment variables. When tests run in record mode and all tests pass, the tool automatically pushes recorded test assets to the assets repo. 8. **Clean up** — Ask the user whether to clean up test resources. If yes, run `eng/common/TestResources/Remove-TestResources.ps1`. If no, inform the user that resources remain deployed for subsequent test runs. See [cleanup details](references/test-resource-cleanup.md). ## Examples From 27caa84a1d177efd6146a459ca41e87f6d1461ce Mon Sep 17 00:00:00 2001 From: Timo van Veenendaal Date: Thu, 23 Apr 2026 13:33:07 -0700 Subject: [PATCH 3/7] Add some troubleshooting to test resource deployment Co-authored-by: Copilot --- .../references/test-resource-deployment.md | 35 ++++++++++++++++++- 1 file changed, 34 insertions(+), 1 deletion(-) diff --git a/.github/skills/azsdk-common-live-and-recorded-tests/references/test-resource-deployment.md b/.github/skills/azsdk-common-live-and-recorded-tests/references/test-resource-deployment.md index 843ece1416aa..6bd16b04489b 100644 --- a/.github/skills/azsdk-common-live-and-recorded-tests/references/test-resource-deployment.md +++ b/.github/skills/azsdk-common-live-and-recorded-tests/references/test-resource-deployment.md @@ -33,7 +33,7 @@ eng/common/TestResources/New-TestResources.ps1 ` -ServiceDirectory 'keyvault' ` -SubscriptionId 'SUBSCRIPTION_ID' ` -ResourceGroupName 'rg-mytest' ` - -Location 'eastus' + -Location 'eastus' ` -OutFile ``` @@ -46,3 +46,36 @@ Environment variables typically include credentials and resource endpoints neede ## Additional Parameters Some services require extra parameters (e.g., `-AdditionalParameters @{enableHsm=$true}` for Key Vault HSM tests). Check the service's `test-resources.bicep` or `test-resources.json` for required parameters. + +## Troubleshooting + +### Resource already exists / deployment conflict + +If deployment fails because a resource or resource group already exists: + +1. Check whether an existing deployment is present by running: + ```powershell + Get-AzResourceGroup -Name '' -ErrorAction SilentlyContinue + ``` +2. Ask the user whether they want to **remove the existing deployment** and redeploy, or **deploy with a different resource group name**. +3. If removing, run `Remove-TestResources.ps1` first (see [cleanup details](test-resource-cleanup.md)), then retry the deployment. +4. If the conflict is on a specific resource (e.g., a Key Vault in soft-deleted state), purge it manually: + ```powershell + Remove-AzKeyVault -VaultName '' -InRemovedState -Force -Location '' + ``` + +### Authentication or authorization failure + +- Verify the signed-in account has **Contributor** or **Owner** role on the target subscription: `Get-AzContext`. +- If no context exists, run `Connect-AzAccount` and select the correct subscription with `Set-AzContext -SubscriptionId `. +- For the TME subscription, ensure the account has been granted access to subscription `4d042dc6-fe17-4698-a23f-ec6a8d1e98f4`. + +### Deployment times out + +- Some services (e.g., Cosmos DB, HDInsight) have long provisioning times. +- Check the Azure portal for the resource group's deployment status to identify which resource is slow. + +### Missing .env file after deployment + +- Ensure `-OutFile` was passed to `New-TestResources.ps1`. +- Verify the language repo's `eng/scripts/LanguageSettings.ps1` sets `$SupportsTestResourcesDotenv = $true`. If it does not, environment variables must be collected from the script's console output instead. From d9a563868e0c2447c2089ab6f55badf0ea0ece89 Mon Sep 17 00:00:00 2001 From: Timo van Veenendaal Date: Thu, 23 Apr 2026 14:37:36 -0700 Subject: [PATCH 4/7] Also check for a .env file at the package level Co-authored-by: Copilot --- .github/skills/azsdk-common-live-and-recorded-tests/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/skills/azsdk-common-live-and-recorded-tests/SKILL.md b/.github/skills/azsdk-common-live-and-recorded-tests/SKILL.md index 32f5376cafee..4c53ad395f60 100644 --- a/.github/skills/azsdk-common-live-and-recorded-tests/SKILL.md +++ b/.github/skills/azsdk-common-live-and-recorded-tests/SKILL.md @@ -29,7 +29,7 @@ compatibility: ## Steps 1. **Identify service directory** — Determine the service directory for the package under test (e.g., `keyvault`, `storage`). This is required by the test resource deployment script. -2. **Check for existing deployment** — Look for a `.env` file at the service directory level (next to `test-resources.bicep`). If one exists, inform the user that a previous deployment appears to be available and ask whether to **reuse the existing deployment** or **redeploy test resources**. If reusing, skip to step 7. +2. **Check for existing deployment** — Look for a `.env` file at either the service directory level (next to `test-resources.bicep`, e.g. `sdk/storage/.env`) or at the package level (e.g. `sdk/storage/storage-blob/.env`). If one exists, inform the user that a previous deployment appears to be available and ask whether to **reuse the existing deployment** or **redeploy test resources**. If reusing, skip to step 7. 3. **Verify Azure context** — Run `Get-AzContext` to check for an active Azure PowerShell session. If no context exists, guide the user through `Connect-AzAccount`. Confirm the correct subscription is selected. 4. **Confirm deployment** — Even if no existing `.env` file is found, confirm with the user before proceeding to deploy test resources. Deployment creates Azure resources that may incur costs. 5. **Deploy test resources** — Run `eng/common/TestResources/New-TestResources.ps1` with the service directory and any user-provided parameters. See [deployment parameters](references/test-resource-deployment.md) for details. The script outputs environment variables needed for live/record test runs. From bc91d2e04b90153efcd77fa6d2ae102c131a8cc6 Mon Sep 17 00:00:00 2001 From: Timo van Veenendaal Date: Mon, 27 Apr 2026 10:53:50 -0700 Subject: [PATCH 5/7] Update .github/skills/azsdk-common-live-and-recorded-tests/SKILL.md Co-authored-by: Maddy Heaps <66138537+m-redding@users.noreply.github.com> --- .github/skills/azsdk-common-live-and-recorded-tests/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/skills/azsdk-common-live-and-recorded-tests/SKILL.md b/.github/skills/azsdk-common-live-and-recorded-tests/SKILL.md index 4c53ad395f60..ff507a3f9dd4 100644 --- a/.github/skills/azsdk-common-live-and-recorded-tests/SKILL.md +++ b/.github/skills/azsdk-common-live-and-recorded-tests/SKILL.md @@ -30,7 +30,7 @@ compatibility: 1. **Identify service directory** — Determine the service directory for the package under test (e.g., `keyvault`, `storage`). This is required by the test resource deployment script. 2. **Check for existing deployment** — Look for a `.env` file at either the service directory level (next to `test-resources.bicep`, e.g. `sdk/storage/.env`) or at the package level (e.g. `sdk/storage/storage-blob/.env`). If one exists, inform the user that a previous deployment appears to be available and ask whether to **reuse the existing deployment** or **redeploy test resources**. If reusing, skip to step 7. -3. **Verify Azure context** — Run `Get-AzContext` to check for an active Azure PowerShell session. If no context exists, guide the user through `Connect-AzAccount`. Confirm the correct subscription is selected. +3. **Verify Azure context** — Run `Get-AzContext` to check for an active Azure PowerShell session. If no context exists, guide the user through `Connect-AzAccount -TenantId [TME tenant ID]`. Confirm the correct subscription is selected. 4. **Confirm deployment** — Even if no existing `.env` file is found, confirm with the user before proceeding to deploy test resources. Deployment creates Azure resources that may incur costs. 5. **Deploy test resources** — Run `eng/common/TestResources/New-TestResources.ps1` with the service directory and any user-provided parameters. See [deployment parameters](references/test-resource-deployment.md) for details. The script outputs environment variables needed for live/record test runs. 6. **Save environment** — Capture the environment variables output by the deployment script. If the script writes a `.env` file, note its path. Otherwise, collect the environment variables from the script output. From aa64ab77c6ef7c527b614cf41b9f2f78ceb4c473 Mon Sep 17 00:00:00 2001 From: Timo van Veenendaal Date: Mon, 27 Apr 2026 10:55:19 -0700 Subject: [PATCH 6/7] Update .github/skills/azsdk-common-live-and-recorded-tests/SKILL.md --- .github/skills/azsdk-common-live-and-recorded-tests/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/skills/azsdk-common-live-and-recorded-tests/SKILL.md b/.github/skills/azsdk-common-live-and-recorded-tests/SKILL.md index ff507a3f9dd4..70342bad9473 100644 --- a/.github/skills/azsdk-common-live-and-recorded-tests/SKILL.md +++ b/.github/skills/azsdk-common-live-and-recorded-tests/SKILL.md @@ -32,7 +32,7 @@ compatibility: 2. **Check for existing deployment** — Look for a `.env` file at either the service directory level (next to `test-resources.bicep`, e.g. `sdk/storage/.env`) or at the package level (e.g. `sdk/storage/storage-blob/.env`). If one exists, inform the user that a previous deployment appears to be available and ask whether to **reuse the existing deployment** or **redeploy test resources**. If reusing, skip to step 7. 3. **Verify Azure context** — Run `Get-AzContext` to check for an active Azure PowerShell session. If no context exists, guide the user through `Connect-AzAccount -TenantId [TME tenant ID]`. Confirm the correct subscription is selected. 4. **Confirm deployment** — Even if no existing `.env` file is found, confirm with the user before proceeding to deploy test resources. Deployment creates Azure resources that may incur costs. -5. **Deploy test resources** — Run `eng/common/TestResources/New-TestResources.ps1` with the service directory and any user-provided parameters. See [deployment parameters](references/test-resource-deployment.md) for details. The script outputs environment variables needed for live/record test runs. +5. **Deploy test resources** — Run `eng/common/TestResources/New-TestResources.ps1` with the service directory, tenant ID, subscription ID, and any user-provided parameters. Use the TME tenant ID and subscription ID if one has not already been provided. See [deployment parameters](references/test-resource-deployment.md) for details. The script outputs environment variables needed for live/record test runs. 6. **Save environment** — Capture the environment variables output by the deployment script. If the script writes a `.env` file, note its path. Otherwise, collect the environment variables from the script output. 7. **Run tests** — Call the `azure-sdk-mcp:azsdk_package_run_tests` MCP tool (do NOT run test commands directly in the terminal). Provide the appropriate test mode (`record`, `live`, or `playback`) and the path to the `.env` file containing test environment variables. When tests run in record mode and all tests pass, the tool automatically pushes recorded test assets to the assets repo. 8. **Clean up** — Ask the user whether to clean up test resources. If yes, run `eng/common/TestResources/Remove-TestResources.ps1`. If no, inform the user that resources remain deployed for subsequent test runs. See [cleanup details](references/test-resource-cleanup.md). From 9d61ee40139f9eac258c281ed3410736f582db5f Mon Sep 17 00:00:00 2001 From: Timo van Veenendaal Date: Mon, 27 Apr 2026 10:56:39 -0700 Subject: [PATCH 7/7] Update .github/skills/azsdk-common-live-and-recorded-tests/SKILL.md Co-authored-by: Maddy Heaps <66138537+m-redding@users.noreply.github.com> --- .github/skills/azsdk-common-live-and-recorded-tests/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/skills/azsdk-common-live-and-recorded-tests/SKILL.md b/.github/skills/azsdk-common-live-and-recorded-tests/SKILL.md index 70342bad9473..cf9b8d310887 100644 --- a/.github/skills/azsdk-common-live-and-recorded-tests/SKILL.md +++ b/.github/skills/azsdk-common-live-and-recorded-tests/SKILL.md @@ -33,7 +33,7 @@ compatibility: 3. **Verify Azure context** — Run `Get-AzContext` to check for an active Azure PowerShell session. If no context exists, guide the user through `Connect-AzAccount -TenantId [TME tenant ID]`. Confirm the correct subscription is selected. 4. **Confirm deployment** — Even if no existing `.env` file is found, confirm with the user before proceeding to deploy test resources. Deployment creates Azure resources that may incur costs. 5. **Deploy test resources** — Run `eng/common/TestResources/New-TestResources.ps1` with the service directory, tenant ID, subscription ID, and any user-provided parameters. Use the TME tenant ID and subscription ID if one has not already been provided. See [deployment parameters](references/test-resource-deployment.md) for details. The script outputs environment variables needed for live/record test runs. -6. **Save environment** — Capture the environment variables output by the deployment script. If the script writes a `.env` file, note its path. Otherwise, collect the environment variables from the script output. +6. **Save environment** — Capture the environment variables output by the deployment script. If the script writes a `.env` file, note its path. Otherwise, collect the environment variables from the script output. In azure-sdk-for-net, if the script outputs a file like `test-resources.bicep.env` instead of a `.env` file, move on to step 7 and call the MCP tool WITHOUT passing an env filepath. The test framework will automatically find and handle this special file type. 7. **Run tests** — Call the `azure-sdk-mcp:azsdk_package_run_tests` MCP tool (do NOT run test commands directly in the terminal). Provide the appropriate test mode (`record`, `live`, or `playback`) and the path to the `.env` file containing test environment variables. When tests run in record mode and all tests pass, the tool automatically pushes recorded test assets to the assets repo. 8. **Clean up** — Ask the user whether to clean up test resources. If yes, run `eng/common/TestResources/Remove-TestResources.ps1`. If no, inform the user that resources remain deployed for subsequent test runs. See [cleanup details](references/test-resource-cleanup.md).