diff --git a/CHANGELOG.md b/CHANGELOG.md index 75a38af..d999b9b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,29 @@ # Changelog +## 2.0.6 (2026-03-28) + +### Fixes + +- Fixed eager rendering of `statecheck` queries that caused hard failures when `this.*` variables were not yet available (e.g. post-create exists re-run fails due to eventual consistency). `statecheck` now uses JIT rendering like `exports`, deferring gracefully when template variables are unresolved. +- When a deferred `statecheck` cannot be rendered post-deploy, the build falls through to `exports`-as-proxy validation or accepts the create/update based on successful execution. +- Applied the same fix to `teardown`, where `statecheck` used as an exists fallback would crash on unresolved variables instead of skipping the resource. +- Fixed `--dry-run` failures for resources that depend on exports from upstream resources. `create` and `update` query rendering now defers gracefully in dry-run mode when upstream exports are unavailable, and placeholder (``) values are injected for unresolved exports so downstream resources can still render. +- When a post-create exists re-run fails to find a newly created resource (eventual consistency), the exists query is automatically retried using the `statecheck` retry settings if available, giving async providers time to make the resource discoverable. + +### Features + +- New optional `troubleshoot` IQL anchor for post-failure diagnostics. When a `build` post-deploy check fails or a `teardown` delete cannot be confirmed, a user-defined diagnostic query is automatically rendered and executed, with results logged as pretty-printed JSON. Supports operation-specific variants (`troubleshoot:create`, `troubleshoot:update`, `troubleshoot:delete`) with fallback to a generic `troubleshoot` anchor. Typically used with `return_vals` to capture an async operation handle (e.g. `RequestToken`) from `RETURNING *` and query its status via `{{ this. }}`. See [resource query files documentation](https://stackql-deploy.io/docs/resource-query-files#troubleshoot) for details. +- The `RETURNING *` log message (`storing RETURNING * result...`) is now logged at `debug` level instead of `info`. + +## 2.0.5 (2026-03-24) + +### Fixes + +- Network and authentication errors (DNS failures, 401/403 responses) are now detected early and surfaced as fatal errors instead of being silently retried. +- Unresolved template variables are caught at render time with a clear error message identifying the missing variable and source template. +- `command` type resources now log query output when using `RETURNING` clauses, matching the behavior of `resource` types. +- Stack level exports (`stack_name`, `stack_env`) are now set as scoped environment variables on the host system for use by external tooling. + ## 2.0.4 (2026-03-18) ### Identifier capture from `exists` queries diff --git a/Cargo.lock b/Cargo.lock index cea14c7..8410613 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1809,7 +1809,7 @@ checksum = "a8f112729512f8e442d81f95a8a7ddf2b7c6b8a1a6f509a95864142b30cab2d3" [[package]] name = "stackql-deploy" -version = "2.0.5" +version = "2.0.6" dependencies = [ "base64", "chrono", diff --git a/Cargo.toml b/Cargo.toml index ada6f91..b8adf41 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "stackql-deploy" -version = "2.0.5" +version = "2.0.6" edition = "2021" rust-version = "1.75" description = "Infrastructure-as-code framework for declarative cloud resource management using StackQL" diff --git a/examples/aws/aws-vpc-webserver/stackql_manifest.yml b/examples/aws/aws-vpc-webserver/stackql_manifest.yml index c335bb0..e5bfaf8 100644 --- a/examples/aws/aws-vpc-webserver/stackql_manifest.yml +++ b/examples/aws/aws-vpc-webserver/stackql_manifest.yml @@ -35,6 +35,7 @@ resources: exports: - vpc_id - vpc_cidr_block + - name: example_subnet props: - name: subnet_cidr_block @@ -53,6 +54,7 @@ resources: exports: - subnet_id - availability_zone + - name: example_inet_gateway props: - name: inet_gateway_tags @@ -62,8 +64,10 @@ resources: merge: ['global_tags'] exports: - internet_gateway_id + - name: example_inet_gw_attachment props: [] + - name: example_route_table props: - name: route_table_tags @@ -73,12 +77,15 @@ resources: merge: ['global_tags'] exports: - route_table_id + - name: example_subnet_rt_assn props: [] exports: - subnet_route_table_assn_id + - name: example_inet_route props: [] + - name: example_security_group props: - name: group_description @@ -111,6 +118,7 @@ resources: IpProtocol: "-1" exports: - security_group_id + - name: example_web_server props: - name: ami_id @@ -141,6 +149,7 @@ resources: - Identifier: instance_id exports: - instance_id + - name: get_web_server_url type: query props: [] diff --git a/examples/aws/sqlserver/README.md b/examples/aws/sqlserver/README.md new file mode 100644 index 0000000..6e0f1d7 --- /dev/null +++ b/examples/aws/sqlserver/README.md @@ -0,0 +1,62 @@ +# `stackql-deploy` starter project for `aws` + +> for starter projects using other providers, try `stackql-deploy init infrastructure/sqlserver/ --provider=azure` or `stackql-deploy init infrastructure/sqlserver/ --provider=google` + +see the following links for more information on `stackql`, `stackql-deploy` and the `awscc` provider: + +- [`awscc` provider docs](https://awscc.stackql.io/providers/awscc/) +- [`stackql`](https://github.com/stackql/stackql) +- [`stackql-deploy` GitHub repo](https://github.com/stackql/stackql-deploy-rs) + +## Overview + +__`stackql-deploy`__ is a stateless, declarative, SQL driven Infrastructure-as-Code (IaC) framework. There is no state file required as the current state is assessed for each resource at runtime. __`stackql-deploy`__ is capable of provisioning, deprovisioning and testing a stack which can include resources across different providers, like a stack spanning `aws` and `azure` for example. + +## Prerequisites + +This example requires `stackql-deploy` to be installed. The host used to run `stackql-deploy` needs the necessary environment variables set to authenticate to your specific provider, in the case of `aws`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and optionally `AWS_SESSION_TOKEN` must be set, for more information on authentication to `aws` see the [`awscc` provider documentation](https://awscc.stackql.io/providers/awscc/). + +## Usage + +Adjust the values in the [__`stackql_manifest.yml`__](stackql_manifest.yml) file if desired. The [__`stackql_manifest.yml`__](stackql_manifest.yml) file contains resource configuration variables to support multiple deployment environments, these will be used for `stackql` queries in the `resources` folder. + +The syntax for the `stackql-deploy` command is as follows: + +```bash +stackql-deploy { build | test | teardown } { stack-directory } { deployment environment} [ optional flags ] +``` + +### Deploying a stack + +For example, to deploy the stack named infrastructure/sqlserver/ to an environment labeled `sit`, run the following: + +```bash +target/release/stackql-deploy build examples/aws/sqlserver dev \ +-e DB_MASTER_PASSWORD=${DB_MASTER_PASSWORD} +``` + +Use the `--dry-run` flag to view the queries to be run without actually running them, for example: + +```bash +stackql-deploy build infrastructure/sqlserver/ sit \ +-e AWS_REGION=us-east-1 \ +--dry-run +``` + +### Testing a stack + +To test a stack to ensure that all resources are present and in the desired state, run the following (in our `sit` deployment example): + +```bash +stackql-deploy test infrastructure/sqlserver/ sit \ +-e AWS_REGION=us-east-1 +``` + +### Tearing down a stack + +To destroy or deprovision all resources in a stack for our `sit` deployment example, run the following: + +```bash +stackql-deploy teardown infrastructure/sqlserver/ sit \ +-e AWS_REGION=us-east-1 +``` diff --git a/examples/aws/sqlserver/resources/db_instance.iql b/examples/aws/sqlserver/resources/db_instance.iql new file mode 100644 index 0000000..9e9db7d --- /dev/null +++ b/examples/aws/sqlserver/resources/db_instance.iql @@ -0,0 +1,82 @@ +/*+ exists */ +SELECT count(*) as count +FROM awscc.rds.db_instances +WHERE region = '{{ region }}' +AND Identifier = '{{ db_instance_identifier }}' + +/*+ create */ +INSERT INTO awscc.rds.db_instances ( + AllocatedStorage, + DBInstanceClass, + DBInstanceIdentifier, + DBSubnetGroupName, + Engine, + EngineVersion, + LicenseModel, + MasterUsername, + MasterUserPassword, + MultiAZ, + BackupRetentionPeriod, + PubliclyAccessible, + StorageType, + Tags, + VPCSecurityGroups, + region +) +SELECT + '{{ allocated_storage }}', + '{{ db_instance_class }}', + '{{ db_instance_identifier }}', + '{{ db_subnet_group_name }}', + '{{ engine }}', + '{{ engine_version }}', + '{{ license_model }}', + '{{ master_username }}', + '{{ master_user_password }}', + {{ multi_az }}, + {{ backup_retention_period }}, + {{ publicly_accessible }}, + '{{ storage_type }}', + '{{ tags }}', + '{{ vpc_security_groups }}', + '{{ region }}' +RETURNING * + +/*+ update */ +UPDATE awscc.rds.db_instances +SET PatchDocument = string('{{ { + "AllocatedStorage": allocated_storage, + "DBInstanceClass": db_instance_class, + "BackupRetentionPeriod": backup_retention_period, + "MasterUserPassword": master_user_password, + "MultiAZ": multi_az, + "PubliclyAccessible": publicly_accessible, + "StorageType": storage_type, + "Tags": tags, + "VPCSecurityGroups": vpc_security_groups +} | generate_patch_document }}') +WHERE region = '{{ region }}' +AND Identifier = '{{ db_instance_identifier }}' + +/*+ statecheck, retries=20, retry_delay=30 */ +SELECT count(*) as count +FROM awscc.rds.db_instances +WHERE region = '{{ region }}' +AND Identifier = '{{ db_instance_identifier }}' + +/*+ exports */ +SELECT endpoint as db_endpoint +FROM awscc.rds.db_instances +WHERE region = '{{ region }}' +AND Identifier = '{{ db_instance_identifier }}' + +/*+ troubleshoot:create */ +SELECT OperationStatus, StatusMessage, ErrorCode +FROM awscc.cloud_control.resource_request +WHERE RequestToken = '{{ this.RequestToken }}' +AND region = '{{ region }}'; + +/*+ delete */ +DELETE FROM awscc.rds.db_instances +WHERE region = '{{ region }}' +AND Identifier = '{{ db_instance_identifier }}' \ No newline at end of file diff --git a/examples/aws/sqlserver/resources/db_subnet_group.iql b/examples/aws/sqlserver/resources/db_subnet_group.iql new file mode 100644 index 0000000..1a0e742 --- /dev/null +++ b/examples/aws/sqlserver/resources/db_subnet_group.iql @@ -0,0 +1,62 @@ +/*+ exists */ +SELECT count(*) as count +FROM awscc.rds.db_subnet_groups +WHERE region = '{{ region }}' AND +Identifier = '{{ db_subnet_group_name }}' +; + +/*+ create */ +INSERT INTO awscc.rds.db_subnet_groups ( + DBSubnetGroupDescription, + DBSubnetGroupName, + SubnetIds, + Tags, + region +) +SELECT + '{{ db_subnet_group_description }}', + '{{ db_subnet_group_name }}', + '{{ subnet_ids }}', + '{{ tags }}', + '{{ region }}' +RETURNING *; + +/*+ update */ +UPDATE awscc.rds.db_subnet_groups +SET PatchDocument = string('{{ { + "DBSubnetGroupDescription": db_subnet_group_description, + "SubnetIds": subnet_ids, + "Tags": tags +} | generate_patch_document }}') +WHERE + region = '{{ region }}' AND + Identifier = '{{ db_subnet_group_name }}'; + +/*+ statecheck, retries=5, retry_delay=10 */ +SELECT count(*) as count FROM +( +SELECT + AWS_POLICY_EQUAL(subnet_ids, '{{ subnet_ids }}') as test_subnet_ids, + AWS_POLICY_EQUAL(tags, '{{ tags }}') as test_tags +FROM awscc.rds.db_subnet_groups +WHERE +region = '{{ region }}' AND +Identifier = '{{ db_subnet_group_name }}' AND +db_subnet_group_description = '{{ db_subnet_group_description }}' +) t +WHERE test_subnet_ids = 1 AND test_tags = 1; + +/*+ troubleshoot:create */ +SELECT OperationStatus, StatusMessage, ErrorCode +FROM awscc.cloud_control.resource_request +WHERE RequestToken = '{{ this.RequestToken }}' +AND region = '{{ region }}'; + +/*+ exports */ +SELECT '{{ db_subnet_group_name }}' as db_subnet_group_name; + +/*+ delete */ +DELETE FROM awscc.rds.db_subnet_groups +WHERE + Identifier = '{{ db_subnet_group_name }}' AND + region = '{{ region }}'; \ No newline at end of file diff --git a/examples/aws/sqlserver/resources/inet_gateway.iql b/examples/aws/sqlserver/resources/inet_gateway.iql new file mode 100644 index 0000000..3ea1265 --- /dev/null +++ b/examples/aws/sqlserver/resources/inet_gateway.iql @@ -0,0 +1,48 @@ +/*+ exists */ +WITH tagged_resources AS +( + SELECT split_part(ResourceARN, '/', 2) as internet_gateway_id + FROM awscc.tagging.tagged_resources + WHERE region = '{{ region }}' + AND TagFilters = '{{ global_tags | to_aws_tag_filters }}' + AND ResourceTypeFilters = '["ec2:internet-gateway"]' +), +internet_gateways AS +( + SELECT internet_gateway_id + FROM awscc.ec2.internet_gateways_list_only + WHERE region = '{{ region }}' +) +SELECT r.internet_gateway_id +FROM internet_gateways r +INNER JOIN tagged_resources tr +ON r.internet_gateway_id = tr.internet_gateway_id; + +/*+ statecheck, retries=5, retry_delay=5 */ +SELECT COUNT(*) as count FROM +( +SELECT +AWS_POLICY_EQUAL(tags, '{{ inet_gateway_tags }}') as test_tags +FROM awscc.ec2.internet_gateways +WHERE Identifier = '{{ this.internet_gateway_id }}' +AND region = '{{ region }}' +AND test_tags = 1 +) t; + +/*+ create */ +INSERT INTO awscc.ec2.internet_gateways ( + Tags, + region +) +SELECT + '{{ inet_gateway_tags }}', + '{{ region }}' +RETURNING *; + +/*+ exports */ +SELECT '{{ this.internet_gateway_id }}' as internet_gateway_id; + +/*+ delete */ +DELETE FROM awscc.ec2.internet_gateways +WHERE Identifier = '{{ internet_gateway_id }}' +AND region = '{{ region }}'; diff --git a/examples/aws/sqlserver/resources/inet_gw_attachment.iql b/examples/aws/sqlserver/resources/inet_gw_attachment.iql new file mode 100644 index 0000000..c3f78aa --- /dev/null +++ b/examples/aws/sqlserver/resources/inet_gw_attachment.iql @@ -0,0 +1,28 @@ +/*+ exists */ +SELECT count(*) as count +FROM awscc.ec2.vpc_gateway_attachments +WHERE Identifier = 'IGW|{{ vpc_id }}' +AND region = '{{ region }}'; + +/*+ create */ +INSERT INTO awscc.ec2.vpc_gateway_attachments ( + InternetGatewayId, + VpcId, + region +) +SELECT + '{{ internet_gateway_id }}', + '{{ vpc_id }}', + '{{ region }}' +RETURNING *; + +/*+ statecheck, retries=3, retry_delay=5 */ +SELECT count(*) as count +FROM awscc.ec2.vpc_gateway_attachments +WHERE Identifier = 'IGW|{{ vpc_id }}' +AND region = '{{ region }}'; + +/*+ delete */ +DELETE FROM awscc.ec2.vpc_gateway_attachments +WHERE Identifier = 'IGW|{{ vpc_id }}' +AND region = '{{ region }}'; diff --git a/examples/aws/sqlserver/resources/inet_route.iql b/examples/aws/sqlserver/resources/inet_route.iql new file mode 100644 index 0000000..e36ded7 --- /dev/null +++ b/examples/aws/sqlserver/resources/inet_route.iql @@ -0,0 +1,24 @@ +/*+ createorupdate */ +INSERT INTO awscc.ec2.routes ( + DestinationCidrBlock, + GatewayId, + RouteTableId, + region +) +SELECT + '0.0.0.0/0', + '{{ internet_gateway_id }}', + '{{ route_table_id }}', + '{{ region }}' +RETURNING *; + +/*+ statecheck, retries=3, retry_delay=5 */ +SELECT count(*) as count +FROM awscc.ec2.routes +WHERE Identifier = '{{ route_table_id }}|0.0.0.0/0' +AND region = '{{ region }}'; + +/*+ delete */ +DELETE FROM awscc.ec2.routes +WHERE Identifier = '{{ route_table_id }}|0.0.0.0/0' +AND region = '{{ region }}'; diff --git a/examples/aws/sqlserver/resources/route_table.iql b/examples/aws/sqlserver/resources/route_table.iql new file mode 100644 index 0000000..c451d88 --- /dev/null +++ b/examples/aws/sqlserver/resources/route_table.iql @@ -0,0 +1,50 @@ +/*+ exists */ +WITH tagged_resources AS +( + SELECT split_part(ResourceARN, '/', 2) as route_table_id + FROM awscc.tagging.tagged_resources + WHERE region = '{{ region }}' + AND TagFilters = '{{ global_tags | to_aws_tag_filters }}' + AND ResourceTypeFilters = '["ec2:route-table"]' +), +route_tables AS +( + SELECT route_table_id + FROM awscc.ec2.route_tables_list_only + WHERE region = '{{ region }}' +) +SELECT r.route_table_id +FROM route_tables r +INNER JOIN tagged_resources tr +ON r.route_table_id = tr.route_table_id; + +/*+ statecheck, retries=5, retry_delay=5 */ +SELECT COUNT(*) as count FROM +( +SELECT +AWS_POLICY_EQUAL(tags, '{{ route_table_tags }}') as test_tags +FROM awscc.ec2.route_tables +WHERE Identifier = '{{ this.route_table_id }}' +AND region = '{{ region }}' +) t +WHERE test_tags = 1; + +/*+ create */ +INSERT INTO awscc.ec2.route_tables ( + Tags, + VpcId, + region +) +SELECT + '{{ route_table_tags }}', + '{{ vpc_id }}', + '{{ region }}' +RETURNING *; + +/*+ exports */ +SELECT '{{ this.route_table_id }}' as route_table_id; + +/*+ delete */ +DELETE FROM awscc.ec2.route_tables +WHERE Identifier = '{{ route_table_id }}' +AND region = '{{ region }}'; diff --git a/examples/aws/sqlserver/resources/security_group.iql b/examples/aws/sqlserver/resources/security_group.iql new file mode 100644 index 0000000..f548b98 --- /dev/null +++ b/examples/aws/sqlserver/resources/security_group.iql @@ -0,0 +1,69 @@ +/*+ exists */ +WITH tagged_resources AS +( + SELECT split_part(ResourceARN, '/', 2) as security_group_id + FROM awscc.tagging.tagged_resources + WHERE region = '{{ region }}' + AND TagFilters = '{{ global_tags | to_aws_tag_filters }}' + AND ResourceTypeFilters = '["ec2:security-group"]' +), +security_groups AS +( + SELECT id as security_group_id + FROM awscc.ec2.security_groups_list_only + WHERE region = '{{ region }}' +) +SELECT r.security_group_id +FROM security_groups r +INNER JOIN tagged_resources tr +ON r.security_group_id = tr.security_group_id; + +/*+ statecheck, retries=5, retry_delay=5 */ +SELECT COUNT(*) as count FROM +( +SELECT +AWS_POLICY_EQUAL(tags, '{{ sg_tags }}') as test_tags, +AWS_POLICY_EQUAL(security_group_ingress, '{{ security_group_ingress }}') as test_ingress +FROM awscc.ec2.security_groups +WHERE Identifier = '{{ this.security_group_id }}' +AND region = '{{ region }}' +AND group_name = '{{ group_name }}' +AND vpc_id = '{{ vpc_id }}' +AND group_description = '{{ group_description }}' +) t +WHERE test_tags = 1 +AND test_ingress = 1; + +/*+ create */ +INSERT INTO awscc.ec2.security_groups ( + GroupName, + GroupDescription, + VpcId, + SecurityGroupIngress, + SecurityGroupEgress, + Tags, + region +) +SELECT + '{{ group_name }}', + '{{ group_description }}', + '{{ vpc_id }}', + '{{ security_group_ingress }}', + '{{ security_group_egress }}', + '{{ sg_tags }}', + '{{ region }}' +RETURNING *; + +/*+ exports */ +SELECT '{{ this.security_group_id }}' as security_group_id; + +/*+ troubleshoot:create */ +SELECT OperationStatus, StatusMessage, ErrorCode +FROM awscc.cloud_control.resource_request +WHERE RequestToken = '{{ this.RequestToken }}' +AND region = '{{ region }}'; + +/*+ delete */ +DELETE FROM awscc.ec2.security_groups +WHERE Identifier = '{{ security_group_id }}' +AND region = '{{ region }}'; diff --git a/examples/aws/sqlserver/resources/subnet.iql b/examples/aws/sqlserver/resources/subnet.iql new file mode 100644 index 0000000..22a4475 --- /dev/null +++ b/examples/aws/sqlserver/resources/subnet.iql @@ -0,0 +1,68 @@ +/*+ exists */ +WITH tagged_resources AS +( + SELECT split_part(ResourceARN, '/', 2) as subnet_id + FROM awscc.tagging.tagged_resources + WHERE region = '{{ region }}' + AND TagFilters = '{{ global_tags | to_aws_tag_filters }}' + AND ResourceTypeFilters = '["ec2:subnet"]' +), +subnets AS +( + SELECT subnet_id + FROM awscc.ec2.subnets_list_only + WHERE region = '{{ region }}' +) +SELECT r.subnet_id +FROM subnets r +INNER JOIN tagged_resources tr +ON r.subnet_id = tr.subnet_id; + +/*+ statecheck, retries=5, retry_delay=5 */ +SELECT COUNT(*) as count FROM +( +SELECT +AWS_POLICY_EQUAL(tags, '{{ subnet_tags }}') as test_tags +FROM awscc.ec2.subnets +WHERE Identifier = '{{ this.subnet_id }}' +AND region = '{{ region }}' +AND cidr_block = '{{ subnet_cidr_block }}' +AND vpc_id = '{{ vpc_id }}' +AND availability_zone = '{{ availability_zone }}' +) t +WHERE test_tags = 1; + +/*+ create */ +INSERT INTO awscc.ec2.subnets ( + VpcId, + CidrBlock, + MapPublicIpOnLaunch, + Tags, + AvailabilityZone, + region +) +SELECT + '{{ vpc_id }}', + '{{ subnet_cidr_block }}', + true, + '{{ subnet_tags }}', + '{{ availability_zone }}', + '{{ region }}' +RETURNING *; + +/*+ exports, retries=5, retry_delay=5 */ +SELECT subnet_id, availability_zone +FROM awscc.ec2.subnets +WHERE Identifier = '{{ this.subnet_id }}' +AND region = '{{ region }}'; + +/*+ troubleshoot:create */ +SELECT OperationStatus, StatusMessage, ErrorCode +FROM awscc.cloud_control.resource_request +WHERE RequestToken = '{{ this.RequestToken }}' +AND region = '{{ region }}'; + +/*+ delete */ +DELETE FROM awscc.ec2.subnets +WHERE Identifier = '{{ subnet_id }}' +AND region = '{{ region }}'; diff --git a/examples/aws/sqlserver/resources/subnet_rt_assn.iql b/examples/aws/sqlserver/resources/subnet_rt_assn.iql new file mode 100644 index 0000000..6a489e8 --- /dev/null +++ b/examples/aws/sqlserver/resources/subnet_rt_assn.iql @@ -0,0 +1,29 @@ +/*+ exists, retries=10, retry_delay=5 */ +SELECT + id as subnet_route_table_assn_id +FROM awscc.ec2.vw_subnet_route_table_associations +WHERE + region = '{{ region }}' + AND route_table_id = '{{ route_table_id }}' + AND subnet_id = '{{ subnet_id }}'; + +/*+ create */ +INSERT INTO awscc.ec2.subnet_route_table_associations ( + RouteTableId, + SubnetId, + region +) +SELECT + '{{ route_table_id }}', + '{{ subnet_id }}', + '{{ region }}' +RETURNING *; + +/*+ exports */ +SELECT '{{ this.subnet_route_table_assn_id }}' as subnet_route_table_assn_id; + +/*+ delete */ +DELETE FROM awscc.ec2.subnet_route_table_associations +WHERE +Identifier = '{{ subnet_route_table_assn_id }}' AND +region = '{{ region }}'; \ No newline at end of file diff --git a/examples/aws/sqlserver/resources/vpc.iql b/examples/aws/sqlserver/resources/vpc.iql new file mode 100644 index 0000000..c63bbdf --- /dev/null +++ b/examples/aws/sqlserver/resources/vpc.iql @@ -0,0 +1,62 @@ +/*+ exists */ +WITH tagged_resources AS +( + SELECT split_part(ResourceARN, '/', 2) as vpc_id + FROM awscc.tagging.tagged_resources + WHERE region = '{{ region }}' + AND TagFilters = '{{ global_tags | to_aws_tag_filters }}' + AND ResourceTypeFilters = '["ec2:vpc"]' +), +vpcs AS +( + SELECT vpc_id + FROM awscc.ec2.vpcs_list_only + WHERE region = '{{ region }}' +) +SELECT r.vpc_id +FROM vpcs r +INNER JOIN tagged_resources tr +ON r.vpc_id = tr.vpc_id; + +/*+ statecheck, retries=5, retry_delay=5 */ +SELECT COUNT(*) as count FROM +( +SELECT +AWS_POLICY_EQUAL(tags, '{{ vpc_tags }}') as test_tags +FROM awscc.ec2.vpcs +WHERE Identifier = '{{ this.vpc_id }}' +AND region = '{{ region }}' +AND cidr_block = '{{ vpc_cidr_block }}' +) t +WHERE test_tags = 1; + +/*+ create */ +INSERT INTO awscc.ec2.vpcs ( + CidrBlock, + Tags, + EnableDnsSupport, + EnableDnsHostnames, + region +) +SELECT + '{{ vpc_cidr_block }}', + '{{ vpc_tags }}', + true, + true, + '{{ region }}' +RETURNING *; + +/*+ exports */ +SELECT '{{ this.vpc_id }}' as vpc_id, +'{{ vpc_cidr_block }}' as vpc_cidr_block; + +/*+ troubleshoot:create */ +SELECT OperationStatus, StatusMessage, ErrorCode +FROM awscc.cloud_control.resource_request +WHERE RequestToken = '{{ this.RequestToken }}' +AND region = '{{ region }}'; + +/*+ delete */ +DELETE FROM awscc.ec2.vpcs +WHERE Identifier = '{{ vpc_id }}' +AND region = '{{ region }}'; diff --git a/examples/aws/sqlserver/stackql_manifest.yml b/examples/aws/sqlserver/stackql_manifest.yml new file mode 100644 index 0000000..c91d968 --- /dev/null +++ b/examples/aws/sqlserver/stackql_manifest.yml @@ -0,0 +1,232 @@ +version: 1 +name: sqlserver-migration-lab +description: SQL Server 2016 Express RDS lab instance for Databricks migration course +providers: + - awscc::v26.03.00379 + +globals: + - name: region + value: ap-southeast-2 + - name: db_master_username + value: admin + - name: db_master_password + value: "{{ DB_MASTER_PASSWORD }}" + - name: global_tags + value: + - Key: 'stackql:stack-name' + Value: "{{ stack_name }}" + - Key: 'stackql:stack-env' + Value: "{{ stack_env }}" + - Key: 'stackql:resource-name' + Value: "{{ resource_name }}" + +resources: + - name: vpc + description: VPC for SQL Server lab + props: + - name: vpc_cidr_block + values: + prd: + value: "10.0.0.0/16" + sit: + value: "10.1.0.0/16" + dev: + value: "10.2.0.0/16" + - name: vpc_tags + value: + - Key: Name + Value: "{{ stack_name }}-{{ stack_env }}-vpc" + merge: + - global_tags + return_vals: + create: + - RequestToken + exports: + - vpc_id + - vpc_cidr_block + + - name: subnet_a + file: subnet.iql + props: + - name: subnet_cidr_block + values: + prd: + value: "10.0.1.0/24" + sit: + value: "10.1.1.0/24" + dev: + value: "10.2.1.0/24" + - name: availability_zone + value: ap-southeast-2a + - name: subnet_tags + value: + - Key: Name + Value: "{{ stack_name }}-{{ stack_env }}-subnet-a" + merge: ['global_tags'] + return_vals: + create: + - RequestToken + exports: + - subnet_id: subnet_a_id + - availability_zone: subnet_a_availability_zone + + - name: subnet_b + file: subnet.iql + props: + - name: subnet_cidr_block + values: + prd: + value: "10.0.2.0/24" + sit: + value: "10.1.2.0/24" + dev: + value: "10.2.2.0/24" + - name: availability_zone + value: ap-southeast-2b + - name: subnet_tags + value: + - Key: Name + Value: "{{ stack_name }}-{{ stack_env }}-subnet-b" + merge: ['global_tags'] + return_vals: + create: + - RequestToken + exports: + - subnet_id: subnet_b_id + - availability_zone: subnet_b_availability_zone + + - name: db_subnet_group + description: RDS subnet group spanning 2 AZs + props: + - name: db_subnet_group_name + value: "{{ stack_name }}-{{ stack_env }}-subnet-group" + - name: db_subnet_group_description + value: SQL Server lab subnet group + - name: subnet_ids + value: + - '{{ subnet_a_id }}' + - '{{ subnet_b_id }}' + - name: tags + value: [] + merge: ['global_tags'] + return_vals: + create: + - RequestToken + exports: + - db_subnet_group_name + + - name: inet_gateway + props: + - name: inet_gateway_tags + value: + - Key: Name + Value: "{{ stack_name }}-{{ stack_env }}-inet-gateway" + merge: ['global_tags'] + exports: + - internet_gateway_id + + - name: inet_gw_attachment + props: [] + + - name: route_table + props: + - name: route_table_tags + value: + - Key: Name + Value: "{{ stack_name }}-{{ stack_env }}-route-table" + merge: ['global_tags'] + exports: + - route_table_id + + - name: subnet_rt_assn_a + file: subnet_rt_assn.iql + props: + - name: subnet_id + value: '{{ subnet_a_id }}' + - name: route_table_id + value: '{{ route_table_id }}' + exports: + - subnet_route_table_assn_id: subnet_a_route_table_assn_id + + - name: subnet_rt_assn_b + file: subnet_rt_assn.iql + props: + - name: subnet_id + value: '{{ subnet_b_id }}' + - name: route_table_id + value: '{{ route_table_id }}' + exports: + - subnet_route_table_assn_id: subnet_b_route_table_assn_id + + - name: inet_route + props: [] + + - name: security_group + props: + - name: group_description + value: SG allowing inbound MSSQL from anywhere + - name: group_name + value: "{{ stack_name }}-{{ stack_env }}-sg" + - name: sg_tags + value: [] + merge: ['global_tags'] + - name: security_group_ingress + value: + - IpProtocol: "tcp" + CidrIp: "0.0.0.0/0" + Description: Allow MSSQL from anywhere + FromPort: 1433 + ToPort: 1433 + - name: security_group_egress + value: + - CidrIp: "0.0.0.0/0" + Description: Allow all outbound traffic + FromPort: -1 + ToPort: -1 + IpProtocol: "-1" + return_vals: + create: + - RequestToken + exports: + - security_group_id + + - name: db_instance + description: SQL Server 2016 Express RDS instance + props: + - name: db_instance_identifier + value: "{{ stack_name }}-{{ stack_env }}" + - name: db_instance_class + value: db.t3.small + - name: engine + value: sqlserver-ex + - name: engine_version + value: "14.00" + - name: allocated_storage + value: "20" + - name: storage_type + value: gp2 + - name: license_model + value: license-included + - name: master_username + value: "{{ db_master_username }}" + - name: master_user_password + value: "{{ db_master_password }}" + - name: db_subnet_group_name + value: "{{ db_subnet_group_name }}" + - name: vpc_security_groups + value: + - "{{ security_group_id }}" + - name: publicly_accessible + value: true + - name: multi_az + value: false + - name: backup_retention_period + value: 0 + - name: tags + value: [] + merge: ['global_tags'] + return_vals: + create: + - RequestToken + exports: + - db_endpoint \ No newline at end of file diff --git a/src/commands/base.rs b/src/commands/base.rs index a7abd02..781ee34 100644 --- a/src/commands/base.rs +++ b/src/commands/base.rs @@ -10,7 +10,7 @@ use std::fs; use std::path::Path; use std::process; -use log::{debug, error, info}; +use log::{debug, error, info, warn}; use crate::core::config::{get_full_context, render_globals, render_string_value}; use crate::core::env::load_env_vars; @@ -373,7 +373,7 @@ impl CommandRunner { return (false, None); } - info!("[{}] does not exist, creating...", resource.name); + info!("creating [{}]...", resource.name); show_query(show_queries, create_query); if has_returning_clause(create_query) { @@ -725,7 +725,7 @@ impl CommandRunner { resource_name: &str, returning_row: &HashMap, ) { - info!( + debug!( "storing RETURNING * result for [{}] in callback context", resource_name ); @@ -797,6 +797,90 @@ impl CommandRunner { ); } + /// Run an optional troubleshoot diagnostic query after a failed operation. + /// + /// Looks for a `troubleshoot:{operation}` anchor first, falling back to a + /// generic `troubleshoot` anchor. If found, the query is rendered with + /// `try_render_query` (tolerant of missing variables) and executed once. + /// Results are logged at `error!()` level so the user sees them alongside + /// the failure message. + /// + /// This method never causes the build/teardown to fail - if the + /// troubleshoot query itself errors, a warning is logged and execution + /// continues to the original error path. + pub fn run_troubleshoot( + &mut self, + resource: &Resource, + resource_queries: &HashMap, + operation: &str, + full_context: &HashMap, + show_queries: bool, + ) { + // Look up operation-specific anchor first, fall back to generic + let specific = format!("troubleshoot:{}", operation); + let anchor = if resource_queries.contains_key(&specific) { + &specific + } else if resource_queries.contains_key("troubleshoot") { + "troubleshoot" + } else { + return; + }; + + let pq = resource_queries.get(anchor).unwrap(); + + let rendered = + match self.try_render_query(&resource.name, anchor, &pq.template, full_context) { + Some(q) => q, + None => { + // Extract referenced variables from template to report + // which ones are missing + let re = regex::Regex::new(r"\{\{\s*([\w.]+)").unwrap(); + let missing: Vec<&str> = re + .captures_iter(&pq.template) + .filter_map(|c| c.get(1).map(|m| m.as_str())) + .filter(|v| !full_context.contains_key(*v)) + .collect(); + warn!( + "[{}] troubleshoot query could not be rendered, missing variables: {:?}", + resource.name, missing + ); + return; + } + }; + + info!( + "running troubleshoot query for [{}] ({})...", + resource.name, operation + ); + show_query(show_queries, &rendered); + + let results = run_stackql_query( + &rendered, + &mut self.client, + true, + pq.options.retries, + pq.options.retry_delay, + ); + + if results.is_empty() { + warn!("[{}] troubleshoot query returned no results", resource.name); + return; + } + + if let Ok(json) = serde_json::to_string_pretty(&results) { + info!( + "[{}] troubleshoot diagnostics ({}):\n\n{}\n", + resource.name, operation, json + ); + } else { + // Fallback if JSON serialization fails + info!( + "[{}] troubleshoot diagnostics ({}): {:?}", + resource.name, operation, results + ); + } + } + /// Run a command-type query. pub fn run_command( &mut self, diff --git a/src/commands/build.rs b/src/commands/build.rs index 14867b3..7a6c948 100644 --- a/src/commands/build.rs +++ b/src/commands/build.rs @@ -18,7 +18,7 @@ use crate::commands::common_args::{ FailureAction, }; use crate::core::config::get_resource_type; -use crate::core::utils::catch_error_and_exit; +use crate::core::utils::{catch_error_and_exit, export_vars}; use crate::utils::connection::create_client; use crate::utils::display::{print_unicode_box, BorderColor}; use crate::utils::server::{check_and_start_server, stop_local_server}; @@ -99,11 +99,21 @@ pub fn execute(matches: &ArgMatches) { } /// Render the statecheck query template with the given context. +/// Uses try_render_query so that unresolved variables (e.g. this.* fields +/// not yet captured) return None instead of a hard error. macro_rules! render_statecheck { ($runner:expr, $resource_queries:expr, $resource:expr, $ctx:expr) => { - $resource_queries.get("statecheck").map(|q| { - let rendered = $runner.render_query(&$resource.name, "statecheck", &q.template, $ctx); - (rendered, q.options.clone()) + $resource_queries.get("statecheck").and_then(|q| { + match $runner.try_render_query(&$resource.name, "statecheck", &q.template, $ctx) { + Some(rendered) => Some((rendered, q.options.clone())), + None => { + debug!( + "statecheck query for [{}] deferred (unresolved variables)", + $resource.name + ); + None + } + } }) }; } @@ -302,20 +312,30 @@ fn run_build( apply_exists_fields(fields, &resource.name, &mut full_context); } } else { - // Use statecheck as exists check (render with current ctx) - let statecheck_query = - render_statecheck!(runner, resource_queries, resource, &full_context); - let sq = statecheck_query.as_ref().unwrap(); - let sq_opts = resource_queries.get("statecheck").unwrap(); - is_correct_state = runner.check_if_resource_is_correct_state( - resource, - &sq.0, - sq_opts.options.retries, - sq_opts.options.retry_delay, - dry_run, - show_queries, - ); - resource_exists = is_correct_state; + // Use statecheck as exists check (render with current ctx). + // If the statecheck template has unresolved variables (e.g. + // this.* fields not yet captured), the resource cannot exist + // yet - treat as not-found. + if let Some(sq) = + render_statecheck!(runner, resource_queries, resource, &full_context) + { + let sq_opts = resource_queries.get("statecheck").unwrap(); + is_correct_state = runner.check_if_resource_is_correct_state( + resource, + &sq.0, + sq_opts.options.retries, + sq_opts.options.retry_delay, + dry_run, + show_queries, + ); + resource_exists = is_correct_state; + } else { + info!( + "[{}] statecheck has unresolved variables, treating as not found", + resource.name + ); + resource_exists = false; + } } // Pre-deployment state check for existing resources @@ -328,18 +348,24 @@ fn run_build( is_correct_state = true; } else { // Re-render statecheck with (possibly enriched) context - let statecheck_query = - render_statecheck!(runner, resource_queries, resource, &full_context); - let sq = statecheck_query.as_ref().unwrap(); - let sq_opts = resource_queries.get("statecheck").unwrap(); - is_correct_state = runner.check_if_resource_is_correct_state( - resource, - &sq.0, - sq_opts.options.retries, - sq_opts.options.retry_delay, - dry_run, - show_queries, - ); + if let Some(sq) = + render_statecheck!(runner, resource_queries, resource, &full_context) + { + let sq_opts = resource_queries.get("statecheck").unwrap(); + is_correct_state = runner.check_if_resource_is_correct_state( + resource, + &sq.0, + sq_opts.options.retries, + sq_opts.options.retry_delay, + dry_run, + show_queries, + ); + } else { + warn!( + "[{}] statecheck has unresolved variables during pre-deploy validation", + resource.name + ); + } } } @@ -461,29 +487,67 @@ fn run_build( let mut is_created_or_updated = false; if !resource_exists { - // JIT render create/createorupdate query + // JIT render create/createorupdate query. + // In dry-run mode, use try_render_query so that unresolved + // variables (from exports not yet available) produce a + // deferral instead of a hard error. let create_query = if has_createorupdate { let cou = resource_queries.get("createorupdate").unwrap(); - runner.render_query( - &resource.name, - "createorupdate", - &cou.template, - &full_context, - ) + if dry_run { + runner.try_render_query( + &resource.name, + "createorupdate", + &cou.template, + &full_context, + ) + } else { + Some(runner.render_query( + &resource.name, + "createorupdate", + &cou.template, + &full_context, + )) + } } else { let cq = resource_queries.get("create").unwrap(); - runner.render_query(&resource.name, "create", &cq.template, &full_context) + if dry_run { + runner.try_render_query( + &resource.name, + "create", + &cq.template, + &full_context, + ) + } else { + Some(runner.render_query( + &resource.name, + "create", + &cq.template, + &full_context, + )) + } }; - let (created, returning_row) = runner.create_resource( - resource, - &create_query, - create_retries, - create_retry_delay, - dry_run, - show_queries, - ignore_errors, - ); + if create_query.is_none() { + info!( + "dry run create for [{}]: query has unresolved variables \ + (upstream exports not yet available), skipping render", + resource.name + ); + } + + let (created, returning_row) = if let Some(ref cq) = create_query { + runner.create_resource( + resource, + cq, + create_retries, + create_retry_delay, + dry_run, + show_queries, + ignore_errors, + ) + } else { + (false, None) + }; is_created_or_updated = created; // Capture RETURNING * result. @@ -569,21 +633,53 @@ fn run_build( } if resource_exists && !is_correct_state { - // JIT render update/createorupdate query + // JIT render update/createorupdate query. + // In dry-run mode, use try_render_query for tolerance. let update_query: Option = if has_createorupdate { let cou = resource_queries.get("createorupdate").unwrap(); - Some(runner.render_query( - &resource.name, - "createorupdate", - &cou.template, - &full_context, - )) + if dry_run { + runner.try_render_query( + &resource.name, + "createorupdate", + &cou.template, + &full_context, + ) + } else { + Some(runner.render_query( + &resource.name, + "createorupdate", + &cou.template, + &full_context, + )) + } } else { - resource_queries.get("update").map(|uq| { - runner.render_query(&resource.name, "update", &uq.template, &full_context) + resource_queries.get("update").and_then(|uq| { + if dry_run { + runner.try_render_query( + &resource.name, + "update", + &uq.template, + &full_context, + ) + } else { + Some(runner.render_query( + &resource.name, + "update", + &uq.template, + &full_context, + )) + } }) }; + if update_query.is_none() && dry_run { + info!( + "dry run update for [{}]: query has unresolved variables \ + (upstream exports not yet available), skipping render", + resource.name + ); + } + let (updated, returning_row) = runner.update_resource( resource, update_query.as_deref(), @@ -682,43 +778,68 @@ fn run_build( // Post-deploy state check if is_created_or_updated { - // Check if return_vals already captured fields from RETURNING. - // If so, skip the post-create exists re-run to save API calls. let op = if !resource_exists { "create" } else { "update" }; - let has_return_vals = !resource.get_return_val_mappings(op).is_empty(); // After create/update, re-run the exists query to capture - // this.* fields (e.g. identifier) that are needed by the - // statecheck and exports queries — but skip this if - // return_vals already provided them. - if !has_return_vals { - if let Some(ref eq) = exists_query { - let eq_opts = resource_queries.get("exists").unwrap(); - let (post_exists, fields) = runner.check_if_resource_exists( + // this.* fields (e.g. identifier) needed by statecheck and + // exports queries. This always runs even when return_vals + // captured some fields, because the exists query discovers + // the resource identifier and waits for the resource to + // become available (async/eventual consistency). + if let Some(ref eq) = exists_query { + // Use statecheck retry settings for the post-create + // exists check when available (async providers need + // time for the resource to become discoverable). + let (post_retries, post_delay) = + if let Some(sc_opts) = resource_queries.get("statecheck") { + (sc_opts.options.retries, sc_opts.options.retry_delay) + } else { + let eq_opts = resource_queries.get("exists").unwrap(); + (eq_opts.options.retries, eq_opts.options.retry_delay) + }; + + let (post_exists, fields) = runner.check_if_resource_exists( + resource, + &eq.0, + post_retries, + post_delay, + dry_run, + show_queries, + false, + ); + + // If exists retries are exhausted and resource still + // not found, run troubleshoot and exit immediately - + // don't attempt statecheck/exports. + if !post_exists && !dry_run { + runner.run_troubleshoot( resource, - &eq.0, - eq_opts.options.retries, - eq_opts.options.retry_delay, - dry_run, + &resource_queries, + op, + &full_context, show_queries, - false, ); - apply_exists_fields(fields, &resource.name, &mut full_context); + catch_error_and_exit(&format!( + "[{}] not found after {} post-deploy check, {} operation may have failed.", + resource.name, op, op + )); + } - // Always try to render exports after post-create exists - exports_query_str = - render_exports!(runner, resource_queries, resource, &full_context); + apply_exists_fields(fields, &resource.name, &mut full_context); - // If exists confirms the resource is present and there is - // no statecheck or exports query, the exists query IS - // the statecheck: a successful re-run confirms the - // resource was created/updated successfully. - if post_exists - && !resource_queries.contains_key("statecheck") - && exports_query_str.is_none() - { - is_correct_state = true; - } + // Always try to render exports after post-create exists + exports_query_str = + render_exports!(runner, resource_queries, resource, &full_context); + + // If exists confirms the resource is present and there is + // no statecheck or exports query, the exists query IS + // the statecheck: a successful re-run confirms the + // resource was created/updated successfully. + if post_exists + && !resource_queries.contains_key("statecheck") + && exports_query_str.is_none() + { + is_correct_state = true; } } @@ -750,6 +871,39 @@ fn run_build( dry_run, show_queries, ); + } else if resource_queries.contains_key("statecheck") { + // Statecheck anchor exists but could not be rendered (unresolved + // this.* variables). Fall through to exports-as-proxy if available, + // otherwise treat as correct (the resource was just created and + // the post-create exists query did not return identifier fields). + if let Some(ref eq_str) = exports_query_str { + info!( + "statecheck deferred for [{}], using exports query as post-deploy statecheck", + resource.name + ); + let post_retries = exports_retries; + let post_delay = exports_retry_delay; + + let (state, proxy) = runner.check_state_using_exports_proxy( + resource, + eq_str, + post_retries, + post_delay, + dry_run, + show_queries, + ); + is_correct_state = state; + if proxy.is_some() { + exports_result_from_proxy = proxy; + } + } else { + info!( + "statecheck deferred for [{}] and no exports available, \ + accepting post-deploy state based on successful create/update", + resource.name + ); + is_correct_state = true; + } } else if let Some(ref eq_str) = exports_query_str { info!( "using exports query as post-deploy statecheck for [{}]", @@ -774,6 +928,14 @@ fn run_build( } if !is_correct_state && !dry_run { + let op = if !resource_exists { "create" } else { "update" }; + runner.run_troubleshoot( + resource, + &resource_queries, + op, + &full_context, + show_queries, + ); catch_error_and_exit(&format!( "deployment failed for {} after post-deploy checks.", resource.name @@ -838,13 +1000,49 @@ fn run_build( if exports_query_str.is_none() && resource_queries.contains_key("exports") && !resource.exports.is_empty() - && !dry_run { - catch_error_and_exit(&format!( - "exports query for [{}] could not be rendered - unresolved template variables. \ - Check that all referenced variables are defined in the manifest or exported by prior resources.", - resource.name - )); + if dry_run { + // In dry-run mode, exports may not render because this.* + // fields are unavailable (no actual API calls). Inject + // placeholder values so downstream resources can still + // render their templates. + let mut placeholder_data = HashMap::new(); + for item in &resource.exports { + if let Some(map) = item.as_mapping() { + for (_, val) in map { + if let Some(v) = val.as_str() { + placeholder_data.insert(v.to_string(), "".to_string()); + } + } + } else if let Some(s) = item.as_str() { + placeholder_data.insert(s.to_string(), "".to_string()); + } + } + info!( + "dry run: injecting placeholder exports for [{}]: {:?}", + resource.name, + placeholder_data.keys().collect::>() + ); + export_vars( + &mut runner.global_context, + &resource.name, + &placeholder_data, + &resource.protected, + ); + } else { + runner.run_troubleshoot( + resource, + &resource_queries, + "create", + &full_context, + show_queries, + ); + catch_error_and_exit(&format!( + "exports query for [{}] could not be rendered - unresolved template variables. \ + Check that all referenced variables are defined in the manifest or exported by prior resources.", + resource.name + )); + } } if !dry_run { diff --git a/src/commands/teardown.rs b/src/commands/teardown.rs index f9c3a1e..42455f5 100644 --- a/src/commands/teardown.rs +++ b/src/commands/teardown.rs @@ -224,26 +224,35 @@ fn run_teardown(runner: &mut CommandRunner, dry_run: bool, show_queries: bool, _ let resource_queries = runner.get_queries(resource, &full_context); // Get exists query (fallback to statecheck) - render JIT - let (exists_query_str, exists_retries, exists_retry_delay) = - if let Some(eq) = resource_queries.get("exists") { - let rendered = - runner.render_query(&resource.name, "exists", &eq.template, &full_context); - (rendered, eq.options.retries, eq.options.retry_delay) - } else if let Some(sq) = resource_queries.get("statecheck") { - info!( - "exists query not defined for [{}], trying statecheck query as exists query.", - resource.name - ); - let rendered = - runner.render_query(&resource.name, "statecheck", &sq.template, &full_context); + let (exists_query_str, exists_retries, exists_retry_delay) = if let Some(eq) = + resource_queries.get("exists") + { + let rendered = + runner.render_query(&resource.name, "exists", &eq.template, &full_context); + (rendered, eq.options.retries, eq.options.retry_delay) + } else if let Some(sq) = resource_queries.get("statecheck") { + info!( + "exists query not defined for [{}], trying statecheck query as exists query.", + resource.name + ); + if let Some(rendered) = + runner.try_render_query(&resource.name, "statecheck", &sq.template, &full_context) + { (rendered, sq.options.retries, sq.options.retry_delay) } else { info!( - "No exists or statecheck query for [{}], skipping...", + "[{}] statecheck has unresolved variables, skipping...", resource.name ); continue; - }; + } + } else { + info!( + "No exists or statecheck query for [{}], skipping...", + resource.name + ); + continue; + }; // Check if delete query template exists (don't render yet — may need // this.* fields from the exists check). @@ -341,6 +350,13 @@ fn run_teardown(runner: &mut CommandRunner, dry_run: bool, show_queries: bool, _ if delete_confirmed { info!("successfully deleted {}", resource.name); } else { + runner.run_troubleshoot( + resource, + &resource_queries, + "delete", + &full_context, + show_queries, + ); info!("[{}] delete could not be confirmed", resource.name); } } else { diff --git a/src/core/utils.rs b/src/core/utils.rs index 8f20085..616c19c 100644 --- a/src/core/utils.rs +++ b/src/core/utils.rs @@ -600,7 +600,7 @@ pub fn export_vars( // --- resource-scoped key (immutable: only written if not already set) --- let scoped_key = format!("{}.{}", resource_name, key); global_context.entry(scoped_key.clone()).or_insert_with(|| { - info!( + debug!( "set {} [{}] to [{}] in exports", if is_protected { "protected variable" diff --git a/website/docs/resource-query-files.md b/website/docs/resource-query-files.md index 3b2a6ff..62da309 100644 --- a/website/docs/resource-query-files.md +++ b/website/docs/resource-query-files.md @@ -314,6 +314,79 @@ Some providers return the final operation status synchronously in the `RETURNING - `RETURNING *` only captures the **first** row of the response. - The `callback.*` shorthand is implicitly scoped to the current resource's `.iql` file. Use the fully-qualified `{resource_name}.callback.*` form in downstream resources. +### `troubleshoot` + +`troubleshoot` blocks are optional diagnostic queries that run automatically when a post-operation verification fails - specifically when a `build` statecheck fails or a `teardown` delete cannot be confirmed. They provide visibility into why an operation failed by querying the provider for status information that is not available in the standard error output. + +This is useful for any provider that returns an asynchronous operation handle (request token, operation ID, job ID, etc.) via `RETURNING *`. When combined with [`return_vals`](manifest-file#resourcereturn_vals) in the manifest, the handle is captured as a `this.*` variable which the troubleshoot query can reference. + +For example, given this manifest configuration: + +```yaml +return_vals: + create: + - RequestToken +``` + +The troubleshoot query can reference the captured token: + +```sql +/*+ troubleshoot:create */ +SELECT OperationStatus, StatusMessage, ErrorCode +FROM awscc.cloud_control.resource_request +WHERE RequestToken = '{{ this.RequestToken }}' +AND region = '{{ region }}'; +``` + +The operation qualifier (`:create`, `:update`, `:delete`) associates the troubleshoot query with a specific operation. A plain `/*+ troubleshoot */` with no qualifier runs after **any** failed operation on the resource. + +#### When troubleshoot runs + +| Command | Trigger | +|---|---| +| `build` | Post-deploy statecheck fails (resource not in desired state after create or update) | +| `teardown` | Delete cannot be confirmed (resource still exists after delete) | + +#### Execution behaviour + +- **Fire-and-forget** - the query executes once (not polled like callbacks) +- **Tolerant rendering** - if template variables are missing the query is silently skipped +- **Non-blocking** - if the troubleshoot query itself fails, a warning is logged and the original error is still reported +- Results are logged as pretty-printed JSON at `info` level so they appear in default output alongside the failure message +- In dry-run mode, troubleshoot queries do not execute + +#### Output format + +When a troubleshoot query returns results, they are logged as pretty-printed JSON: + +``` +[vpc] troubleshoot diagnostics (create): + +[ + { + "OperationStatus": "FAILED", + "StatusMessage": "The maximum number of VPCs has been reached.", + "ErrorCode": "GeneralServiceException" + } +] +``` + +#### Context + +The troubleshoot query is rendered with the full template context at the point of failure, including: + +- All global and resource properties from the manifest +- `callback.*` variables from `RETURNING *` data (if the preceding DML included `RETURNING *`) +- `this.*` fields captured by the `exists` query (if available) +- All variables exported by upstream resources + +#### Troubleshoot options + +| Option | Required | Default | Description | +|---|---|---|---| +| `retries` | no | 1 | Number of query attempts | +| `retry_delay` | no | 0 | Seconds to wait between attempts | + ## Query options Query options are used with query anchors to provide options for the execution of the query.