Context
System benchmarks can define a corpus generator (corpora.generator in the benchmark YAML) and a service deployer, including Terraform (deployer: tf under _dev/benchmark/system/deploy/).
Some data streams are exercised with cloud inputs (for example aws-s3), where Terraform is used to create buckets, objects, queues, and related resources. The objects Terraform uploads are defined with paths on disk at apply time (for example aws_s3_object with a local source file).
Problem
The benchmark runner writes generated corpus files at a different phase and location than Terraform uses when applying infrastructure.
Terraform runs during service setup, before the benchmark run phase that performs corpus generation. The generator writes its output under the service logs directory (the path surfaced as {{SERVICE_LOGS_DIR}} in scenario YAML). That path is aligned with inputs that read from the agent-accessible log directory (for example filestream), not with Terraform’s staged module directory where *.tf and referenced local files are copied for terraform apply.
As a result, a benchmark cannot use the corpus generator to produce the same bytes that Terraform uploads in the same run. Teams that need Terraform for realistic cloud wiring and also want template-driven, per-run event counts from the generator hit this limitation: the generator and the Terraform source paths are not connected by the tool today.
Impact
Packages with S3-backed (or similar) inputs cannot mirror the generator + SERVICE_LOGS_DIR workflow that works for file and api based benchmarks.
Context
System benchmarks can define a corpus generator (
corpora.generatorin the benchmark YAML) and a service deployer, including Terraform (deployer: tfunder_dev/benchmark/system/deploy/).Some data streams are exercised with cloud inputs (for example aws-s3), where Terraform is used to create buckets, objects, queues, and related resources. The objects Terraform uploads are defined with paths on disk at apply time (for example
aws_s3_objectwith a localsourcefile).Problem
The benchmark runner writes generated corpus files at a different phase and location than Terraform uses when applying infrastructure.
Terraform runs during service setup, before the benchmark run phase that performs corpus generation. The generator writes its output under the service logs directory (the path surfaced as
{{SERVICE_LOGS_DIR}}in scenario YAML). That path is aligned with inputs that read from the agent-accessible log directory (for example filestream), not with Terraform’s staged module directory where*.tfand referenced local files are copied forterraform apply.As a result, a benchmark cannot use the corpus generator to produce the same bytes that Terraform uploads in the same run. Teams that need Terraform for realistic cloud wiring and also want template-driven, per-run event counts from the generator hit this limitation: the generator and the Terraform
sourcepaths are not connected by the tool today.Impact
Packages with S3-backed (or similar) inputs cannot mirror the generator +
SERVICE_LOGS_DIRworkflow that works for file and api based benchmarks.