appsec: add smoke tests for apm standalone

[florentinl]

Motivation

Test that AppSec is correctly enabled and functional in APM Standalone mode (with infra disabled at the agent level).

To do so, I setup a testing class that performs AAP smoke tests using only data intercepted at the agent level. This testing class can then be used through inheritance to create different test suites linked to different features and manifest entries.

Changes

• Add appsec helpers in interfaces.agent to perform assertions on traces intercepted after the agent
• Add smoke tests to check:
  • Threat detection
  • RASP
  • Remote configuration of rules
  • Telemetry emission
  • API Security
    => This ensures that all communication protocols used by AAP (traces, trace stats, telemetry and RC) keep getting proxied by the agent.
• Add two scenarios and a feature for APM Standalone and ASM Standalone + APM Standalone

Workflow

1. ⚠️ Create your PR as draft ⚠️
2. Work on you PR until the CI passes
3. Mark it as ready for review
   • Test logic is modified? -> Get a review from RFC owner.
   • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

• Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
• A docker base image is modified?
  • the relevant build-XXX-image label is present
• A scenario is added, removed or renamed?
  • Get a review from R&P team

[github-actions]

CODEOWNERS have been resolved as:

tests/appsec/test_agent_level_smoke_tests.py                            @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_apm_standalone.py                                     @DataDog/asm-libraries @DataDog/system-tests-core
.github/workflows/run-end-to-end.yml                                    @DataDog/system-tests-core
docs/edit/agent-interface-validation-methods.md                         @DataDog/system-tests-core
manifests/agent.yml                                                     @DataDog/system-tests-core
manifests/cpp_nginx.yml                                                 @DataDog/dd-trace-cpp
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
utils/_context/_scenarios/__init__.py                                   @DataDog/system-tests-core
utils/_features.py                                                      @DataDog/system-tests-core
utils/interfaces/_agent.py                                              @DataDog/system-tests-core
utils/proxy/_deserializer.py                                            @DataDog/system-tests-core
utils/proxy/traces/trace_v1.py                                          @DataDog/system-tests-core

[datadog-official]

⚠️ Tests

✨ Fix all issues with BitsAI or with Cursor

⚠️ Warnings

❄️ 3 New flaky tests detected

tests.appsec.test_apm_standalone.Test_AppSecAPMStandalone_ApiSecurity.test_api_security_smoke[spring-boot-wildfly] from system_tests_suite     (Fix with Cursor)

assert False
 +  where False = any(<generator object ApiSecuritySmokeTests.test_api_security_smoke.<locals>.<genexpr> at 0x7ff492800820>)

self = <tests.appsec.test_apm_standalone.Test_AppSecAPMStandalone_ApiSecurity object at 0x7ff4afd17710>

    def test_api_security_smoke(self) -> None:
>       assert any(
            any(key.startswith("_dd.appsec.s.") for key in span.meta) for _, span in interfaces.agent.get_spans(self.r)
        )
E       assert False
...

tests.appsec.test_apm_standalone.Test_AppSecAPMStandalone_Rasp.test_lfi_smoke[spring-boot-wildfly] from system_tests_suite     (Fix with Cursor)

AssertionError: No RASP attack found for rule rasp-930-100

self = <tests.appsec.test_apm_standalone.Test_AppSecAPMStandalone_Rasp object at 0x7ff4afd16d80>

    def test_lfi_smoke(self) -> None:
>       _assert_rasp_attack(
            self.r,
            "rasp-930-100",
            {
                "resource": {"address": "server.io.fs.file", "value": "../etc/passwd"},
...

tests.appsec.test_apm_standalone.Test_AppSecAPMStandalone_Threats.test_attack_detection_smoke[spring-boot-wildfly] from system_tests_suite     (Fix with Cursor)

AssertionError: Agent should forward detected attacks in span metadata
assert False

self = <tests.appsec.test_apm_standalone.Test_AppSecAPMStandalone_Threats object at 0x7ff4afd16c30>

    def test_attack_detection_smoke(self) -> None:
        found_attack = False
        has_appsec_data = False
    
        for _, span, appsec_data in interfaces.agent.get_appsec_data(self.r):
...

ℹ️ Info

No other issues found (see more)

🧪 All tests passed

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: f753448 | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 03d3a84078

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 03d3a84078

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 733fcc0d42

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4042a1951a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[cbeauchesne]

Could you move this deserialization inside the proxy? ping me if you need help (it may not be obvious to do).

[florentinl]

Makes sense. Thanks. I pushed a version with the deserialization in the proxy but it looks like it could go in many different locations so I am not sure if I put it in the most idiomatic place.