feat(ci): add GitHub Actions integration smoke test#149
Open
srmanda-cs wants to merge 5 commits intomicrosoft:mainfrom
Open
feat(ci): add GitHub Actions integration smoke test#149srmanda-cs wants to merge 5 commits intomicrosoft:mainfrom
srmanda-cs wants to merge 5 commits intomicrosoft:mainfrom
Conversation
Move vllm, flwr, autogen-agentchat, transformers, groq, azure-ai-ml into [tool.poetry.group.clients] so CI can install without CUDA/ML runtime dependencies: poetry install --without clients --with dev Declare packages used by aiopslab/ core modules that were previously only available as undeclared transitive dependencies: docker - aiopslab/service/dock.py pandas - aiopslab/orchestrator/actions/base.py, observer/*.py pytz - aiopslab/observer/__init__.py, metric_api.py requests - aiopslab/observer/trace_api.py, service/telemetry/loki.py pyyaml - aiopslab/config.py, generators/fault/*.py, service/*.py tiktoken - aiopslab/orchestrator/evaluators/quantitative.py Add [tool.poetry.group.dev] with pytest + pytest-asyncio and register the 'integration' pytest marker for the new smoke test.
Adds tests/integration/smoke_test.py — a zero-cost integration test
that exercises the full orchestrator pipeline using a DummyAgent that
makes no LLM or API calls.
Problem: noop_detection_hotel_reservation-1 (lightest in the registry)
Pipeline: deploy HotelReservation → inject no-op fault → run workload
→ DummyAgent calls submit("No") → evaluate → recover → cleanup
The test asserts:
- init_problem() returns non-empty desc, instructions, and actions
- 'submit' is present in available actions
- Detection Accuracy == 'Correct' (no fault injected, answer is 'No')
Run locally after cluster setup:
poetry run pytest tests/integration/smoke_test.py -v -s -m integration
Triggers on every push and pull request to any branch, ensuring
regressions in the core orchestrator pipeline are caught before merge.
Setup:
- Checks out repo with submodules (aiopslab-applications)
- Installs kind v0.27.0 (to /tmp to avoid collision with kind/ dir)
- Creates two-node kind cluster with kind/kind-config-x86.yaml
- Creates /run/udev as a directory so OpenEBS NDM hostPath mount
succeeds on runners where it may be absent or a socket file
- Pre-installs OpenEBS and Prometheus before pytest to avoid hitting
the 5-minute wait_for_ready() hard timeout on cold runners
- Installs Python deps excluding CUDA clients (--without clients)
- Generates aiopslab/config.yml (k8s_host: kind)
- Runs: pytest tests/integration/smoke_test.py -m integration
On failure: dumps cluster state and uploads kind logs as an artifact
(retention: 7 days).
srmanda-cs
changed the title
feat(ci): add GitHub Actions integration smoke test
Contributor
Author
|
Latest run for the PR available at: https://github.com/srmanda-cs/AIOpsLab/actions/runs/22465817762/job/65071423830 |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a zero-cost GitHub Actions smoke test that validates the complete orchestrator pipeline without requiring LLM APIs or CUDA dependencies. The test deploys a kind cluster, runs the HotelReservation application with no-op fault injection, and verifies correctness using a lightweight DummyAgent.
Changes:
- Added GitHub Actions workflow with pre-installed OpenEBS/Prometheus to avoid timeout issues during cold image pulls
- Created integration smoke test using DummyAgent against the lightest problem in the registry
- Restructured dependencies to separate heavy ML packages into optional
[clients]group
Reviewed changes
Copilot reviewed 3 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
.github/workflows/integration-test.yml |
New CI workflow that sets up kind cluster, pre-installs infrastructure, and runs smoke test with failure diagnostics |
tests/integration/smoke_test.py |
New smoke test exercising full orchestrator pipeline with DummyAgent requiring no external APIs |
pyproject.toml |
Reorganized dependencies: moved ML packages to optional [clients] group, declared previously implicit core deps, added [dev] group |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
gaganso
reviewed
Feb 27, 2026
| # Trigger on every push to any branch and on every pull request. | ||
| # This ensures regressions are caught before merging to main. | ||
| on: | ||
| push: |
Collaborator
There was a problem hiding this comment.
Let us restrict the triggers
on:
push:
branches: [main]
pull_request:
branches: [main]
I think the current trigger is too broad.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #19
Adds a GitHub Actions workflow (
.github/workflows/integration-test.yml) that runs ano-cost, end-to-end integration smoke test on every push and pull request.
The test deploys a real kind cluster, runs the full orchestrator pipeline
(HotelReservation app → no-op fault injection → workload → evaluation → teardown),
and asserts correctness — using a
DummyAgentthat requires no LLM or API keys.Changes
.github/workflows/integration-test.ymltests/integration/smoke_test.pyDummyAgentagainstnoop_detection_hotel_reservation-1pyproject.toml[clients]group; declared previously-implicit core deps (docker, pandas, pytz, requests, pyyaml, tiktoken); added[dev]group with pytestpoetry.lockpyproject.tomlBreaking Changes
None. All changes are purely additive. The
[clients]group is installed by default(
poetry install) so existing developer workflows are unaffected. CI usespoetry install --without clients --with devto avoid pulling CUDA/ML runtimes.Notes for Reviewers
Happy to revise anything before it's considered ready!
A few things I'd specifically appreciate input on:
wait_for_ready()has a hardmax_wait=300s. To handle cold Docker Hub pulls on CI runners, the workflow pre-installs OpenEBS and Prometheus before pytest runs. An alternative would be to increasemax_waitor make it configurable — open to whichever approach the team prefers.kind/kind-config-x86.yamldirectly. I initially created a separatekind-config-ci.yamlbut removed it to avoid duplication. Let me know if a CI-specific config is preferred.noop_detection_hotel_reservation-1was chosen as the lightest problem in the registry (no real fault, minimal workload). If there's a more idiomatic choice for a smoke test, I'm happy to change it.pyproject.tomldeps: The implicit-dependency declarations I added (docker,pandas, etc.) were missing from the file but working on developer machines via transitive deps. I can split these into a separate PR if preferred.