Skip to content

feat: Add AI-powered issue triage and PR review bot#260

Open
sudsali wants to merge 17 commits intoawslabs:masterfrom
sudsali:master
Open

feat: Add AI-powered issue triage and PR review bot#260
sudsali wants to merge 17 commits intoawslabs:masterfrom
sudsali:master

Conversation

@sudsali
Copy link
Copy Markdown

@sudsali sudsali commented Apr 20, 2026

Summary

Adds a GitHub Actions bot that automatically triages issues and reviews PRs using Amazon Bedrock. Runs on issue open/reopen, issue comments, and PR open/reopen/synchronize events.

Issue Handling

  • Classifies issues (bug, question, feature request, off-topic) using a two-phase LLM call
  • First pass classifies and selects source files to read; second pass responds with full code context
  • Follows up on user replies (capped at 2 bot replies per issue)
  • Detects user dissatisfaction and escalates to maintainers via Slack
  • Escalates reopened issues directly to Slack
  • Labels issues automatically

PR Review

  • Posts inline review comments on specific diff lines via GitHub review API
  • Validates line numbers against diff hunks before submitting
  • Uses REQUEST_CHANGES when findings exist
  • Re-reviews on new commits (synchronize trigger)
  • Reads existing review comments to avoid duplicates

Architecture

GitHub Event → Job 1: analyze (read-only, AWS creds)
                ├── Load KB from S3
                ├── Generate codebase map
                ├── Fetch issue/PR + comments
                ├── Bedrock Converse API (system prompt cached, user content guarded)
                ├── Structured JSON output
                ├── Sanitize output
                └── Write JSON artifact
              → Job 2: act (write-only, no AWS creds)
                ├── Post comment/review to GitHub
                ├── Add labels
                └── Notify Slack

Security

  • Bedrock Guardrails — prompt injection detection + sensitive info blocking (replaces regex-based sanitizer)
  • Trusted/untrusted split — prompt instructions go as system prompt (cached, not guarded); user content (issue body, PR diff, comments) wrapped in guardContent and run through guardrail.
  • Template injection preventionstring.Template.safe_substitute instead of .format()
  • Two-job architecture — read-only analyze job (AWS creds, no GitHub write), write-only act job (GitHub write, no AWS creds)
  • Fork PR blocking — fork PRs are not processed
  • Output sanitization — backup injection markers + #N references wrapped in backticks to prevent auto-linking

Files

File Purpose
issue-bot.yml GitHub Actions workflow
main.py Two-phase orchestrator (analyze/act)
config.py Environment variable loading + allowed labels
bedrock_client.py Bedrock Converse API with prompt caching, guardrails, structured output, circuit breaker
github_client.py GitHub API + local file reads + codebase map + diff line validation
knowledge_base.py S3-backed KB with relevance truncation
slack_client.py Slack webhook notifications
sanitizer.py Backup injection blocking + issue ref formatting
prompts.py Prompt loading from env vars
schemas/*.json JSON schemas for structured LLM output (issue, PR review, followup)
tests/test_bot.py 42 unit tests for parsing, validation, and sanitization

Required GitHub Secrets

AWS_ROLE_ARN, KB_S3_BUCKET, KB_S3_KEY, BEDROCK_MODEL_ID, BEDROCK_API_VERSION, SLACK_WEBHOOK_URL, ISSUE_CLASSIFY_PROMPT, ISSUE_RESPOND_PROMPT, PR_FILE_REVIEW_PROMPT, FOLLOWUP_PROMPT, GUARDRAIL_ID

Testing

Supports workflow_dispatch with dry_run: true. Tested on the fork against bug reports, usage questions, feature requests, follow-ups, dissatisfaction escalation, prompt injection, duplicate runs, PR inline reviews, and re-reviews on new commits.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Comment thread scripts/issue_bot/sanitizer.py
Comment thread .github/workflows/issue-bot.yml
Comment thread scripts/issue_bot/prompts.py
Comment thread scripts/issue_bot/main.py Outdated
Comment thread scripts/issue_bot/sanitizer.py
Comment thread scripts/issue_bot/sanitizer.py
Comment thread scripts/issue_bot/bedrock_client.py Outdated
Comment thread scripts/issue_bot/bedrock_client.py Outdated
Comment thread .github/workflows/issue-bot.yml
Comment thread scripts/issue_bot/github_client.py
sudsali added 14 commits April 22, 2026 14:30
…ials false, metadata footer, os.walk, retention 30d
… diffs

Bedrock Guardrails scan all content (system prompt AND user message).
PR diffs containing the bot's own security code (e.g. 'ignore previous
instructions' in sanitizer.py) trigger guardrail false positives.

Fix: add use_guardrail parameter to invoke(). PR reviews pass
use_guardrail=False since diffs are code, not user-generated attacks.
Issue/followup prompts still use the guardrail (default True).

Reverts _split_prompt to the original simple <knowledge_base> split.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants