diff --git a/_posts/2026-06-22-oncall-agent-aws-i.md b/_posts/2026-06-22-oncall-agent-aws-i.md new file mode 100644 index 0000000..f583208 --- /dev/null +++ b/_posts/2026-06-22-oncall-agent-aws-i.md @@ -0,0 +1,496 @@ +--- +layout: post +comments: true +title: Building an oncall agent on AWS - Part 1 +excerpt: Architecture and AWS infrastructure for a production-grade oncall agent built with Bedrock AgentCore, MCP, Lambda, and Terraform +categories: genai +tags: [aws,bedrock,agent,mcp,sre] +toc: true +img_excerpt: +--- + +For an oncall agent to be useful, it needs to inspect live operational data, respect permissions, run reliably during incidents, and produce answers based on facts. This two-part article walks through building such an agent on AWS, by leveraging **AWS Bedrock AgentCore** for hosting agents, **MCP** for tool access, **Lambda** for executing tools, **S3** for deployment artifacts and runtime manifests, **Secrets Manager** for credentials, and **Terraform** as the source of truth. + +In this first part, we focus on the system architecture and AWS infrastructure. In [Part 2]({{ "genai/2026/06/22/oncall-agent-aws-ii/" | absolute_url }}), we implement the different components of the system: agents MCP tools, agent-to-agent orchestration, tests, and deployment workflow. + +## Overall Architecture + +The core idea is to split the platform into four layers: + +1. **Interaction layer**: Slack, an internal assistant, a web UI, or local AI tooling sends user requests. +2. **Agent runtime layer**: one or more AgentCore runtimes host the actual agents. +3. **Tool gateway layer**: an MCP gateway exposes operational tools through a standard protocol. +4. **System adapter layer**: Lambda functions call PagerDuty, Grafana, Elasticsearch, Backstage, Sourcegraph, deployment APIs, Snowflake, or any other SRE system. + +```mermaid +flowchart TD + U[User in Slack, Web UI, or IDE] --> C[Client or bot service] + + subgraph AGENTCORE_RUNTIME[AWS Bedrock AgentCore Runtime] + O[Orchestrating agent] + A1[Incident triage agent] + A2[Ownership agent] + A3[Code search agent] + end + + subgraph AGENTCORE_GATEWAY[AWS Bedrock AgentCore Gateway] + G[MCP gateway] + end + + subgraph LAMBDA[AWS Lambda] + I[Gateway request interceptor] + L1[PagerDuty MCP target] + L2[Grafana MCP target] + L3[Elasticsearch MCP target] + L4[Backstage MCP target] + end + + subgraph S3[Amazon S3] + M[Agent runtime manifest] + Z[Agent deployment ZIPs] + TS[MCP tool schemas] + end + + subgraph SECRETS[AWS Secrets Manager] + GS[Gateway OAuth client credentials] + BS[Backend API credentials] + end + + subgraph CACHE[Amazon ElastiCache for Redis] + R[Tool rate-limit counters] + end + + subgraph EXTERNAL[External operational systems] + S1[PagerDuty API] + S2[Grafana API] + S3L[Log search] + S4[Service catalog] + end + + C --> O + O --> A1 + O --> A2 + O --> A3 + O --> M + A1 --> M + A2 --> M + A3 --> M + Z --> O + Z --> A1 + Z --> A2 + Z --> A3 + + A1 --> G + A2 --> G + A3 --> G + GS --> A1 + GS --> A2 + GS --> A3 + + G --> I + I --> R + I --> M + I --> L1 + I --> L2 + I --> L3 + I --> L4 + TS --> G + + BS --> L1 + BS --> L2 + BS --> L3 + BS --> L4 + L1 --> S1 + L2 --> S2 + L3 --> S3L + L4 --> S4 +``` + +Each subgraph names the AWS service that hosts or runs that part of the platform. This design keeps the model-facing part small. The LLM reasons, chooses tools, and synthesizes the answer. The gateway, Lambda targets, and IAM policies decide what the model can touch. + +### Why split agents and tools? + +It is tempting to create one large agent with every tool and every instruction. That becomes hard to evaluate and easy to confuse. A better pattern is to create a small number of specialized agents: + +- an **orchestrating agent** that plans and delegates +- an **incident triage agent** for PagerDuty, Grafana, logs, and deploy context +- an **ownership agent** for service catalog and team routing +- a **code search agent** for source search and file inspection +- domain agents for specific teams or platforms + +Each specialist has narrower prompts, fewer tools, and clearer evaluation cases. The orchestrator can expose those specialists as tools through an agent-to-agent protocol such as A2A. + +### Use configuration as the control plane + +The reviewed project uses a central `config.yml` file as the source of truth for deployed agents and MCP tool policy. A simplified version looks like this: + +```yaml +agents: + oncall-agent: + path: agents/oncall-agent + entrypoint: main.py + description: Orchestrates specialist agents and answers users. + + sre-agent: + path: agents/sre-agent + entrypoint: main.py + description: Investigates incidents, alerts, deployments, and logs. + +tools: + pagerduty: + list_open_incidents: + rate_limit: + calls_per_min: 60 + get_incident_detail: + rate_limit: + calls_per_min: 120 + + elasticsearch: + search_logs: + disabled: true + get_log_context: + rate_limit: + calls_per_min: 120 +``` + +This gives you one place to answer operational questions: + +- Which agents are deployed? +- Where is each agent entrypoint? +- Which tools are enabled? +- Which tools are rate limited? +- Which tools are intentionally disabled? + +Terraform can decode this file, build deployment artifacts for each agent, create AgentCore runtimes, publish a manifest to S3, and configure gateway policy. The application code can read the same manifest at runtime. + +### Repository layout + +With the building blocks defined, the repository layout can be simple: + +```text +oncall-agent/ + agents/ + oncall-agent/ + ownership-agent/ + tools/ + pagerduty/ + handler.py + schema.json + grafana/ + handler.py + schema.json + common/ + interceptors/ + handler.py + terraform/ + agentcore/ + dev/ + prod/ + scripts/ + build_deployment_zips.py + setup_lambdas.sh + config.yml +``` + +## AWS Infrastructure + +The AWS side has a small number of durable building blocks. The goal is to make every agent deployable, discoverable, and able to call tools without embedding secrets in prompts or source code. + +### VPC and networking + +Run the agents and Lambda targets inside a VPC when they need private network access. The common shape is: + +- one VPC dedicated to the agent platform +- private subnets across supported availability zones +- public subnets only for NAT gateways +- a security group for AgentCore runtimes +- a security group for Lambda targets +- VPC endpoints for S3, CloudWatch Logs, and Bedrock Runtime +- NAT egress for external SaaS APIs such as PagerDuty or Slack + +In Terraform: + +```hcl +resource "aws_vpc" "agent_vpc" { + cidr_block = var.vpc_cidr + enable_dns_support = true + enable_dns_hostnames = true +} + +resource "aws_subnet" "private" { + count = length(var.availability_zones) + vpc_id = aws_vpc.agent_vpc.id + cidr_block = cidrsubnet(var.vpc_cidr, 8, count.index) + availability_zone = var.availability_zones[count.index] + map_public_ip_on_launch = false +} + +resource "aws_security_group" "agent_runtime" { + vpc_id = aws_vpc.agent_vpc.id + + egress { + from_port = 0 + to_port = 0 + protocol = "-1" + cidr_blocks = ["0.0.0.0/0"] + } +} +``` + +Do not skip this layer. Incident agents usually need to reach a mix of private systems and external SaaS APIs. The VPC design should make that possible without placing runtimes on public networks. + +### S3 buckets for agent artifacts and manifests + +AgentCore direct-code deployment needs code artifacts. Store those in a versioned S3 bucket: + +```hcl +resource "aws_s3_bucket" "agent_code" { + bucket = "sre-agents-agent-code-${var.environment}-${var.aws_region}" +} + +resource "aws_s3_bucket_versioning" "agent_code" { + bucket = aws_s3_bucket.agent_code.id + + versioning_configuration { + status = "Enabled" + } +} +``` + +Then upload one ZIP per configured agent: + +```hcl +resource "aws_s3_object" "agent_code_objects" { + for_each = local.agents + + bucket = aws_s3_bucket.agent_code.bucket + key = "agents/${each.key}-code.zip" + source = "${path.module}/../../agent_build/${each.key}-code.zip" + source_hash = filemd5("${path.module}/../../agent_build/${each.key}-code.zip") +} +``` + +Also publish a runtime manifest. The orchestrating agent can read it to discover subagents: + +```hcl +resource "aws_s3_object" "agent_manifest" { + bucket = aws_s3_bucket.agent_code.bucket + key = "agent_manifest.json" + content_type = "application/json" + + content = jsonencode({ + agents = { + for name, info in local.agents : name => merge(info, { + runtime_name = aws_bedrockagentcore_agent_runtime.agent[name].agent_runtime_name + runtime_arn = aws_bedrockagentcore_agent_runtime.agent[name].agent_runtime_arn + aws_region = var.aws_region + }) + } + tools = local.tools + }) +} +``` + +This manifest is the bridge between Terraform and application code. It lets an agent discover the rest of the fleet without hard-coded ARNs. + +### Bedrock AgentCore runtimes + +Create one AgentCore runtime per configured agent: + +```hcl +resource "aws_bedrockagentcore_agent_runtime" "agent" { + for_each = local.agents + agent_runtime_name = replace(each.key, "-", "_") + role_arn = aws_iam_role.agent_runtime_execution.arn + + protocol_configuration { + server_protocol = "A2A" + } + + agent_runtime_artifact { + code_configuration { + entry_point = [each.value.entrypoint] + runtime = "PYTHON_3_12" + + code { + s3 { + bucket = aws_s3_bucket.agent_code.bucket + prefix = aws_s3_object.agent_code_objects[each.key].key + version_id = aws_s3_object.agent_code_objects[each.key].version_id + } + } + } + } + + environment_variables = { + AGENT_NAME = each.key + CONFIGURATION_BUCKET_NAME = aws_s3_bucket.agent_code.bucket + AGENT_MANIFEST_S3_KEY = "agent_manifest.json" + BEDROCK_MCP_GATEWAY_REGION = var.aws_region + BEDROCK_MCP_GATEWAY_ID = aws_bedrockagentcore_gateway.sre_tools.gateway_id + BEDROCK_MCP_SECRET_ID = aws_secretsmanager_secret.gateway_oauth.name + } + + network_configuration { + network_mode = "VPC" + network_mode_config { + subnets = aws_subnet.private[*].id + security_groups = [aws_security_group.agent_runtime.id] + } + } +} +``` + +Two practical details matter: + +- AgentCore runtime names may have stricter naming rules than your repository paths, so normalize names. +- Use S3 object version IDs in the runtime artifact. That gives Terraform a clean reason to redeploy when code changes. + +### MCP gateway + +The gateway is the model-safe front door to operational tools. It speaks MCP to agents and invokes Lambda targets behind the scenes. + +```hcl +resource "aws_bedrockagentcore_gateway" "sre_tools" { + name = var.tools_gateway_name + description = "Shared SRE MCP gateway" + role_arn = aws_iam_role.tools_gateway.arn + + authorizer_type = "CUSTOM_JWT" + authorizer_configuration { + custom_jwt_authorizer { + discovery_url = "${var.gateway_oauth_issuer}/.well-known/openid-configuration" + allowed_audience = [ + var.gateway_oauth_user_client_id, + "api://default", + ] + } + } + + protocol_type = "MCP" + protocol_configuration { + mcp { + instructions = "Expose SRE tools for incident management, observability, deploy data, ownership, code search, and log analysis." + search_type = "SEMANTIC" + supported_versions = ["2025-03-26"] + } + } +} +``` + +The gateway should not know PagerDuty or Grafana secrets. It only routes tool calls. The Lambda target owns the backend-specific auth. + +### Lambda-backed MCP targets + +Each tool namespace becomes a Lambda target with a JSON schema: + +```hcl +resource "aws_bedrockagentcore_gateway_target" "pagerduty" { + name = "pagerduty-target" + gateway_identifier = aws_bedrockagentcore_gateway.sre_tools.gateway_id + description = "PagerDuty incident management tools" + + credential_provider_configuration { + gateway_iam_role {} + } + + target_configuration { + mcp { + lambda { + lambda_arn = aws_lambda_function.pagerduty.arn + + tool_schema { + s3 { + uri = "s3://${aws_s3_bucket.tool_schemas.bucket}/pagerduty/schema.json" + } + } + } + } + } +} +``` + +The schema is not documentation only. It is the contract the model sees. Keep tool names, descriptions, input schemas, and output schemas precise. + +### Secrets Manager + +Store service credentials in Secrets Manager and inject only secret names into runtimes or Lambda environment variables: + +```hcl +resource "aws_secretsmanager_secret" "pagerduty" { + name = "${var.environment}/sre-tools/pagerduty" +} + +resource "aws_lambda_function" "pagerduty" { + function_name = "pagerduty-mcp-target" + role = aws_iam_role.lambda_role.arn + handler = "handler.lambda_handler" + runtime = "python3.12" + + environment { + variables = { + PAGERDUTY_SECRET_NAME = aws_secretsmanager_secret.pagerduty.name + } + } +} +``` + +Then grant the Lambda role permission to read only that secret. + +### Request interceptor and rate limits + +The reviewed architecture uses a gateway request interceptor backed by Redis. It handles: + +- MCP `initialize` responses +- disabled tools +- per-tool fixed-window rate limits +- gateway request logging + +That gives you a policy enforcement point before tool Lambda invocation. The policy comes from the same manifest that Terraform generated from `config.yml`. + +At a high level: + +```python +namespace, tool = tool_name.split("___") +tool_configuration = load_tool_configuration_from_s3() + +if tool_configuration[namespace][tool].get("disabled"): + return disabled_response() + +limit = tool_configuration[namespace][tool].get("rate_limit", {}).get("calls_per_min", 120) +current = redis.get(f"{tool_name}:{current_minute}") or 0 + +if int(current) >= limit: + return rate_limited_response() + +redis.incr(f"{tool_name}:{current_minute}") +return allow_request() +``` + +This is especially useful for expensive log searches, high-cardinality metrics queries, and tools backed by vendor APIs with strict limits. + +### 8. IAM + +Keep IAM split by responsibility: + +- **Agent runtime role**: invoke Bedrock models, read the agent manifest from S3, read gateway OAuth credentials, discover the gateway, invoke other AgentCore runtimes. +- **Gateway role**: invoke only the registered Lambda targets. +- **Lambda target roles**: read only their own secrets, write logs, and access VPC resources if needed. +- **Invocation access role**: allow approved external clients or bots to invoke AgentCore runtimes. + +The most important rule is that the agent runtime should not receive broad cloud permissions. Let tools encapsulate privileged actions and expose only safe, typed operations. + +## What we have built so far + +At this point the platform has: + +- a private AWS network for agents and tools +- a versioned S3 deployment bucket +- one AgentCore runtime per configured agent +- an MCP gateway with custom authorization +- Lambda-backed tool targets +- Secrets Manager integration +- Redis-backed request interception and rate limiting +- a generated runtime manifest that agents can use for discovery + +In [Part 2]({{ "genai/2026/06/22/oncall-agent-aws-ii/" | absolute_url }}), we will implement the Python agent itself, expose SRE systems as MCP tools, create an orchestrating agent that delegates to specialists, and set up tests and CI for safe rollout. diff --git a/_posts/2026-06-22-oncall-agent-aws-ii.md b/_posts/2026-06-22-oncall-agent-aws-ii.md new file mode 100644 index 0000000..424e5bc --- /dev/null +++ b/_posts/2026-06-22-oncall-agent-aws-ii.md @@ -0,0 +1,668 @@ +--- +layout: post +comments: true +title: Building an oncall agent on AWS - Part 2 +excerpt: Implementing the Python agent, MCP tools, agent-to-agent orchestration, tests, and deployment workflow +categories: genai +tags: [aws,bedrock,agent,mcp,sre] +toc: true +img_excerpt: +mermaid: true +--- + +In [Part 1]({{ "genai/2026/06/22/oncall-agent-aws-i/" | absolute_url }}), we designed the overall architecture and AWS infrastructure for a production-grade oncall agent. We created the AgentCore runtimes, MCP gateway, Lambda tool targets, secrets, networking, runtime manifest, and rate-limit interceptor. + +This second part focuses on implementation. We will build the agent entrypoint, load tools from the MCP gateway, compose prompts, expose SRE systems as Lambda-backed MCP tools, orchestrate specialist agents, and add tests and CI gates. + +## Implementation + +The final repository should make the happy path boring: + +```bash +make setup-env +make create incident-agent +make dev incident-agent +make test agent=incident-agent +``` + +Behind those commands, the repo should keep a clear separation between agents, tools, shared libraries, infrastructure, and evaluation data: + +```text +oncall-agent/ + agents/ + squire-agent/ + main.py + pyproject.toml + incident-agent/ + main.py + prompts/ + tests/ + evals/ + tools/ + pagerduty/ + handler.py + schema.json + grafana/ + handler.py + schema.json + common/ + mcp_handler.py + tool_policy.py + sre-agent-common/ + a2a_support/ + agentcore/ + input/ + prompt/ + tool/ + scripts/ + terraform/ + config.yml +``` + +The shared package is important. If every agent reimplements payload parsing, A2A server setup, MCP loading, and prompt loading, the platform becomes inconsistent fast. + +## Building the Agent Runtime + +An AgentCore direct-code agent can be a small Python program. The application creates a Bedrock AgentCore app, creates a LangChain agent backed by Bedrock, loads tools, wraps the LangChain agent in an A2A server, and mounts that server. + +```python +from bedrock_agentcore import BedrockAgentCoreApp +from langchain.agents import create_agent +from langchain_aws import ChatBedrockConverse + +from sre_agent_common.a2a_support.server import a2a_server_from_environment +from sre_agent_common.tool.langchain_mcp_tools import load_mcp_registry_tools_sync + +MODEL_ID = "us.anthropic.claude-sonnet-4-5-20250929-v1:0" + +app = BedrockAgentCoreApp() + +agent = create_agent( + model=ChatBedrockConverse(model=MODEL_ID), + system_prompt="You are an SRE investigation agent. Gather evidence before answering.", + tools=load_mcp_registry_tools_sync(), +) + +a2a_server = a2a_server_from_environment( + langchain_agent=agent, + local_host="0.0.0.0", + local_port=9000, +) + +app.mount("/", a2a_server.asgi_server) + + +def main(): + app.run(host="0.0.0.0", port=9000) + + +if __name__ == "__main__": + main() +``` + +This is the minimum useful shape. In production, add: + +- prompt files instead of one inline prompt +- context-window trimming +- transport retry middleware for tool calls +- structured response models for reports +- health endpoint support +- tests around payload conversion and prompt assembly + +### Compose prompts from reusable files + +Oncall agents need domain context: what "shard" means, what systems own deployment state, when to use logs, and how to report confidence. Store that context as files rather than a large string in `main.py`. + +```python +from pathlib import Path + +PROMPTS_DIR = Path(__file__).resolve().parent / "prompts" + + +def load_prompt(name: str) -> str: + return (PROMPTS_DIR / name).read_text(encoding="utf-8").strip() + + +def build_system_prompt() -> str: + sections = [ + load_prompt("identity.md"), + load_prompt("investigation_workflow.md"), + load_prompt("tool_guidance.md"), + load_prompt("triage_report_format.md"), + ] + return "\n\n".join(section for section in sections if section) +``` + +The investigation prompt should be explicit: + +```markdown +# Operating Rules + +- Always gather evidence with tools before answering. +- Prefer targeted queries over broad scans. +- Separate facts from inference. +- Label uncertain conclusions with confidence. +- Never invent commands, URLs, owners, rollbacks, or environment details. + +# Investigation Method + +1. Identify the concrete entity: incident ID, service, cluster, shard, host, or alert. +2. Query the highest-signal system first. +3. Correlate evidence across tools before forming conclusions. +4. Highlight impact, blast radius, next action, and missing evidence. +``` + +This is not just style. It reduces unsupported answers during incidents. + +### Convert client payloads into model messages + +Most production agents receive more than a raw string. They receive: + +- the next user request +- prior conversation history +- optional system context +- channel metadata +- user identity or scope + +Normalize that into the input shape your agent framework expects: + +```python +def convert_invocation_payload_into_langchain_input(payload, history_to_inject=None): + history = [] + + messages = payload.history.messages if payload.history else [] + for message in messages: + if message.role == "user": + history.append({"role": "user", "content": message.text}) + elif message.role == "assistant": + history.append({"role": "assistant", "content": message.text}) + + if history_to_inject: + history.extend(history_to_inject) + + history.append({"role": "user", "content": payload.next_request}) + return {"messages": history} +``` + +Also guard the context window. If the request is too large, trim older history and insert a notice: + +```python +MAX_INPUT_CHARS = 350_000 +TRUNCATED_HISTORY_NOTICE = "Earlier conversation history was truncated." + + +def estimate_input_chars(system_prompt, messages): + return len(system_prompt) + len(json.dumps(messages, separators=(",", ":"))) + + +def trim_messages_to_budget(system_prompt, messages): + messages = list(messages) + trimmed = False + + while len(messages) > 1 and estimate_input_chars(system_prompt, messages) > MAX_INPUT_CHARS: + messages.pop(0) + trimmed = True + + if trimmed: + messages = messages[:-1] + [ + {"role": "system", "content": TRUNCATED_HISTORY_NOTICE}, + messages[-1], + ] + + if estimate_input_chars(system_prompt, messages) > MAX_INPUT_CHARS: + raise ValueError("Request exceeds the configured model input budget.") + + return messages +``` + +Failing cleanly is better than letting a long Slack thread turn into a model invocation error. + +## Loading MCP Tools + +Agents should not hard-code every tool endpoint. They should discover the gateway endpoint from AgentCore, exchange OAuth client credentials for a bearer token, and create MCP client connections. + +```python +from langchain_mcp_adapters.client import MultiServerMCPClient +from langchain_mcp_adapters.sessions import StreamableHttpConnection + + +async def load_mcp_tools(gateway_url: str, bearer_token: str, allowed_tools=None): + client = MultiServerMCPClient( + { + "sre_tools": StreamableHttpConnection( + url=gateway_url, + transport="streamable_http", + headers={"Authorization": f"Bearer {bearer_token}"}, + ) + } + ) + tools = await client.get_tools() + + if allowed_tools is None: + return tools + + return [ + tool + for tool in tools + if any(allowed_name in tool.name for allowed_name in allowed_tools) + ] +``` + +In AgentCore, resolve the bearer token from Secrets Manager: + +```python +async def get_bearer_token(secret_id, region): + secret = await load_gateway_oauth_secret_from_secrets_manager(secret_id, region) + return await exchange_client_credentials_for_access_token( + client_id=secret.client_id, + client_secret=secret.client_secret, + token_url=secret.token_url, + scope=secret.scope, + ) +``` + +This keeps gateway credentials out of the agent prompt, tool schemas, and code. + +## Implementing MCP Tool Targets + +Each tool target has two files: + +- `schema.json`: the contract the model sees +- `handler.py`: the Lambda implementation + +### Define a precise schema + +For a PagerDuty target, a schema might expose only safe read operations: + +```json +[ + { + "name": "list_open_incidents", + "description": "List PagerDuty incidents that are currently triggered or acknowledged.", + "inputSchema": { + "type": "object", + "properties": { + "service_ids": { + "type": "array", + "items": { "type": "string" }, + "description": "Optional PagerDuty service IDs to scope the query." + }, + "urgency": { + "type": "string", + "description": "Optional urgency filter such as high or low." + }, + "limit": { + "type": "integer", + "minimum": 1, + "maximum": 100 + } + } + } + } +] +``` + +Avoid vague tools like `run_query` or `call_api` unless the user is trusted and the backend has strong authorization. Oncall agents are most reliable when tools encode operational intent. + +### Write a small Lambda handler + +The tool implementation should focus on backend logic: + +```python +import json +import os +from urllib import parse, request + +import boto3 + +from common.mcp_handler import create_lambda_handler + +PAGERDUTY_API_BASE_URL = "https://api.pagerduty.com" +PAGERDUTY_SECRET_NAME = os.environ["PAGERDUTY_SECRET_NAME"] + + +def get_pagerduty_api_token(): + client = boto3.client("secretsmanager") + response = client.get_secret_value(SecretId=PAGERDUTY_SECRET_NAME) + secret_string = response["SecretString"] + payload = json.loads(secret_string) + return payload["api_token"] + + +def pagerduty_get(path, params=None): + query = parse.urlencode(params or {}, doseq=True) + url = f"{PAGERDUTY_API_BASE_URL}{path}" + if query: + url = f"{url}?{query}" + + req = request.Request( + url, + headers={ + "Authorization": f"Token token={get_pagerduty_api_token()}", + "Accept": "application/vnd.pagerduty+json;version=2", + }, + ) + with request.urlopen(req, timeout=10) as response: + return json.loads(response.read().decode("utf-8")) + + +def list_open_incidents(service_ids=None, urgency=None, limit=100): + params = { + "statuses[]": ["triggered", "acknowledged"], + "limit": min(int(limit), 100), + } + if service_ids: + params["service_ids[]"] = service_ids + if urgency: + params["urgencies[]"] = [urgency] + + data = pagerduty_get("/incidents", params) + return { + "count": len(data.get("incidents", [])), + "incidents": [ + { + "id": incident.get("id"), + "title": incident.get("title") or incident.get("summary"), + "status": incident.get("status"), + "urgency": incident.get("urgency"), + "service": (incident.get("service") or {}).get("summary"), + "html_url": incident.get("html_url"), + } + for incident in data.get("incidents", []) + ], + } + + +lambda_handler = create_lambda_handler( + {"list_open_incidents": list_open_incidents}, + handler_name="pagerduty_mcp_lambda", +) +``` + +The shared `create_lambda_handler` handles gateway event parsing, tool-name extraction, argument extraction, response wrapping, and error wrapping. + +### Keep disabled tools out of the schema + +When building Lambda artifacts, generate the deployed schema from `config.yml`: + +```python +def filter_schema_tools(schema_tools, namespace_policy): + filtered = [] + for tool in schema_tools: + policy = namespace_policy.get(tool["name"], {}) + if policy.get("disabled"): + continue + filtered.append(tool) + return filtered +``` + +This gives you two layers of safety: + +- disabled tools are not advertised to agents +- the interceptor can still reject disabled tool calls + +## Implementing the Orchestrator + +Once you have multiple specialist agents, expose each one as a tool to the orchestrating agent. + +```mermaid +flowchart LR + U[User] --> O[Squire or orchestrator] + O --> T1[Tool: incident_agent] + O --> T2[Tool: ownership_agent] + O --> T3[Tool: code_search_agent] + T1 --> A1[AgentCore runtime] + T2 --> A2[AgentCore runtime] + T3 --> A3[AgentCore runtime] +``` + +The orchestrator loads the manifest from S3: + +```python +def load_agent_manifest_from_s3(bucket, key): + response = boto3.client("s3").get_object(Bucket=bucket, Key=key) + return json.loads(response["Body"].read().decode("utf-8")) +``` + +Then it turns each subagent into a LangChain tool: + +```python +from langchain_core.tools import tool + + +def subagent_tool_generator(definition, session_id): + @tool( + definition.runtime_name, + description=definition.description, + args_schema=definition.input_jsonschema, + ) + async def subagent_tool(**kwargs): + return await invoke_a2a_client(definition, kwargs, session_id) + + return subagent_tool +``` + +The orchestrator prompt should teach routing behavior: + +```markdown +# Routing + +- Use the incident agent for active PagerDuty incidents, alerts, metrics, logs, and deploy context. +- Use the ownership agent for team, service, escalation, and Slack channel questions. +- Use the code search agent for implementation details, configuration, and ownership clues in source code. +- Ask a clarifying question only when there is no concrete entity to investigate. +``` + +This turns "agent sprawl" into a manageable platform. + +## Local Development + +A good local workflow should set up credentials, fetch non-secret runtime config, and run AgentCore's local development server. + +```makefile +dev: + aws sts get-caller-identity >/dev/null + aws s3 cp s3://$(DEV_AGENT_BUCKET)/agent_env_configuration - > .agent-env + export AGENT_NAME=$(agent); \ + export AGENT_MANIFEST_LOCAL_PATH=$(PWD)/config.yml; \ + cd agents/$(agent) && uv run agentcore dev +``` + +Then run a sample client: + +```bash +make dev incident-agent +make client +``` + +For a deployed runtime: + +```bash +make client agent_runtime_arn="arn:aws:bedrock-agentcore:us-west-2:123456789012:runtime/incident_agent" +``` + +## Scaffolding a New Agent + +Do not make engineers remember every file to create. Provide a command: + +```bash +make create incident-agent +``` + +The scaffolder should: + +1. Add the agent to `config.yml`. +2. Create `agents/incident-agent/`. +3. Generate `pyproject.toml`. +4. Add the shared agent package dependency. +5. Create `main.py`. +6. Generate AgentCore local config. +7. Print the next command to run. + +The generated `main.py` can start as a small template: + +```python +from bedrock_agentcore import BedrockAgentCoreApp +from langchain.agents import create_agent +from langchain_aws import ChatBedrockConverse + +from sre_agent_common.a2a_support.server import a2a_server_from_environment +from sre_agent_common.tool.langchain_mcp_tools import load_mcp_registry_tools_sync + +app = BedrockAgentCoreApp() + +agent = create_agent( + model=ChatBedrockConverse(model="us.anthropic.claude-sonnet-4-5-20250929-v1:0"), + system_prompt="Edit this prompt for your agent.", + tools=load_mcp_registry_tools_sync({"backstage", "pagerduty"}), +) + +server = a2a_server_from_environment(agent, "0.0.0.0", 9000) +app.mount("/", server.asgi_server) + + +def main(): + app.run(host="0.0.0.0", port=9000) +``` + +## Tests and Evaluations + +Test the platform in layers. + +### Unit tests + +Unit-test deterministic code: + +- prompt assembly +- payload parsing +- context trimming +- schema filtering +- tool policy loading +- Lambda event parsing + +Example: + +```python +def test_trim_messages_preserves_latest_request(): + messages = [ + {"role": "user", "content": "old" * 100_000}, + {"role": "user", "content": "what is broken right now?"}, + ] + + trimmed = trim_messages_to_budget("system prompt", messages) + + assert trimmed[-1]["content"] == "what is broken right now?" +``` + +### Tool tests + +For every Lambda target, test: + +- unknown tools return a structured error +- missing required arguments return an error +- backend responses are normalized +- secrets are loaded from the expected source +- disabled tools are filtered + +### Agent evals + +Agent behavior needs evals beyond unit tests. Useful layers are: + +- **deterministic tool-call assertions**: the agent should call `get_incident_detail` when given an incident ID +- **golden snapshots**: a known prompt should produce a stable report structure +- **degradation cases**: poor inputs should not produce invented facts +- **LLM-as-judge rubrics**: judge evidence quality, concision, and actionability + +Keep eval data next to the agent: + +```text +agents/incident-agent/evals/ + test_cases.yaml + degradation_cases.yaml + rubric.yaml + golden/ + test_deterministic.py + test_golden_snapshots.py + test_llm_judge.py +``` + +## CI and Deployment + +The CI pipeline should follow the same shape as local development: + +1. Lint Python. +2. Format-check Terraform. +3. Build direct-code deployment ZIPs. +4. Build Lambda target artifacts. +5. Run unit tests. +6. Run targeted evals. +7. Run Terraform tests. +8. Plan development infrastructure. +9. Apply development infrastructure. +10. Plan production infrastructure. +11. Manually apply production. + +Example pipeline jobs: + +```yaml +build:agent-zips: + stage: build + image: python:3.12-slim + script: + - pip install uv pyyaml + - python scripts/build_deployment_zips.py + artifacts: + paths: + - agent_build/* + +test:repo: + stage: test + image: python:3.12-slim + script: + - pip install uv + - uv sync --extra dev + - uv run pytest --cov=tools --cov=agents + +terraform:plan:dev: + stage: plan + script: + - ./scripts/setup_lambdas.sh + - cd terraform/dev && terraform plan +``` + +Print runtime metadata after apply so humans and bots can smoke test the deployment: + +```bash +terraform output -json agent_runtime_metadata \ + | python scripts/print_agent_runtime_metadata.py --label dev-apply +``` + +## Production Checklist + +Before giving an oncall agent broad access, check the following: + +- **Tool boundaries**: every tool has a narrow schema and a safe output shape. +- **Secrets**: credentials live in Secrets Manager, not prompts or config files. +- **IAM**: agents cannot call arbitrary AWS APIs. +- **Rate limits**: expensive or vendor-limited tools have per-minute limits. +- **Prompt rules**: the agent must gather evidence before answering. +- **Evidence citations**: final reports identify the tool and query behind important claims. +- **Context limits**: long conversations are trimmed or rejected cleanly. +- **Evals**: incident, ownership, code search, and failure cases have test coverage. +- **Rollout**: development apply is automatic or easy; production apply is manual. +- **Observability**: CloudWatch logs, latency, error rate, and tool failures are visible. + +## Final Thoughts + +The most reusable idea is the separation of responsibilities: + +- AgentCore hosts agents. +- MCP standardizes tool access. +- Lambda adapts internal systems into typed tools. +- Terraform owns deployment state. +- S3 publishes runtime manifests. +- Secrets Manager owns credentials. +- Redis-backed interception enforces runtime tool policy. +- Evals keep behavior from drifting. + +That architecture lets you add new SRE capabilities without continuously rewriting the agent. Add a tool schema and Lambda handler for a new system, register it in Terraform, expose it through the gateway, and give the right specialist agent clear instructions for when to use it.