Architecture & Components
k8s-autopilot is built on the production-grade Deep Agent pattern — a multi-tier hierarchy of agents, MCP servers, and HITL gates. This page covers the architectural components that power the system. For domain-specific capabilities, see the Capabilities pages.
1. 🎯 Supervisor Agent
The Supervisor Agent is a pure router that delegates ALL Kubernetes infrastructure requests to the appropriate domain coordinator. It never performs operations directly.
Routing Table
| Request Type | Tool | Target Coordinator |
|---|---|---|
| Helm chart generation/update/ops | transfer_to_helm_operator | Helm Operator Coordinator |
| K8s cluster ops (pods, scaling, exec) | transfer_to_k8s_operator | K8s Operator Coordinator |
| ArgoCD / Argo Rollouts / Traefik | transfer_to_app_operator | App Operator Coordinator |
| Prometheus / Alertmanager / OTel / Loki / Tempo | transfer_to_observability_operator | Observability Coordinator |
| Clarification / out-of-scope | request_human_feedback | User |
Natural Language Mapping
The Supervisor translates non-technical language into domain-aware routing:
| User Says | Maps To | Coordinator |
|---|---|---|
| "deploy", "ship", "release" | ArgoCD sync | App Operator |
| "zero downtime", "gradual" | Argo Rollouts canary/blue-green | App Operator |
| "split traffic", "A/B test" | Traefik weighted routing | App Operator |
| "scale up", "more capacity" | K8s scaling | K8s Operator |
| "what's firing", "on-call" | Alert triage | Observability |
| "silence", "mute" | Create silence | Observability |
| "metrics", "PromQL", "CPU spikes" | Prometheus query | Observability |
| "trace", "latency", "bottleneck" | Tempo distributed tracing | Observability |
| "logs", "error patterns", "LogQL" | Loki log aggregation | Observability |
| "instrument service", "collector" | OpenTelemetry pipelines | Observability |
Context Engineering
The Supervisor uses a 3-layer middleware stack to maintain routing accuracy across long sessions:
| Middleware | Purpose |
|---|---|
SupervisorContextMiddleware | Re-injects accumulated domain summaries as a SystemMessage before every model call — ensures cross-domain awareness survives summarization |
SummarizationMiddleware | Auto-compresses conversation history when it exceeds ~75% of context budget (default: 4000 tokens), keeping only the last 6 messages |
ModelCallLimitMiddleware | Caps model calls at 15 per turn to prevent runaway routing loops |
2. 🧩 Domain Coordinators
Each coordinator is a Deep Agent — a LangGraph-based orchestrator that manages its own team of sub-agents. Coordinators handle:
- Intent extraction: Translating user requests into DevOps-aware parameters
- Sub-agent delegation: Routing to the correct sub-agent with
[PLAN-LOCKED]context - HITL orchestration: Presenting plans and collecting approval before delegating execution
- Operations journaling: Logging every operation for context persistence
| Coordinator | Sub-Agents | Capabilities Page |
|---|---|---|
helm-operator-coordinator | 7 (planner, skill-builder, generator, validator, updater, operation, github) | Helm Operator |
app-operator-coordinator | 3 (argocd-onboarder, argo-rollouts-onboarder, traefik-edge-router) | App Operator |
k8s-operator-coordinator | 1 (k8s-cluster-ops) | K8s Operator |
observability-coordinator | 5 (prometheus-operator, alertmanager-operator, opentelemetry-operator, loki-operator, tempo-operator) | Observability |
3. 🔌 JIT MCP Connections
Sub-agents that interact with external systems use a Just-In-Time (JIT) connection pattern. Instead of holding open connections to all 10 MCP servers for the entire session, each sub-agent is wrapped in a CompiledSubAgent that only opens its MCP connection when that specific node is executed. The connection is closed immediately after the sub-agent completes.
Connection Types
| Type | Description | Used By |
|---|---|---|
| JIT MCP | Opens MCP connection lazily, closes after execution | All MCP-connected sub-agents |
| Static Dict | Simple dict spec, no MCP — uses filesystem and in-memory tools | helm-skill-builder, helm-generator, helm-updater, helm-validator |
| Compiled Subgraph | LangGraph subgraph with its own internal supervisor | helm-planner |
4. 🛡️ Human-in-the-Loop Governance
AI shouldn't arbitrarily execute state-modifying operations on your cluster. k8s-autopilot enforces strict HITL governance at multiple layers.
The [PLAN-LOCKED] Delegation Protocol
Every state-modifying operation follows a mandatory Intent → Plan → Approve → Execute lifecycle:
- Intent Extraction: The coordinator translates the user's request into DevOps-aware parameters
- Plan Presentation: A structured plan is presented via
request_user_inputwith action details, resource names, namespaces, and impact assessment - User Approval: The LangGraph execution pauses (
interrupt()) and the UI renders an approval card [PLAN-LOCKED]Execution: After approval, the coordinator delegates to the sub-agent with the[PLAN-LOCKED]prefix — the sub-agent skips its own planning phase and executes pre-approved parameters directly- Verification: The sub-agent confirms the operation's success independently
Operation Classification
| Operation Type | Approval Required | Example |
|---|---|---|
| Read-Only | No — instant execution | "List pods", "check sync status", "query metrics", "TraceQL lookup" |
| State-Modifying | Yes — full HITL pipeline | "Deploy app", "scale deployment", "create silence", "install exporter" |
| Commit Gates | Yes — explicit confirmation | "Push chart to GitHub", "sync ArgoCD app" |
Rejection Protocol
If a user rejects a plan:
- The agent does not retry autonomously with modified parameters
- It asks the user what to adjust
- Maximum of 2 plan presentations per request before asking the user to rephrase
5. ⚡ Advanced Middleware Stack
k8s-autopilot implements a dedicated middleware stack to protect cluster stability and prevent LLM context degradation:
A2UI Buffer Interceptor (A2UIBufferMiddleware)
Complex observability queries (e.g. distributed trace trees, 500-line log streams, PromQL range matrices) return megabytes of raw JSON that can exhaust the LLM's context window.
The A2UIBufferMiddleware intercepts raw data from MCP tools, stores it in the LangChain tool artifact store, and provides the LLM with a lightweight pointer string. The sub-agent then calls build_obs_a2ui to stream interactive charts, timelines, and log tables directly to the UI via the A2UI Protocol without loading payload bytes into the prompt.
Operations Context Injection (ObsOperationContextMiddleware)
Survives standard LangGraph message summarization by actively prepending the recent operations log as a SystemMessage just-in-time before each model invocation.
Code Interpreter Middleware (PTC Allowlist)
Sub-agents are equipped with Python Tool Calling (PTC) within a sandbox. Instead of issuing dozens of sequential tool calls to inspect individual pods or targets, sub-agents can run a programmatic loop to query read-only endpoints and aggregate results in a single step.
6. 🔄 Cross-Domain Handoff Protocol
When a coordinator determines that a user's request belongs to a different domain, it emits a structured signal:
"This is outside my scope. Please use the appropriate operator.
User Request: [The user's specific request]
Context: [What was previously discovered]"
The Supervisor detects this via pattern matching, extracts the structured context, and immediately re-routes to the correct coordinator with a [CROSS-DOMAIN] prefix — injecting the prior coordinator's findings:
[CROSS-DOMAIN] Source: observability.
Prior findings: 5 critical alerts for checkout service.
User Request: Check pod status for checkout service
This "blackboard pattern" enables seamless multi-domain investigations without the user repeating themselves.
7. 🧠 Skills, Memory & Context Engineering
k8s-autopilot maintains persistence and context awareness using a multi-layered virtual filesystem backed by a CompositeBackend:
| Virtual Path | Backend | Purpose |
|---|---|---|
/skills/ | StateBackend (LangGraph state) | Operational workflow instructions loaded by sub-agents |
/memories/ | StoreBackend (InMemoryStore, org-scoped) | Governance files and operations journals |
/workspace/ | FilesystemBackend (real disk) | Generated chart files, synced via sync_workspace_to_disk |
/shared/ | StoreBackend (shared namespace) | Cross-domain shared context |
Skills
Skills are strict operational playbooks that dictate exactly how each sub-agent must interact with MCP servers. Each skill directory contains:
SKILL.md— YAML frontmatter + step-by-step workflowreferences/— Domain-specific patterns and templates
| Domain | Skill Directories | Sub-Agents |
|---|---|---|
| Helm | 6 directories (generator, skill-builder, operation, validator, updater, github-agent) | 7 sub-agents |
| App | 3 directories (argocd-gitops, argo-rollouts-gitops, traefik-edge-routing) | 3 sub-agents |
| K8s | 1 directory (kubernetes-cluster-ops) | 1 sub-agent |
| Observability | 5 directories (prometheus, alertmanager, opentelemetry, loki, tempo) | 5 sub-agents |
8. ⚙️ State Management
Each agent operates on its own dedicated state schema, optimized for its specific task:
| State Schema | Used By | Key Fields |
|---|---|---|
MainSupervisorState | Supervisor | user_query, workflow_state, active_phase, domain_summaries, cross_domain_context |
| Coordinator States | Each coordinator | messages, user_query, domain-specific fields |
| Sub-Agent States | Each sub-agent | Inherited from coordinator via input_transform |
State Transformers
Data is not shared blindly. A StateTransformer middleware explicitly converts data when moving between the Supervisor and coordinators. This ensures context isolation — sub-agents see only what they need, preventing hallucination from irrelevant history.
9. 🛠 Tech Stack & MCP Servers
| Component | Technology | Purpose |
|---|---|---|
| Agent Framework | deepagents / LangGraph | State machine, orchestration, sub-graph routing |
| LLM Interface | LangChain Core | Tool execution, message schemas |
| Tools/Integrations | Model Context Protocol (MCP) | Standardized protocol for external systems |
| User Interface | A2UI / TalkOps A2A | Real-time streaming, HITL approval cards, markdown rendering |
| Persistence | PostgreSQL (AsyncPg / Psycopg) | Durable state machine checkpointing across sessions |
| Runtime | Python 3.12+ | Core agent backend |
MCP Servers
k8s-autopilot connects to 10 MCP servers — TalkOps-native servers (PyPI packages, stdio transport) and external tools:
| MCP Server | Package / Command | Transport | Domain |
|---|---|---|---|
helm_mcp_server | helm-mcp-server | stdio | Helm chart operations & releases |
argocd_mcp_server | argocd-mcp-server | stdio | ArgoCD GitOps sync & app projects |
traefik_mcp_server | traefik-mcp-server | stdio | Traefik IngressRoutes & middlewares |
argo_rollout_mcp_server | argo-rollout-mcp-server | stdio | Progressive delivery canary & blue-green |
prometheus-mcp-server | prometheus-mcp-server | stdio | Prometheus metrics & PromQL |
alertmanager-mcp-server | alertmanager-mcp-server | stdio | Alertmanager triage & silences |
opentelemetry-mcp-server | opentelemetry-mcp-server | stdio | OTel Collector & auto-instrumentation |
loki-mcp-server | loki-mcp-server | stdio | Loki log aggregation & LogQL |
tempo-mcp-server | tempo-mcp-server | stdio | Tempo distributed tracing & TraceQL |
kubernetes_mcp_server | npx kubernetes-mcp-server@latest | stdio | Raw Kubernetes cluster ops |