Building a Production-Ready AI Agent on the JVM
Build an enterprise operations agent with Java 25, Spring Boot 4.1, Spring AI 2.0, Kotlin MCP tools, RAG, OAuth, evals, and observability.
Chapters
What we are building: an enterprise operations agent
The architecture, scope, and threat model for a JVM AI agent that answers runbook questions, calls secured MCP tools, and never writes without approval.
Scaffold the multi-module build: Gradle 9, convention plugins, and architecture tests
Build the ops-agent-platform skeleton: Gradle wrapper, version catalog, four convention plugins, Java 25 toolchain, and ArchUnit rules that run from day one.
Build the deterministic operations simulator
Implement the fictional ops backend: Flyway migrations, JdbcClient persistence, idempotent incident creation, and controllable latency, flaky, malformed, and outage fault modes.
Create the agent API: ChatClient, provider profiles, and a deterministic stub model
Stand up agent-api with Spring AI ChatClient, Ollama and hosted-provider profiles, REST and SSE endpoints, and a stub ChatModel that keeps CI fully offline.
Build the runbook ingestion pipeline: Markdown to pgvector
Parse versioned Markdown runbooks into semantic chunks, embed them with Ollama, store them in pgvector with tenant metadata, and re-ingest only what changed.
Add access-controlled RAG and citations
Tenant-filtered pgvector retrieval, delimited context construction, deterministic citation mapping, and a real abstention path when the evidence is not there.
Add typed, self-correcting outputs: the IncidentAssessment schema
Turn model prose into a validated IncidentAssessment record: JSON schema generation, provider-native output where available, bounded retries, and semantic checks beyond syntax.
Build the Kotlin MCP operations server: tools as a secured boundary
Implement the mcp-operations-server in Kotlin: Streamable HTTP transport, @McpTool read tools, a typed simulator client, structured errors, and contract tests.
Connect the agent to remote MCP tools: policy registry and bounded loop
Configure the Spring AI MCP client, wrap tool execution in a per-tool policy registry, inject tenant context the model cannot choose, and bound the agent loop.
Secure both boundaries: Keycloak, JWT validation, and tenant authorization
Replace trusted headers with real OIDC: Keycloak realm import, issuer/audience/scope validation on agent-api and the MCP server, tenant claims, and the negative security matrix.
Add write tools and human approval: the gate the model cannot cross
Arm create_incident and append_incident_note behind scope checks, idempotency keys, and a single-use approval token bound to user, tenant, and a hash of the exact arguments.
Add guardrails and bounded orchestration: budgets, timeouts, and degraded modes
Prompt size limits, retry classification, deadline propagation, bulkheads, and graceful degradation — the layer that keeps a sick dependency from becoming an incident.
End-to-end observability: one trace from HTTP to tool call
OpenTelemetry across agent, MCP server, and simulator; Prometheus metrics with low-cardinality labels; Loki logs; Tempo traces; Grafana dashboards — with redaction built in.
Build the evaluation suite: testing what the model actually does
Versioned eval datasets, deterministic graders for citations and tool choice, pinned LLM-as-judge where needed, and a CI gate that separates hard failures from probabilistic drift.
Test performance and resilience: measuring the framework, not the model
Gatling scenarios against the stub model isolate application overhead; controlled fault modes separate framework latency from downstream sickness; a results template keeps numbers honest.
Containerize and deploy: Compose for real, Kubernetes without lies
Production-shaped container images, the full Compose stack, Kustomize overlays, probes, NetworkPolicies, resource limits, and rollout guidance that matches what we actually built.
Production hardening and the final exercise: prove it, then ship it
The final architecture review, threat-model update, failure drills, secret rotation, upgrade strategy, and the incident scenario that exercises every mechanism at once — v1.0.0.