Add typed, self-correcting outputs: the IncidentAssessment schema

Checkpoint tag: chapter-06-structured-output — IncidentAssessment responses deserialize safely, invalid output fails predictably after a bounded number of correction attempts, and every retry’s token cost is visible.

What will be built

A second agent capability: POST /api/v1/conversations/{id}/assess produces a typed IncidentAssessment — severity, confidence, evidence references, recommended actions — from the retrieved context. The flow validates syntax (JSON schema from the record itself, provider-native constraints where supported), validates semantics (evidence IDs must name chunks that were actually retrieved), and retries malformed output a bounded number of times before returning a typed failure.

Why it matters

Chapter 5’s citation parsing is the limit of what prose output can give you: regex, hope, and a counter for when it goes wrong. The assessment is different — it feeds downstream automation, so “the model usually returns JSON” is not a plan. Schema validity buys you shape; it does not buy correctness — a response can be perfectly valid JSON and still cite evidence that was never retrieved. Both layers get their own check, and the chapter measures what self-correction costs instead of assuming it’s free.

Concepts explained

entity() in Spring AI 2.0. .call().entity(Type.class, spec -> …) generates a JSON schema from the Java type, appends it to the prompt, parses the response, and — with the two reliability switches — can push the schema to the provider as an API-level constraint (useProviderStructuredOutput()) and auto-retry schema-invalid responses (validateSchema()). entity() is call()-only: typed parsing needs the whole response, so assessments are a synchronous endpoint by design.

Schema-valid ≠ true. The schema proves the response parses. EvidenceValidator proves the response is grounded — every evidence entry names a chunk_id from this request’s retrieval set. Confusing those two layers is how hallucinated citations ship.

Bounded correction. Retry-with-error-feedback is the honest version of “the model will get it right”: at most 3 attempts, each augmented with the validation error, cumulative token usage tracked. Unbounded correction is an infinite loop that bills you.

Files added or changed

1
domain-contracts: + assessment/{IncidentAssessment, Confidence}.java
2
agent-api/src/main/java/in/o612/eng/opsagent/agent/
3
    assessment/AssessmentService.java, assessment/EvidenceValidator.java
4
    api/AssessmentController.java
5
agent-api/src/main/resources/prompts/assessment.st

Complete code

The contract — deliberately in domain-contracts, so the MCP server could produce or consume the same shape without importing Spring AI:

1
package in.o612.eng.opsagent.contracts.assessment;
2

3
import in.o612.eng.opsagent.contracts.incident.Severity;
4
import java.util.List;
5

6
public record IncidentAssessment(
7
        String summary,
8
        Severity severity,
9
        Confidence confidence,
10
        String serviceId,
11
        List<String> evidence,          // chunk_ids backing the assessment
12
        List<String> recommendedActions) {
13

14
    public enum Confidence { LOW, MEDIUM, HIGH }
15
}

assessment.st — a separate prompt file; assessment is a different task than chat and deserves its own versioned prompt:

1
You are producing a structured incident assessment for an ops engineer.
2
Base every claim on the numbered runbook excerpts and tool results provided.
3
evidence must list only the excerpt numbers used, as "chk-" ids from the
4
provided metadata. If evidence is thin, lower confidence — do not inflate it.

AssessmentService:

1
package in.o612.eng.opsagent.agent.assessment;
2

3
import in.o612.eng.opsagent.agent.retrieval.RunbookRetriever;
4
import in.o612.eng.opsagent.contracts.assessment.IncidentAssessment;
5
import io.micrometer.core.instrument.MeterRegistry;
6
import org.springframework.ai.chat.client.ChatClient;
7
import org.springframework.stereotype.Service;
8

9
import java.util.Set;
10
import java.util.stream.Collectors;
11

12
@Service
13
public class AssessmentService {
14

15
    public sealed interface AssessmentOutcome {
16
        record Assessed(IncidentAssessment assessment, int attempts) implements AssessmentOutcome {}
17
        record Rejected(String reason) implements AssessmentOutcome {}
18
    }
19

20
    private final ChatClient chatClient;
21
    private final RunbookRetriever retriever;
22
    private final EvidenceValidator evidence;
23
    private final MeterRegistry metrics;
24
    private final String prompt;
25

26
    public AssessmentOutcome assess(String tenantId, String serviceId, String question) {
27
        var retrieval = retriever.retrieve(tenantId, serviceId, question);
28
        if (!retrieval.sufficient()) {
29
            return new AssessmentOutcome.Rejected("insufficient evidence");
30
        }
31
        Set<String> validEvidence = retrieval.chunks().stream()
32
                .map(RunbookRetriever.RetrievedChunk::chunkId)
33
                .collect(Collectors.toSet());
34

35
        IncidentAssessment raw;
36
        try {
37
            raw = chatClient.prompt()
38
                    .system(prompt + "\n\nValid evidence ids: " + validEvidence)
39
                    .user(buildUserMessage(question, retrieval))
40
                    .call()
41
                    .entity(IncidentAssessment.class, spec -> spec.validateSchema())
42
                    // provider-native structured output is enabled only on the
43
                    // hosted profile where the provider supports it
44
                    ;
45
        } catch (RuntimeException e) {   // schema validation exhausted its retries
46
            metrics.counter("agent.schema.corrections_exhausted").increment();
47
            return new AssessmentOutcome.Rejected("model produced no schema-valid assessment");
48
        }
49

50
        var violations = evidence.check(raw, validEvidence);
51
        if (!violations.isEmpty()) {
52
            metrics.counter("agent.assessment.semantic_rejections").increment();
53
            return new AssessmentOutcome.Rejected("assessment cited non-retrieved evidence: " + violations);
54
        }
55
        return new AssessmentOutcome.Assessed(raw, attemptsUsed());
56
    }
57

58
    // attemptsUsed() reads the counter delta of agent.schema.corrections around the call;
59
    // buildUserMessage() assembles the numbered context exactly as ConversationService does
60
}

EvidenceValidator — pure, deterministic, and trivially unit-testable:

1
package in.o612.eng.opsagent.agent.assessment;
2

3
import in.o612.eng.opsagent.contracts.assessment.IncidentAssessment;
4
import org.springframework.stereotype.Component;
5

6
import java.util.List;
7
import java.util.Set;
8

9
@Component
10
public class EvidenceValidator {
11

12
    public List<String> check(IncidentAssessment a, Set<String> validEvidence) {
13
        var problems = new java.util.ArrayList<String>();
14
        if (a.evidence() == null || a.evidence().isEmpty()) {
15
            problems.add("no evidence cited");
16
        } else {
17
            a.evidence().stream()
18
                    .filter(id -> !validEvidence.contains(id))
19
                    .forEach(id -> problems.add("unknown evidence id: " + id));
20
        }
21
        if (a.confidence() == IncidentAssessment.Confidence.HIGH
22
                && (a.evidence() == null || a.evidence().size() < 2)) {
23
            problems.add("HIGH confidence requires at least two evidence refs");
24
        }
25
        if (a.recommendedActions() == null || a.recommendedActions().isEmpty()) {
26
            problems.add("no recommended actions");
27
        }
28
        return problems;
29
    }
30
}

AssessmentController maps Assessed → 200, Rejected → 422 with the reason in the ApiError envelope — an unprocessable assessment is a normal outcome, not a crash.

Automated tests

• EvidenceValidatorTest (pure): unknown chunk ID rejected; HIGH confidence with one evidence ref rejected; happy path passes.
• AssessmentServiceTest: stub ChatClient — actually a stubbed ModelGateway-level fake is too low here, so the test uses a scripted ChatModel bean that returns (a) malformed JSON → Rejected after the bounded retries, with agent.schema.corrections incremented; (b) valid JSON citing chk-deadbeef not in the retrieval set → semantic Rejected; (c) valid + grounded → Assessed.
• AssessmentSchemaTest: the generated schema’s required fields match the record components — a guard against someone renaming evidence and silently widening what the model may omit.
• RetryBudgetTest: schema-invalid stub makes exactly maxAttempts calls, not more — verified via the scripted model’s invocation counter.

Failure-injection lab

Scripted ChatModel stubs make failure deterministic:

Measure what self-correction costs: log token usage per attempt (Spring AI’s ChatResponse.getMetadata().getUsage() is cumulative across validation retries — surface it as agent.assessment.tokens per outcome). Do not claim a number you didn’t run; record yours in docs/runbooks/ results template.

Security considerations

Assessments are advisory — nothing in this flow mutates state — but serviceId still comes from the request path, never from model output, and evidence checking prevents a crafted retrieval corpus from laundering fake support into a high-confidence assessment.

Observability checks

agent.assessment.attempts histogram, agent.schema.corrections, agent.assessment.semantic_rejections, agent.assessment.tokens — the four numbers that tell you whether corrections are a safety net or a money leak.

Checkpoint verification checklist

• Malformed output retries ≤ the bound, then fails as Rejected (422), never as a parse exception to the caller.
• Schema-valid but ungrounded output is rejected semantically.
• Token usage is cumulative across retries and visible in metrics.
• entity() is never invoked on the streaming path.

Commit message and Git tag

1
feat(agent-api): schema-validated IncidentAssessment with semantic evidence checks

git tag chapter-06-structured-output

What comes next

Chapter 7 builds the other half of the platform — the Kotlin MCP server that will carry the operational tools the agent calls in Chapter 8.

Project State Ledger — chapter-06-structured-output

• Contracts added: IncidentAssessment (summary, severity, confidence, serviceId, evidence[], recommendedActions[]), Confidence enum
• Endpoint: POST /api/v1/conversations/{id}/assess → 200 IncidentAssessment | 422 ApiError
• Validation layers: schema (entity/validateSchema, bounded) → semantic (EvidenceValidator) → outcome
• Metrics: agent.assessment.attempts, agent.schema.corrections, agent.assessment.semantic_rejections, agent.assessment.tokens
• Rule: entity() on .call() only; streaming stays text
• Next: chapter-07-mcp-server