Operating it safely: monitoring, snapshots, security, and personal data

A search projection over 300 million user profiles is a production system holding personal data. This chapter covers what that means day to day. It lists the signals that warn you before users do, takes snapshots and restores them with a retention policy, and replaces the elastic superuser in both applications with API keys that can do only their own jobs. Then it turns to the people behind the profiles: which data the projection should not hold, where personal data leaks into URLs and logs, how erasure reaches every copy, and why Elasticsearch cannot decide what a caller is allowed to see.

That last part includes a bug found while writing this chapter. A support agent restricted to one state could read profile counts for every other state through chapter 12’s facets. You reproduce it and fix it in the application, which is where the fix belongs.

You need the lab and the user-search project from chapter 15. The chapter takes about 75 minutes.

Health states

Chapter 02 introduced them; here is what each means for operations.

A single-node lab with zero replicas is always green, which makes it a poor rehearsal for yellow. On a production cluster, the most common cause of yellow is a node that has left, and the most dangerous is a disk watermark that stops replicas from being allocated.

Monitoring signals

Every signal in this table comes from an API you can call on the lab. The “lab reading” column shows the values observed on the lab node; they show the shape of each response, not healthy targets. Counters such as index_total are cumulative since the node started, so monitoring systems compute rates from the difference between two readings.

The last row is not an Elasticsearch metric, and it is the most important one for this architecture. It is the direct measure of how stale the projection is. Each API is documented in the nodes stats reference; Elastic’s troubleshooting guides cover high JVM memory pressure, watermark errors, and rejected requests.

Needs validation: the thresholds in the last column are starting points. Set real alert thresholds from your own baselines. Stack monitoring collects these metrics continuously into a separate monitoring cluster, which is the usual production setup.

Production note — Measure search latency where users feel it: in the Search API, per endpoint, as a histogram. Elasticsearch’s took excludes network time, deserialisation, and your own code. The Micrometer series on this site covers application metrics.

Snapshots and restore

What it is. A snapshot is a backup of indices, stored in a repository outside the cluster: a shared file system, or object storage such as S3, GCS, or Azure. Snapshots are incremental: each one copies only segments that earlier snapshots in the same repository do not already hold. See snapshot and restore.

Why it matters at 300 million profiles. The projection can always be rebuilt from PostgreSQL, so snapshots are not the only recovery path; they are the faster one. Chapter 14’s backfill rebuilt one million profiles in about 96 seconds in the lab. At 300 million that is hours, even on a large cluster. A restore copies files instead of indexing documents.

Example. A file-system repository needs a path that Elasticsearch is allowed to write to, declared in path.repo. Add it to compose.yaml, with a named volume for the snapshots:

1
      xpack.license.self_generated.type: basic
2
      # Where file-system snapshot repositories may live (chapter 16).
3
      path.repo: /usr/share/elasticsearch/snapshots
4
      ES_JAVA_OPTS: -Xms1g -Xmx1g

1
      - esdata:/usr/share/elasticsearch/data
2
      - essnapshots:/usr/share/elasticsearch/snapshots
3
      - ./certs:/usr/share/elasticsearch/config/certs:ro

and essnapshots: under the top-level volumes. On the first attempt, registering the repository failed with access_denied_exception: Docker creates a new named volume owned by root, and Elasticsearch runs as uid 1000. A one-shot service fixes the ownership before Elasticsearch starts. Add it before the elasticsearch service, and make elasticsearch depend on it:

1
  # One-shot job: a new named volume is owned by root; Elasticsearch runs as uid 1000.
2
  snapshots-init:
3
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
4
    user: "0"
5
    volumes:
6
      - essnapshots:/usr/share/elasticsearch/snapshots
7
    command: ["chown", "-R", "1000:0", "/usr/share/elasticsearch/snapshots"]
8

9
  elasticsearch:
10
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
11
    depends_on:
12
      certs: { condition: service_completed_successfully }
13
      snapshots-init: { condition: service_completed_successfully }

Run docker compose up -d, then register a shared file system repository and take a snapshot of the live index:

1
PUT _snapshot/lab-backups
2
{ "type": "fs", "settings": { "location": "/usr/share/elasticsearch/snapshots/lab-backups" } }
3

4
PUT _snapshot/lab-backups/profiles-1?wait_for_completion=true
5
{ "indices": "user-profile-v2", "include_global_state": false }

1
{"snapshot":{"snapshot":"profiles-1","indices":["user-profile-v2"],"state":"SUCCESS","shards":{"total":1,"failed":0,"successful":1}}}

Restore it next to the live index, under another name, which is how you check a backup without touching production:

1
POST _snapshot/lab-backups/profiles-1/_restore?wait_for_completion=true
2
{
3
  "indices": "user-profile-v2",
4
  "rename_pattern": "user-profile-(.+)",
5
  "rename_replacement": "restored-user-profile-$1",
6
  "include_aliases": false
7
}

1
index                    docs.count store.size
2
restored-user-profile-v2     999997    150.5mb
3
user-profile-v2              999997    150.5mb

_cat/snapshots/lab-backups reported the snapshot’s duration as 4.8 seconds. include_aliases: false matters: restoring the aliases would point user-profile-read at the copy. Delete the restored index when you have checked it.

Schedule snapshots with a lifecycle policy, which also deletes old ones:

1
PUT _slm/policy/nightly-profiles
2
{
3
  "schedule": "0 30 1 * * ?",
4
  "name": "<profiles-{now/d}>",
5
  "repository": "lab-backups",
6
  "config": { "indices": ["user-profile-*"], "include_global_state": false },
7
  "retention": { "expire_after": "14d", "min_count": 3, "max_count": 20 }
8
}

POST _slm/policy/nightly-profiles/_execute runs it immediately; the lab’s run produced a snapshot named profiles-2026.09.25-… covering both index versions.

Trade-off. Choose the recovery path by what you need back. Restoring a snapshot is fast, but it brings back the index as it was, including profiles changed or deleted since. Rebuilding from PostgreSQL is slower and always correct. A realistic plan uses both: restore to serve searches quickly, then reconcile against PostgreSQL, by replaying the outbox from before the snapshot or by running a backfill into the restored index, whose external versions make it safe to repeat.

Common mistake. Never testing a restore. A snapshot you have not restored is a hope, not a backup.

Least privilege with API keys

Both applications have used the elastic superuser since chapter 09. Anyone who obtains that password controls the cluster. Each application should instead hold an API key with only the privileges its job needs. API keys are included in Elastic’s free Basic tier.

The Search API’s key

Create a key that can read the read alias and nothing else:

1
POST _security/api_key
2
{
3
  "name": "search-api",
4
  "expiration": "30d",
5
  "role_descriptors": {
6
    "search_api": { "indices": [ { "names": ["user-profile-read"], "privileges": ["read"] } ] }
7
  },
8
  "metadata": { "service": "search-api" }
9
}

The response contains an encoded value, the credential. Test it with curl -H "Authorization: ApiKey <encoded>":

Exactly as intended, until the Search API runs a stable, point-in-time search with it. Opening the PIT succeeds; the search against the PIT fails with 403 for indices:data/read/search on user-profile-v2. A PIT is bound to the concrete indices behind the alias when it is opened, and searches against it are authorised on those indices, not on the alias name. A key that may only use the alias cannot page with a PIT.

The service’s key therefore also needs read access to the concrete index versions:

1
POST _security/api_key
2
{
3
  "name": "search-api",
4
  "expiration": "30d",
5
  "role_descriptors": {
6
    "search_api": { "indices": [ { "names": ["user-profile-read", "user-profile-v*"], "privileges": ["read"] } ] }
7
  },
8
  "metadata": { "service": "search-api" }
9
}

With this key, a stable walk over 117 profiles returns all 117, and writes through user-profile-write are still refused with 403.

Trade-off. The wider key can also read an old index version kept for rollback, which the narrow one could not. The alternative is to drop stable=true from the Search API and keep the narrow key. Choose deliberately, and note that either way the key cannot write, cannot touch other indices, and cannot read cluster state.

Revoke the first key with the invalidate API, DELETE _security/api_key with {"ids": ["<id>"]}; the response lists it under invalidated_api_keys.

The indexer’s key

The indexer writes, manages index versions and aliases, and creates the synonyms set on an empty cluster:

1
POST _security/api_key
2
{
3
  "name": "indexer",
4
  "expiration": "30d",
5
  "role_descriptors": {
6
    "indexer": {
7
      "cluster": ["manage_search_synonyms"],
8
      "indices": [ { "names": ["user-profile-*"], "privileges": ["read", "write", "manage"] } ]
9
    }
10
  },
11
  "metadata": { "service": "indexer" }
12
}

With this key, creating, configuring, aliasing, and deleting a user-profile-vtest index all succeed; PUT other-index and GET _cat/nodes return 403. The relay and the validate command run unchanged with it.

Configure both applications with keys

Spring Boot 4.1 accepts an API key directly as spring.elasticsearch.api-key, and sends it as Authorization: ApiKey <value>. In search-api/src/main/resources/application.yaml, replace the username and password lines under spring.elasticsearch:

1
    # A key restricted to reading user-profile-read (chapter 16). Never the elastic superuser.
2
    api-key: ${ELASTICSEARCH_API_KEY}

and in indexer/src/main/resources/application.yaml:

1
    # Write and index management on user-profile-*, and the synonyms API (chapter 16).
2
    api-key: ${ELASTICSEARCH_API_KEY}

Each application now receives its own key through ELASTICSEARCH_API_KEY, and ELASTICSEARCH_USERNAME and ELASTICSEARCH_PASSWORD are no longer used. The relay, run with the indexer’s key, applied a change to profile 7 within seconds, and the Search API’s composed search returned the same seven profiles as in chapter 11.

Security note — Keys expire in 30 days here, so rotation is forced rather than forgotten. Rotate by creating the new key, deploying it, and invalidating the old one. Store keys in your platform’s secret store, never in application.yaml or the repository. Keep the elastic superuser for break-glass administration only.

Network isolation and audit

The lab binds every port to 127.0.0.1. In production, Elasticsearch’s HTTP port should be reachable only from the Search API, the indexer, Kibana, and monitoring, on a private network; Kibana itself belongs behind your organisation’s single sign-on. Transport TLS between nodes, which a single node does not need, is required for a multi-node cluster (chapter 10).

Audit logging records authentication events and data access inside Elasticsearch, but it is not part of the free Basic tier: Elastic’s subscriptions page lists it, together with field- and document-level security, for Platinum and Enterprise. On Basic, the Search API must record who searched for what, which it has to do anyway, because only it knows the human caller.

Personal data in the projection

Chapter 01 made PostgreSQL the source of truth. For personal data, that rule has a corollary: every copy outside PostgreSQL is a liability to minimise, protect, and erase.

Data minimisation. The index holds only the fields the capability matrix needs (chapter 05). Question every field again with privacy in mind. dateOfBirth is stored for display but never searched; if no screen needs it, remove it from the next index version and serve it from PostgreSQL on the detail page.

Direct identifiers never go in URLs. Chapter 10’s lookups were GET /api/users/by-email?email=... and GET /by-mobile?mobileNumber=.... That puts email addresses and phone numbers in URLs, which access logs, proxies, monitoring tools, and browser histories keep. This chapter replaces both with a single POST /api/users/lookup whose identifier travels in the request body. Name searches still use query parameters; where URLs are logged by intermediaries you do not control, apply the same change to /search.

Logs are copies too. The Elasticsearch slow log records full query sources (chapter 15). The Search API’s own logs must not contain request bodies or identifiers; its error handling in chapter 10 logs error types, not queries. Give Elasticsearch and application logs the same access controls and retention rules as the data.

Erasure reaches every copy. When a profile is erased in PostgreSQL:

Right to erasure. The table is the operational meaning of an erasure request for this system. A restore from a snapshot taken before the erasure brings the profile back into search results until reconciliation deletes it again. Document that window, and keep it short.

Elasticsearch is not your authorisation system

Elasticsearch’s security controls decide what the service may do: the Search API’s key can read profiles and nothing else. They know nothing about the person using the Search API. Whether a support agent may see profiles outside their region, or see contact details at all, is a decision only the application can make. Field- and document-level security in Elasticsearch cannot replace it either: it is not available on the Basic tier, and even where it is, it would restrict the service account, not the human behind the request.

Stage 1 — Authenticate callers and assign roles

Add Spring Security to search-api/build.gradle.kts:

1
    implementation("org.springframework.boot:spring-boot-starter-validation")
2
    implementation("org.springframework.boot:spring-boot-starter-security")

For the lab, two in-memory users stand in for your identity provider. Add their passwords to application.yaml, read from the environment:

1
  # LOCAL-DEV SHORTCUT: two in-memory users for the lab. Production authenticates callers with your
2
  # identity provider (for example as an OAuth2 resource server) and maps its claims to these roles.
3
  lab-users:
4
    agent-password: ${LAB_AGENT_PASSWORD}
5
    supervisor-password: ${LAB_SUPERVISOR_PASSWORD}

That block goes under user-search:, next to index:. The security configuration grants search and facets to agents and supervisors, and contact details only to supervisors:

1
package `in`.o612.eng.usersearch.api.security
2

3
import org.springframework.beans.factory.annotation.Value
4
import org.springframework.context.annotation.Bean
5
import org.springframework.context.annotation.Configuration
6
import org.springframework.http.HttpMethod
7
import org.springframework.security.config.annotation.web.builders.HttpSecurity
8
import org.springframework.security.config.http.SessionCreationPolicy
9
import org.springframework.security.core.userdetails.User
10
import org.springframework.security.core.userdetails.UserDetailsService
11
import org.springframework.security.crypto.factory.PasswordEncoderFactories
12
import org.springframework.security.provisioning.InMemoryUserDetailsManager
13
import org.springframework.security.web.SecurityFilterChain
14

15
/**
16
 * Who may call what. Elasticsearch only knows the service's API key; these rules are about the person
17
 * using the Search API, and Elasticsearch cannot enforce them.
18
 */
19
@Configuration
20
class SecurityConfig {
21

22
    @Bean
23
    fun securityFilterChain(http: HttpSecurity): SecurityFilterChain {
24
        http
25
            .authorizeHttpRequests { auth ->
26
                auth
27
                    // Spring renders errors, including validation failures, through /error.
28
                    .requestMatchers("/error").permitAll()
29
                    .requestMatchers(HttpMethod.GET, "/api/users/search", "/api/users/facets").hasAnyRole("AGENT", "SUPERVISOR")
30
                    .requestMatchers(HttpMethod.POST, "/api/users/lookup").hasRole("SUPERVISOR")
31
                    .requestMatchers(HttpMethod.GET, "/api/users/*").hasRole("SUPERVISOR")
32
                    .anyRequest().denyAll()
33
            }
34
            .httpBasic { }
35
            .csrf { it.disable() } // stateless API: no browser session, no cookies to forge
36
            .sessionManagement { it.sessionCreationPolicy(SessionCreationPolicy.STATELESS) }
37
        return http.build()
38
    }
39

40
    /** LOCAL-DEV SHORTCUT: fixed lab users. The agent may only see profiles in Bihar. */
41
    @Bean
42
    fun labUsers(
43
        @Value("\${user-search.lab-users.agent-password}") agentPassword: String,
44
        @Value("\${user-search.lab-users.supervisor-password}") supervisorPassword: String,
45
    ): UserDetailsService {
46
        val encoder = PasswordEncoderFactories.createDelegatingPasswordEncoder()
47
        return InMemoryUserDetailsManager(
48
            User.withUsername("agent").password(encoder.encode(agentPassword))
49
                .authorities("ROLE_AGENT", "${AccessScope.STATE_PREFIX}bihar").build(),
50
            User.withUsername("supervisor").password(encoder.encode(supervisorPassword))
51
                .authorities("ROLE_SUPERVISOR").build(),
52
        )
53
    }
54
}

The agent carries an extra authority, STATE_bihar: this agent may only see profiles in Bihar. /error is permitted because Spring renders error responses, including validation failures, through it; without that line, every 400 became a 403 in testing.

Stage 2 — The facet leak

The first implementation applied the agent’s restriction by setting the request’s state parameter to bihar before searching. Searches were correctly restricted. Then the agent asked for facets:

1
curl -s -u agent:$LAB_AGENT_PASSWORD 'localhost:8080/api/users/facets'

1
{"facets":{"state":[{"value":"Maharashtra","count":140007},{"value":"Bihar","count":139925},{"value":"Karnataka","count":70316},{"value":"West Bengal","count":70255},...]}}

Every state, with counts. Chapter 12 designed facets to ignore their own filter, so that users can see alternatives. A state filter is a facet filter, so the state facet ignored it, as designed, and exposed exactly what the restriction was meant to hide. Counts are data: in a small region, “3 profiles named X in state Y” can identify a person.

Principle. An access restriction is never a user-selectable filter. It belongs to a different category, applied to every query and every aggregation, where no facet logic, cursor, or parameter can reach it.

Stage 3 — Scope as a base filter

Add a type for constraints that come from the caller:

1
package `in`.o612.eng.usersearch.api.search
2

3
/**
4
 * Constraints that come from who is asking, not from what they asked. Applied as a base filter to every
5
 * query and every facet, so no facet, cursor, or parameter can reach outside it.
6
 */
7
data class SearchScope(val states: Set<String>) {
8
    companion object {
9
        val UNRESTRICTED = SearchScope(emptySet())
10
    }
11
}

Derive it from the authenticated principal, and refuse requests that try to leave it:

1
package `in`.o612.eng.usersearch.api.security
2

3
import `in`.o612.eng.usersearch.api.search.SearchScope
4
import `in`.o612.eng.usersearch.api.web.UserSearchRequest
5
import org.springframework.security.access.AccessDeniedException
6
import org.springframework.security.core.Authentication
7

8
/**
9
 * What the caller may see, from the authenticated principal, never from request parameters,
10
 * so a caller cannot widen it by editing the query string.
11
 */
12
object AccessScope {
13
    const val STATE_PREFIX = "STATE_"
14

15
    fun of(caller: Authentication, request: UserSearchRequest): SearchScope {
16
        val states = caller.authorities
17
            .mapNotNull { it.authority }
18
            .filter { it.startsWith(STATE_PREFIX) }
19
            .map { it.removePrefix(STATE_PREFIX) }
20
            .toSet()
21
        if (states.isNotEmpty() && request.state != null && states.none { it.equals(request.state, ignoreCase = true) }) {
22
            throw AccessDeniedException("Not permitted to search outside $states")
23
        }
24
        return SearchScope(states)
25
    }
26
}

In UserQueryBuilder, thread the scope through build, query, filters, and baseFilters, and apply it as the first base filter:

1
    /** Constraints that are never faceted, including the caller's scope (chapter 16). */
2
    fun baseFilters(request: UserSearchRequest, scope: SearchScope = SearchScope.UNRESTRICTED): List<Query> = buildList {
3
        if (scope.states.isNotEmpty()) add(terms("state", scope.states))
4
        add(term("accountStatus", request.accountStatus.name))

1
    private fun terms(field: String, values: Collection<String>): Query =
2
        Query.of { q -> q.terms { t -> t.field(field).terms { tv -> tv.value(values.map { FieldValue.of(it) }) } } }

build gains a scope: SearchScope = SearchScope.UNRESTRICTED parameter after request and passes it to query(request, scope), which passes it to filters(request, scope). UserFacetsBuilder.build gains the same parameter and passes it to UserQueryBuilder.baseFilters(request, scope), so the scope is in the facets’ shared base query, not in any facet’s own filter. UserSearchService.search and facets accept the scope and hand it to the builders.

Finally, the controller derives the scope from the caller and moves identifier lookups to a POST body:

1
package `in`.o612.eng.usersearch.api.web
2

3
import `in`.o612.eng.usersearch.api.search.UserSearchService
4
import `in`.o612.eng.usersearch.api.security.AccessScope
5
import jakarta.validation.Valid
6
import jakarta.validation.constraints.Pattern
7
import org.springframework.http.ResponseEntity
8
import org.springframework.security.core.Authentication
9
import org.springframework.validation.annotation.Validated
10
import org.springframework.web.bind.annotation.GetMapping
11
import org.springframework.web.bind.annotation.PathVariable
12
import org.springframework.web.bind.annotation.PostMapping
13
import org.springframework.web.bind.annotation.RequestBody
14
import org.springframework.web.bind.annotation.RequestMapping
15
import org.springframework.web.bind.annotation.RestController
16

17
@RestController
18
@RequestMapping("/api/users")
19
@Validated
20
class UserSearchController(private val service: UserSearchService) {
21

22
    @GetMapping("/search")
23
    fun search(@Valid request: UserSearchRequest, caller: Authentication): UserSearchResponse =
24
        service.search(request, AccessScope.of(caller, request))
25

26
    @GetMapping("/facets")
27
    fun facets(@Valid request: UserSearchRequest, caller: Authentication): FacetsResponse =
28
        service.facets(request, AccessScope.of(caller, request))
29

30
    @GetMapping("/{userId}")
31
    fun byId(@PathVariable @Pattern(regexp = "\\d{1,19}") userId: String): ResponseEntity<ProfileDetail> =
32
        ResponseEntity.ofNullable(service.findById(userId))
33

34
    /** Lookup by a direct identifier. POST, so the email or mobile number never appears in a URL or an access log. */
35
    @PostMapping("/lookup")
36
    fun lookup(@Valid @RequestBody request: LookupRequest): ResponseEntity<ProfileDetail> =
37
        ResponseEntity.ofNullable(
38
            request.email?.let { service.findByEmail(it) } ?: request.mobileNumber?.let { service.findByMobile(it) },
39
        )
40
}

Add the lookup request to SearchDtos.kt, with AssertTrue, Email, and Pattern imported from jakarta.validation.constraints:

1
/** Exactly one direct identifier. Sent in a POST body, never in a URL (chapter 16). */
2
data class LookupRequest(
3
    @field:Email val email: String? = null,
4
    @field:Pattern(regexp = "\\d{10}") val mobileNumber: String? = null,
5
) {
6
    @AssertTrue(message = "Provide exactly one of email or mobileNumber")
7
    fun isExactlyOneIdentifier(): Boolean = (email == null) != (mobileNumber == null)
8
}

and map AccessDeniedException to 403 in ErrorHandling, importing it from org.springframework.security.access:

1
    @ExceptionHandler(AccessDeniedException::class)
2
    fun accessDenied(e: AccessDeniedException): ProblemDetail =
3
        ProblemDetail.forStatusAndDetail(HttpStatus.FORBIDDEN, e.message ?: "Access denied")

Stage 4 — Verify the rules

Start search-api with the Search API’s key and two lab passwords of your choice:

1
export ELASTICSEARCH_API_KEY=<encoded search-api key>
2
export LAB_AGENT_PASSWORD=<choose one> LAB_SUPERVISOR_PASSWORD=<choose another>
3
./gradlew :search-api:bootRun

The agent’s facets now show Bihar alone, and a supervisor’s facets are unchanged.

Checkpoint

An agent’s facets contain only the states in their scope, and no request parameter changes that. If an agent’s state facet lists other states, the scope has been added as a facet filter instead of a base filter.

What you built, and what comes next

The lab has a snapshot repository, a tested restore, and a snapshot lifecycle policy with retention. Both applications authenticate with API keys scoped to their jobs, and the superuser is out of the application path. The Search API authenticates its callers, restricts contact details to supervisors, takes identifiers only in request bodies, and enforces each caller’s scope in every query and facet, after a facet leak showed why the scope cannot be an ordinary filter.

The lab users are in-memory, and there is no audit trail of searches yet. Production needs your identity provider and a per-request audit log in the Search API.

Chapter 17 collects the failure modes from the whole series into one reference, and builds the test suite that guards against them: unit tests for the query builders, including the scope, and Testcontainers integration tests for mappings, autocomplete, pagination, and the alias swap.