The REST API surface, from documents to the cluster

Every client library, including the Java API Client in chapter 09, is a typed wrapper around the Elasticsearch REST API. This chapter works through the parts of that API a search projection depends on: writing and reading single documents, optimistic concurrency, external versioning, bulk requests and their partial failures, the by-query APIs, background tasks, and the inspection APIs you reach for when something looks wrong. It finishes with a short look at ES|QL.

Two findings in this chapter shape the synchronisation design in chapter 04. External versioning rejects stale events only while a delete’s tombstone is remembered. And _update_by_query silently changes external versions, so the next legitimate event is rejected. You reproduce both.

You need the chapter 01 lab running. Requests use Kibana Dev Tools syntax, as in chapter 02; every one also works with curl. The chapter takes about 40 minutes and uses a scratch index that you delete at the end.

Conventions shared by every endpoint

A few rules apply across the whole API, and knowing them saves time in every later chapter.

• Paths name the target, then the action. ch03-profiles/_doc/1 is a document; ch03-profiles/_search, _count, and _bulk are actions on an index; _cluster/..., _cat/..., and _tasks act on the cluster.
• Status codes carry meaning. 200 and 201 succeed. 400 is a request the cluster will never accept, such as a mapping violation. 404 is a missing index or document. 409 is a version conflict. 429 means a thread pool queue is full and the request was rejected, so retry later with backoff. 5xx means the cluster failed.
• Errors share one envelope. Every error has an error object with a type, a reason, and usually a root_cause array, plus a top-level status. Error responses in this chapter are trimmed to type, reason, and status.
• filter_path trims responses. It is one of the common options every endpoint accepts. The responses shown here use it heavily. It does not apply to error bodies.

Create the scratch index

Chapter 02 showed what dynamic mapping guesses. This time, declare a small mapping and make it strict, so that unknown fields are rejected. Chapter 05 designs the real profile mapping; this one only has to be predictable.

1
PUT ch03-profiles
2
{
3
  "settings": { "number_of_shards": 1, "number_of_replicas": 0 },
4
  "mappings": {
5
    "dynamic": "strict",
6
    "properties": {
7
      "userId":        { "type": "long" },
8
      "fullName":      { "type": "text" },
9
      "city":          { "type": "keyword" },
10
      "accountStatus": { "type": "keyword" },
11
      "updatedAt":     { "type": "date" }
12
    }
13
  }
14
}

1
{"acknowledged":true,"shards_acknowledged":true,"index":"ch03-profiles"}

The index has a single shard, so the sequence numbers in the rest of the chapter match yours exactly.

Write a document three ways: create, index, update

What it is. Three endpoints write a single document, and they differ in what they do when the document already exists:

Why it matters at 300 million profiles. A projection is fed by events that can arrive twice. The endpoint you choose decides whether a duplicate is harmless (_doc replaces with the same content), an error you must handle (_create), or a merge that depends on what was there before (_update).

Example. Create profile 1, then try to create it again:

1
PUT ch03-profiles/_create/1
2
{"userId":1,"fullName":"Prashant Kumar","city":"Patna","accountStatus":"ACTIVE","updatedAt":"2026-09-01T10:15:00Z"}

1
{"_index":"ch03-profiles","_id":"1","_version":1,"result":"created",
2
 "_shards":{"total":1,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}

1
version_conflict_engine_exception: [1]: version conflict, document already exists (current version [1])
2
status: 409

Replace it with the index endpoint, and add profile 2:

1
PUT ch03-profiles/_doc/1?filter_path=result,_version,_seq_no
2
{"userId":1,"fullName":"Prashant Kumar","city":"Gaya","accountStatus":"ACTIVE","updatedAt":"2026-09-02T08:00:00Z"}

1
{"_version":2,"result":"updated","_seq_no":1}

1
PUT ch03-profiles/_doc/2?filter_path=result,_version,_seq_no
2
{"userId":2,"fullName":"Priya Sharma","city":"Mumbai","accountStatus":"ACTIVE","updatedAt":"2026-08-11T09:00:00Z"}

1
{"_version":1,"result":"created","_seq_no":2}

Now change one field with the update API, and then send the identical update again:

1
POST ch03-profiles/_update/1?filter_path=result,_version,_seq_no
2
{ "doc": { "accountStatus": "SUSPENDED" } }

1
{"_version":3,"result":"updated","_seq_no":3}

1
{"_version":3,"result":"noop","_seq_no":3}

The second call returned noop: the merged document was identical to the stored one, so nothing was written and the version did not move. Noop detection is on by default for partial updates. A full PUT _doc has no such check and always writes a new version.

An update can also create a missing document when you set doc_as_upsert:

1
POST ch03-profiles/_update/3?filter_path=result,_version
2
{
3
  "doc": {"userId":3,"fullName":"José Fernandes","city":"Kochi","accountStatus":"ACTIVE","updatedAt":"2026-07-02T12:30:00Z"},
4
  "doc_as_upsert": true
5
}

1
{"_version":1,"result":"created"}

Common mistake. Using _update with partial documents to apply change events. If an event is lost or reordered, a partial update merges onto the wrong base and the document silently drifts from PostgreSQL. A projection is simplest when every write is a complete document built from the source row, sent with PUT _doc.

Read documents: get, exists, and multi-get

What it is. GET index/_doc/<id> returns one document. It is real time: it sees the latest write even before a refresh, as chapter 02 showed. HEAD checks existence without a body. _mget fetches many IDs in one round trip.

Why it matters at 300 million profiles. Exact lookup by userId (Q1 in chapter 01) never needs a search. A get goes to one shard, chosen by routing, instead of fanning out to all of them.

Example. Fetch profile 1 with only two fields of its source:

1
GET ch03-profiles/_doc/1?_source_includes=fullName,city

1
{"_index":"ch03-profiles","_id":"1","_version":3,"_seq_no":3,"_primary_term":1,"found":true,
2
 "_source":{"fullName":"Prashant Kumar","city":"Gaya"}}

HEAD ch03-profiles/_doc/1 returns status 200; HEAD ch03-profiles/_doc/99 returns 404. Fetch several at once:

1
GET ch03-profiles/_mget?filter_path=docs._id,docs.found,docs._source.city
2
{ "ids": ["1", "2", "99"] }

1
{"docs":[{"_id":"1","found":true,"_source":{"city":"Gaya"}},
2
         {"_id":"2","found":true,"_source":{"city":"Mumbai"}},
3
         {"_id":"99","found":false}]}

Common mistake. Returning _source unfiltered to API callers. For user profiles, _source contains every indexed field, including personal data the caller may not be allowed to see. Always name the fields you return: _source_includes here, and source filtering on searches in chapter 11.

Optimistic concurrency with sequence numbers

What it is. Every write to a shard gets a sequence number, _seq_no, and the shard’s current _primary_term. Pass both back as if_seq_no and if_primary_term, and the write succeeds only if the document has not changed since you read it. The optimistic concurrency guide describes the mechanism. It is the Elasticsearch equivalent of UPDATE ... WHERE version = :read_version in PostgreSQL.

Why it matters at 300 million profiles. Any read-modify-write against the index needs it, such as an admin tool that edits a document directly, or a job that recalculates a field. For the projection itself, external versioning in the next section is usually the better fit, because the version comes from PostgreSQL rather than from Elasticsearch.

Example. Read the current sequence number of profile 2, then write with it:

1
GET ch03-profiles/_doc/2?filter_path=_seq_no,_primary_term

1
{"_seq_no":2,"_primary_term":1}

1
PUT ch03-profiles/_doc/2?if_seq_no=2&if_primary_term=1&filter_path=result,_seq_no
2
{"userId":2,"fullName":"Priya Sharma","city":"Pune","accountStatus":"ACTIVE","updatedAt":"2026-09-03T09:00:00Z"}

1
{"result":"updated","_seq_no":5}

A second writer that read the same sequence number now loses:

1
PUT ch03-profiles/_doc/2?if_seq_no=2&if_primary_term=1
2
{"userId":2,"fullName":"Priya Sharma","city":"Nagpur","accountStatus":"ACTIVE","updatedAt":"2026-09-03T09:05:00Z"}

1
version_conflict_engine_exception: [2]: version conflict, required seqNo [2], primary term [1].
2
current document has seqNo [5] and primary term [1]
3
status: 409

The sequence number jumped from 2 to 5 because it counts every operation on the shard, not only those on this document.

Common mistake. Retrying a 409 by blindly re-sending the same write. A conflict means your view of the document is stale: read it again, reapply the change, and write with the new sequence number, or give up.

External versioning: letting PostgreSQL decide what is newer

What it is. With version_type=external, you supply the version number yourself, and Elasticsearch accepts a write only if its version is greater than the stored one. The version usually comes from the source system: a row version column, or a change-log sequence number.

Why it matters at 300 million profiles. Change events for the same profile can arrive twice or out of order: a retry, a replay after an outage, a consumer restart. With external versions, a stale event is rejected by Elasticsearch itself, so the indexer does not need to read before it writes. This is the idempotency mechanism chapter 04 builds on.

Example. In this example the version is a row version from PostgreSQL that increases by one on every change to that row. Index version 5 of profile 10:

1
PUT ch03-profiles/_doc/10?version=5&version_type=external&filter_path=result,_version
2
{"userId":10,"fullName":"Rahul Verma","city":"Lucknow","accountStatus":"ACTIVE","updatedAt":"2026-09-05T00:00:00Z"}

1
{"_version":5,"result":"created"}

A late event carrying version 4 is rejected, and so is a duplicate of version 5:

1
version_conflict_engine_exception: [10]: version conflict, current version [5] is higher or equal to the one provided [4]
2
status: 409

1
version_conflict_engine_exception: [10]: version conflict, current version [5] is higher or equal to the one provided [5]
2
status: 409

For an indexer, both 409s mean the same thing: Elasticsearch already has this change or a newer one. Treat them as success, not as errors.

Deletes take a version too. Delete profile 10 at version 7, then deliver a stale update at version 6:

1
DELETE ch03-profiles/_doc/10?version=7&version_type=external&filter_path=result,_version

1
{"_version":7,"result":"deleted"}

1
version_conflict_engine_exception: [10]: version conflict, current version [7] is higher or equal to the one provided [6]
2
status: 409

The deleted profile stayed deleted, because Elasticsearch remembers the version of a deleted document for a while. That memory is a tombstone, and the next experiment shows how long it lasts.

The tombstone expires

The retention period is the index.gc_deletes setting, which defaults to 60 seconds. Shorten it to one second so you can watch it expire:

1
PUT ch03-profiles/_settings
2
{ "index": { "gc_deletes": "1s" } }

Index profile 11 at version 1, delete it at version 3, wait a few seconds, refresh, and then deliver a stale event at version 2:

1
PUT ch03-profiles/_doc/11?version=1&version_type=external
2
{"userId":11,"fullName":"Meera Iyer","city":"Chennai","accountStatus":"ACTIVE","updatedAt":"2026-09-01T00:00:00Z"}
3

4
DELETE ch03-profiles/_doc/11?version=3&version_type=external
5

6
POST ch03-profiles/_refresh
7

8
PUT ch03-profiles/_doc/11?version=2&version_type=external&filter_path=result,_version
9
{"userId":11,"fullName":"Meera Iyer","city":"Chennai","accountStatus":"ACTIVE","updatedAt":"2026-09-02T00:00:00Z"}

1
{"_version":2,"result":"created"}

The profile is back. Version 2 is older than the delete at version 3, but once the tombstone was gone, Elasticsearch had nothing to compare it against. Restore the default before you continue:

1
PUT ch03-profiles/_settings
2
{ "index": { "gc_deletes": null } }

Principle. External versioning protects against stale updates indefinitely, but against stale updates after a delete only for gc_deletes. Any path that can deliver an old event more than 60 seconds after a delete, such as a replay from the start of a topic or a backfill racing a delete, can resurrect a deleted profile. For user profiles, that is a privacy defect, not a cosmetic one. Chapter 04 designs around it.

Common mistake. Using updatedAt in milliseconds as the external version. Two updates in the same millisecond get the same version and the second is dropped, and clock differences between application servers can make a newer change look older. Use a counter that the database increments.

Bulk requests and partial failure

What it is. The _bulk API sends many index, create, update, and delete operations in one request. The body is newline-delimited JSON: an action line, followed by a document line for everything except delete. Each operation succeeds or fails on its own.

Why it matters at 300 million profiles. Single-document requests cannot backfill 300 million rows in reasonable time. Every serious ingestion path uses bulk, and every bulk client must read the response per item, because a bulk request can return HTTP 200 while some of its operations failed.

Example. Send four operations. Two are designed to fail: one document carries a field that the strict mapping does not allow, and one tries to create a document that already exists.

1
POST ch03-profiles/_bulk?filter_path=errors,items.*._id,items.*.status,items.*.result,items.*.error.type,items.*.error.reason
2
{"index":{"_id":"20","version":1,"version_type":"external"}}
3
{"userId":20,"fullName":"Anjali Nair","city":"Kochi","accountStatus":"ACTIVE","updatedAt":"2026-09-10T10:00:00Z"}
4
{"index":{"_id":"21","version":1,"version_type":"external"}}
5
{"userId":21,"fullName":"Amit Das","city":"Kolkata","accountStatus":"ACTIVE","nickname":"Amu","updatedAt":"2026-09-10T10:00:00Z"}
6
{"create":{"_id":"3"}}
7
{"userId":3,"fullName":"José Fernandes","city":"Kochi","accountStatus":"ACTIVE","updatedAt":"2026-07-02T12:30:00Z"}
8
{"delete":{"_id":"2"}}

1
{
2
  "errors": true,
3
  "items": [
4
    {"index":  {"_id":"20","result":"created","status":201}},
5
    {"index":  {"_id":"21","status":400,"error":{"type":"strict_dynamic_mapping_exception",
6
                "reason":"[1:89] mapping set to strict, dynamic introduction of [nickname] within [_doc] is not allowed"}}},
7
    {"create": {"_id":"3","status":409,"error":{"type":"version_conflict_engine_exception",
8
                "reason":"[3]: version conflict, document already exists (current version [1])"}}},
9
    {"delete": {"_id":"2","result":"deleted","status":200}}
10
  ]
11
}

The HTTP status of this response is 200. The only request-level hint is "errors": true. The items come back in the same order as the operations, and each has its own status. They fall into three classes, and a bulk client needs a rule for each:

Chapter 14 implements exactly this table in Kotlin, on top of the Java client’s bulk helper.

Common mistake. Checking only the HTTP status. The nickname document was never indexed, and nothing at the HTTP level says so.

Count documents

_count runs a query and returns only the number of matches. Refresh first so that recent writes are visible:

1
POST ch03-profiles/_refresh
2

3
GET ch03-profiles/_count?filter_path=count
4
{ "query": { "term": { "accountStatus": "ACTIVE" } } }

1
{"count":3}

The three active profiles are 3, 11, and 20. Profile 1 is suspended, and profiles 2 and 10 are deleted. Chapter 14 uses _count to compare an index against PostgreSQL before an alias swap.

Change documents in place with the by-query APIs

What it is. _update_by_query runs a query and applies a script to every match. _delete_by_query deletes every match. Both take a point-in-time view of the matching documents and process them in batches.

Why it matters at 300 million profiles. They are the tempting shortcut for a data fix: “rename a city everywhere”. At this scale they run for a long time, create many deleted documents and merge work, and, as the next experiment shows, can break the versioning your sync pipeline relies on.

Example. Rename a city with a Painless script:

1
POST ch03-profiles/_update_by_query?conflicts=proceed&refresh=true&filter_path=total,updated,version_conflicts,failures
2
{
3
  "query":  { "term": { "city": "Kochi" } },
4
  "script": { "source": "ctx._source.city = params.to", "params": { "to": "Kochi (Ernakulam)" } }
5
}

1
{"total":2,"updated":2,"version_conflicts":0,"failures":[]}

conflicts=proceed counts version conflicts instead of aborting on the first one. Now look at profile 20, which the bulk request wrote at external version 1:

1
GET ch03-profiles/_doc/20?filter_path=_version,_source.city

1
{"_version":2,"_source":{"city":"Kochi (Ernakulam)"}}

The update-by-query bumped the version to 2. When PostgreSQL’s next real change to profile 20 arrives as external version 2, Elasticsearch rejects it:

1
PUT ch03-profiles/_doc/20?version=2&version_type=external
2
{"userId":20,"fullName":"Anjali Nair","city":"Kochi","accountStatus":"INACTIVE","updatedAt":"2026-09-11T10:00:00Z"}

1
version_conflict_engine_exception: [20]: version conflict, current version [2] is higher or equal to the one provided [2]
2
status: 409

The indexer treats this 409 as “already applied”, so the change to INACTIVE is lost without any error. The projection now disagrees with PostgreSQL and nothing will correct it until the next change to that profile.

Principle. In a projection with external versions, never modify documents inside Elasticsearch. Fix the data in PostgreSQL and let the change flow through the pipeline, or rebuild the index from the source (chapter 14).

Common mistake. Running a large by-query request synchronously from a script or a tool with an HTTP timeout. The request keeps running on the cluster after the client gives up, and you lose the result. Run it as a task instead, which the next section shows.

Long-running work runs as a task

Any request that accepts wait_for_completion=false returns a task ID immediately and runs in the background:

1
POST ch03-profiles/_delete_by_query?conflicts=proceed&wait_for_completion=false
2
{ "query": { "term": { "accountStatus": "SUSPENDED" } } }

1
{"task":"zTj-BSTiRmeOpfWK_7mHaQ:12569"}

Poll it with the tasks API, using the ID from your own response:

1
GET _tasks/zTj-BSTiRmeOpfWK_7mHaQ:12569?filter_path=completed,task.action,response.total,response.deleted,response.failures

1
{"completed":true,"task":{"action":"indices:data/write/delete/byquery"},
2
 "response":{"total":1,"deleted":1,"failures":[]}}

The same pattern applies to _reindex, which chapter 14 uses to copy data between index versions.

Inspect the cluster with the cat and cluster APIs

The _cat APIs return aligned text tables for humans. Every one accepts v for headers and h to choose columns. These are the ones used most often in this series:

Look at the scratch index after this chapter’s writes:

1
GET _cat/indices/ch03-profiles?v&h=health,index,pri,rep,docs.count,docs.deleted

1
health index         pri rep docs.count docs.deleted
2
green  ch03-profiles   1   0          3           13

Three live documents, and thirteen deleted ones still occupying space in this run: every replace, update, and delete in this chapter left one behind until a merge removes it. Your deleted count may be lower if a merge has already run. That ratio is what a high update rate looks like, and chapter 15 discusses when it matters.

When a shard will not allocate, _cat tells you that, and the allocation explain API tells you why. Create an index that wants a replica the single node cannot host, and ask:

1
PUT ch03-yellow
2
{ "settings": { "number_of_shards": 1, "number_of_replicas": 1 } }
3

4
GET _cluster/allocation/explain
5
{ "index": "ch03-yellow", "shard": 0, "primary": false }

1
{
2
  "index": "ch03-yellow", "shard": 0, "primary": false,
3
  "current_state": "unassigned",
4
  "can_allocate": "no",
5
  "node_allocation_decisions": [{
6
    "deciders": [{
7
      "decider": "same_shard",
8
      "explanation": "a copy of this shard is already allocated to this node [[ch03-yellow][0], node[...], [P], s[STARTED], ...]"
9
    }]
10
  }],
11
  ...
12
}

The same_shard decider forbids placing a replica on the node that holds its primary: the rule behind chapter 02’s yellow cluster. In production, the same API explains shards blocked by disk watermarks, allocation filters, or a missing node. Delete the index:

1
DELETE ch03-yellow

Production note — _cat output is for people. Scripts and monitoring should call the JSON APIs, such as _cluster/health, _nodes/stats, and _stats, because _cat column formats are not a stable contract. Chapter 16 builds monitoring on those.

Query with ES|QL

ES|QL is a piped query language that runs on the _query endpoint. It is well suited to ad-hoc analysis and aggregation while you explore data:

1
POST _query?format=txt
2
{ "query": "FROM ch03-profiles | STATS profiles = COUNT(*) BY city | SORT city" }

1
   profiles    |      city
2
---------------+-----------------
3
1              |Chennai
4
2              |Kochi (Ernakulam)

The user-facing search in this series uses the Query DSL through _search, because it needs relevance scoring, highlighting, and cursor pagination, which chapters 11 and 13 build. Treat ES|QL as the tool you use to look at the projection, not the API behind the search box.

Clean up

Warning — DELETE removes the index and all its documents permanently. Everything in ch03-profiles was created in this chapter.

1
DELETE ch03-profiles

What you can do now, and what comes next

You can write, read, and version single documents; send bulk requests and classify every item’s result; run by-query changes as tasks; and inspect indices, shards, and allocation decisions. You have also reproduced the two traps that decide how a projection is kept in sync: delete tombstones that expire after gc_deletes, and by-query changes that break external versioning.

This chapter did not cover search itself. The Query DSL gets its own chapter (11), after the mapping that decides what queries can match (chapters 05 to 07).

Chapter 04 designs the synchronisation pipeline from PostgreSQL to Elasticsearch: why a dual write is unsafe, how a transactional outbox and change data capture compare, and how external versions, per-profile ordering, and delete handling fit together into an idempotent indexer.