Elasticsearch Limits and Recommended Configurations

Posted on Views: Word count in article: 3.8k Reading time ≈ 3 mins.
Elasticsearch limits and recommended settings: array and document size caps, a reference index template, shard balancing, delayed allocation, and concurrent recoveries.

A collection of Elasticsearch usage limits and recommended configurations, covering best practices for key parameters such as array size, document limits, and shard counts.

Limits

Array fields: no limit on array size

There is no hard limit but it’s definitely recommended to keep those arrays “reasonable”. When performing an update, Elasticsearch needs to fetch the entire doc, apply the update, then index the updated document and replicate the entire updated document to the replica, so very large arrays would come with a performance penalty indeed.

Document size limit: 2GB

Given that the default http.max_content_length is set to 100MB, Elasticsearch will refuse to index any document that is larger than that. You might decide to increase that particular setting, but Lucene still has a limit of about 2GB.

https://stackoverflow.com/questions/28841221/what-is-the-maximum-elasticsearch-document-size/28841656#28841656

https://www.elastic.co/guide/en/elasticsearch/reference/current/general-recommendations.html

Single-field size limit

No information found; my working assumption is that as long as the document as a whole stays within its limit, this is fine.

Reference index template

Since ES 5.x, index-level settings have to be written into a template, or specified when the index is created. We put the settings shared by all of our indices into one template. This template matches every index and carries the lowest order, so user-defined templates get higher priority and can override the settings in it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
{
  "template": "*",
  "order": 0,
  "settings": {
    "index.merge.policy.max_merged_segment": "2gb",
    "index.merge.policy.segments_per_tier": "24",
    "index.number_of_replicas": "1",
    "index.number_of_shards": "24",
    "index.optimize_auto_generated_id": "true",
    "index.refresh_interval": "120s",
    "index.translog.durability": "async",
    "index.translog.flush_threshold_size": "1000mb",
    "index.translog.sync_interval": "120s",
    "index.unassigned.node_left.delayed_timeout": "5d"
  }
}

Avoiding uneven shards on hot indices

By default, ES’s shard balancing strategy tries to keep the shard count roughly equal across nodes. But when the cluster scales out, the newly added nodes hold no shards, so the shards of newly created indices pile up on the new nodes. That gives a new node too much hot data, and it may face enormous write pressure. So for all the shards of an index, we need to cap how many shards of that index a single node may store, so that the index’s shards spread more evenly across nodes.

For example, in a 10-node cluster where an index has 5 primary shards and 1 replica, each node should hold (5×2)/10=1 shard on average; allowing for node failures and shard migrations, you can set the total shards per node to 2:

1
2
3
4
5
6
7
curl --location --request PUT 'http://127.0.0.1:9200/myindex/_settings' \
--header 'Content-Type: application/json' \
--data '{
  "index": {
    "routing.allocation.total_shards_per_node": "2"
  }
}'

Delayed shard allocation (delaying allocation when a node leaves)

When a node leaves the cluster for any reason — human error or a system failure — the master node reacts in the following steps (labeled “step X” below to make the rest of this easier to read):

  • Step 1: promote replica shards to primary to replace any primary shards on that node.
  • Step 2: allocate replicas to replace the lost replicas (given enough nodes).
  • Step 3: rebalance shards evenly across the remaining nodes.

The benefit of all this: it prevents data loss and keeps the cluster highly available.

But the side effects are equally obvious. First, it puts extra load on the cluster — shard allocation is very resource-hungry. Second, if the node that left comes back quickly, the necessity of the whole mechanism is debatable.

This is where delayed shard allocation earns its keep. Configure it like this:

1
2
3
4
5
6
PUT _all/_settings
{
  "settings": {
    "index.unassigned.node_left.delayed_timeout": "5m"
  }
}

The essence of delayed shard allocation:

When a node leaves the cluster and we know (we set the window ourselves) that it will come back online quickly, only step 1 runs while it is gone: the replica shards that were on the departing node get promoted to primary. The cluster is then at least yellow rather than red. Steps 2 and 3 do not happen; the cluster stays available, and once the node rejoins within the configured delay, the promoted shards turn back into replicas, returning the cluster to green.

This process effectively avoids the shard allocations of steps 2 and 3, ensuring the cluster’s high availability in the shortest possible time overall.

https://www.elastic.co/guide/en/elasticsearch/reference/current/delayed-allocation.html

Number of concurrent shard recoveries per node

1
2
3
4
5
6
PUT _cluster/settings
{
  "transient": {
    "cluster.routing.allocation.node_concurrent_recoveries": 3
  }
}

The setting above makes node_concurrent_incoming_recoveries and node_concurrent_outgoing_recoveries take effect at the same time.

incoming_recoveries: usually replica shards on other nodes recovering onto this node. outgoing_recoveries: usually primary shards on this node recovering their replicas onto other nodes.

https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html

This article is the English edition of a post originally published in Chinese.