The Challenge
A large process intelligence platform can place very different demands on Elasticsearch or OpenSearch. Some workloads continuously ingest large datasets; others require interactive searches with low latency; historical data must remain searchable but is accessed infrequently.
As the platform grows, the simplest response is often to add nodes — but distributed search systems do not scale efficiently when underlying data design and workload placement remain unchanged. Shard counts grow, cluster state becomes heavier, and a single workload can consume resources required by another.
The question becomes: does the platform need more capacity, or does the workload need a different architecture?
Profiling Workloads Before Resizing the Cluster
The first step is to stop treating the cluster as one homogeneous system. Ingestion-heavy workloads, interactive queries, historical searches and background processing are analyzed separately for data growth, query behavior, latency requirements, retention, shard distribution and mapping design.
This often exposes a mismatch between historical cluster configuration and current usage — index patterns and shard strategies that reflect old assumptions, or mappings that index fields rarely searched.
Redesigning Shards and Index Lifecycles
Shard strategy is one of the highest-leverage decisions in a large Elasticsearch or OpenSearch environment. Rather than selecting a universal shard target, rollover and shard allocation are designed around actual workload behavior and operational constraints.
Lifecycle policies are reviewed at the same time: frequently queried data remains on infrastructure optimized for active search, while older data moves to storage and compute profiles appropriate for less frequent access. Retention becomes an architectural decision rather than a default setting.
Controlling Mapping Growth and Isolating Workloads
Mappings are reviewed to determine whether every field needs to be indexed and whether field types reflect how the data is actually queried, with templates providing consistency across recurring indices.
Workload isolation can prevent one class of activity from destabilizing another. Depending on requirements, this may mean separate clusters, dedicated node roles, routing strategies or a hybrid Elasticsearch/OpenSearch architecture — the technology choice comes after the workload analysis, not before it.
The Outcome
The target architecture is designed to make scale predictable rather than simply larger:
- Shard strategy aligned with real ingestion patterns
- Lifecycle and retention policies tied to access requirements
- Mapping growth placed under control
- Interactive and background workloads isolated where appropriate
- Capacity models based on workload demand
- Lower recovery and rebalancing pressure
- Clearer decisions about where Elasticsearch or OpenSearch provides the best fit
When a search platform becomes expensive to scale, adding nodes should be the conclusion of the analysis — not the starting point.