The Challenge
Hitachi was operating a large-scale Elastic environment processing approximately 1 TB of observability data every day, with 30 days of retention. The platform ran on Azure Kubernetes Service and consisted of roughly 100 nodes distributed across hot, warm and cold tiers.
Over time, the environment had grown to support several very different workloads through the same architecture: operational and application logs, cybersecurity data used by SIEM, and application performance monitoring.
At that scale, cost was no longer determined by the Elastic subscription alone. Compute, storage, replication, shard movement and the operational overhead of a large distributed cluster all contributed to the total cost of the platform.
But replacing Elastic entirely would have created a different problem. Some workloads were deeply integrated with the Elastic ecosystem: cybersecurity teams relied on the existing SIEM capabilities, and APM depended on the Fleet and Elastic Agent architecture already in place.
The question was therefore not how to migrate away from Elastic.
It was: which workloads genuinely required Elastic, and which could run more efficiently elsewhere?
Segmenting Workloads Instead of Replacing the Platform
The first step was to stop treating the observability environment as a single workload. Each data flow was assessed against its operational requirements, retention, search patterns, integrations and dependency on Elastic-specific capabilities.
The analysis showed a clear separation.
Cybersecurity logs and APM benefited directly from capabilities already deployed within Elastic. Moving them would have meant rebuilding integrations and operational processes for little benefit.
General logging workloads were different. They represented approximately 60% of the platform's data volume, but required none of the SIEM, APM, Fleet or Elastic Agent capabilities.
Those workloads became the candidates for OpenSearch. Rather than a full migration, Hitachi moved toward a hybrid Elastic and OpenSearch architecture, with each platform supporting the workloads for which it provided the most value.
Moving Selected Logging Workloads to OpenSearch
Approximately 60% of the logging volume was migrated to Amazon OpenSearch Service — a pragmatic choice within the organization's multi-cloud footprint. At an overall ingestion rate of roughly 1 TB per day, this removed a substantial share of data from the original Elastic environment.
The migration was not treated as a lift-and-shift exercise. Copying an inefficient architecture into OpenSearch would only have transferred the same cost and reliability problems to another platform.
The architecture was therefore reviewed as part of the move. Shard strategy was reconsidered around the actual workload rather than historical index patterns. Retention and storage policies were aligned with how frequently data actually needed to be queried. Mapping and indexing behavior were reviewed to avoid unnecessary resource consumption.
These decisions are interconnected: reducing platform cost while leaving an inefficient shard strategy unchanged does not solve the underlying problem, and cheaper storage provides limited value if indexing patterns continue to consume resources unnecessarily.
The objective was to give the OpenSearch environment a narrow, predictable responsibility — efficiently ingest, retain and search the logging workloads that did not need additional Elastic capabilities — and to allow the remaining Elastic environment to become significantly more focused.
Keeping Elastic Where It Still Created Value
The project was deliberately not an Elastic replacement.
Cybersecurity and SIEM workloads remained on Elastic, preserving the existing security capabilities and operational processes. APM remained on Elastic as well, together with the existing Fleet and Elastic Agent integrations.
This avoided a common migration mistake: forcing every workload onto the new platform simply to achieve technology standardization. For Hitachi, the hybrid architecture was economically more rational. OpenSearch handled high-volume logging where Elastic-specific capabilities were not required; Elastic continued to support the workloads where its integrated capabilities justified the platform.
The decision was based on workload economics, not vendor preference.
The Result: An Estimated 60% Reduction in Elastic Costs
Moving the majority of general logging volume away from the original environment substantially reduced the capacity required on Elastic. The engagement resulted in an estimated 60% reduction in Elastic costs, while Hitachi continued to use Elastic for its cybersecurity and APM workloads.
The benefit was not only financial. With the logging workload running separately, both platforms could be sized and operated around clearer requirements instead of competing for resources within the same architecture. Operational stability improved across both environments: the Elastic platform became smaller and more focused, and the OpenSearch platform could be tuned specifically for large-scale log ingestion and search.
The organization also avoided the risk and complexity of a full-platform migration.
The Outcome
- ~1 TB/day of observability data processed, 30-day retention
- ~60% of log volume moved to Amazon OpenSearch Service
- SIEM and APM retained on Elastic, preserving Fleet and Elastic Agent integrations
- ~60% estimated reduction in Elastic costs
- Improved operational stability across both platforms
The project demonstrated that observability cost optimization does not always require choosing between Elasticsearch and OpenSearch. At enterprise scale, the better question is:
Which platform should run each workload?