Search & AI Engineer at KMW Technology
The Problem
It is a generally accepted best practice to upgrade software to newer versions to take advantage of things like bug fixes, security patches, new features and performance improvements. On occasion, those newer features have unexpected (and sometimes unwelcome) impacts. When upgrades don’t go smoothly, we have to quickly shift into investigation mode to figure out the root cause and how to fix it.
After a customer upgraded an Elasticsearch cluster from 8.14.3 → 8.17.8, they observed clear signs of query performance degradation during testing. Because their production queries are especially latency-sensitive, even modest regressions posed real risk.
Our concern was that changes introduced in newer versions of Elasticsearch, such as index compatibility adjustments, might end up negatively affecting query throughout, error rate, and reliability. Before making any decisions to adopt Elasticsearch versions beyond 8.14, we needed to validate whether these risks were real and, if so, under what conditions they occurred.
Validation methodology
We measured query performance across Elasticsearch 8.14.3 → 8.17.8 → 8.18.6 on two index shapes:
- Non-nested index: documents contain only top-level fields, no nested arrays/objects.
- Heavily-nested index: documents contain the same top-level fields plus a nested array containing a variable number of objects (1-100 per document), each carrying three short text fields populated by string generators.
Controls
All runs were executed under consistent clusters and workload conditions (hardware, JVM, node count, data volume, and JMeter settings held constant). The only variables were the Elasticsearch version and index structure.
Data Generation (Lucille)
Document bodies were generated with a sequence connector pipeline using Lucille, an open-source ETL framework purpose-built for search engines like Elasticsearch. For nested data we implemented a new stage, AddRandomNestedField, which:
- Writes a JSON array at a target field with either a fixed size or random range per document.
- Supports mapping of nested destinations to either an existing document field or a previously defined Lucille random generation stage.
- Guarantees valid dotted paths and a consistent structure among documents.
We didn’t use real customer data due to security and privacy constraints, which made large-scale testing with production data unrealistic. We recommend validating upgrades with your own data when policy allows. When that’s not possible, Lucille’s generators mirror schema and distribution characteristics closely enough to produce reliable upgrade metrics.
Workload (Queries)
The testing was driven by Apache JMeter using a precomputed CSV of 100,000 queries. To ensure apples-to-apples comparisons, the exact same query set was reused across all runs. Thread groups were tuned to push the cluster to saturation without extreme overload, and those settings were held constant across Elasticsearch versions.
Evaluation
We tracked three primary metrics:
- Latency: the end-to-end response time per request.
- Error rate: the fraction of requests that failed, including timeouts.
- Effective QPS: the realized successful throughput, computed by reducing the total queries per second by the fraction of requests that failed.
Results
Non-Nested Index
For the index with only top-level fields, upgrading from 8.14.3 → 8.17.8 showed no performance regression. Latency remained steady, and at higher load thresholds, error rates actually improved compared to 8.14.3.
Across target rates, average latency changed by only -5.9 % → +13.7 %, while error rate fell by ≈ 40 – 45 % and effective QPS improved by up to 15 %.
Reindexing was also uneventful for this shape. Using the Elasticsearch reindex API, indices created in 8.14.3 reindexed successfully after upgrading to 8.17.8, with no issues encountered.
Heavily-Nested Index
For the index with heavily-nested fields, the results were drastically different. Upgrading from 8.14.3 → 8.17.8 produced:
- Higher error rates: a sharp increase in timeouts and failed responses under load.
- Lower effective QPS: even when raw throughput held steady, the rising error rate pulled down realized throughput.
- Latency inflation: average and high-percentile response times increased significantly compared to 8.14.3.
Average latency ballooned by +55 – 950 %, error rates surged by +30 – 530 %, and effective QPS dropped 8 – 12 % across all load levels.
The Culprit
This regression aligned with the introduction of Zstd compression in Elasticsearch 8.16. Nested indices created in 8.14.3 and upgraded to 8.17.8 failed to reindex with Zstd-related errors, while fresh indices created directly on 8.17.8 did not, reinforcing that the issue was tied to the upgrade path rather than nested data alone.
We couldn’t find a public issue or release note confirming this, so we tested 8.18.6 to see if it had been quietly resolved. In 8.18.6, reindexing nested indices succeeded, and although pre-reindex query performance still trailed the 8.14.3 baseline, reindexing to the default compression restored healthy latency and error rates.
In 8.18.6 before reindexing, latency stayed higher than 8.14.3 by ≈ 30 – 130 % at moderate loads and high-load errors appeared at ≈ 12 – 21 %. After a default reindex in 8.18.6, errors fell to ≈ 0 % across 20 – 100 qps, average latency dropped ≈ 25 – 50 % below the 8.14.3 baseline at 20 – 100 qps, and effective QPS recovered to baseline or slightly above by ≈ 10 – 15%.
Conclusion & Recommendations
Across our tests, upgrading Elasticsearch behaved acceptably for non-nested indices but produced clear regressions for heavily-nested indices when moving to 8.17.8 (elevated error rates, lower effective QPS, and inflated latency). Those issues were no longer present after moving to 8.18.6 and reindexing.
The regression was driven by Zstd compression changes in newer versions, where upgraded indices inherit older compression metadata and pay a decompression penalty at query time. A default reindex fixes this and restores healthy latency and error rates, and is also the most stable option, but it uses roughly 65% more storage than the best option. Choose best only when that space reduction is worth the trade-off in latency and variance.
Uniform Recommendation
- Test performance before upgrading. Run representative queries and verify latency, errors, and QPS.
- Skip Elasticsearch 8.17.8 for all clusters and upgrade directly from 8.14.3 → 8.18.6+.
- After upgrading, reindex (especially nested indices) to ensure compression/format settings are normalized and to restore healthy latency and error profiles.
If You’re Already on Elasticsearch 8.17.8
- Move to 8.18.6+ and reindex affected indices.
- Monitor latency, error rate, and effective QPS under load to confirm recovery.
Lastly, the most important recommendation is to always contact us at KMW if you need help diagnosing your search performance issues! If you’d like to use Lucille to generate test documents, check out this example and let us know what you think.
