{ "name": "brick/math", "description": "Arbitrary-precision arithmetic library", "type": "library", "keywords": [ "Brick", "Math", "Mathematics", "Arbitrary-precision", "Arithmetic", "BigInteger", "BigDecimal", "BigRational", "BigNumber", "Bignum", "Decimal", "Rational", "Integer" ], "license": "MIT", "require": { "php": "^7.2" }, "require-dev": { "phpunit/phpunit": "^10.1", "php-coveralls/php-coveralls": "^2.2", "vimeo/psalm": "5.25.0" }, "autoload": { "psr-4": { "Brick\\Math\\": "src/" } }, "autoload-dev": { "psr-4": { "Brick\\Math\\Tests\\": "tests/" } } } KMW Technology https://kmwllc.com Search Professional Services Thu, 15 Jan 2026 02:42:09 +0000 en-US hourly 1 https://wordpress.org/?v=7.0.3 https://kmwllc.com/wp-content/uploads/2022/12/Black-circles.png KMW Technology https://kmwllc.com 32 32 The Mystery of Elasticsearch 8.17 Query Performance Degradation https://kmwllc.com/index.php/2026/01/10/the-mystery-of-elasticsearch-8-17-query-performance-degradation/?utm_source=rss&utm_medium=rss&utm_campaign=the-mystery-of-elasticsearch-8-17-query-performance-degradation Fri, 09 Jan 2026 22:33:43 +0000 https://kmwllc.com/?p=30279 We benchmarked Elasticsearch 8.14–8.18 as part of an upgrade we helped a customer with. We uncovered a hidden performance regression in nested indices and will share how we discovered this, along with the exact steps needed to fix it.

The post The Mystery of Elasticsearch 8.17 Query Performance Degradation first appeared on KMW Technology.

]]>
We benchmarked Elasticsearch 8.14–8.18 as part of an upgrade we helped a customer with. We uncovered a hidden performance regression in nested indices and will share how we discovered this, along with the exact steps needed to fix it.
Picture of Henry Caldwell
Henry Caldwell

Search & AI Engineer at KMW Technology

The Problem

It is a generally accepted best practice to upgrade software to newer versions to take advantage of things like bug fixes, security patches, new features and performance improvements.  On occasion, those newer features have unexpected (and sometimes unwelcome) impacts.  When upgrades don’t go smoothly, we have to quickly shift into investigation mode to figure out the root cause and how to fix it.

After a customer upgraded an Elasticsearch cluster from 8.14.3 → 8.17.8, they observed clear signs of query performance degradation during testing. Because their production queries are especially latency-sensitive, even modest regressions posed real risk.

Our concern was that changes introduced in newer versions of Elasticsearch, such as index compatibility adjustments, might end up negatively affecting query throughout, error rate, and reliability. Before making any decisions to adopt Elasticsearch versions beyond 8.14, we needed to validate whether these risks were real and, if so, under what conditions they occurred.

Validation methodology

We measured query performance across Elasticsearch 8.14.3 → 8.17.8 → 8.18.6 on two index shapes:

  • Non-nested index: documents contain only top-level fields, no nested arrays/objects.
  • Heavily-nested index: documents contain the same top-level fields plus a nested array containing a variable number of objects (1-100 per document), each carrying three short text fields populated by string generators.
Controls

All runs were executed under consistent clusters and workload conditions (hardware, JVM, node count, data volume, and JMeter settings held constant). The only variables were the Elasticsearch version and index structure.

Data Generation (Lucille)

Document bodies were generated with a sequence connector pipeline using Lucille, an open-source ETL framework purpose-built for search engines like Elasticsearch. For nested data we implemented a new stage, AddRandomNestedField, which:

  • Writes a JSON array at a target field with either a fixed size or random range per document.
  • Supports mapping of nested destinations to either an existing document field or a previously defined Lucille random generation stage.
  • Guarantees valid dotted paths and a consistent structure among documents.

We didn’t use real customer data due to security and privacy constraints, which made large-scale testing with production data unrealistic. We recommend validating upgrades with your own data when policy allows. When that’s not possible, Lucille’s generators mirror schema and distribution characteristics closely enough to produce reliable upgrade metrics. 

Workload (Queries)

The testing was driven by Apache JMeter using a precomputed CSV of 100,000 queries. To ensure apples-to-apples comparisons, the exact same query set was reused across all runs. Thread groups were tuned to push the cluster to saturation without extreme overload, and those settings were held constant across Elasticsearch versions.

Evaluation

We tracked three primary metrics:

  • Latency: the end-to-end response time per request.
  • Error rate: the fraction of requests that failed, including timeouts.
  • Effective QPS: the realized successful throughput, computed by reducing the total queries per second by the fraction of requests that failed.

Results

Non-Nested Index

For the index with only top-level fields, upgrading from 8.14.3 → 8.17.8 showed no performance regression. Latency remained steady, and at higher load thresholds, error rates actually improved compared to 8.14.3.

Across target rates, average latency changed by only -5.9 % → +13.7 %, while error rate fell by ≈ 40 – 45 % and effective QPS improved by up to 15 %.

Reindexing was also uneventful for this shape. Using the Elasticsearch reindex API, indices created in 8.14.3 reindexed successfully after upgrading to 8.17.8, with no issues encountered.

Heavily-Nested Index

For the index with heavily-nested fields, the results were drastically different. Upgrading from 8.14.3 → 8.17.8 produced:

  • Higher error rates: a sharp increase in timeouts and failed responses under load.
  • Lower effective QPS: even when raw throughput held steady, the rising error rate pulled down realized throughput.
  • Latency inflation: average and high-percentile response times increased significantly compared to 8.14.3.

Average latency ballooned by +55 – 950 %, error rates surged by +30 – 530 %, and effective QPS dropped 8 – 12 % across all load levels.

 

The Culprit

This regression aligned with the introduction of Zstd compression in Elasticsearch 8.16. Nested indices created in 8.14.3 and upgraded to 8.17.8 failed to reindex with Zstd-related errors, while fresh indices created directly on 8.17.8 did not, reinforcing that the issue was tied to the upgrade path rather than nested data alone.

We couldn’t find a public issue or release note confirming this, so we tested 8.18.6 to see if it had been quietly resolved. In 8.18.6, reindexing nested indices succeeded, and although pre-reindex query performance still trailed the 8.14.3 baseline, reindexing to the default compression restored healthy latency and error rates.

In 8.18.6 before reindexing, latency stayed higher than 8.14.3 by ≈ 30 – 130 % at moderate loads and high-load errors appeared at ≈ 12 – 21 %. After a default reindex in 8.18.6, errors fell to ≈ 0 % across 20 – 100 qps, average latency dropped ≈ 25 – 50 % below the 8.14.3 baseline at 20 – 100 qps, and effective QPS recovered to baseline or slightly above by ≈ 10 – 15%.

Conclusion & Recommendations

Across our tests, upgrading Elasticsearch behaved acceptably for non-nested indices but produced clear regressions for heavily-nested indices when moving to 8.17.8 (elevated error rates, lower effective QPS, and inflated latency). Those issues were no longer present after moving to 8.18.6 and reindexing.

The regression was driven by Zstd compression changes in newer versions, where upgraded indices inherit older compression metadata and pay a decompression penalty at query time. A default reindex fixes this and restores healthy latency and error rates, and is also the most stable option, but it uses roughly 65% more storage than the best option. Choose best only when that space reduction is worth the trade-off in latency and variance.

Uniform Recommendation
  • Test performance before upgrading. Run representative queries and verify latency, errors, and QPS.
  • Skip Elasticsearch 8.17.8 for all clusters and upgrade directly from 8.14.3 → 8.18.6+.
  • After upgrading, reindex (especially nested indices) to ensure compression/format settings are normalized and to restore healthy latency and error profiles.
If You’re Already on Elasticsearch 8.17.8
  • Move to 8.18.6+ and reindex affected indices.
  • Monitor latency, error rate, and effective QPS under load to confirm recovery.

Lastly, the most important recommendation is to always contact us at KMW if you need help diagnosing your search performance issues! If you’d like to use Lucille to generate test documents, check out this example and let us know what you think.

Share post
More From KMW

The post The Mystery of Elasticsearch 8.17 Query Performance Degradation first appeared on KMW Technology.

]]> 30279 What’s the best way to do entity extraction for search? https://kmwllc.com/index.php/2025/10/04/whats-the-best-way-to-do-entity-extraction-for-search/?utm_source=rss&utm_medium=rss&utm_campaign=whats-the-best-way-to-do-entity-extraction-for-search Sat, 04 Oct 2025 00:52:34 +0000 https://kmwllc.com/?p=30125 Comparing the Effectiveness of Entity Extraction between NLP and LLMs

The post What’s the best way to do entity extraction for search? first appeared on KMW Technology.

]]>
Comparing the Effectiveness of Entity Extraction between NLP and LLMs
Picture of Jacob Squatrito
Jacob Squatrito

Search & AI Engineer at KMW Technology

Entities are really important for search, but what’s the best way to do it?

The ability to analyze a piece of text and identify the key entities within it can have lots of practical search uses like improving search relevancy; enabling faceting and filtering classifying documents and even obfuscation of sensitive data.

As a natural language problem, entity extraction is not new but it’s always been tricky to do well. There are lots of traditional NLP models for entity extraction and recently LLMs have shown promising abilities too. For search applications we are typically trying to balance excellent entity extraction along with operational needs like running fast and not consuming too many resources. So what is the best way to do entity extraction for modern search applications?

In this blog post we are going to compare traditional NLP models with LLMs to see how they measure up. We’ll focus on extracting the names of people, organizations, or locations within a body of text. We’ll discuss ways entity extraction can improve the search experience, analyze the performance of traditional models and large language models, run a few experiments, and conclude with a review of our findings

A Simple Example

Imagine we have four documents we want to index:

				
					{
  “id”: 1
  “text”: “Will, are you going to the store today?”
}

{
  “id”: 2
  “text”: “Will you go to the store today?”
}

{
  “id”: 3
  “text”: “I hope you will join us.”
}

{
  “id”: 4
  “text”: “Is Hope going to be joining us?”
}

Let’s say you want to search for documents referencing a certain person. In some cases, you might get away with just searching for their name, as-is, in text. But, when you search for names like Will or Hope, your search engine will likely return documents that use these words in a different context. As you can see, “will” is found in documents 1, 2, and 3, but only document 1 actually references someone named Will. We run into a similar issue when searching for “hope” as well. 

Having some irrelevant documents returned is, of course, not ideal. But in practice, the problem may be more than just a minor nuisance. Terms like Will and Hope could be used more often as English words rather than names. Documents that naturally use the word “will” multiple times might score higher than documents mentioning a person named Will. Making matters even worse, the text might reference Will with pronouns instead of stating his name repeatedly, further decreasing the document’s search score.

So… just running a search on “text” won’t always suffice. To avoid manually pruning through your search results to remove the irrelevant documents, you’ll want to enrich your content before indexing. 

With entity extraction, we can enrich our documents like so:

				
					{
  “id”: 1
  “text”: “Will, are you going to the store today?”
  “people”: [“Will”]
}

{
  “id”: 2
  “text”: “Will you go to the store today?”
}

{
  “id”: 3
  “text”: “I hope you will join us.”
}

{
  “id”: 4
  “text”: “Is Hope going to be joining us?”
  “people”: [“Hope”]
}

If we extract entities from the text, we can attach lists of names mentioned to the document. Now, if we want to search for documents that reference somebody named Will, we would search on “people” instead of “text”. This allows us to ensure documents like 2 and 3 don’t hinder our search process or contaminate our results. This is definitely a simple use case – different use cases will certainly have unique challenges to overcome.

Our Entity Extraction Approach

There exist many statistical/rules-based models for entity extraction and named entity recognition. For our testing, we used Apache OpenNLP’s pretrained models and Stanford CoreNLP’s built-in named entity recognition models. But they aren’t perfect, and with the ever increasing popularity of large language models (LLMs), we thought it would be interesting to see how an LLM performs entity extraction compared against these legacy approaches.

We theorized that an LLM had the potential to extract certain names that more traditional models may be likely to miss. (In particular, we figured an LLM would be more likely to extract a “newer” name that was uncommon when the traditional models were trained.) But, we also believed an LLM could potentially get “distracted” and underperform the traditional models on longer pieces of text.

In order to test our theories, we ran four different experiments:

  1. Baseline: We started by evaluating the performance of OpenNLP, CoreNLP, and Google’s gemma3, a popular and capable open-source LLM. The default version of the model is about 3 GB in size and has roughly 4 billion parameters, making it suitable for use on modern hardware. For each model, we evaluated its precision, recall, and F1 score.
  2. Number of Parameters: We introduced two new variants of gemma3 with a different number of parameters. We discussed how using more / less parameters appeared to affect the results.
  3. Two Pass: Using the two strongest models – CoreNLP and gemma3 – we instructed gemma3 to observe and edit the output of CoreNLP as it saw fit. 
  4. Alternate Model: We evaluated another popular model, deepseek-r1, against OpenNLP and CoreNLP, to see if there were any notable differences.

For all experiments, we used a publicly available and fully annotated wikigold dataset, allowing us to evaluate the models against a source of truth. Across the 140+ wiki articles, there were roughly 3,000 words annotated as a person, organization, or location. 

As part of our evaluation, we worked with Lucille, our open-source Search ETL solution that allowed us to pass text through entity extraction processes and update each document with the output. We created a custom Connector to process the wikigold dataset into Lucille, including the article’s text as well as lists of the annotated (or “gold”) people, organization, and location names. 

To perform entity extraction, we used three different Lucille Stages. We created a custom Stage to extract people, organization, and location names using OpenNLP. We did the same for CoreNLP as well. To work with an LLM, we used Lucille’s PromptOllama Stage, which allows you to provide parts (or all) of a document to a compatible LLM for generic enrichment. The model was instructed to read the source text and output a JSON object including the names of people, organizations, and locations mentioned in the document. Lucille then integrated the model’s JSON response into the document. The models did not have access to the output of other Stages – they only saw the source text. 

Here’s an example of what a finalized document looked like, after we normalized the output for evaluation:

				
					{
  “text”: “010 is the tenth album from Japanese Punk Techno band The Mad Capsule Markets . This album proved to be more commercial and more techno-based than Osc-Dis , with heavily synthesized songs like Introduction 010 and Come . Founding member Kojima Minoru played guitar on Good Day , and Wardanceis cover of a song by UK post punk industrial band Killing Joke . XXX can of This had a different meaning , and most people did n't understand what the song was about . it was later explained that the song was about Cannabis ( ' can of this ' sounding like Cannabis when said faster ) it is uncertain if they were told to change the lyric like they did on P.O.P and HUMANITY . UK Edition came with the OSC-DIS video , and most of the tracks were re-engineered .”
  “openNLP_people”: []
  “coreNLP_people”: [“kojima”, “minoru”]
  “ollama_people”: []
  “gold_people”: [“kojima”, “minoru”]
  “openNLP_organizations”: [“uk”, “killing”, “joke”, “founding”]
  “coreNLP_organizations”: []
  “ollama_organizations”: [“the”, “mad”, “capsule”, “markets”, “meta”, “killing”, “joke”]
  “gold_organizations”: [“the”, “killing”, “mad”, “markets”, “capsule”, “joke”]
  “openNLP_locations”: []
  “coreNLP_locations”: [“uk”]
  “ollama_locations”: [“uk”]
  “gold_locations”: [“uk”]
}

For the purposes of evaluation, we did not index the documents into a search engine. Instead, they were indexed into a CSV, which stored the original text, annotated entities, and output from each model. We then ran a custom script to analyze the models’ performance. Using the annotated people, organizations, and locations from the wikigold dataset, we were able to compute some key metrics for each model:

  • Precision – What percentage of the person/organization/location names output by a model were annotated as such in the dataset?
  • Recall – What percentage of the annotated person/organization/location names in the dataset were output by the model?
  • F1 – The “harmonic mean” of precision and recall. Considered a solid overall indicator of a model’s performance. 
  • Unique Gold Words – How many gold names did a model mention that no other model did?

ExperIminets & Results

Experiment 1: Number of Parameters

We began by creating a pipeline that used three different models: OpenNLP, CoreNLP, and gemma3. Each model ran independently of the other, meaning they were not aware of each other’s output. Here are the precision, recall, and F1 scores for each model:

exp1gemma3
exp1opennlp
exp1corenlp

Clearly, CoreNLP is the strongest contender here, with gemma3 in a close second. Both had similar F1 scores of about ~0.75. OpenNLP wasn’t the strongest contender with a lower F1 score. We should also consider the latency associated with running the LLM.

On average, gemma3 took about 8 seconds to respond per Document, slowing the pipeline down substantially. (The experiment was run on an Apple M1 Pro with 16 GB of RAM.)

We also analyzed the effect of text length on model performance. Here, we considered just the top performing models – CoreNLP and gemma3. We calculated the same metrics (precision, recall, F1) for each wiki article. There were a few articles with 900+ words that we excluded to avoid skewing the results. We also excluded articles with less than 100 words. Since these documents were very short, they usually just didn’t reference names of a certain type. As a result, the model scores were primarily either zero or one, which made the results very volatile and difficult to observe:

exp1textlength_gemma3_f1
exp1textlength_corenlp_f1
exp1textlength_gemma3_recall
exp1textlength_corenlp_recall
exp1textlength_gemma3_precision
exp1textlength_corenlp_precision

As expected, the data here was a bit scattered. Since many points overlapped at the top and bottom of each chart, we did include lines of best fit to help visualize the overall trend. However, these lines should be interpreted cautiously, as they all had a very low R-squared value. In other words, the length of a piece of text shouldn’t be used to singularly predict the recall, precision, or F1 score you’ll get from a model.

Again, these charts should be observed with caution, as there were a multitude of factors at play here. But, it does seem reasonable to suggest that there was some sort of relationship between longer text and LLM underperformance. CoreNLP, on the other hand, appears to have been remarkably consistent.

Experiment 2: Number of Parameters

Our next pipeline included five different models (OpenNLP, CoreNLP, and three variants of gemma3). In addition to the previous pipeline’s models, we added a smaller variant of gemma3, gemma3:1b, and a larger variant of gemma3, gemma3:12b. As we mentioned above, gemma3, had 4 billion parameters and was a little more than 3 GB in size. The smaller variant, gemma3:1b, had 1 billion parameters and was less than 1 GB in size. The larger variant, gemma3:12b, had 12 billion parameters and was about 8 GB in size.

Here are the results we found:

exp2_gemma3_12b
exp2_gemma3
exp2_gemma3_1b
exp2_opennlp
exp2_corenlp

Again, gemma3 and CoreNLP are the strongest contenders here. The smallest LLM, gemma3:1b, didn’t do well – not only did it have a poor F1 score, but we found it was actually struggling to follow our instructions. Surprisingly, the largest LLM, gemma3:12b, was actually a bit worse than the medium variant, gemma3. Compared to gemma3, gemma3:12b had a somewhat higher precision but a notably lower recall. It seems that this larger model was a bit too cautious when engaging with the source text.

For this experiment, we also calculated the number of “gold” words that were uniquely mentioned by each model.

As expected, CoreNLP and gemma3 pick up on the most unique gold words. Interestingly, the small and large gemma variants had the fewest unique gold words – even less than OpenNLP.

Experiment 3: Two Pass

In the previous experiment, we saw that gemma3 listed ~85 “gold” words that no other model did. CoreNLP uniquely listed ~115 “gold” words. We wondered if a better overall result could be achieved by having the two models actually work together to improve their output. Ideally, an LLM could catch some of these “additional” names (increasing recall) and make some minor changes to CoreNLP’s output (increasing precision).

				
					(LLM Request)
{
  “text”: “The 38th NAACP Image Awards televised live on FOX in Hollywood, California, hosted by LL Cool J.”
  “organizations”: [“FOX in”],
  “locations”: [“Hollywood, California”]
}

(LLM Response)
{
  “people”: [“LL Cool J”]
  “organizations”: [“FOX”, “NAACP”],
  “locations”: [“Hollywood, California”]
}

Our modified Lucille ETL pipeline used only two models. First, CoreNLP extracted entity names, as usual. Then, we used gemma3 again, but with a new system prompt and a different PromptOllama configuration. Now, gemma3 was instructed to “edit” the results from CoreNLP as needed. The stage’s configuration ensured the request included the source text and the people, organization, and location names extracted by CoreNLP. (This was the only pipeline where an LLM was provided results from a previous model.) Together, the models had the following scores:

Unfortunately, this approach was actually less performant, with an F1 score lower than CoreNLP or gemma3 operating individually. Instead of finding a way to include those “unique” gold words, it looks like the LLM had an inclination to delete the names output by CoreNLP. (In a later chart, you’ll see the total number of words output in this “two pass” pipeline is very similar to the number output by gemma3 alone.) While there are certainly a variety of ways to tweak the pipeline, it seemed we weren’t going to obtain the results we were looking for with this approach.

Experiment 4: Alternate Model

Lastly, we wanted to measure the performance of an alternate LLM. We created a pipeline similar to the first experiment, but PromptOllama used deepseek-r1 instead of gemma3. deepseek-r1 is a “reasoning” model, which could potentially yield different results. The variant we used, deepseek-r1:14b, had 14 billion parameters, and was roughly 9 GB in size. This made it slightly larger than gemma3:12b, the “large” model used in the first pipeline.

The “unique” count only considered these three models, so the results aren’t directly comparable to the same chart from Experiment 2.

We can see that deepseek-r1’s performance was roughly in line with gemma3’s performance from earlier, with an F1 score of roughly 0.7.

Again, we noticed that CoreNLP and the LLM were each picking up on many gold words that the other models weren’t. We still wanted to find a way to capture as many gold words as possible. So, instead of running another pipeline, we decided to just calculate the scores associated with combining the outputs of every model:

As you could imagine, we got higher recall at the cost of reduced precision. We picked up on more of the gold words, but less of the words listed were actually gold words from the original dataset. Interestingly, the F1 score remained roughly the same, despite pronounced shifts in precision and recall.

If you’re looking to run enhanced searches on your documents, a higher recall will help ensure you don’t miss out on any names. But, if you’re looking to run aggregations or facets on the extracted entities, these extra non-gold entries could undermine the quality of your insights.

Again, we noticed that CoreNLP and the LLM were each picking up on many gold words that the other models weren’t. We still wanted to find a way to capture as many gold words as possible. So, instead of running another pipeline, we decided to just calculate the scores associated with combining the outputs of every model:

Pulling The Data Together

Lastly, here are some higher level results comparing all of the models.

CPU / GPU: Apple M1 Pro RAM: 16 GB.

Conclusion

Overall, it looks like traditional NLP models are still valuable, even as LLMs are frequently touted as the solution to all of life’s problems. CoreNLP generally led the way with the highest F1 scores and only a fraction of the LLMs’ high latency. But some of the LLMs we tested were still very strong alternatives. They had high F1 scores and picked up on some words that CoreNLP didn’t. OpenNLP did underperform, but we were using its pretrained models. Training a custom model with OpenNLP’s architecture could yield improved results.

Though CoreNLP outperformed, LLMs could still play a vital role in many entity extraction solutions, as they are extremely versatile and require minimal setup. If you don’t have the time to find a training dataset, cleanse it, and then train and evaluate a model, an LLM is certainly a viable option. Additionally, an LLM could handle data in a variety of languages without any additional configuration or training needed. If our data was in multiple languages, we would have had to completely overhaul our pipeline to support this data.

As such, any entity extraction solution you build should be tailored to your use case. While you can’t really go wrong with a traditional model, you may want to consider integrating an LLM into your process. Are your documents in multiple languages? Do they have very long pieces of text? How many documents do you have? How much compute is available to you? You’ll have to take a holistic approach to designing your solution.

Based on our findings, even in a world filled with LLMs, it looks like traditional models still have a place in addressing classic NLP problems.

Share Post
More From the KMW Blog

The post What’s the best way to do entity extraction for search? first appeared on KMW Technology.

]]> 30125 MCP in LLM Apps: Overkill or Integral? https://kmwllc.com/index.php/2025/05/20/mcp-in-llm-apps-overkill-or-integral/?utm_source=rss&utm_medium=rss&utm_campaign=mcp-in-llm-apps-overkill-or-integral Tue, 20 May 2025 15:55:54 +0000 https://kmwllc.com/?p=30155 As with any tech decision, the right tool depends on the context. But as LLM development matures, the trend is clear: we’re moving toward simplicity, agility, and tight integration.

The post MCP in LLM Apps: Overkill or Integral? first appeared on KMW Technology.

]]>
As with any tech decision, the right tool depends on the context. But as LLM development matures, the trend is clear: we’re moving toward simplicity, agility, and tight integration.
Picture of Kevin Butler
Kevin Butler

Search & AI Engineer at KMW Technology

Model Control Protocol (MCP) in LLM Apps: Overkill or Integral?

As large language models (LLMs) become more central to app development, we’re starting to see a surge in new “standards” that aim to streamline the way we work with them. One of those is Model Control Protocol (MCP)—a specification that introduces structured patterns for how LLMs can interact with external tools and services.

As LLM use evolves, MCP has the potential to meet a real need: a consistent interface, reusable patterns, well-defined roles for inputs and tools. After trying it in a few real-world projects, I’ve landed on a pretty firm opinion:

MCP is overkill for some self-contained LLM applications.

Let’s break that down.

WHAT MCP Tries to SOLVE

MCP offers a standardized format for:

  • Defining tools and functions an LLM can use
  • Structuring prompts and inputs
  • Managing model responses
  • Integrating external systems into a unified orchestration layer

If you’re building a platform where multiple apps or teams need to access a model with consistent behavior, then yes, MCP can be helpful. It makes your model “discoverable” and “programmable” in the same way REST and GraphQL made APIs predictable.

But here’s the catch…

In self-contained apps, it’s just extra weight

If you’re building an app where you own the model, you control the tools, and you define the prompt context—you don’t need MCP. You already know what the LLM needs. You’ve got the orchestration baked into your app’s API layer.

Let’s take a typical modern setup:

  • A Next.js app with built-in API routes
  • A custom orchestrator to manage model calls and tool routing
  • Locally hosted context (embeddings, history, memory)
  • Internal tools for summarization, tagging, or database lookup

     

In this environment, introducing MCP means:

  • Creating new wrappers around tools
  • Translating your existing structure into a generic spec
  • Debugging new abstraction layers
  • Maintaining protocol compliance on top of your working logic

All of which gives you… what? A clear contract between the LLM and the MCP service? 

That’s not simplification. That’s ceremony.

Where MCP Does Make Sense

MCP shines when you’re not the only one calling the model. For example:

  • Multi-tenant LLM services that need a consistent interface for different consumers
  • Third-party apps that interact with a centralized orchestration layer
  • Teams with shared infrastructure, where models and tools are exposed through a shared gateway

In those cases, it’s worth it. Standardization reduces friction. It enables reuse. It helps with governance, monitoring, and security. MCP becomes a contract between producers and consumers of LLM capability.

But most apps aren’t there. Most LLM-enabled apps today are still early, experimental, and owned end-to-end by a single team. In those scenarios, MCP’s benefits are marginal.

The Real Game-Changer: Function Calling

What’s actually changing the landscape isn’t protocol—it’s capability.

LLMs are getting much better at tool calling (aka function calling). With structured outputs, schema validation, and multi-step reasoning, we’re approaching the point where your model can:

  • Choose the right tool on its own
  • Execute functions with arguments
  • Chain results intelligently
  • Ask follow-up questions when needed

That’s the future. And it’s pretty easy to imagine a world where a new generation of LLMs are trained and tuned for MCP based tool use, making this protocol even more powerful. 

We are already seeing existing application providers looking to expose their data and services via MCP to hook into AI applications more easily. There are certainly benefits to adopting early in order to play nice in a wider AI ecosystem.

Final Thought

Is MCP overkill or integral? As with any tech decision, it depends. As LLM development matures, we’re moving toward simplicity, agility, and tight integration and each layer of protocol is going to need to justify its own weight. At the same time, we are now seeing the next wave of LLM applications where external integration and tool calling are absolutely essential. At KMW, we are watching MCP closely and experimenting with more complex tools while keeping a close eye on industry adoption of the standard. These are exciting times in AI development and we’ll have more to come soon

TL;DR

  • MCP is useful when you’re exposing models to external consumers or services and need a standardized interface.
  • It’s overkill for internal, self-contained apps where you already manage the model’s full context and environment.
  • Function calling and tool use are the real potential for MCP, and they are only going to gain in importance and capability going forward.
  • Assess whether the complexity is warranted, and if your use case actually requires it. Your LLM doesn’t need a passport to move around in its own country.
Share Post
More From the KMW Blog

The post MCP in LLM Apps: Overkill or Integral? first appeared on KMW Technology.

]]> 30155 RAG Question Answering System for Solr and OpenSearch  https://kmwllc.com/index.php/2024/06/23/rag-question-answering-system-for-solr-and-opensearch/?utm_source=rss&utm_medium=rss&utm_campaign=rag-question-answering-system-for-solr-and-opensearch Sun, 23 Jun 2024 16:01:28 +0000 https://kmwllc.com/?p=29895 We describe the process of using retrieval-augmented generation (RAG) to create a question-answering system about Solr and OpenSearch using an assortment of LLMs from HuggingFace and OpenAI.

The post RAG Question Answering System for Solr and OpenSearch  first appeared on KMW Technology.

]]>
We describe the process of using retrieval-augmented generation (RAG) to create a question-answering system about Solr and OpenSearch using an assortment of LLMs from HuggingFace and OpenAI.
Picture of Akul Sethi
Akul Sethi

Search Engineer at KMW Technology

What We’ve Accomplished

With the recent advances in Large Language Models (LLMs), a very natural intersection with traditional Search has arisen. Historically, the only way to “ask” a search engine a question is to provide some search keywords and the system responds with a set of ordered documents which hopefully contain an answer. In a RAG system however, you ask your question in natural language, it retrieves documents, but then goes the additional step to synthesize an answer from those documents so that the user does not have to comb through them themselves. This is done by feeding the search results (documents) as context to an LLM which may not necessarily have trained on that information. In this way, an LLM can “learn” about a subject at inference time. 

In this blog post, we describe the process of creating a RAG question-answering system to answer questions about Solr and OpenSearch based on technical documentation. We used an OpenSearch instance as the search backend and investigated an assortment of HuggingFace and OpenAI model LLMs.

Architecture

The system can be separated into 3 logical components: 

  1. Search engine & ingestion
  2. Large Language Models
  3. UI website to interact with the system

All of these components are dockerized and are orchestrated using docker compose. The search engine is used to store documents and retrieve relevant ones during generation. The LLM component hosts a REST API which contains the heart of the RAG logic. It hits both the search engine as well as externally hosted models to produce a response. Finally, the UI website is a static front end to allow a user to interact with the system and visualize results.

Search Engine & Ingestion

We cannot just throw all of the Solr or OpenSearch documentation at an LLM and expect it to produce improved results. These excess, nonrelevant documents will just convolute the context window and for lack of a better word “confuse” the LLM. We additionally would prefer to provide the documents to the LLM in order of relevance as it helps the LLM understand which documents to prioritize. There is no better device for this than a search engine: it stores documents and allows for fast retrieval of ordered, relevant documents. 

We decided to use an OpenSearch cluster for this purpose. Traditionally, search engines have used a lexical process combined with Term Frequency – Inverse Document Frequency to determine the relevance of documents. Without going into too much detail, this means that documents are split into tokens, such as words, and then for a given query, a score is calculated for each document with the following properties: the score for a document is positively correlated with how many times the query appears in it and negatively correlated with the number of times the query appears in other documents. In this way, common words such as “the”, “of”, or “there” which appear in most documents do not contribute to the score as much as rare words since a less common word is more likely relevant to the document it appears in. 

While this approach works, it does not take into consideration the semantic meaning of words. The search engine actually does not know how words relate to each other or even have a concept of language. What if there was a technique that did? This is where vector search comes to the rescue. Like the name implies, in vector search, documents are embedded into vectors which have the property that the more similar two documents are, the closer their respective vectors will lie in the embedding space.

Again, the details are complicated, but some sort of neural network is generally used to approximate this function. 

In order to investigate how both lexical and neural retrieval affect RAG, our demo supports both. We used the Neural Search plugin as it comes natively with OpenSearch. 

To ingest documents into the engine, we used our own production grade, open-source ETL solution, Lucille. This allowed us to easily specify separate sources for both OpenSearch and Solr each with their own processing pipelines in a very simple config. Since Solr has its documentation in raw web pages and OpenSearch has it in markup, different preprocessing must be applied to both. This makes the task great for Lucille. We constructed two separate stages for the task: a HTML extraction stage which uses JSoup to extract relevant portions of a webpage and a Markup extraction stage which does the same for markup. 

Large Language Models

Having a system to retrieve documents is all well and good, but there cannot be any RAG without Large Language Models. Since these models are extremely computationally expensive we used externally hosted REST APIs to run them. We chose two of the more popular platforms for this project: the HuggingFace inference and OpenAI APIs. 

When RAG was first developed it was used with question-answering LLMs. These are LLMs which, given a context and question, are trained to produce an answer from the context. What’s special about them however is rather than producing an answer “from scratch” they can only return a span from the context. For example:

Context: I’m a search engineer at KMW Technology and my name is Akul Sethi. I love to go hiking and play chess.

Question: What is my name?

Answer: Akul Sethi

However, with the recent explosion in LLM development there are now LLMs specifically suited for various tasks. Our demo supports multiple LLMs to allow a user to experiment with different ways in which RAG can be used. Keep in mind that since they are all trained differently, they take prompts differently. The demo adheres to this by changing the placeholder prompt to provide an example of how the selected model receives input.

Prompting Roberta

deepset/roberta-base-squad2

Roberta is a model that has been trained for the task of Question-Answering as described above. It is rather small at 124M parameters but for demonstration purposes it works well. This model takes its prompt in a simple question form, such as “How do I make a collection in Solr?”.

Prompting Mistral

mistralai/Mistral-7B-v0.1

Mistral is a model that has been trained for the task of Text Generation. These models are trained by giving them a portion of a sentence and asking them to complete it. Because of this, if you want Mistral to tell you how to make a collection in Solr it would have to be phrased as: “A collection can be made in Solr by…”. Mistral is also a larger model than Roberta at 7.24B parameters making it second best to GPT-3 in our demo.

Prompting GPT-3

gpt-3.5-turbo-0125

Of course no, RAG demo would be complete without the most quintessential LLM of them all: GPT-3. This is OpenAI’s proprietary LLM which is generally considered to be a conversational LLM. It is the most flexible of the 3 models in how it can take its prompts but is usually used similarly to Roberta. The big difference of course is that GPT does produce an answer “from scratch” rather than a snippet from the context meaning that it can produce “hallucinations”, or completely false answers. While GPT does produce the best answers, most likely due to its size, it is something to keep in mind (the exact size of the model is not known as it isn’t something which OpenAI has made public at the time of writing, but it is estimated to be in the hundreds of billions of parameters).

UI Website

The final component to tie it all together is the UI Website. This allows a user to query the system while controlling the various parameters.

Model panel
This is for the main controls which includes the model, number of documents to retrieve, whether to use neural search, and if we want an AI response to be generated. 

Source panel
As the name suggests this panel allows a user to select which document sources (Solr or OpenSearch) should be queried. 

Categories panel
This panel allows a user to more efficiently sift through the documents by applying filters based on the category of the document. This is populated dynamically by the front end based on the categories of the returned documents.

Selection
The front end also supports a selection mode allowing the user to manually select which documents are used for generation. To enable, toggle the “select documents” button. Check boxes will appear next to the documents indicating if they will be used the next time the “Search” button is clicked. NOTE: when this mode is on, fresh documents will not be returned. 

Key Takeaways

Retrieval

We performed a relevancy test of the results using the open source tool Quepid. This test compared RAG results using lexical vs neural search and we noticed some significant differences. Since this is primarily a question-answering system, queries will generally be formulated as questions. No surprise there. However, documentation is generally not written in that voice. For example, take the following two queries:

How do I make a collection in Solr? 

How do I use facets in Solr? 

In the first query we would like to retrieve documents pertaining to collections and in the second retrieve ones pertaining to facets. However, using lexical search we noticed that both queries would just return the same set of documents from the FAQ section of the documentation. Upon further analysis we realized that due to the nature of the documentation; all the words in those queries are abundant in the documents other than the word “I”. Thus, TF/IDF incorrectly assigns a lot of weight to the word “I” which is why we only see documents from the FAQ section: the only section which contains user questions. 

Neural search does not succumb to this problem. It is harder to see why, since explainability does not yet exist with neural networks, but it seems as though embedding models are better able to detect the significant words in the above queries in this case. 

Prompting

We found that prompting was another unexpected hurdle. The original plan was that the user question would be used as both the search query and the LLM. However, the question format which produces the best results for retrieval does not necessarily produce the best results for generation and vice versa. In fact, as each model is trained differently there is not even a format that consistently produces the best results across models.

This is definitely a point for further improvement but currently we found that optimal results can be achieved by first using the system as a search engine with a search optimized query, selecting these documents in selection mode, and then re-running with a generation optimized query. 

Conclusion

Overall, we were able to show how recent developments in Large Language Models can improve how we interact with traditional search engines using RAG. Along the way, we explored various challenges which may come up while working with RAG systems and our recommendations in dealing with them. 

Our final demonstration can be used both to query Solr and OpenSearch documentation, but more importantly, allows beginners in RAG to experiment with various hyperparameters and models to get a better understanding for how this cutting edge technology works. 

Interested in seeing a Demo?

If you’d like to see what we’ve built in action,  Contact Us!

Share Post
More From the KMW Blog

The post RAG Question Answering System for Solr and OpenSearch  first appeared on KMW Technology.

]]> 29895 Duplicate Terms Aggregation Plug-in for OpenSearch https://kmwllc.com/index.php/2024/05/30/duplicate-terms-aggregation-plug-in-for-opensearch/?utm_source=rss&utm_medium=rss&utm_campaign=duplicate-terms-aggregation-plug-in-for-opensearch Thu, 30 May 2024 01:25:59 +0000 https://kmwllc.com/?p=29639 In Lucene-based search engines like OpenSearch and Solr, keyword aggregations ignore duplicate values that occur within a multi-valued field. We built an OpenSearch plugin to overcome this limitation.

The post Duplicate Terms Aggregation Plug-in for OpenSearch first appeared on KMW Technology.

]]>
In Lucene-based search engines like OpenSearch and Solr, keyword aggregations ignore duplicate values that occur within a multi-valued field. We built an OpenSearch plugin to overcome this limitation.
Picture of Abijit Rangesh
Abijit Rangesh

Search Engineer at KMW Technology

What We’ve Accomplished

In Lucene-based search engines like OpenSearch and Solr, keyword aggregations ignore duplicate values that occur within a multi-valued field. If a document has a multi-valued field containing the values [“foo”, “foo”, “bar”] then an aggregation would increment the count for the “foo” and “bar” bucket once, even though “foo” occurs twice. We built an OpenSearch plugin to overcome this limitation.

In this blog post, we detail the investigation process into duplicate terms aggregation solutions and the subsequent development process by which a plug-in was built for OpenSearch. Previously, only the nested field type and scripted aggregation methods seemed to be viable solutions. Our plug-in aims to be a third option to solve this problem. This plug-in serves to be the fastest solution for a duplicate terms aggregation on a multivalued keyword field in OpenSearch without compromising index size.

The link to the plug-in repository can be found here. Please follow the steps in the README.md to install this plug-in into an OpenSearch distribution and interact with the custom aggregation. 

The Problem

Imagine you’re rummaging through your attic at your family home and you’ve found a large recipe book. It seems to be something your great-grandmother cherished. As both a devoted great-grandchild and a fan of search solutions, you’d like to catalogue the data found in this recipe book in a search engine, specifically OpenSearch. 

You wish to represent each recipe as a document and each document would store fields for different aspects of the recipe: cook_time, steps, technique, ingredients, etc. Ingredients could have duplicate values, designating how many of them to include i.e. [carrot, carrot, apple, cucumber] would represent 2 carrots, 1 apple, and 1 cucumber for our recipe. 

Upon completion of your index, you wish to aggregate upon all the ingredients and find out how many of each ingredient to purchase in order to make all of your great grandmother’s recipes. You notice a problem quite quickly: OpenSearch aggregations do not take into account duplicate values on multi-valued fields. While the recipes may have called for a total of three carrots (two for one recipe and one for a different recipe), the aggregation result only tells you to buy two (one for each recipe).

				
					Documents & Aggregation: 

curl -XPUT "http://localhost:9200/recipe-book/_doc/1" -H 'Content-Type: application/json' -d'
{
  "favorite_foods": ["carrot", "carrot", "apple", "cucumber"]
}'

curl -XPUT "http://localhost:9200/recipe-book/_doc/2" -H 'Content-Type: application/json' -d'
{
  "favorite_foods": ["carrot", "apple", "cucumber"]
}'

curl -XPUT "http://localhost:9200/recipe-book/_doc/3" -H 'Content-Type: application/json' -d'
{
  "favorite_foods": ["cucumber", "cucumber", "cucumber"]
}'

curl -XGET "http://localhost:9200/recipe-book/_search" -H 'Content-Type: application/json' -d'
{
  "size": 0,
  "aggregations": {
    "ingredient_frequency": {
      "terms": {
   	   "field": "favorite_foods"
      }
    }
  }
}'

Result: 
"aggregations":{
  	"ingredient_frequency":{
     	  "doc_count_error_upper_bound":0,
     	  "sum_other_doc_count":0,
     	  "buckets":[
        	{
           	"key":"cucumber",
           	"doc_count":3
        	},
        	{
           	"key":"apple",
           	"doc_count":2
        	},
        	{
           	"key":"carrot",
           	"doc_count":2
        	}
     	  ]
  	}
}

Delving Deeper Into the Problem

Why is it that OpenSearch is unable to perform aggregations on duplicate values in keyword fields? Behind the scenes, OpenSearch is using a doc_values data structure for operations like aggregations, as they perform far better than the traditional inverted-index data structure. The specific type of doc_values supported by Lucene for keyword fields do not store duplicate values. Conversely, a match_all query would return us duplicate values as it would be using a segment lookup unlike aggregations which would be using the aforementioned doc_values data structure. 

The following link provides more information about specific field types and whether they support duplicates or not. Note that while this links to Solr documentation and presents Solr specific information, the section on docValues types is a Lucene specific implementation detail that applies to OpenSearch as well. 

Scripting

OpenSearch offers users the ability to include scripts with aggregations via the “scripted_metric” aggregation type. This leverages the painless scripting language, built specifically for Elasticsearch (and subsequently OpenSearch). Painless has Java-like syntax and compiles directly into JVM bytecode, leveraging any optimizations that the JVM has. 

With painless, we have access to the “_source” variable that offers us the ability to view the segment data on specific fields. With this approach, we should be able to access duplicates on our keyword field type and aggregate them together via a hashmap defined by our script. 

Here’s what the script looked like inside of an aggregation query:

				
					{
	"size": 0,
	"aggs": {
    	"ingredient_frequency": {
      	"scripted_metric": {
        	"init_script": "state.foodFreq = new HashMap();",
        	"map_script": "for (f in params._source.favorite_foods) { if (state.foodFreq.get(f) == null) { state.foodFreq.put(f, 1); } else { state.foodFreq.put(f, state.foodFreq.get(f) + 1); }}",
        	"combine_script": "return state.foodFreq;",
        	"reduce_script": """
Map finalMap = new HashMap(); 
for (map in states) { 
 for (key in map.keySet()) { 
  if (finalMap.get(key) == null) { 
   def val = map[key]; 
   finalMap.put(key, map[key]); 
  } 
  else {
   def prevVal = finalMap[key]; 
   finalMap.put(key, prevVal + map[key]); 
  }
  }
} 
return finalMap;
"""
      	}
    	}
  	    }
  }

Notice that this is quite brute-force. We have to account for each and every value, place them in a hashmap, and then combine these hashmaps across shards via the “reduce_script”. Also, we don’t leverage the highly performant doc_values data structure, built for aggregations. We know this, as we are accessing “_source” which takes information directly from Lucene segments, rather than from doc_values. In the benchmarking section, these observations will be substantiated by our gathered performance statistics. 

For more information on the scripted_metric aggregation, here’s a link to official documentation.

Nested Fields

Scripting worked within the confines of our original organisation of the recipe book. We wanted our ingredients to be represented in a multi-valued field. However, if we can’t see duplicate values across multi-valued fields due to the doc_value limitation, why not try to change how the documents themselves are indexed?

 OpenSearch offers users the ability to create inner or “nested” documents within other documents. Each nested document is underlyingly treated as a “hidden” Lucene document, allowing for a nested aggregation that would guarantee that we would be able to see duplicate documents. 

We can implement this, simply, with the following:

				
					PUT /recipe-book
{
  "mappings": {
    "properties": {
      "favorite_foods": {
   	 "type": "nested",
   	 "properties": {
 		 "food": {
 		 "type": "keyword"
 		 }
   	  }
     }
    }
  }
}   

PUT /recipe-book/_doc/1
{
  "favorite_foods": [
    {
      "food": "carrot"
    },
    {
      "food": "carrot"
    },
    {
      "food": "apple"
    },
    {
      "food": "cucumber"
    }
  ]
}

PUT /recipe-book/_doc/2
{
  "favorite_foods": [
    {
      "food": "carrot"
    },
    {
      "food": "apple"
    },
    {
      "food": "cucumber"
    }
  ]
}

PUT /recipe-book/_doc/3
{
  "favorite_foods": [
    {
      "food": "cucumber"
    },
    {
      "food": "cucumber"
    },
    {
      "food": "cucumber"
    }
  ]
}

{
  "size": 0,
  "aggs": {
	"favorite_foods": {
  	"nested": {
    	"path": "favorite_foods"
  	},
  	"aggs": {
    	"food_freq": {
      	"terms": {
        	"size": 10000,
        	"field": "favorite_foods.data"
      	}
    	}
    	}
    	}
  }
}

"aggregations": {
	"favorite_foods": {
  	"doc_count": 10,
  	"food_freq": {
    	"doc_count_error_upper_bound": 0,
    	"sum_other_doc_count": 0,
    	"buckets": [
      	{
        	"key": "cucumber",
        	"doc_count": 5
      	},
      	{
        	"key": "carrot",
        	"doc_count": 3
      	},
      	{
        	"key": "apple",
        	"doc_count": 2
      	}
    	]
  	    }
	    }
}

This implementation is simple at first glance, but this approach can quickly cause problems at scale. Nested documents are inherently documents, therefore they take up their own space in the index. 200 recipes with 20 average ingredients in each would lead to over 4000 documents. Again, we’ll see how this implementation fares in our benchmarks. 

For more information on the nested field aggregations, here’s a link to official documentation.

Underscore Representation

Let’s try something different. Let’s go back to using multi-valued fields, but this time, let’s look at how the ingredients themselves are represented. What if we changed the representation from [carrots, carrots, apple, cucumber] to [carrots_2, apple_1, cucumber_1]? By appending our counts, we retain information about duplicates, without exposing duplicates to the underlying doc_values data structure. We can perform this conversion at index time well in advance. The question now remains as to how we can aggregate upon those “suffixes” or numbers after our underscore delimiter. 

We can try to write code directly within a cloned repository of the OpenSearch codebase  and test to see if we can create a performant solution with regards to this representation of our ingredients. If this solution proves to be a winner in our benchmarks, we can move this code over to a plug-in architecture that’s easily packageable and can be installed via a single command by other users. 

Here’s how the example from before might look like with our new representation:

				
					curl -XPUT "http://localhost:9200/recipe-book" -H 'Content-Type: application/json' -d' 	 
{
  "mappings": {
    "properties": {
    	 "favorite_foods": {
   			 "type": "keyword"
     	 }
     }
   }
}'

curl -XPUT "http://localhost:9200/recipe-book/_doc/1" -H 'Content-Type: application/json' -d'
{
  "favorite_foods": ["apple_1", "carrot_2", "cucumber_1"]
}'

curl -XPUT "http://localhost:9200/recipe-book/_doc/2" -H 'Content-Type: application/json' -d'
{
  "favorite_foods": ["apple_1", "carrot_1", "cucumber_1"]
}'

curl -XPUT "http://localhost:9200/recipe-book/_doc/3" -H 'Content-Type: application/json' -d'
{
  "favorite_foods": ["cucumber_3"]
}'

We took a look at the OpenSearch codebase, finding the code path by which traditional terms aggregations were performed. There were two “aggregator” classes that OpenSearch uses in a terms aggregation, either the GlobalOrdinalsStringTermsAggregator or the MapStringTermsAggregator. The Global Ordinals aggregation takes advantage of global ordinals, mappings of segment ordinals to their original locations at a “global” level. While this is the traditional aggregator used for terms aggregations, seeing that aggregations occur across shards and segments, we opted to base our custom implementation in the MapStringTermsAggregator. 

The GlobalOrdinalsStringTermsAggregator deals in globalOrds represented as longs, which are not easily convertible into Strings. We need to deal with Strings in order to separate the suffix count from the delimiter. The MapStringTermsAggregator uses a representation of doc values that enables for ordinal representations to be converted to BytesRefs and subsequently to Strings. 

Following further down the code path, we then modified the doc values typing used by the MapStringTermsAggregator, specifically the SortedBinaryDocValues type, and created a custom implementation that separated the suffix from our term (apple_3 → apple, 3) and created new buckets with these new terms as keys and suffixes as counts. We then sent “fake” values to the LeafCollector method that would mimic the bucket counts that we required. For example, the value apple_3 would be seen by the collector as apple, apple(fake), apple(fake). 

On further iteration, caching via Hashmap was supported to prevent multiple ordinal lookups and multiple conversions from BytesRef to Strings, improving performance in the process. 

Benchmarks & Takeaways

We now have three implementations that each net us the expected result. It now remains to see which implementation will offer us the best solution in accordance with our guidelines. 

We used JMeter to test the query performance. Here are important points regarding how the implementations were benchmarked:

  • Documents were indexed via Lucille, an open-source Java framework for ETL pipelines created by KMW Technology; Lucille supports a benchmarking workflow with randomised document creation
  • 1 index was created per implementation, each within their own instance of OS
  • 100k documents were indexed into each index
  • Each document averaged at 1000 values in its field, with the range spanning from 750 to 1250
  • All queries had random boolean match filters alongside the aggregations so as to prevent the impact of caches
    • The underscore representation queries used a “prefix” match seeing that their values had an additional delimiter and count attached
    • This “prefix” analysis may potentially have had ramifications for query performance for the underscore implementation
  • 50 users sent queries simultaneously with a 1 second ramp-up time

 Scripted Imp.Underscore Imp.Nested Imp.
# of samples505050
Average Latency (ms)103907 ms34505 ms11877 ms
Min Latency (ms)0 ms0 ms0 ms
Max Latency (ms)147451 ms41623 ms16203 ms
Std. Dev39522.057823.512761.71
Error %0%0%0%
Throughput (requests per second)0%0%0%

The scripted implementation saw the longest average query time at around 104 seconds. The underscore implementation sat in the middle with an average query time of around 35 seconds.The nested implementation was the fastest with an average query time of around 12 seconds.

With this information, it may seem clear that the nested approach is the best approach for handling duplicate terms aggregations in OpenSearch. However, taking a closer look at our proposed guidelines, we notice that we’ve yet to evaluate index size. Looking at index size benchmarks for both the nested and underscore implementations, we see a potentially new conclusion. The nested document index holds a large 1.4 GB index size. Meanwhile, the underscore index boasts a significantly smaller 187 MB index size. This translates to a 7.5x smaller index size for the underscore implementation, with a 2.9x slower average query time. 

While we didn’t explicitly weigh each guideline’s importance, it was clear to us that a significantly larger index would present more problems than the query time performance hit, especially with the magnitudes mentioned before. This led us to believe that the underscore implementation was the best fit for our needs and the guidelines that we set forth. If index size is not a consideration for users, then the nested approach would be the best, seeing that it offers extremely fast lookup / query times.

The Plug-in

It now makes sense to package this solution into a plug-in. While OpenSearch does offer an easy-to-use template for developers to get started with plug-in development, the convenience stops shortly thereafter. We encountered some problems while developing the plug-in.

Difficulties in Plug-in Creation

The most challenging issue in plug-in development with OpenSearch has to do with Java class loaders. When installing a plug-in into an OpenSearch distribution, OpenSearch uses a parent class loader to load in OpenSearch related files and a child class loader to load in our plug-in files. OpenSearch files are largely package-private, preventing the child class loader from having access to their methods, constructors, etc. This causes large amounts of code duplication and requires a deep understanding of the inner workings of OpenSearch aggregations. Here is a link to a code example that illustrates the issues we encountered with class loaders: link.

Another issue faced was a lack of clear testing protocol for plug-ins. Tests situated within the OpenSearch core codebase, specifically aggregation tests, relied heavily on helper classes and methods found within the core codebase. Our plug-in code, due to the aforementioned class loader issue, did not have access to these assisting classes. This made it extremely difficult to write unit tests, leaving us only with integration tests.

Future Points of Interest

To say that our investigation exhausted all avenues for how to perform a duplicate terms aggregation would be naive. A few ideas were brainstormed (and even tested), but didn’t seem promising or did not fit our original guidelines. Here are a few:

Our plug-in implementation leveraged the SortedBinaryDocValues, the primary doc values data type for keyword field types. If we were open to changing the field type of our data, we could have experimented with the SortedOrderedNumericDocValues which stores duplicates for the numeric field type. This type is the only doc values field type in Lucene that retains duplicates. However, the issue would arise for a need to convert from numeric type to string at query time, leading to definite performance loss. 

Another potential solution would be to utilise Lucene payloads. Payloads are metadata that can be stored with terms and accessed via Lucene. The concern regarding this solution was that payloads would likely not be accessible via doc_values, leaving them as a likely slower alternative.

One solution that was explored thoroughly was term vectors. Term vectors work by offering statistics for all the terms in the fields of a specific document, provided by the user. This already raised a few concerns, especially since we would require an artificial “master” document that contained all possible terms and fields so that its ‘id’ could be provided. If using this artificial document approach, term vectors would gather statistics from a randomly selected shard. The user must then choose to use only a single-shard instance or total the counts across all shards and create “master” documents within each shard. Despite this, we created code against Lucene itself comparing doc values to term vectors. For 10 million documents, term vectors seemed to be averaging 50 seconds slower than doc values. 

Conclusion

Overall, we believe our investigation to be a thorough analysis of the existing possible tools to perform a duplicate terms aggregation. The plug-in we’ve created seems likely to be the best solution for duplicate terms aggregation depending on the individual context and use-case. 

With this solution under our belt, we can now make our shopping list and make our great-grandmother proud! 

Share Post
More From the KMW Blog

The post Duplicate Terms Aggregation Plug-in for OpenSearch first appeared on KMW Technology.

]]> 29639 Building A Vector Search Application on OpenSearch https://kmwllc.com/index.php/2023/03/29/building-vector-search-on-opensearch/?utm_source=rss&utm_medium=rss&utm_campaign=building-vector-search-on-opensearch Tue, 28 Mar 2023 22:49:39 +0000 https://kmwllc.com/?p=28464 We created a POC vector search application using OpenSearch. In this post, we discuss what we did to get it working as well as investigate how popular search features like sorting, aggregating and filtering can be utilized in vector search.

The post Building A Vector Search Application on OpenSearch first appeared on KMW Technology.

]]>
We created a POC vector search application using OpenSearch. In this post, we discuss what we did to get it working as well as investigate how popular search features like sorting, aggregating and filtering can be utilized in vector search.
Picture of Jake Horban
Jake Horban

Search Engineer at KMW Technology

Introduction

Vector search has the potential to uncover the semantic meaning of a body of text and provide an ability to match documents from different domains. We thought an interesting use case for semantic understanding/vector search would be talent acquisition – matching a job description with a candidate’s resume and vice versa. To satisfy this use case, we created a reference implementation of an OpenSearch application able to retrieve documents based on k-Nearest Neighbors search between a job description embedding and resume embedding.

Our main goal was to create a vector search application that we can use to evaluate the technology and document what’s necessary to get it working.  We wanted to uncover how vector search may or may not work with standard lexical search features like faceting, sorting and filtering.  We were also interested in solving some of the operational challenges of working with a vector search application such as how to generate embeddings and how to fine-tune sentence embedding models.

Ingest Architecture

To leverage approximate kNN search in OpenSearch, sentence embeddings must be included as document fields during indexing and as a search parameter during querying. Moreover, both of these vectors must have the same dimensionality and be generated by the same fine-tuned model. These requirements motivated us to create a RESTful service to generate embeddings for both documents (at ingest time) and queries (at query time) using a sentence transformer model such as SBERT.

Architecture diagram

1. Embedding Service

The embedding service was created in Python using FastAPI and the Hugging Face transformer library. When the service starts, it loads the pre-trained models like SBERT from Hugging Face or from local files based on a configurable list. The service accepts text and returns embeddings at ingest time for the document and at query time for the query.  A request can specify which model should be used to generate the embeddings. Otherwise, a default model is used. 

2. Lucille (ETL) Stage

Documents are indexed into OpenSearch with Lucille, an open-source Java framework for ETL pipelines created by KMW Technology.  An embedding stage was added to Lucille to connect to the embedding service and retrieve the embeddings for the specified fields during indexing. This stage allows specifying multiple pairs of field mappings and will retrieve the embeddings for the source fields and add them to the respective target fields.

				
					pipelines: [
  {
    name: "pipeline1",
    stages: [
      {
        class:"com.kmwllc.lucille.stage.EmbedText",
        connection:"http://127.0.0.1:8000",
        fieldMapping {
          "resume": "resume_embedded",
        }
      }
    ]
  }
]

3. OpenSearch

Before indexing the documents, we need to enable the kNN index, specify the field type for holding the embeddings as knn_vector, and set the dimensionality to the same size as the embeddings generated by the language model used by the embedding service during ingest.

				
					{
  "settings": {
    "index.knn": true
  },
  "mappings": {
    "properties": {
      "resume_embedded": {
        "type": "knn_vector",
        "dimension": 384
      }
    }
  }
}

OpenSearch also allows additional parameters such as specifying the nearest neighbors indexing algorithm preferred for the dataset and the type of computing resources available in the cluster, which you can view here.

Once the documents are indexed, we will be ready to make an approximate kNN query.

				
					{
  "size": 300,
  "query": {
    "knn": {
      "resume_embedded": {
        "k": 50,
        "vector": [
          -0.04631288722157478,
          -0.03802000731229782,
          ...
        ]
      }
    }
  }
}

Approximate KNN Search

A kNN search query traditionally uses a brute-force approach to compute similarity, which produces exact results but can be slow for large, high-dimensional datasets. Approximate kNN search methods can improve efficiency by restructuring indexes and reducing dimensionality. This approach reduces the accuracy of the results but increases search processing speeds significantly. 

OpenSearch offers several different search methods that support approximate kNN. If a search method is not specified at index creation, OpenSearch will use the nmslib  engine to create Hierarchical Navigable Small World (HNSW) graphs. HNSW graphs support more efficient approximate kNN search. For more information about the implementation of approximate kNN search in OpenSearch, refer to their documentation.

Traditional Search Feature Support

Computing facets/aggregations, sorting, and filtering are some of the most common search features used in lexical search. As part of our POC, we wanted to investigate how these features worked in conjunction with kNN queries.

Aggregations

As is typical in lexical search, the aggregation will be computed for the documents in the approximate kNN query result set. Here is an example query:

				
					{
  "size":10,
  "aggregations": {
    "category": {
      "terms": {
        "field": "category.keyword"
      }
    }
  },
  "query": {
    "knn": {
      "resume_embedded": {
        "k": 10
        "vector": [
             0.02731507644057274,
             0.010414771735668182,
            ...
            ],
               
            }
        }
    }
}

Sorting

kNN query results can also be sorted according to a keyword field:

				
					{
  "size":10,
  "sort": [
    {
     "category.keyword": {
      "order": "asc"
      }
    }
  ],
  "query": {
    "knn": {
      "resume_embedded": {
        "k": 10
        "vector": [
             0.02731507644057274,
             0.010414771735668182,
            ...
            ],
            }
        }
    }
}

Filters

The typical filter behavior for lexical queries is to apply a filter before the query is executed, thereby narrowing the scope of documents that need to be searched. This is the default behavior for most search engines and generally known as a ‘pre-filter.’ For an OpenSearch approximate KNN query, pre-filter queries are only supported if the kNN index is constructed using the Lucene engine to build HNSW graphs. 

As we previously mentioned, the default engine used to construct HNSW graphs in OpenSearch is nmslib, which only supports post-filtering of results using the post_filter parameter. This default behavior is important to keep in mind when setting up a vector index in OpenSearch, as post-filtering can impact what results are actually returned for a filter query.  

When a post-filter query is executed, the kNN query will be computed first. Then the filter will be applied, potentially reducing the number of returned documents. It is essential to keep this order in mind when selecting parameters for the query. For example, we can increase the size and k parameters to ensure that the query returns a sufficiently broad set of results (i.e., to ensure that recall is high enough) before the filter is applied. In most cases, increasing k will improve the accuracy of the approximate k-NN search but will also increase the computation time.

Example kNN query using the post_filter parameter:

				
					{
  "size": 300,
  "post_filter": {
    "match": {
      "location": "New York"
    }
  },
  "query": {
    "knn": {
      "resume_embedded": {
        "k": 5,
        "vector": [
          -0.07738623768091202,
          -0.06183512881398201,
          ...
        ]
      }
    }
  }
}

If you would like to know more about  different kinds of filter queries, their use cases, and their performance in combination with kNN, take a look at this documentation from OpenSearch. One of the examples included is a Boolean filter query that allows filtering documents based on must,  must_not, and should parameters. This type of query can be used to combine lexical and vector search.

				
					{
  "size": 3,
  "query": {
    "bool": {
      "filter": {
        "bool": {
          "must": [
            {
              "match": {
                "location": "New York"
              }
            }
          ]
        }
      },
      "must": [
        {
          "knn": {
            "resume_embedded": {
              "k": 20,
              "vector": [
                -0.07738623768091202,
                -0.06183512881398201,
                ...
              ]
            }
          }
        }
      ]
    }
  }
}

Our examples here are based on using the approximate nearest neighbor approach  because it scales well for larger datasets. The painless scripting approach is preferred if you need to use a distance function as part of your scoring method. On small datasets, the performance impact of brute force kNN may not be as consequential and can potentially provide more accurate results. In this case, one can use the custom script scoring approach with a pre-filter and brute force kNN instead of approximate kNN and post-filter.

Model Selection and Fine-Tuning

Since our queries and the corpus are about the same length, we decided to use a symmetric semantic search to have the ability to do two-way matching between resumes and job descriptions. The pre-trained model dimensionality and statistics can be found here.  We started our experiments with all-MiniLM-L6-v2 and all-MiniLM-L12-v2 models with dimensions 384.

To evaluate the models, we created a small dataset with job descriptions, resumes, and the expected cosine similarity (.9 for good pairs and .1 for poor pairs). We then generated the embeddings and ran brute force kNN, recording the number of documents that appear in the result set with the expected rank. The results were evaluated with top_1 and top_k accuracy ∈[0, 1]. In the future, we would like to extend our evaluation utility to use the normalized discounted cumulative gain (NDCG) method.

To fine-tune our models, we followed the example from the SBERT documentation.  The models were trained by creating (job_description, resume, score) tuples and using cosine similarity loss for n number of epochs. From our experiments, the training data’s size and quality are essential for successful tuning, and a large number of training epochs is required for the model to return top_k correct results. The table above demonstrates the improvement of the evaluation scores for models trained for 50 and 100 epochs.

Below is the documentation for the fine-tuning utility.

				
					usage: tune.py [-h] -f FILE -o OUTPUT [-m MODEL] [-e EPOCHS] [-d DEVICE]

SentenceTransformer tuning utility

optional arguments:
  -h, --help            show this help message and exit
  -f FILE, --file FILE  path to csv file with training data
  -o OUTPUT, --output OUTPUT
                        output directory for the trained model
  -m MODEL, --model MODEL
                        base model name or path
  -e EPOCHS, --epochs EPOCHS
                        number of epochs
  -d DEVICE, --device DEVICE
                        device to use ("cuda" / "cpu"). If None, checks if a
                        GPU can be used.

Future Work

The next step is to evaluate the impact to relevancy of using vector search The Home Depot dataset from Kaggle is a good candidate for test data because it includes many query/result document pairs that are labeled (numerically ranked as relevant or irrelevant). By training and evaluating our application on this data set, we can further investigate whether pure vector search provides an improvement compared to lexical search.

We will also explore utilizing Quepid to visualize queries, search results, and their scores for a vector search application. Finally, we would like to create a user interface that provides traditional search options like filtering & aggregations; simplifies the queries’ embedding generation; and displays the results in a readable format.

Share Post
More From the KMW Blog

The post Building A Vector Search Application on OpenSearch first appeared on KMW Technology.

]]> 28464 Ingesting Solr Logs with the ELK Stack https://kmwllc.com/index.php/2022/12/17/ingesting-solr-logs-with-the-elk-stack/?utm_source=rss&utm_medium=rss&utm_campaign=ingesting-solr-logs-with-the-elk-stack Fri, 16 Dec 2022 23:01:28 +0000 https://kmwllc.com/?p=28075 When it comes to analyzing Solr logs, Solr does have some out of the box tools. However, we’ve found that those tools don’t give a lot of options for creating rich visual analysis, and don't offer a way to analyze logs in real time. So what do we do? We turn to another open-source platform: Elastic.

The post Ingesting Solr Logs with the ELK Stack first appeared on KMW Technology.

]]>
When it comes to analyzing Solr logs, Solr does have some out of the box tools. However, we’ve found that those tools don’t give a lot of options for creating rich visual analysis, and don't offer a way to analyze logs in real time. So what do we do? We turn to another open-source platform: Elastic.
Picture of Kira Traynor
Kira Traynor

Search Engineer at KMW Technology

Introduction

Have you ever needed to find out more about what’s going on with your Solr deployment?  The Solr Admin UI is great at communicating the overall health of the cluster, how the cores are doing and validating configuration. But sometimes you need to go a bit deeper to understand:

  • What’s my query latency?
  • How long are commits taking?
  • Is Solr throwing any errors?
  • What queries are returning zero results?

When we need to know more, we need to look at the logs.

The ability to analyze log files is foundational to monitoring the success of your Solr cloud deployment. Log files consist of events that are logged with a date, timestamp, event level (warning, error, info, etc.) and event detail. The information contained in the logs give you insight into what is happening within your system. 

At KMW Technology, we focus on utilizing open-source software in our search solutions in order to support and contribute to community-driven development. As such, we have a lot of expertise in working with Solr. When it comes to analyzing Solr logs, Solr does have some out of the box tools. However, we’ve found that those tools don’t give a lot of options for creating rich visual analysis. We’ve also found there’s no great way to analyze logs in real time. So what do we do? We turn to another open-source platform: Elastic. Using Elastic’s ELK stack, we can ingest Solr log files and leverage tools like Kibana to query and visualize what’s happening in Solr.

In this post, we’ll go over how to use Elasticsearch and its tools within the ELK stack to query, analyze and visualize your Solr logs. It’s easier than you might think!

The ELK Stack

The four components of the ELK stack are: Elasticsearch, Logstash, Kibana, and Beats. The process begins with Beats, a platform with multiple different data shippers. Filebeat is one of these data shippers, and setting up Filebeat is the first step to ingesting your logs. Once Filebeat is hooked into your Solr logs, your log data can then be shipped to Logstash which will ingest the logs. When Logstash ingests and parses the log data, create an index in Elasticsearch and add the logs to this index. Once Elasticsearch has all the data, you can use Kibana to query your log data and create visualizations that aid in your analysis.

The Process

Installing & Configuring ELK

Install the following products:

Ensure that each of the downloaded products are compatible with each other, i.e. all have the version 8.4.2. You can find the compatibility matrix here.

Pointing Filebeat at Your Logs

Filebeat’s role will be to monitor the files that are in a defined input location and send them to a defined output location. In this case, the input will be the path to your Solr logs and the output will be Logstash.

You can choose to either actively monitor your logs in real time or ingest a saved set of logs that came from a certain time period. In either scenario, the Filebeat setup will be the same. However, if you are not monitoring your logs in real time Filebeat only has to run once and can be terminated when it has finished. Otherwise, Filebeat should be left running so that it can continue to send log updates in real time.

Within the downloaded Filebeat package, find filebeat.yml.

  1. Add the path(s) to your Solr logs under the filebeat.inputs section and set enabled to true. You can use glob to match multiple logs.
				
					filebeat.inputs:
- type: log
  # Change to true to enable this input configuration.
  enabled: true
  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /Downloads/Logs/SolrLogs/solr.log*
  1. Under the Kibana section, make sure that the Kibana host is set to your specific Kibana host. You do not need to set anything for the Elasticsearch output.
				
					setup.kibana:
  # Kibana Host
  host: "localhost:5601"
  1. Since we want to connect it to Logstash, set the output accordingly. Make sure output.elasticsearch is not set to anything and output.logstash is set:
				
					output.logstash:
  # The Logstash hosts
  hosts: ["0.0.0.0:5044"]

Note: there are other sections of Filebeat that can be configured, but for this example we are leaving these sections set according to the values that are pre-loaded when you first install Filebeat.

Configuring Logstash

Logstash is an ETL tool that requires some initial configuration. It is the pipeline that takes files from Filebeat, ingests and transforms the data so that it can be indexed, and sends it to Elasticsearch to be made searchable. 

Logstash has to be configured accordingly to ensure that the data that is in your logs is captured and made searchable as fits your needs. This means identifying what content from the Solr log files is important to retain and what might not be necessary. In the below example, we will walk through what we consider a basic Logstash configuration for Solr log ingestion– but be aware that your use case might be different.

Setting up the Pipeline

The first step is to set up the pipeline for Logstash. As you’ll see below, you will use Grok to match and filter the content in the logs. Grok is similar to regular expression in that it is a search pattern that can be matched to text. This will allow you to set values to fields. Some documentation and examples of Grok statements from Elasticsearch can be found here.

Within the installed Logstash package, locate the conf/logstash-sample.conf file. There should be an inputs and an outputs section.

  1. Add a filter section after the inputs. This will be where you can create Grok statements to filter and match the data that you want from your logs. The following code matches the time and log level from Solr logs and sets those values to the LogTime and level fields. Use the Grok debugger from Kibana to check if the Grok statements are matching the correct data.

     Additionally, you can add Grok that looks like the example below, which will match the basic log configuration for Solr logs. Keep in mind you can also match error level logs and garbage collection logs.
				
					 if "INFO" in [level] {
   grok {
     match => [
       "message", "%{DATESTAMP} %{LOGLEVEL} (%{DATA}) \[(c:%{DATA:collection}| ) (s:%{DATA}|)\] %{DATA} \[%{WORD:core_node_name_s}\] %{SPACE} webapp=\/?%{WORD:webapp} path=%{DATA:path_s} params=\{%{DATA:params}\} status=%{NUMBER:status_i} QTime=%{NUMBER:qtime_i}",
       "message", "%{DATESTAMP} %{LOGLEVEL} (%{DATA}) \[(c:%{DATA:collection}| ) (s:%{DATA}|)\] %{DATA} \[%{WORD:core_node_name_s}\] %{SPACE} webapp=\/?%{WORD:webapp} path=%{DATA:path_s} params=\{%{DATA:params}\} hits=%{NUMBER:hits_i} status=%{NUMBER:status_i} QTime=%{NUMBER:qtime_i}",
       "message", "%{DATESTAMP} %{LOGLEVEL} (%{DATA}) \[(c:%{DATA:collection}| ) (s:%{DATA}|)\] %{DATA} \[%{WORD:core_node_name_s}\] %{SPACE} webapp=\/?%{WORD:webapp} path=%{DATA:path_s} params=\{%{GREEDYDATA:params}\} %{NUMBER:status_i} %{NUMBER:qtime_i}"
     ]
     tag_on_failure => []
   }
   if [params] {
     kv {
       field_split_pattern => "&|}{"
       source => "params"
     }
}
  1. Configure the output.
    1. Set Elasticsearch host to "https://localhost:9200"“
    2. Set the template to the path. We will set up the template (mapping) after this. 
    3. Set index name
    4. Set the user and password from your elasticsearch or ssl_certificate_verification

Defining the Mappings (Index Template)

Since you’re familiar with Solr, you know that a collection schema declares the fields and corresponding data types per field. In Elasticsearch, a schema is referred to as a mapping, and the mapping is applied to a specific index. A Logstash index template is needed in order to define the mappings that Elasticsearch will use to create an index of your Solr log files.

Keep in mind that index templates are only applied at index creation or during a re-index.

If you don’t specify mappings for each of the fields that you are matching from the Grok statements, Elasticsearch will still ingest the logs and assume a type for each field. This can be problematic if Elasticsearch assumes the wrong field type. For example, if a field with an integer type is ingested as a string type you will not be able to represent it in the correct way in a Kibana graph using minimums, maximums, averages or other mathematical operations.

To start making a template, create a JSON file using the example template below. The name and path has to be whatever you set your template to in the pipeline above. There are two main components to the index template.

  1. Include your index_patterns to match the indices you want.
  2. Include the mappings map. This part includes the mapping of the fields to their data type. Since you already set up your pipeline in the section above, you know what fields you are matching from the logs. For each of the fields, determine what data type they should be. For example, if you are matching query time values in your log and you called the field qtime_i because it is an integer value, you should add this to your mappings. From Elasticsearch’s documentation, here are all the different types that you can include in your mapping. Each field can only have one data type.
				
					"qtime_i":{
    "type": "integer",
    "fields":{
        "keyword":{
            "type": "keyword", 
            "ignore_above": 256
        }
    }
}

 

The whole template will look something like what is below.

				
					{
 "template": "solr-logs-template",
 "index_patterns": ["solr-logs*"],
 "mappings" : {
   "properties" : {
     "@timestamp" : {
       "type" : "date"
     },
     "@version" : {
       "type" : "text",
       "fields" : {
         "keyword" : {
           "type" : "keyword",
           "ignore_above" : 256
         }
       }
     },
     "LogTime" : {
       "type" : "date",
       "format" : "yy-MM-dd HH:mm:ss.SSS",
       "fields" : {
         "keyword" : {
           "type" : "keyword",
           "ignore_above" : 256
         }
       }
     },
     "agent" : {
       "properties" : {
         "ephemeral_id" : {
           "type" : "text",
           "fields" : {
             "keyword" : {
               "type" : "keyword",
               "ignore_above" : 256
             }
           }
         },
         "hostname" : {
           "type" : "text",
           "fields" : {
             "keyword" : {
               "type" : "keyword",
               "ignore_above" : 256
             }
           }
         },
         "id" : {
           "type" : "text",
           "fields" : {
             "keyword" : {
               "type" : "keyword",
               "ignore_above" : 256
             }
           }
         },
         "type" : {
           "type" : "text",
           "fields" : {
             "keyword" : {
               "type" : "keyword",
               "ignore_above" : 256
             }
           }
         },
         "version" : {
           "type" : "text",
           "fields" : {
             "keyword" : {
               "type" : "keyword",
               "ignore_above" : 256
             }
           }
         }
       }
     },
     "commit" : {
       "type" : "text",
       "fields" : {
         "keyword" : {
           "type" : "keyword",
           "ignore_above" : 256
         }
       }
     },
     "core_node_name_s" : {
       "type" : "text",
       "fields" : {
         "keyword" : {
           "type" : "keyword",
           "ignore_above" : 256
         }
       }
     },
     "ecs" : {
       "properties" : {
         "version" : {
           "type" : "text",
           "fields" : {
             "keyword" : {
               "type" : "keyword",
               "ignore_above" : 256
             }
           }
         }
       }
     },
     "file" : {
       "type" : "text",
       "fields" : {
         "keyword" : {
           "type" : "keyword",
           "ignore_above" : 256
         }
       }
     },
     "fl" : {
       "type" : "text",
       "fields" : {
         "keyword" : {
           "type" : "keyword",
           "ignore_above" : 256
         }
       }
     },
     "hits_i" : {
       "type" : "integer",
       "fields" : {
         "keyword" : {
           "type" : "keyword",
           "ignore_above" : 256
         }
       }
     },
     "host" : {
       "properties" : {
         "architecture" : {
           "type" : "text",
           "fields" : {
             "keyword" : {
               "type" : "keyword",
               "ignore_above" : 256
             }
           }
         },
         "hostname" : {
           "type" : "text",
           "fields" : {
             "keyword" : {
               "type" : "keyword",
               "ignore_above" : 256
             }
           }
         },
         "id" : {
           "type" : "text",
           "fields" : {
             "keyword" : {
               "type" : "keyword",
               "ignore_above" : 256
             }
           }
         },
         "name" : {
           "type" : "text",
           "fields" : {
             "keyword" : {
               "type" : "keyword",
               "ignore_above" : 256
             }
           }
         },
         "os" : {
           "properties" : {
             "build" : {
               "type" : "text",
               "fields" : {
                 "keyword" : {
                   "type" : "keyword",
                   "ignore_above" : 256
                 }
               }
             },
             "family" : {
               "type" : "text",
               "fields" : {
                 "keyword" : {
                   "type" : "keyword",
                   "ignore_above" : 256
                 }
               }
             },
             "kernel" : {
               "type" : "text",
               "fields" : {
                 "keyword" : {
                   "type" : "keyword",
                   "ignore_above" : 256
                 }
               }
             },
             "name" : {
               "type" : "text",
               "fields" : {
                 "keyword" : {
                   "type" : "keyword",
                   "ignore_above" : 256
                 }
               }
             },
             "platform" : {
               "type" : "text",
               "fields" : {
                 "keyword" : {
                   "type" : "keyword",
                   "ignore_above" : 256
                 }
               }
             },
             "version" : {
               "type" : "text",
               "fields" : {
                 "keyword" : {
                   "type" : "keyword",
                   "ignore_above" : 256
                 }
               }
             }
           }
         }
       }
     },
     "input" : {
       "properties" : {
         "type" : {
           "type" : "text",
           "fields" : {
             "keyword" : {
               "type" : "keyword",
               "ignore_above" : 256
             }
           }
         }
       }
     },
     "level" : {
       "type" : "text",
       "fields" : {
         "keyword" : {
           "type" : "keyword",
           "ignore_above" : 256
         }
       }
     },
     "log" : {
       "properties" : {
         "file" : {
           "properties" : {
             "path" : {
               "type" : "text",
               "fields" : {
                 "keyword" : {
                   "type" : "keyword",
                   "ignore_above" : 256
                 }
               }
             }
           }
         },
         "flags" : {
           "type" : "text",
           "fields" : {
             "keyword" : {
               "type" : "keyword",
               "ignore_above" : 256
             }
           }
         },
         "offset" : {
           "type" : "long"
         }
       }
     },
     "message" : {
       "type" : "text",
       "fields" : {
         "keyword" : {
           "type" : "keyword",
           "ignore_above" : 256
         }
       }
     },
     "params" : {
       "type" : "text",
       "fields" : {
         "keyword" : {
           "type" : "keyword",
           "ignore_above" : 256
         }
       }
     },
     "path_s" : {
       "type" : "text",
       "fields" : {
         "keyword" : {
           "type" : "keyword",
           "ignore_above" : 256
         }
       }
     },
     "q" : {
       "type" : "text",
       "fields" : {
         "keyword" : {
           "type" : "keyword",
           "ignore_above" : 256
         }
       }
     },
     "qt" : {
       "type" : "text",
       "fields" : {
         "keyword" : {
           "type" : "keyword",
           "ignore_above" : 256
         }
       }
     },
     "qtime_i" : {
       "type" : "integer",
       "fields" : {
         "keyword" : {
           "type" : "keyword",
           "ignore_above" : 256
         }
       }
     },
     "rows" : {
       "type" : "integer",
       "fields" : {
         "keyword" : {
           "type" : "keyword",
           "ignore_above" : 256
         }
       }
     },
     "status_i" : {
       "type" : "integer",
       "fields" : {
         "keyword" : {
           "type" : "keyword",
           "ignore_above" : 256
         }
       }
     },
     "tags" : {
       "type" : "text",
       "fields" : {
         "keyword" : {
           "type" : "keyword",
           "ignore_above" : 256
         }
       }
     },
     "version" : {
       "type" : "text",
       "fields" : {
         "keyword" : {
           "type" : "keyword",
           "ignore_above" : 256
         }
       }
     },
     "webapp_s" : {
       "type" : "text",
       "fields" : {
         "keyword" : {
           "type" : "keyword",
           "ignore_above" : 256
         }
       }
     },
     "threads_stopped_for_seconds_i" : {
       "type" : "float",
       "fields" : {
         "keyword" : {
           "type" : "keyword",
           "ignore_above" : 256
         }
       }
     },
     "wt" : {
       "type" : "text",
       "fields" : {
         "keyword" : {
           "type" : "keyword",
           "ignore_above" : 256
         }
       }
     }
   }
 }
}

Running ELK

Now that setup has been completed, you’re ready to run all four parts: Elasticsearch, Kibana, Logstash, Filebeat.

  • Run Elasticsearch.  From the Elasticsearch package run [./bin/elasticsearch]. Elasticsearch will be found at [https://localhost:9200]
  • Running Kibana may be useful during the configuration of Logstash for the Grok debugger. Be aware that you need to have Elasticsearch running in order to run Kibana. From the Kibana package run [./bin/kibana]. Kibana will be found at [http://localhost:5601].This is where you will be doing the data querying and visualization.
  • Run Logstash to create the index. From the Logstash package run [./bin/logstash -f logstash.conf] where logstash.conf is the configuration file we created above.
  • Run Filebeat to monitor the logs and send to Logstash. From the Filebeat package run [./filebeat -e]. 

Filebeat and Logstash only need to run once unless you are monitoring logs in real time.

Querying and Visualizing

Goals for Analyzing Logs

Depending on the use case, you can focus on different things when querying and visualizing logs. Some questions can be answered by querying Kibana, while in other circumstances setting up a visualization is more helpful. Since you are already interested in log analysis you probably have some specific metrics in mind, but some common analysis goals include:

  • Seeing how long garbage collection takes
  • Knowing how many searches have been run over a given time (per minute/hour/week)?
  • Identifying the queries that take the longest time to execute
  • Identifying most common queries issued to a collection
  • Visualizing spikes in query traffic 
  • Seeing how often commits are occurring

Example Queries against Elasticsearch Index

Querying against your newly created index is simple with Kibana. Go to the menu on the top left and scroll all the way down to Management/Dev Tools. From here you can create queries in the Console and test your Grok statements in the Grok Debugger. For help understanding the specific query syntax, here is some documentation.

There are some simple queries that you can use to start off with:

To get all the indices so you can ensure the index was created: 

				
					GET _cat/indices

 

To get all information from one index:  

				
					GET /<index-name>/_search 
{
  "query": {
    "match_all": {}
  }
}

 

Finding the longest-running search:

				
					GET /<index-name>/_search 
{
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "path_s.keyword": {
              "value": "/select"
            }
          }
        }
      ]
    }
  },
  "aggs": {
    "doc_with_max_qTime": {
      "top_hits": {
        "sort": [
          {
            "qtime_i": {
              "order": "desc"
            }
          }
        ],
        "size": 1
      }
    }
  },
  "size": 0
}

 

Finding the most common query:

				
					GET /<index-name>/_search 
{
  "aggs": {
    "frequent_query": {
      "terms": {
        "field": "q.keyword"
      }
    }
  },
  "size": 0
}

 

Finding percentiles (aggregating on query time):

				
					GET /<index-name>/_search 
{
  "aggs": {
    "qTime_percentiles": {
      "percentiles": {
        "field": "qtime_i",
        "percents": [
          90,
          95,
          99
        ]
      }
    }
  },
  "size": 0
}

Example Visualizations with Kibana

Once you understand the data that you are looking at, you can create a dashboard with visualizations. Create a visualization by navigating to Analytics -> Discover or Analytics  -> Dashboard if you already know what you’d like to make up a dashboard.

The ability to create visualizations with Kibana is one of our favorite reasons to look at Solr logs using Elastic stack. It is easy to create dashboards that convey a lot of information in an easily digestible manner. While it is possible to use grep commands in a console to see commits per collection, you can see in this example that a visualization is a lot easier to understand than the results you would get from grep.

Conclusion

While there is a bit of up-front work required with this approach, the payoff is having a great way to look at your Solr logs both in real time or as needed. Let us know what type of questions you hope to answer when looking at your Solr logs, and if you have other approaches that you prefer.

If you’re experiencing issues with your Solr (or Elasticsearch, or Opensearch) cluster or need help interpreting your logs, please contact us! 

Share post
More From the KMW Blog

The post Ingesting Solr Logs with the ELK Stack first appeared on KMW Technology.

]]> 28075 Solr’s query elevation component now supports filter exclusions https://kmwllc.com/index.php/2022/11/17/solrs-query-elevation-component-now-supports-filter-exclusions/?utm_source=rss&utm_medium=rss&utm_campaign=solrs-query-elevation-component-now-supports-filter-exclusions Thu, 17 Nov 2022 15:06:37 +0000 https://kmwllc.com/?p=27467 New in Solr 9.2! We created a way for the Query Elevation Component to exclude filters. Read about how we did this and what you should know about this new feature.

The post Solr’s query elevation component now supports filter exclusions first appeared on KMW Technology.

]]>
New in Solr 9.2! We created a way for the Query Elevation Component to exclude filters. Read about how we did this and what you should know about this new feature.
Picture of Rudi Seitz
Rudi Seitz

Solr Contributor & Senior Search Engineer at KMW Technologoy

The Problem

If you’ve ever needed to editorially override the top results for a Solr query, you’ve probably looked at the Query Elevation Component (QEC). Using QEC, you can indicate that certain documents should appear as top results for a given query, even if those documents would have had a lower position based on natural scoring, or would have been absent entirely.

In Solr 9.1 and before, filters always took precedence over elevation. For example, you might have configured QEC to return the document with id=1 whenever a user searched for foo. However, if the query also included an “in stock” filter, like this:

q=foo
fq=in_stock:true

then id=1 would only be elevated if it happened to be in stock.

Of course, this might have been the behavior you wanted. But what if you needed to elevate out-of-stock items too – maybe so you could accept preorders? We had a customer who wanted to support this use case – applying an fq to non-elevated documents but bypassing the same fq for elevated documents. There wasn’t a way to do this, so we implemented the feature.

The Solution in Solr 9.2

Starting in Solr 9.2, QEC supports filter exclusions. You can use the following syntax to assign tags to specific filters and to indicate that QEC should let elevated documents bypass those tagged filters.

q=foo
fq={!tag=t1}in_stock:true
elevate.excludeTags=t1

The example above assigns the tag t1 to the “in stock” filter and excludes it for elevated documents. Note that the syntax is similar to the way you can tag and exclude filters while faceting.

In the rest of this post, we’ll discuss the implementation details of QEC’s new filter exclusion feature. To understand those details, we first need to understand QEC’s basic design.

QEC Design Background

If you were building the Query Elevation Component from scratch, your first thought might be to use an additive approach. The component would run the user’s query to get an initial result set. Then it would run a second query to retrieve the elevated documents. Finally it would merge those two results sets, placing the elevated documents on top.

There are a few drawbacks to this possible design. First, we’d be incurring the overhead of running two queries instead of one. Second, we’d have to find a way of preventing duplicates. We don’t want to include any document in the result set twice, so we’d have to figure out if an elevated document existed in the original result set before we could add it. If an elevated document turned out to be present already, we’d need a way of moving it to the top. And third, we’d need to find a way of inserting the elevated documents into facets as well as the primary result set.

To avoid all these complications, QEC takes a different approach:

  1. It broadens the user’s original query to make sure that it matches all the elevated documents. The broadened query is a Boolean OR of the original query with a disjunction across the elevated document IDs. So if the original query was q=XYZ, the new query would be something like q=XYZ OR (id:1 OR id:2 OR id:3).
  2. It adds a new sort criterion to the query that makes the elevated docs appear at the top of the sort order.

This approach allows QEC to achieve its goals with a single query, eliminating the complex piecing-together of multiple queries. But earlier versions of QEC applied this strategy to the q parameter only, leaving all fq instances unmodified. Since fq always take precedence over q in Solr, elevated documents still had to match the filters in order to be included.

Implementing Exclusions

To improve QEC so that elevated documents can bypass specific filters, we can reuse the same strategy that QEC applies to the primary query. Indeed, that’s how our Solr 9.2 changes work. To “exclude” a given filter, we transform it into a Boolean OR of the original filter with a disjunction across the elevated document IDs. So a filter like fq=a:b would become

fq=a:b OR (id:1 OR id:2 OR id3)

where and 1, 2, 3 are the IDs of the documents that should be elevated for the incoming q. It’s important to clarify we’re not removing or disabling the filter altogether; rather, we’re broadening it to let the elevated documents through.

Caching Considerations

There are some subtleties that come up as we try to make this new feature as good as it could be. One of the advantages of using filters in Solr is that they can be very fast because they can take advantage of the filter cache. We’d hope to still benefit from filter caching when using QEC with excluded filters.

But if the user’s original filter was fq=a:b and it’s in the cache, we’re still going to get a cache miss the first time we execute the modified filter fq=a:b OR (id:1 OR id:2 OR id:3).

And even if the modified filter eventually gets cached, the set of elevated documents can change for different values of q, so the next time the filter is applied it might be modified as fq=a:b OR (id:5 OR id:6 OR id:7).

As you can see, we could start filling up the filter cache with different variants of the original filter, still without any guarantee of a cache hit for our fq if the accompanying q hasn’t been seen before.

Fortunately, Solr has a mechanism for decomposing a filter query into separate clauses that can be cached independently. This mechanism is exposed to users via the filter() syntax. If you have a filter like a:b AND c:d, you can write:

fq=filter(a:b) AND filter(c:d)

This means that a:b and c:d each get their own entries in the filter cache. If we execute this fq, and later execute a different fq=filter(a:b) AND filter(e:f), we can read the first clause a:b from the cache, even though the second clause is different.

What this means for QEC is that when we’re modifying a filter like fq=a:b to allow the elevated documents through, we can mark the original filter for independent caching. QEC will transform the original fq into the equivalent of this:

fq={!cache=false}filter(a:b) OR (id:1 OR id:2 OR id:3)

Here are the key points to notice about this strategy for modifying the filter:

  1. We set the entire modified filter to be non-caching. This prevents the cache from filling up with variants of the same filter with different sets of elevation IDs.
  2. We wrap the user’s original fq in filter() syntax to guarantee that it is always cached as an independent clause.
  3. We don’t wrap the elevation IDs in filter() syntax. The thinking is that a simple set of doc IDs is fast enough that it doesn’t benefit much from being cached.

There are a few other details to consider:

  1. If the user had set their filter to be non-caching via {!cache=false}then we respect this and we don’t wrap their original filter in filter() syntax.
  2. If the user had already wrapped their filter in filter() syntax, we don’t doubly wrap it.
  3. If the user had associated a cost with a filter via fq={!cost=120} then we copy this cost to the top level of the new, broadened filter.

Conclusion

Editorial boosting is a common use case in search, but Solr’s Query Elevation Component lacked the flexibility to handle scenarios where documents should be elevated “no matter what.” We  hope the new support for filter exclusions in Solr 9.2 will make QEC usable in a wider range of scenarios, in a way that maintains good performance.

For further details, see: SOLR-16496.

Share Post
More from the KMW Blog

The post Solr’s query elevation component now supports filter exclusions first appeared on KMW Technology.

]]> 27467 The KMW Search Audit https://kmwllc.com/index.php/2022/09/30/the-kmw-search-audit/?utm_source=rss&utm_medium=rss&utm_campaign=the-kmw-search-audit https://kmwllc.com/index.php/2022/09/30/the-kmw-search-audit/#comments Thu, 29 Sep 2022 22:09:09 +0000 https://kmwllc.com/?p=26659 Learn about our most popular service, where we take a deep dive into what may not be working for your Solr, Elasticsearch, or OpenSearch instance.

The post The KMW Search Audit first appeared on KMW Technology.

]]>

KMW Technology is a leading provider of Search consulting and professional services.  For over a decade, KMW Technology has worked with some of the largest names in e-commerce, financial services, life sciences, higher education and IT to resolve their complex search problems and unlock new opportunities.  The KMW team brings a wide range of search experience and a deep knowledge of Search internals to ensure successful projects that bring value to customers.

KMW’s most popular service offering is the KMW Search Audit – a quick time-to-value audit that covers all aspects of a Search Platform in order to realize the maximum value from a Search-based deployment.  The KMW Search Audit focuses on quickly resolving pain points, simplifying operations and unlocking new opportunities by leveraging our accumulated experience, industry best practices and the latest offerings from the open source community.  The KMW Search Audit is best suited for organizations currently running a search solution in production, but don’t believe they are getting everything they can out of their search platform.

The KMW Search Audit is available for organizations using OpenSearch, Solr or  Elasticsearch as their search platform.

Customer issues addressed by a KMW Search Audit include:

  • Cluster instability
  • Query throughput and latency issues
  • Relevancy issues / poor recall or precision
  • Ingestion / index latency concerns
  • Planning for a Search upgrade or major platform update
  • Problematic Hardware/cluster sizing and/or scaling issues
  • Defunct operational practices

THE SEARCH AUDIT OFFERING

The KMW Search Audit is a deep dive into an existing Search platform performed by technical experts. A team of KMW consultants will begin by reviewing the search engine configurations, schemas/mappings, query logs, plugins, cluster sizing, web application features and full application architecture. Each Search Audit is tailored to the customer’s specific needs to ensure maximum value for the engagement. Subject matter experts within the customer’s organization are interviewed to identify specific pain points and areas of interest to ensure that the audit addresses the top concerns.

Areas of Focus

Architecture

  • Search engine configuration
  • Schema / mappings and analysis chains
  • Custom plugin review
  • Indexing and data transformations
  • Content enrichment and entity extraction
  • Query syntax review
  • Scaling and data volume review
  • Disaster recovery planning
  • Fault Tolerance / High Availability
  • Security (SSL / Authentication / Authorization)

User Experience

  • Search UI review
  • Query parameter reviews
  • Spell checking
  • Autocomplete / Typeahead
  • Faceting and filtering
  • Teasers (static and dynamic)
  • User behavior tracking
  • Relevancy tuning
  • User/customer specific synonym dictionaries and management
  • Multi-language search and localization

Infrastructure & Operations

  • Query volume and sizing
  • Update/Commit volume and sizing
  • Monitoring
  • Query log analytics
  • Index snapshot/backup
  • CI/CD & DevOps
  • Cluster sizing and scaling

Product / IT Roadmap Review

  • Review of roadmap features / goals
  • Map existing search engine features to roadmap
  • Recommendations / implementation strategies
  • Sizing of features – Levels of Effort
  • Assistance with planning and prioritization

 What’s Delivered

The key deliverable from the KMW Search Audit is a consulting report containing  all the key findings & recommendations from the audit. Recommendations are detailed to include a summary of their positive impact and include any associated config samples, links or commentary.  The recommendations are also scored according to complexity of implementation and impact to the business to help with prioritization and planning.  

Search Audits have led to impactful findings and recommendations for KMW customers.  Actual customer outcomes have included:

  • Decreased operational costs due to proper sizing of search clusters to take better advantage of the hardware.
  • Recommended memory and configuration optimizations fixed search cluster instability and prevented severity 1 issues for the prod support team.
  • Decrease in average query latency due to enhanced use of autowarming and caching
  • Increased stability of Production search clusters by implementing a unified commit strategy
  • Major improvements to relevancy and precision by replacing wildcards and fuzzy search operators with ngram filters
  • Identification of most engaged user segments and geo-location from query logs
  • Resolution of top queries that previously returned zero search results

If you think a Search Audit would be a good fit for your organization, contact us to learn more.

Contact Us – Search Audit

The post The KMW Search Audit first appeared on KMW Technology.

]]> https://kmwllc.com/index.php/2022/09/30/the-kmw-search-audit/feed/ 3 26659 Search Engine Upgrade https://kmwllc.com/index.php/2022/07/02/search-engine-upgrade/?utm_source=rss&utm_medium=rss&utm_campaign=search-engine-upgrade https://kmwllc.com/index.php/2022/07/02/search-engine-upgrade/#comments Fri, 01 Jul 2022 19:00:41 +0000 https://kmwllc.com/?p=26507 Open-source search engines are constantly being updated to add features, improve existing features, and fix vulnerabilities. Here are some more reasons you would want to update accordingly and how we can help.

The post Search Engine Upgrade first appeared on KMW Technology.

]]>

Open source search engines are constantly being updated. New releases come out on a regular basis.  It’s easy for companies to fall behind on releases because if the search engine is working, why touch it?   Open source communities put a lot of effort into ensuring that projects like Solr, Elasticsearch and OpenSearch address issues like security vulnerabilities and bug fixes that are reported, and these improvements accumulate with each release.  The longer companies wait to upgrade, the harder it is to upgrade.  One example is that data formats can change causing unexpected incompatibilities. It’s especially important to consider upgrading frequently to ensure data formats don’t cause unexpected incompatibilities and that custom plugins continue to work.

Over time, software interfaces change. This  can make old plugin code incompatible without some code changes.  Usually these changes are pretty small, even non-existent, between minor point releases.  However, the changes in a minor release have a cumulative effect, eventually manifesting in a significant API shift between major release versions. The net result is that the level of effort to maintain the custom plugins across major version releases only grows over time and could result in having to re-implement those plugins.  Upgrading frequently and keeping up to date with the latest release will allow the business to identify breaking API changes early, when the open source community has made a change to the project.

Old software that works is still vulnerable to new security exploits that occur.  Hackers are constantly looking for attack vectors in software that can be exploited and potentially expose sensitive data.  The older the software is, the longer the hackers have to analyze the code to identify the vulnerabilities and leak them to the dark web.  It’s possible that the search engine that is currently running has many well known vulnerabilities that can leave a business’s data exposed to hackers that want to take over the servers and exploit them for their own illegal means.

Another benefit to upgrading your search engine technology is to take advantage of bug fixes submitted by the open source community. These fixes and improvements  may allow your  business to remove some work-arounds that were put in place to address the previous limitations. Removing technical debt makes the search engine easier and cheaper to maintain in the long run.  

It’s also entirely possible that new features might exist in the release that eliminate the need to have custom code. This also decreases the complexity and cost of ownership, especially when custom plugins are at play.

KMW Technologies has a long history of performing successful upgrades and migrations of search engine platforms.  If you’d like to learn more, contact us below for more information on how you can keep your technology stack up to date.

Contact Us – Search Audit

The post Search Engine Upgrade first appeared on KMW Technology.

]]> https://kmwllc.com/index.php/2022/07/02/search-engine-upgrade/feed/ 1 26507