What is hybrid search?

Hybrid search combines keyword search and meaning-based search into one ranked list. Keyword search, also called lexical search, matches the words in a query. Meaning-based search, also called vector or semantic search, matches what the query is about. Each misses what the other finds, so a hybrid system runs both and merges the results.

What separates it from its neighbors is the merge. Keyword search with a synonym list is still one search. Vector search on its own has no exact match. A reranker reorders a single list after the fact. Hybrid search starts from two lists and has to decide how to make them one. Elastic’s explainer notes that the term can cover other pairs, such as text and image search, but keyword with meaning is the common one.

Keyword retrieval breaks each record into terms and builds an index from every term to the records that contain it. Keyword engines commonly rank the matches with BM25, a scoring function that Stephen Robertson and Hugo Zaragoza reviewed in 2009.

BM25 follows three rules. A rare term counts for more than a common one. A term counts for less each time it repeats. In the review’s words, any one term’s “contribution to the document score cannot exceed a saturation point”. And a long record is damped, so it cannot win by size alone.

Two properties make keyword search hard to replace. It finds an exact string such as A320-2741-08, and it can say why a record matched: these are the terms it shares with the query. Its weakness is vocabulary. A field report about an aft door seal weeping fluid shares no word with a ticket about a hydraulic leak on door 4.

Meaning-based search turns each record, or each passage of a long one, into a vector: a list of numbers produced by a language model. Texts about the same thing land close together, whatever words their authors chose. The query becomes a vector the same way, and the search returns the records nearest to it. An index such as HNSW finds those neighbors without comparing the query with every record.

That is right for descriptions and wrong for identifiers. Part 2741-08 and part 2741-09 sit close in meaning, and they are different parts. Without a minimum similarity, a vector search also returns its nearest records when none of them is relevant, and the cutoff has to be tuned. A match is a distance, so it cannot say which words made two records close. Our post on identifiers and similarity shows each of these with invented part numbers.

The two searches score on different scales. A BM25 score has no fixed top, and a vector similarity usually falls between 0 and 1. Elastic’s explainer puts the problem plainly: “The main challenge lies in merging multiple ranked lists into a single, coherent ranking.”

There are two common answers. The first normalizes both scores to the same range and adds them with a weight, so a setting decides how much keyword counts against meaning. It can be tuned well, but the right weight depends on the data and has to be measured.

The second ignores the scores and uses only the ranks. Reciprocal rank fusion, published by Cormack, Clarke and Büttcher in 2009, gives each record 1 / (k + rank) from every list it appears in and adds them up. They fixed k at 60 in a pilot study. In their words, “The constant k mitigates the impact of high rankings by outlier systems.”

In the drawing above, the ticket is second in both lists and scores 1/62 + 1/62, about 0.0323. The field report tops the meaning list but is missing from the keyword list. It scores 1/61, about 0.0164, and drops to fourth. A record that both searches rank well comes out on top.

Fusion rewards agreement, which is not always what the reader needs. In the drawing, the maintenance log for part 2741-09 ranks third for a query about 2741-08. It shares the words fluid loss and sits close in meaning. For a technician it is the wrong part.

So treat identifiers as their own part of the query. Spot them with a pattern for the company’s part numbers, or with a list of known ones. Match them exactly against the field that holds them. Then rank the descriptive part of the query, with both searches, among the records that carry that identifier.

When the identifier matches nothing, return nothing and say so. Offer the close matches as a separate list, marked as close, so the engineer chooses knowingly.

• Judge real queries. Take queries from your search logs, and have people who know the records mark the right answers. 80 to 120 is a workable start.
• Split them by kind. Score identifier queries and descriptive queries separately. An average can hide a ranking that got better at descriptions and worse at part numbers.
• Measure the merge. Compare keyword alone, meaning alone and the merged list on the same queries. NDCG at 10 is a common measure: it rewards putting the best records in the first ten. If the merged list does not beat both, the second search is not earning its cost.
• Judge what is new. A new search surfaces records nobody judged, and unjudged records count as wrong. Judge the new ones near the top before you trust the comparison.
• Check permissions on both sides. Both searches have to run as the person who asked. A record one search may not return must not arrive through the other.

Our post on measuring search relevance works through these measures, and the unjudged records problem, with examples.

In Curiosity Studio, keyword and vector search run in one query over the same graph: see search for developers. Vector matches are added to the keyword results or rerank them, each above a similarity cutoff you set, so exact part numbers still match. The cutoffs are yours to measure against a judged set.

Studio’s similarity engine can merge its sources by weighted score or by reciprocal rank fusion, which its documentation shows with k set to 60. A graph link can be one of those sources, so records that mention the same part number score together, as the post on identifiers shows.

The vectors are stored beside the records they describe. There is no second store to keep in sync, and no second set of permissions: see vector search for developers. Access is checked per record at query time. Studio ships local models that turn text into vectors in process, with no network, and hosted providers work too. The Modern enterprise search whitepaper puts hybrid search among the six layers of search, and What is enterprise search? covers the rest.