Improving Retrieval Quality - Hybrid Search, Filtering, Reranking, and Tool Loops

A vector store does not automatically produce high-quality retrieval. It provides one way to compare the meaning of a query with the meaning of stored chunks. That is useful for paraphrases and conceptual questions, but it is weaker when the user provides an exact model number, a product code, a technical term, or a rare phrase that should match literally.

Keyword search has the opposite profile. It is strong when the same words appear in the query and the document. It can identify exact identifiers, part numbers, names, and uncommon phrases. It is weaker when the user paraphrases the source or uses a related concept rather than the same vocabulary.

Hybrid search combines the two. The retrieval pipeline runs semantic vector search and lexical full-text search, then merges the candidate sets. The purpose is not to choose one universal winner. It is to let each method contribute where it is strongest.

A practical hybrid pipeline works in stages:

1. Normalize the user query.
2. Run vector similarity search against chunk embeddings.
3. Run keyword or full-text search against chunk content.
4. Collect a candidate set from both methods.
5. Fuse the rankings.
6. Optionally rerank the fused candidates with a separate model.
7. Return the best evidence to the main agent.

The vector search score and keyword rank are not directly comparable. Reciprocal rank fusion solves this by combining positions rather than raw scores. A result that ranks highly in either list receives a useful contribution. A result that ranks well in both receives a stronger combined score. This is valuable because the semantic and lexical systems may use different score scales.

Reciprocal rank fusion is a candidate-merging method, not a final proof of relevance. It rewards agreement between rankings, but it does not deeply compare the user question with the full candidate passage. That is the role of reranking.

A reranker receives the query and a candidate passage together. It assigns a relevance score based on the pair. This second-stage evaluation is more expensive than the first search, so it should run only on a limited candidate set. Running a reranker across every chunk in the knowledge base would be slow and unnecessary.

The application can support a cloud reranker such as Cohere and a local reranking endpoint. A Qwen reranking model is one local option used in the build. The administrator should be able to enable or disable reranking and configure the provider, model, URL, and credential.

The trace should show whether reranking actually occurred. Before the reranker was configured, a search result exposed similarity and reciprocal-rank-fusion values. After the reranker was enabled, the trace also contained a rerank score. This is stronger evidence than assuming the feature worked because the final answer looked plausible.

A useful result record can expose:

• Vector similarity score.
• Keyword or lexical rank.
• Reciprocal-rank-fusion score.
• Reranker score when enabled.
• Document identifier.
• File name.
• Chunk index.
• Selected metadata.

The application does not need to display every score to ordinary users, but operators need them for evaluation and debugging.

Hybrid search improves retrieval, but metadata filters can still override it. A hard filter is applied before or during candidate selection. If the agent chooses the wrong document type or topic, the correct chunk never reaches the fusion or reranking stage.

The distinction between ranking and filtering is important:

• Ranking orders candidates by estimated relevance.
• Filtering removes candidates from consideration.

When the system is uncertain, ranking is safer than filtering. A broad search can retrieve several candidates and let the reranker decide. A guessed filter can return zero results.

A practical filter policy should distinguish explicit constraints from inferred constraints.

Explicit constraints include a product number supplied by the user, a clearly named language, or a known document category. Inferred constraints include a guessed topic or document type based only on the wording of the question. Explicit values can be applied early when the metadata is reliable. Inferred values should be used cautiously or only after an initial broad search.

The agent prompt can include rules such as:

• Do not apply metadata filters unless the user supplied the value or previous retrieval established it.
• Use exact identifiers when available.
• Avoid guessing document type.
• Start broad when the correct category is uncertain.
• Retry without filters when a filtered search returns no results.
• Do not conclude that the knowledge base lacks information until a broader search has been attempted.

This policy addresses the Samsung smart-home example. The topic value was correct, but the guessed document type was wrong. A fallback search without that document-type filter would have recovered the document.

Search mode should also be explicit. The tool can accept semantic, keyword, or hybrid mode. Hybrid can be the default, but there are cases where a specific mode is useful. An exact identifier may justify keyword-heavy retrieval. A conceptual question may benefit from semantic search. The agent can choose, but the system prompt should discourage unnecessary mode switching without evidence.

Retrieval quality also depends on the query passed to the tool. The user message is not always the best search query. A user may ask, "Can this be used outside?" after discussing a specific oven. The retrieval query needs the appliance identity from conversation history. The agent should combine the follow-up with the known subject rather than search the entire knowledge base for the word "outside."

Agentic retrieval allows multiple tool calls. The main agent can search by product code, inspect the top document, search within the topic, and then delegate full-document analysis. The system prompt should tell the agent to develop and execute a retrieval strategy rather than assume one tool call is sufficient.

A retrieval strategy may include:

• One broad hybrid search to identify the document.
• A second search using an exact product identifier.
• A filtered search after metadata is confirmed.
• A whole-document analysis after the correct file is known.
• A SQL query for structured facts.
• A web search when the information is outside the private corpus.

Tool loops need a limit. An early implementation used a maximum of three rounds. That can be too small for a complex request. The limit was made configurable and increased to ten. The system prompt can refer to the active limit and instruct the model to finish its research and answer before the limit is reached.

A higher limit is not a license to search indefinitely. Some models, especially smaller local models, can repeat similar searches. The application should detect the end condition and retain a user stop control. Traces should reveal when the same tool and query are being called repeatedly.

Another failure occurs when the model ends on a tool call. It retrieves evidence but never converts the result into a user-facing answer. The application can solve this by issuing one final model call with tools disabled when the last event is a tool result. The final call receives the accumulated evidence and must synthesize the answer.

This creates a bounded-autonomy pattern:

• The model can use tools flexibly.
• The number of tool rounds is limited.
• Repeated behavior is observable.
• The user can stop the process.
• The application guarantees a final synthesis step.

Retrieval evaluation should use realistic questions. A single successful answer is weak evidence. The test set should include:

• Exact product codes.
• Paraphrased questions.
• Similar manuals from different products.
• Questions whose answer spans several chunks.
• Questions that should return no answer.
• Queries with correct metadata.
• Queries with misleading metadata.
• Follow-up questions that depend on conversation history.
• Questions where keyword and semantic search disagree.
• Questions where reranking should change the top result.

The operator should compare the retrieved evidence with the original document, not only judge the fluency of the answer. A fluent answer can still be grounded in the wrong manual.

Observability completes the retrieval loop. For every question, the operator should be able to inspect:

1. The search query generated by the agent.
2. The search mode.
3. The metadata filters.
4. The candidate chunks.
5. The fusion scores.
6. The reranking scores.
7. The evidence placed in context.
8. The final model response.

Without this, retrieval tuning becomes guesswork. With it, a failure can be classified: wrong query, wrong filter, weak candidate generation, poor reranking, missing context, or bad synthesis.

Practical retrieval-quality rules:

• Use semantic and lexical retrieval as complementary signals.
• Fuse rankings before reranking.
• Rerank a limited candidate set, not the whole collection.
• Treat metadata filters as hard exclusions.
• Apply only justified filters.
• Retry broadly after a zero-result filtered search.
• Include conversation context in follow-up search queries.
• Limit tool rounds and force final synthesis.
• Inspect retrieved evidence and scores, not only the final prose.
• Evaluate on difficult, realistic queries with similar documents.