Hybrid tool search in Fabric's orchestrator

The first time we connected our orchestrator to a customer's full toolkit, we hit a wall.

They had 15 MCP servers configured. GitHub, Jira, Slack, Linear, Context7 for docs, Firecrawl for scraping and custom internal tools, the works. Every time a user asked a simple question, we were stuffing 77,000 tokens of tool definitions into context.

The response was slow. The cost was astronomical. And half the time, the LLM picked the wrong tool anyway because it was drowning in options.

We needed a smarter approach.

---

The Challenge: Finding a Needle in a Haystack of Tools

Modern AI applications don't just generate text, they take action. They create Jira tickets, query databases, send Slack messages, and scrape websites. But as the number of available tools grows, a critical question emerges:

How do you efficiently find the right tool among hundreds of options without burning through your token budget?

At Fabric, our orchestrator manages tools from:

• MCP Servers (Model Context Protocol) - user-configured tool providers
• Registered Agents - specialized AI agents with specific capabilities
• Integrations - Slack, GitHub, Linear, Notion, and more
• Workflows - user-defined Temporal workflows
• Built-in Tools - RAG queries, web scraping, YouTube processing

Loading all tools into context for every request would consume ~77,000 tokens. That's expensive, slow, and often unnecessary when the user just wants to "send a Slack message."

---

Our Solution: Hybrid BM25 + Semantic Search

We implemented a multi-stage search pipeline inspired by Anthropic's Tool Search Tool pattern. The key insight: most queries can be resolved with fast keyword matching, reserving expensive semantic search for ambiguous cases.

This pipeline achieves 85% token reduction (77K → 8.7K tokens) while maintaining intelligent tool selection.

---

Deep Dive: Each Stage Explained

Stage 1: Always-Available Tools

Some tools are so frequently used that they should always be considered. We maintain a small set (3-5) of "always-available" capabilities that bypass the search entirely.

const ALWAYS_AVAILABLE_CAPABILITIES = [
  {
    name: "workspace_rag_query",
    keywords: ["document", "knowledge", "search workspace", "find in docs"],
    description: "Query workspace documents and knowledge base"
  },
  {
    name: "web_search",
    keywords: ["search web", "google", "look up", "find online"],
    description: "Search the internet for information"
  }
];

function matchAlwaysAvailableCapabilities(query: string): ToolMatch[] {
  return ALWAYS_AVAILABLE_CAPABILITIES.filter(cap =>
    cap.keywords.some(kw => query.toLowerCase().includes(kw))
  );
}

Why it matters: For queries like "search my documents for the Q4 report," we skip all search overhead and directly returnworkspace_rag_query.

---

Stage 2: Explicit Server Detection

Users often know exactly which tool provider they want. We parse natural language patterns to detect explicit server mentions:

function detectExplicitServerMention(query: string): string | null {
  const patterns = [
    /\buse\s+(\w+(?:\s+mcp)?)\s+(?:to|for)\b/i,
    /\bwith\s+(\w+)\b/i,
    /\bvia\s+(\w+)\b/i,
    /\bfrom\s+(\w+)\s+(?:mcp|server)\b/i,
    /\busing\s+(\w+)\b/i
  ];

  for (const pattern of patterns) {
    const match = query.match(pattern);
    if (match) {
      return normalizeServerName(match[1]);
    }
  }
  return null;
}

Why it matters: If the user says "use context7 to look up React hooks," we connect only to the Context7 MCP server, avoiding connections to potentially dozens of other configured servers.

---

Stage 3: BM25 Keyword Search

BM25 (Best Matching 25) is a battle-tested ranking function used by search engines. It's deterministic, fast, and surprisingly effective for tool matching.

Our BM25 implementation extracts keywords from multiple sources for each tool:

1. Server name (highest priority): Both the full name and individual segments (splitting on hyphens/shows)
2. Tool name decomposition: Breaking apart camelCase and snake_case names into individual words
3. Description keywords: The first 20 words of the tool description, filtered for stop words

All keywords are deduplicated and lowercased for consistent matching.

Why it matters: BM25 resolves ~50% of queries with high confidence (>0.7), completely bypassing the more expensive semantic search.

---

Stage 4: Semantic Search with Qdrant

When keyword matching isn't confident enough, we fall back to semantic search using vector embeddings stored in Qdrant.

The semantic search process works as follows:

1. Embed the query: Convert the user's natural language query into a vector embedding
2. Search the vector database: Find the top 10 most similar tool descriptions, filtered by the user's tenant context (personal or organization)
3. Return scored matches: Each result includes the server name, tool name, and a confidence score based on cosine similarity

Why it matters: Semantic search catches conceptually similar queries that keyword matching misses. "Help me track my tasks" might not contain "kanban" or "card," but semantic similarity connects it to project management tools.

---

Stage 5: Hybrid Result Merging

The final stage combines results from all sources with weighted scoring.

The merge algorithm follows these steps:

1. Assign priority-based confidence floors: Always-available tools get 1.0, explicit server matches get 0.95, and keyword/semantic matches keep their computed scores
2. Deduplicate by tool identity: If the same tool appears from multiple sources, keep the highest confidence score
3. Apply hybrid scoring: When a tool is found by both keyword and semantic search, the scores are blended (60% weight to the higher score, 40% to the lower) for more reliable ranking
4. Sort and limit: Return the top 10 tools sorted by final confidence score

---

The Complete Architecture

Here's how all the pieces fit together in our orchestrator:

---

Optimizations That Matter

1. Qdrant Cache Loading

Instead of connecting to MCP servers on every request, we cache tool metadata in Qdrant:

Instead of connecting to every MCP server on each request, we scroll through the cached tool metadata in the vector database, filtered by the user's tenant context. Each cached entry includes the server name, tool name, description, keywords, and input schema -- everything needed for search and planning without a live MCP connection.

Impact: Eliminates MCP connection overhead for cached tools. A typical MCP connection takes 200-500ms; loading from the cache takes ~20ms.

2. Lazy MCP Loading

We defer actual tool fetching until execution time:

3. Result Reuse Across Phases

Matched tools from routing are passed directly to planning, no redundant searches:

During the routing phase, matched tools are discovered once and then passed directly into the planning phase -- no redundant searches needed. The planner receives the full list of matched tools from routing and uses them directly to decompose tasks and assign executors.

Impact: Saves ~500ms+ per request by avoiding redundant MCP connections and search operations.

4. Token Budget Enforcement

We track token usage per step and enforce limits:

interface TokenBudget {
  maxTotalTokens: 100_000;
  maxTokensPerStep: 16_000;
  warningThreshold: 0.8;  // Warn at 80%
  reserveForSynthesis: 4_000;
}

function checkBudget(
  currentUsage: number,
  config: TokenBudget
): BudgetStatus {
  const remaining = config.maxTotalTokens - currentUsage - config.reserveForSynthesis;
  const usagePercent = currentUsage / config.maxTotalTokens;

  return {
    allowed: remaining > config.maxTokensPerStep,
    remainingTokens: remaining,
    usagePercentage: usagePercent,
    shouldWarn: usagePercent > config.warningThreshold
  };
}

---

Results: By the Numbers

Metric	Before	After	Improvement
Tokens per request	77,000	8,700	85% reduction
Average latency	3.2s	0.8s	75% faster
MCP connections	All servers	1-2 servers	90% fewer
Semantic searches	Every request	~50% of requests	50% reduction

---

Key Takeaways

1. Hybrid search beats pure semantic search. BM25 keyword matching resolves most queries faster and cheaper than embeddings.

2. User intent detection saves resources. Parsing "use X to..." patterns eliminates unnecessary server connections.

3. Caching is critical. Qdrant persistence lets us search tools without MCP connections on every request.

4. Reuse across phases. Don't search twice, pass matched results from routing to planning.

5. Lazy loading wins. Defer expensive operations (MCP connections) until absolutely necessary.

---

What's Next

We're exploring several improvements:

• Learning from usage patterns: Tools used together frequently should be suggested together
• Predictive pre-warming: Anticipate likely tools based on conversation context
• Federated search: Query multiple Qdrant collections in parallel
• Confidence calibration: ML-based threshold tuning for BM25 → semantic fallback

Frequently asked questions

Does the orchestrator still connect to MCP servers at all?

Yes, but only when it runs a tool, and only to the server that owns it. Search and planning both read the tool metadata cached in Qdrant, so the 200 to 500ms a connection costs is paid for one or two servers a request rather than for every configured one. With 15 servers set up, that's the gap between one or two handshakes and fifteen.

Why not send every query straight to semantic search?

Because BM25 settles about half of them on its own, and it does it without embedding the query or calling Qdrant. It's also deterministic, so the same wording picks the same tool every time, which makes a wrong pick easy to reproduce. Semantic search is kept for the queries whose words don't appear in any tool's name or description.

Which tools skip the search completely?

The always-available set, which is three to five tools such as workspace document search and web search. A plain substring match on their keywords is enough, and the merge gives them a confidence of 1.0, so they rank above a named server's tools (0.95) and anything BM25 or Qdrant scored. Naming a server with "use", "with", "via", "from" or "using" is the other shortcut, since the orchestrator then looks only at that server.

Can a search return tools from someone else's tenant?

No. The Qdrant similarity search and the cache load are both filtered by the user's tenant context, so a search only sees the tools set up in that context. The always-available tools are the one shared piece, because they're a fixed list in code rather than anything a user configures.

What stops a long multi-step plan from running out of tokens?

The token budget, which is checked before each step. A request gets 100,000 tokens, 4,000 of them held back for the final synthesis, and a step may only start while more than its 16,000 limit is left over. In practice that means no new step starts once a request has used about 80,000 tokens, the same point at which the 80% warning fires.