Claude Code: Why You Must Switch to Local RAG (and Ditch Grep)

Using AI agents in the terminal, like Claude Code, is a revelation for many developers. But this magic quickly hits a physical limit: context management.

On a small codebase, everything is fluid. But as soon as your project exceeds a few dozen files, performance collapses.

Why? Because by default, Claude operates like a database without an index.

This article explores how to implement a local RAG (Retrieval-Augmented Generation) architecture to transform your CLI assistant into a surgical tool, reducing latency and token consumption by over 90%.

The Bottleneck: "Full Table Scan" vs Index

When you ask Claude: "How is authentication handled?", the agent has no pre-existing knowledge of your code. It has to discover it.

By default, its strategy is brute force:

1. It lists all files (ls -R / glob).
2. It searches for text occurrences (grep).
3. It reads entire files to validate their relevance.

In database terms, this is a Full Table Scan. It is inefficient, slow, and expensive. For a simple question, the agent might read 3000 tokens of useless code just to find the 50 relevant tokens.

The Solution: Local Indexing via MCP

The solution is to pre-calculate knowledge of the project. Instead of letting the AI rummage through files for every request, we will use qmd, an open-source tool that acts as an MCP (Model Context Protocol) server.

qmd generates two types of indexes on your machine:

• Lexical (BM25): For exact keyword searches.
• Semantic (Embeddings): To understand concepts ("error handling" will find code even if the word "error" is not present).

The paradigm shift is radical: the agent queries the index, receives a relevant snippet, and responds.

Technical Implementation

Here is how to set up this architecture on your development environment.

1. Installing the Stack

We use bun as the runtime for its speed, and qmd as the indexing engine.

# Install the engine
curl -fsSL https://bun.sh/install | bash
bun install -g https://github.com/tobi/qmd

2. Creating Vectors

In your project root, initialize the collection. This is where qmd will "read" your code for the first time and create its mental map.

# Create collection and filter relevant files
qmd collection add . --name backend-api --mask "**/*.{ts,tsx,go,rs,md}"

# Generate embeddings (crucial step for semantic search)
qmd embed

Note: Unlike grep which is instant but dumb, this step takes a few seconds initially, but will make all future queries instant.

3. The MCP Bridge

Claude Code needs to know this server exists. Modify your ~/.claude/mcp.json configuration:

{
  "mcpServers": {
    "qmd": {
      "command": "qmd",
      "args": ["mcp"]
    }
  }
}

4. Prompt Engineering (The CLAUDE.md File)

This is the decisive step. Without instruction, Claude will continue to use its native tools out of habit. We must force the use of the index via a system rule in the CLAUDE.md file at the project root.

## System Rule: Indexing Priority

**Critical Directive:** Never scan the file system (ls/grep/read) for information discovery.

1. Use EXCLUSIVELY the `qmd` tool for any context search.
2. Use `qmd search` for precise identifiers (function names, constants).
3. Use `qmd vsearch` for abstract concepts or functional exploration.
4. Only read the full file (`Read`) if the snippet provided by `qmd` is insufficient.

Impact Analysis: Before/After

On a medium-sized TypeScript project (200 files), the metrics are undeniable:

Conclusion

Optimizing tokens isn't just about saving money. It's about workflow.

By switching to local RAG, you remove the friction between your thought and the AI's response. Claude stops being an intern rummaging through archives to become a senior engineer who knows the exact location of every function.