A Three-Layer Caching Architecture for Low-Latency LLM Web Search on Commodity CPU Hardware

We’re excited to share OreoLook, an open-source answer engine, and the caching architecture we built to make agentic web search faster and more affordable on commodity CPU infrastructure.

Most LLM caching approaches focus on reusing final responses. Our work instead addresses redundancy across the full search pipeline through three coordinated layers:

• Session Context Window: keeps recent conversation turns in Redis and moves overflow into Huffman-compressed disk archives.
• Semantic Query Cache: detects equivalent or rephrased queries using embedding similarity, allowing expensive search and synthesis work to be skipped.
• URL Embedding Cache: reuses embeddings for URLs encountered across different sessions.

In our evaluated deployment—a single 8-vCPU Intel Cascade Lake server with 32 GB RAM, running 30 Hypercorn workers across three replicas—the system reported an 89.3% aggregate Redis keyspace hit rate, approximately 0.1 ms Redis read latency, and only 1.38 MB of measured Redis memory overhead.

The local web-search, session-management, caching, and embedding stack runs on CPU hardware, while final answer synthesis uses provider-routed remote inference. We describe the architecture honestly as a production-derived system, including its limitations and the distinction between Redis keyspace hit rate and end-to-end query avoidance.

The broader goal is simple: make live, conversational, source-backed AI search practical without requiring expensive local accelerators or repeated work at every turn.

We’d especially welcome feedback on semantic-cache isolation, cache evaluation methodology, and extending the architecture toward long-lived agent memory.

🔗 Try OreoLook
💻 Source code
📄 Read the paper