Status: [UNTESTED]

Endpoints found:
http://localhost:8000/v1/chat/completions

Source: https://github.com/mitochondria-28/semantic-caching-layer

# Semantic Caching Layer for LLM APIs

A production-quality middleware service that sits between your application and LLM providers (OpenAI, Anthropic, Ollama). It intercepts requests, checks semantic similarity against cached responses, and returns cached results — **without any changes to your existing client code**.

## Features

- **Drop-in OpenAI compatibility** — change only `base_url` in your client
- **Semantic cache lookup** — vector similarity search via Redis Stack; catches rephrased questions
- **Streaming support** — SSE streams are buffered and cached transparently
- **Multi-provider routing** — OpenAI → Anthropic → Ollama fallback chain
- **TTL tiers** — STABLE (24h) for factual, NORMAL (6h) for general, DYNAMIC (1h) for version-specific queries
- **Cache invalidation API** — invalidate by provider, model, or system prompt hash
- **Prometheus metrics** — hit rate, latency percentiles, token usage, cost incurred vs saved
- **Grafana dashboard** — pre-provisioned; auto-loads on `docker compose up`
- **Fail-open design** — Redis or embedding outages never block your app
- **Near-miss tracking** — queries just below the similarity threshold are logged for threshold tuning

## Architecture

```
Client App
    │
    │  POST /v1/chat/completions  (OpenAI-compatible)
    ▼
┌─────────────────────────────────────────────┐
│           Semantic Cache Proxy               │
│                                             │
│  1. Embed query (text-embedding-3-small)    │
│  2. KNN search in Redis with exact filters  │
│     (provider + model + system_prompt_hash  │
│      + param_fingerprint)                   │
│                                             │
│  HIT  ──▶  return cached response (~8ms)   │
│  MISS ──▶  call LLM provider               │
│       ──▶  store response in Redis