The Match Cascade

The invariant: no LLM in the hot path

Lead with the contract, because everything else follows from it. When a request arrives in MOCK mode, the only work the request thread is allowed to do is deterministic lookup: hash comparisons, an in-memory store probe, a structural scan, and at most one O(1) cache read. It never blocks on an inference call. The model can be loading, warming, rate-limited, or behind a tripped circuit breaker, and the request still returns at lookup speed.

This is enforced structurally, not by convention. The blocking generation call was moved off the request path entirely: a cache miss enqueues a pre-warm job onto a bounded queue and falls straight through to the next deterministic tier. A single background task owns the inference call, populates the cache, and the nextrequest for the same key gets a cache hit. The inference call never contributes to a request's wall-clock latency.

Generative tiers are also gated by subscription tier (Pro and above) and are off by default. See the AI pipeline for the inference-server details.

The five tiers, in order

Each request descends the cascade until a tier produces a response. Earlier tiers are cheaper and more faithful; later tiers are more capable but more synthetic. The tier that answered is recorded on the response via an X-Ghost-* header and counted in the ghost_requests_total{match_type} metric.

¹ Smart-swap is granted on every licensed tier (Free included). The SMART_SWAP_ENABLED=true opt-in only applies to unlicensed boots, where agent features default off; the license capability sets the ceiling.

If no tier produces a response, the request is recorded as a miss (match_type="miss") and the unmatched request is tracked so you can record or seed coverage for it later.

How generation stays off the hot path

The AI tier is split into two halves: a synchronous half that runs on the request thread, and an asynchronous half that runs on a background worker. The split is the whole point.

Where the model itself runs depends on configuration. Generation is off by default (ENABLE_GENERATION=false) and routes through the local ghost-llamacpp sidecar; retrieval-augmented matching (ENABLE_RAG) is on by default. When fine-tuning is used, LoRA adapters train only on PII-scrubbed rows, are served from cache, and stay self-hosted. None of this changes the invariant — the request thread never waits on a model.

What the cascade can serve — and what it can't

The cascade records and replays HTTP and HTTPS — HTTP/1.1 and HTTP/2 over TLS. Plain HTTP is served on :8080; the TLS-MITM listener binds on :8443 only when ENABLE_TLS_INTERCEPTION is set (tri-state: default off; true/lax; or strict). With interception on, GET /ca.crt serves the MITM CA for the per-OS trust-install step (it returns a 503 while interception is off).

Two product behaviours are worth distinguishing from the cascade itself. Drift watches recorded responses against live traffic and emits drift events plus a 0–100 freshness score with a sparkline trend, so you know when a mock has gone stale. AI mock-repair turns drift into operator proposals you approve or reject; it is off by default (ENABLE_AI_MOCK_REPAIR) and never edits a mock without approval. Neither sits on the request hot path.

The cascade runs in MOCK mode — the other three modes

The five-tier cascade is what the proxy does in MOCK mode. The agent has four modes total:

You don't have to record from scratch to fill the library. Cold-start seeding accepts a HAR, Postman collection, or OpenAPI spec via the dashboard's drag-and-drop, which posts to POST /v1/seed/{har,postman,openapi}. Seeded entries feed the same cascade as recorded traffic.

# Switch to MOCK mode (the cascade is now active)
curl -X POST http://localhost:8000/v1/mode \
  -H "X-API-Key: $GHOST_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"mode": "MOCK"}'

Observing which tier answered

Every match-path outcome is metered. The agent exposes Prometheus metrics on /metrics; the cascade's primary signal is the match_type label on ghost_requests_total:

# Per-tier request counts (one increment per match-path outcome)
ghost_requests_total{match_type="session_verbatim"}
ghost_requests_total{match_type="exact"}
ghost_requests_total{match_type="resource_store"}
ghost_requests_total{match_type="smart_swap"}
ghost_requests_total{match_type="generated_cached"}
ghost_requests_total{match_type="miss"}

# Library + I/O health
ghost_mock_library_size
ghost_io_errors_total{operation="..."}

# HTTP rate + latency, and the TLS-MITM subsystem
axum_http_requests_total
axum_http_requests_duration_seconds
gostly_tls_listener_state{state="..."}

Reading the match_type distribution tells you how your coverage is composed — a healthy library answers most traffic at the exact or smart-swap tiers, with generated_cached as the long tail. A rising miss rate is your signal to record or seed more.

OSS proxy vs. the licensed product

The deterministic tiers — session-verbatim, exact, resource store, and smart-swap — are the core matching engine and run everywhere. The generative tier is the line between products: