Deterministic LLM mocks — recorded from the real provider, replayed in your perimeter

llmock is a clean answer to a real problem: agent tests that hit a live model are slow, expensive, and flaky, and they drift every time the provider ships a new checkpoint. So llmock (and the related aimock) runs a local mock server you can reach from any process on a port, records real provider responses VCR-style the first time, and replays them deterministically after that. It supports the LLM call shape end to end — OpenAI, Claude, Gemini, Bedrock, and more — including full streaming, preserving time-to-first-token and inter-frame cadence. If your need is “make one test file deterministic without a network call,” a fixture-based mock server or a VCR cassette is the lighter, more direct tool, and often the right one.

Gostly starts from a different scope. It is not a per-language library you drop into a test file; it is a self-hosted proxy that sits in front of an upstream for your whole stack. You point your agent’s provider base URL at it, run LEARN to record an hour of real responses, then flip to MOCK and it replays them byte-for-byte — in your perimeter, with no model in the serving path. Because it lives at the proxy layer, every service and every language in your system gets the same recorded library; there is nothing to wire per test framework.

Both approaches keep your secrets local. llmock replays from on-disk fixtures with no live call; Gostly records into a library on your own host and never sends recorded traffic, mocks, or PII off the box — credential headers are redacted before anything is written to disk. For a team whose blocker is “our prompts and completions can’t touch a third-party service,” that perimeter property is the whole point, and it holds across every upstream your agent calls, not just the model.

The pain that grows over time is drift. A recorded cassette keeps returning the same tokens long after the provider changed its response shape — the test stays green, and the agent breaks in production against the new format. Gostly re-records the upstream and diffs it against what you captured, so it emits a drift event and a 0–100 freshness score before an incident does. And for multi-step tool flows, a recorded statechart means a POST followed by a GET-by-id behaves like the real resource lifecycle instead of a flat per-request fixture.

Because Gostly runs as your own single-tenant deployment, the controls a security review asks for are built in: SSO via SAML or OIDC, role-based access control, and an append-only audit log on Team, all self-hosted in the container. Each deployment is single-tenant, so your data is isolated by running in your own perimeter, never shared with another customer’s. The trade-off is real, too: llmock and aimock are purpose-built for the model-call shape — they reproduce streaming token cadence and the provider-specific protocol details in a way a general HTTP recorder does not, and a single-file VCR cassette is lighter to adopt for one suite.