Run Claude Skills against deterministic mock services

What you’ll build

By the end of this runbook you’ll have:

• A Gostly proxy running locally with a recorded mock library for the GitHub REST API (the running example below — the same pattern works for any HTTP-speaking upstream).
• A Claude Skill whose backend HTTP base URL points at the local Gostly proxy, not the live upstream.
• A reproducible test harness for the skill: same input → same recorded response → same observed behavior, every run.
• A pattern you can extend to every other skill in your workspace.

Prerequisites

• Docker (to run the Gostly stack locally — the customer image ships as a Compose file).
• Claude Code or the Claude Agent SDK, with the ability to define custom skills in your local config.
• An upstream API you want to mock.We’ll use GitHub’s REST API (api.github.com) as the running example. The same pattern applies to any HTTP-speaking upstream — internal service, Stripe, Twilio, OpenAI, anything.

Start Gostly in LEARN mode and route a handful of representative API calls through it. Each request/response pair lands in a local JSONL recording, scrubbed of auth-class headers before the bytes ever touch disk.

# Start the Gostly stack in LEARN mode
docker compose up -d
curl -X POST http://localhost:8000/v1/mode -d '{"mode":"learn"}'

# Point your code at the proxy instead of api.github.com
export GITHUB_API_BASE=http://localhost:8080

# Exercise a few representative paths
your-app fetch-pr      --owner octocat --repo Hello-World --number 1
your-app list-issues   --owner octocat --repo Hello-World --state open
your-app fetch-checks  --owner octocat --repo Hello-World --ref main

# Recordings now live in ./data/mocks/github.jsonl

Flip the proxy to MOCK mode. From here on, every matching request gets the recorded response. The live upstream is never touched again until you explicitly switch back.

curl -X POST http://localhost:8000/v1/mode -d '{"mode":"mock"}'

# Same request as before — but now served from the recording
your-app fetch-pr --owner octocat --repo Hello-World --number 1
# → byte-equivalent response from data/mocks/github.jsonl
# → no api.github.com call made, no quota consumed, no PAT needed

Define the skill so its backend HTTP base URL is the local Gostly proxy. The skill code is unchanged from how it would target the live upstream — only the base URL differs. The proxy is transparent to the skill itself.

# ~/.claude/skills/pr-status/skill.md
---
name: pr-status
description: Look up the status of a GitHub pull request by owner/repo/number.
---

When the user asks about a PR, call
${GITHUB_API_BASE}/repos/{owner}/{repo}/pulls/{number}
and return: state, mergeable status, reviewer list, and a check-run
summary. If the PR is mergeable, suggest the next action.

With GITHUB_API_BASE pointing at the local proxy, the skill calls Gostly. Gostly serves the recorded response. Claude reasons about it. The skill returns its summary. Nothing on GitHub’s side moves — no API quota consumed, no PAT required.

Invoke the skill the same way three times. The recorded response comes back identically each time. Diff the outputs to confirm byte-equivalence. This is the property that makes regression tests possible: a previously-green skill that turns red points at something the skill changed, not something the upstream changed.

# Run 1
$ claude /pr-status octocat/Hello-World 1
> PR #1: open · mergeable · 2 reviewers · 3/3 checks passing

# Run 2 (five minutes later)
$ claude /pr-status octocat/Hello-World 1
> PR #1: open · mergeable · 2 reviewers · 3/3 checks passing

# Run 3 (in CI tomorrow morning)
$ claude /pr-status octocat/Hello-World 1
> PR #1: open · mergeable · 2 reviewers · 3/3 checks passing

Commit the recordings to your repo (or to a shared mock library your team draws from). CI brings up the Gostly stack from those recordings — no upstream credentials needed in CI, no live API allowed. Skill regression tests run hermetically.

# .github/workflows/skill-tests.yml (illustrative)
- name: Boot Gostly with committed recordings
  run: |
    docker compose up -d
    curl -X POST http://localhost:8000/v1/mode \
      -d '{"mode":"mock"}'

- name: Run skill regression tests
  env:
    GITHUB_API_BASE: http://localhost:8080
    # No GITHUB_TOKEN needed — every call is served from the recording.
  run: pytest tests/skills/

What you’ve gained — the three pillars

Failure modes the clean room blocks