Metrics & Observability

Scraping the endpoint

The endpoint is plain Prometheus text exposition. Hit it directly to confirm the agent is emitting:

curl http://localhost:8080/metrics

A minimal Prometheus scrape config:

scrape_configs:
  - job_name: gostly-agent
    metrics_path: /metrics
    static_configs:
      - targets: ["localhost:8080"]

Metric descriptions are registered at process boot, so the scrape page carries the full schema for every family — including the gostly_tls_* series — even before a counter has ticked or the TLS listener has spawned. Dashboards that template panels over a metric family will not see missing-series gaps on a fresh agent.

ghost_requests_total{match_type}

A counter incremented exactly once per request, on the path that ultimately handled it. The match_type label is the single most useful signal the agent emits: it tells you which tier of the match cascade served each MOCK-mode request, and which operating mode handled the rest.

The cascade order on a MOCK-mode request is session_verbatim → exact → resource_store → statechart → smart_swap → cached generation, ending in miss if nothing matched.

Useful queries — the mock hit rate, and the share of traffic that fell all the way through to a miss:

# Fraction of MOCK-mode requests that hit a deterministic tier
sum(rate(ghost_requests_total{match_type=~"session_verbatim|exact|resource_store|resource_transition|smart_swap"}[5m]))
  / sum(rate(ghost_requests_total{match_type=~"session_verbatim|exact|resource_store|resource_transition|smart_swap|generated_cached|miss"}[5m]))

# Miss rate — the signal that your library has coverage gaps
sum(rate(ghost_requests_total{match_type="miss"}[5m]))

ghost_mock_library_size

An unlabeled gauge holding the total number of mock entries the agent is currently serving from — summed across all services. It is set when the in-memory serving index is read (for example on a /health probe and after a library reload), so it tracks the live serving index, not the full traffic history on disk.

# Alert if the library empties unexpectedly (e.g. a bad reload)
ghost_mock_library_size == 0

ghost_io_errors_total{operation}

A counter for disk-sink failures — opening, writing, or flushing the JSONL files the agent persists to. Any non-zero rate means recorded data is being lost: a recorded request, a served-mock log line, or a captured webhook did not reach disk. The usual causes are a full volume, wrong mount permissions, or a missing mount. The operation label tells you which sink failed:

# Any disk-sink failure in the last 5 minutes is actionable
sum(rate(ghost_io_errors_total[5m])) by (operation) > 0

Each increment is paired with an ERROR-level structured log line carrying the path and the underlying OS error, so the metric tells you that a sink is failing and the log tells you why.

axum_http_requests_* (control-plane HTTP)

RED-method (Rate, Errors, Duration) instrumentation over the agent's HTTP surface, emitted by the axum-prometheus layer:

# p95 latency of the control-plane HTTP surface
histogram_quantile(0.95,
  sum(rate(axum_http_requests_duration_seconds_bucket[5m])) by (le, endpoint))

gostly_tls_* (TLS MITM subsystem)

The TLS family is emitted by the optional :8443 MITM listener (see TLS Interception). The descriptions are registered at boot even when ENABLE_TLS_INTERCEPTION is off, so the series are present on every scrape — with the listener-state gauge reporting state="off".

# Which TLS listener state is live right now
max by (state) (gostly_tls_listener_state)

# Cert-cache hit ratio — alert if it drops below the SLO
sum(rate(gostly_tls_cert_cache_hit_total[5m]))
  / sum(rate(gostly_tls_cert_cache_hit_total[5m]) + rate(gostly_tls_cert_cache_miss_total[5m]))

A note on metric prefixes

You will see three prefixes on the scrape page:

Both ghost_ and gostly_ product prefixes appear on the scrape page. When you build dashboards, match on the exact metric names documented above rather than on a prefix glob, since the two namespaces are not interchangeable.