Seeing inside OpsAPI: probes, metrics, errors and audit trails

OpsAPI runs inside nginx workers, so the usual “attach a profiler to the app server” doesn’t apply. What you get instead is a small set of endpoints and records built for the people who run it: three health endpoints with distinct jobs, Prometheus metrics from every worker, one error envelope with a correlation id, and tables that record every sign-in and every user action.

Three health endpoints, three jobs

Point the liveness probe at /live, not /ready: a database blip should take a pod out of rotation, not restart it.

$ curl -s https://identity.demo.opsapi.org/ready
{"ready":true,"version":"1.0.281","timestamp":1793195573}

$ curl -s https://identity.demo.opsapi.org/health
{"status":"degraded","services":{"database":{"status":"healthy"},
 "minio":{"status":"unhealthy","error":"MinIO not reachable on any endpoint"}}, ...}

“Degraded” still returns 200: the demo has no object storage, which matters for uploads but not for sign-in. Use /health for humans and dashboards and /ready for anything that routes traffic.

Metrics

Every worker records into a shared dictionary through nginx-lua-prometheus, and GET /metrics exposes the lot. The sample Prometheus config scrapes the API every five seconds.

The Grafana dashboards read Prometheus; reports about users read a set of database views through a read-only role that can't see passwords or tokens.

Metric	Labels	Use it for
`nginx_http_requests_total`	host, status, method, endpoint	Traffic and error ratio per route
`nginx_http_request_duration_seconds`	host, method, endpoint	Latency percentiles
`nginx_http_5xx_errors_total`	host, status, endpoint	The first alert to set
`api_auth_failures_total`	reason	Credential stuffing, broken clients
`api_database_query_duration_seconds`	operation	Slow queries
`nginx_http_rate_limited_total`	host, ip	Who is hitting the limits
`opsapi_auth_events_total`	event, result, method	Sign-in volume by outcome
`opsapi_activity_dropped_total`	reason	Activity rows lost under load (should be 0)

The repo’s alert rules include high error rate, slow responses, slow database queries, high authentication failure rate and request-rate spikes. Start with the 5xx rate and authentication failures; add the rest once you know your baseline.

One error envelope, with a correlation id

Route handlers return errors through Errors.response(self, CODE, opts). The helper resolves the reader’s locale, looks the code up in the message catalog, writes one sanitised row to error_occurrences and returns the same shape every time:

HTTP/1.1 401
{"error":{
  "code":"AUTH_INVALID_CREDENTIALS",
  "title":"Invalid credentials",
  "message":"The email or password you entered is incorrect.",
  "category":"error",
  "correlation_id":"627512de-3867-4a88-9606-bc75ad743d1d",
  "occurrence_uuid":"43555f92-2f0c-45d1-a342-c5c1d8219212"
}}

If the caller sends X-Request-ID, that becomes the correlation id, so one id can follow a request from the edge through the API and into the database row. Support asks the user for the id; the occurrence row has the endpoint, method, status and request context with passwords redacted.

The normal path along the top. Along the bottom, what happened when the catalog tables didn't exist.

Lesson: anything every request needs belongs in the core

The catalog tables were created by the tax module’s migrations, but the error helper is used by every module. A deployment without tax, such as the minimal identity build behind the demo, had no catalog. Every error, including an ordinary wrong password, came back as a generic SYSTEM_500 “Something went wrong”, and the real cause (a failed catalog read, then a failed occurrence insert) only showed in the nginx debug log. The fix moved those three migrations into the core set; deployments that already had them don’t run them again.

When you build a slim deployment, test an error path, not just a success path: a wrong password should be a 401 with a readable message.

Who signed in, and what they did

Table	Holds	Kept for
`auth_events`	Every sign-in success or failure, 2FA, refresh, sign-out, password change	1 year by default
`user_login_stats`	One row per user: last sign-in, count, consecutive failures, last seen	Life of the account
`user_activity`	What signed-in users did, partitioned by month	90 days; whole partitions dropped
`user_activity_daily`	Per workspace, day, user and action counters	For the in-app Activity page

Login details are kept out of the users table on purpose: user rows are returned by many APIs, and new columns there would leak IP addresses. Deleting a user deletes their activity and auth events whatever path deleted them, which covers erasure requests.

Events that survive a broker outage

Audit and business events go to Kafka when it’s configured. If the producer can’t reach a broker, the event is written to a kafka_outbox table instead, and the consumer side works through the outbox and marks rows sent. A broker outage delays events; it doesn’t lose them, and it never fails the business write that produced them.

The outbox turns an outage into a delay.

A starting runbook

Pods not ready: read /ready from inside the pod. The reason names the failing dependency.
Users report “something went wrong”: ask for the correlation id and find the occurrence row.
Error rate up: break nginx_http_5xx_errors_total down by endpoint before reading logs.
Sign-in complaints: auth_events for that user shows each attempt and why it failed.
Events missing downstream: count unsent rows in kafka_outbox.