Monitoring and alerting
You can operate the Model Hub like any other production web service: watch process health, resources, logs, and model-sync failures.
Health checks and readiness
The Model Hub process listens on port 3000 and exposes a GET /health and GET /ready endpoint. A successful check is an HTTP 200 response.
Use this endpoint for:
- Docker / Compose health checks (already defined in the official image)
- Kubernetes liveness and readiness probes
- Load balancer health checks
See Install and start the server for the reference health-check command.
Also monitor the database migration job on install and upgrade. A non-zero exit from model-hub-*-database-migration would mean that the schema was not applied successfully; do not ignore failed migration containers.
Resource metrics
Collect at least:
- CPU and memory usage for the model-hub container (or pod)
- Disk usage for the filestore volume (/home/node/.hub)
- Database CPU, storage, connection count, and free space (on the external PostgreSQL or Oracle instance)
Alert when the Model Hub container restarts repeatedly, when memory grows without bound under sync load, or when the filestore volume is close to full.
Logs
The Model HUB writes **JSON logs** to the container’s standard output (stdout). Ship them to your central logging platform.
You will see:
- Access / request logs for HTTP traffic
- Warning and error logs for unexpected conditions (for example database connectivity problems or Git provider failures)
Useful alert ideas on log streams:
- Spikes in HTTP **5xx** responses
- Repeated **401** / **403** after authentication was enabled (often an IdP or redirect URI misconfiguration)
- Messages indicating database connection or pool errors
- Messages indicating Git provider or webhook processing failures
Model sync errors
When the Model Hub fails to process a webhook or to sync a model file, it records the failure in the `sync_errors` table in the Model HUB database. Monitor this table in addition to logs.
Useful columns
| Column | Meaning |
|---|---|
| error_type | webhook_error failure when the Git provider called the webhook endpoint; model_sync failure while processing model files |
| error_message | Human-readable error text for troubleshooting |
| created_at | When the error was recorded |
| git_request_id | Identifier from the Git provider webhook, when available |
| git_provider | Provider that triggered the sync |
| repository_name | Repository involved, when available |
| branch_name | Branch involved, when available |
| model_path | Model file path involved, when available |
| http_status_code | HTTP status related to the failure, when applicable |
Example queries
PostgreSQL (schema hck_hub)
SELECT error_type, error_message, git_provider, repository_name, model_path, created_at
FROM hck_hub.sync_errors
WHERE created_at > now() - interval '24 hours'
ORDER BY created_at DESC
LIMIT 100;
Count recent webhook failures:
SELECT count(*) AS webhook_errors_last_24h
FROM hck_hub.sync_errors
WHERE error_type = 'webhook_error'
AND created_at > now() - interval '24 hours';
Oracle
SELECT error_type, error_message, git_provider, repository_name, model_path, created_at
FROM sync_errors
WHERE created_at > SYSTIMESTAMP - INTERVAL '1' DAY
ORDER BY created_at DESC
FETCH FIRST 100 ROWS ONLY;
Alert when the rate of new sync_errors rows rises suddenly, or when webhook_error rows appear continuously (often a bad webhook URL, secret, or network path from the Git provider to the Model Hub).
Suggested alert checklist
- /health failing or Model Hubcontainer/pod not ready
- Repeated container restarts
- Elevated HTTP 5xx rate
- Repeated 401/403 after authentication was enabled
- Database connectivity errors in logs
- Growth of rows in sync_errors (especially webhook_error spikes)
- Filestore / disk usage high on /home/node/.hub