DevOps runbook

Processes

In production you typically run:

1. Web: npm run start
2. Worker: npm run worker
3. (Optional) MCP server: npm run mcp

A basic systemd/pm2 setup is recommended.

Common operations

Restart worker

If crawlers are stuck or you changed worker code:

• Restart the worker process.

Note: schedule changes (cron) are now picked up automatically by the worker within a few minutes.

Purge logs

Logs are stored in registry.job_logs for UI visibility.

• UI: Management → App admin → Logs → Purge
• API: POST /api/admin/logs/purge (admin token required)

Rotate API keys

• UI: App admin → API keys → Create
• Disable old keys when callers are updated.

DB reset (testing only)

• UI: App admin → Danger zone → Reset registry DB schema
• CLI: npm run db:reset

This drops schema registry (datasets + run history).

Troubleshooting

Doctor says DB down

1. Verify DB credentials (UI: test DB)
2. Check Postgres is reachable from the host
3. Ensure schema exists / migrations applied:

npm run db:migrate

Worker not running crawls

1. Verify worker process is up
2. Check pg-boss tables exist (schema pgboss)
3. Check logs:

• UI: Dashboard → Recent logs
• API: GET /api/app/logs (admin token)

Crawls failing (HTTP errors)

• Lower RPS/concurrency per project
• Confirm the portal doesn't require auth/app tokens
• Some portals block aggressive scraping; add delay and a polite user-agent.

Backups

Back up the Postgres database. For a minimal deployment:

• Daily logical dump (pg_dump) to durable storage
• Weekly retention policy