Help & documentation
This is the runbook + user guide. If you step away and something breaks, start here.
All docs
Project overview
What this app does, how it works, and who it’s for.
/help/overview
Quick start
Install, configure Postgres, run migrations, seed projects, and start web + worker.
/help/quickstart
DevOps runbook
Restart, recover, rotate keys, purge logs, and troubleshoot common failures.
/help/devops/runbook
Deploy on Hostinger / LiteSpeed
Practical, step-by-step deployment guidance for a beginner-friendly server setup.
/help/devops/deploy-hostinger-litespeed
Troubleshooting
Common failure modes and how to recover quickly.
/help/devops/troubleshooting
Backup & restore
How to back up Postgres and recover the registry quickly.
/help/devops/backup-restore
It’s down… now what?
A simple recovery checklist for non-experts.
/help/devops/its-down
API reference
Public APIs, admin APIs, and agent APIs (with curl examples).
/help/api/reference
Provider notes
What the crawler extracts per provider and how to add new provider implementations.
/help/providers/overview
API overview
Public read APIs, admin APIs, and how to explore endpoints via OpenAPI.
/help/api/overview
Analytics import
Store your context analytics engine output and extract field-level profiles.
/help/analytics-import
Management UI guide
What each page does and the safest way to operate the crawlers.
/help/admin/management-ui
Configuration
Config file, environment variables, DB overrides, and how to safely change settings.
/help/admin/configuration
Registry DB reset
Safely destroying and recreating the registry schema for testing.
/help/admin/db-reset
ArcGIS portal discovery
How ArcGIS Hub/Open Data portal discovery works (sitemap + orgId fallback).
/help/providers/arcgis-discovery
MCP server for agents
Run the MCP server and expose tools like datasets_search, fields_search, and crawler_run_project.
/help/agents/mcp
Test sources
Extra URLs you can use to validate each provider if a seeded link stops working.
/help/examples/test-sources
Agent events
Recording reads/searches and other actions via POST /api/events.
/help/agents/events
Other portal discovery
How the generic portal discovery works (sitemap + link heuristics) and when to use explicit sources.
/help/providers/other-discovery
Socrata (and Tyler Technologies)
Crawl Socrata portals (including Tyler Data & Insights instances).
/help/providers/socrata
ArcGIS Hub / Open Data
Add ArcGIS open data portals and dataset URLs.
/help/providers/arcgis
CKAN
Crawl CKAN catalogs and dataset pages.
/help/providers/ckan
OpenDataSoft
Crawl OpenDataSoft dataset pages.
/help/providers/opendatasoft
DCAT / data.json feeds
Crawl DCAT-US feeds like data.json to discover many datasets.
/help/providers/dcat
Other providers & sitemap discovery
Use robots.txt + sitemap.xml heuristics to discover dataset pages on unknown sites.
/help/providers/other-sitemap
Filesystem / network share
Crawl local directories or mounted shares (with strict allow-listing).
/help/providers/filesystem
deploy-library-myorg-ai
/help/devops/deploy-library-myorg-ai