PublicCrawler-poweredPostgresAgent-ready
DataShield Public Dataset Library
Find datasets for projects. This site continuously discovers and tracks public datasets across government portals (Socrata, ArcGIS Hub, CKAN, OpenDataSoft, DCAT feeds, sitemaps, and more) and stores clean, structured metadata + stable links in Postgres.
SEO keyword: datasets for projects
Library status
Best-effort live stats from Postgres.
Datasets
4168
Active
4168
Last changed: 3/9/2026
OTHER · 4158DCAT · 6SOCRATA · 1ARCGIS · 1CKAN · 1OPENDATASOFT · 1
Searchable catalog
Full-text search + tags, providers, and freshness metadata.
- Dataset pages with resources (API / downloads)
- Human notes + raw JSON payload fields
- Usage analytics: reads + searches tracked
Crawlers & scheduling
Provider-aware ingestion with per-project rate limits and cron schedules.
- Separate crawler projects per provider (or sub-projects)
- Run now, schedule, monitor runs, inspect errors
- Easy DB reset for testing
Built for agents
Clean APIs + an MCP server script so tools can discover datasets later.
- Public read APIs
- Admin + API keys (scoped)
- MCP server (stdio) for tool-based access
Supported providers
Start with big government portal engines. Add more as you go.
SocrataTyler Data & InsightsArcGIS Hub / Open DataCKANOpenDataSoftDCAT-US (data.json)Filesystem / ShareOther / Sitemap