ArcGIS Hub / Open Data

What it supports

• Dataset URLs like:
  • https://data-usdot.opendata.arcgis.com/datasets/<slug>/about
• Portal roots (discovery):
  • https://geodata.bts.gov/
  • The crawler uses the portal’s /api/search/v1 endpoint to enumerate datasets.

Notes

• Many ArcGIS sites use custom domains. Provider detection includes a fast probe to confirm ArcGIS Open Data.
• Ingestion uses the ArcGIS Hub API to pull structured metadata and resource links.

Tips

• Start with a single dataset URL to validate crawling.
• Then add the portal root to discover many datasets.
• Tune RPS/concurrency if you hit throttling.