Download Datasets

NoteAt a glance
  • Input: 15 source layers from public providers and collaborator drops
  • Output: data/01_inputs/source/, organized by dataset ID
  • Helper: src/download_input_layers.py

Goal

The notebook downloads the source layers used to build the Master Parcel Attribute Table (MPAT) into a source input directory. Most layers download programmatically from public providers. Building footprints and the cesspool inventory data layers are from collaborators and are added to the directory manually.

Inputs

The table below lists and links the 15 source layers. The Access column indicates whether the layer downloads programmatically or arrives from a collaborator.

Dataset ID Description Provider Coverage File Type Resolution Access
coastline_hi_dbedt Coastline Hawaii DBEDT Office of Planning Statewide Shapefile Vector Automated (ZIP)
sma_hi_dbedt Special Management Area (SMA) boundaries Hawaii DBEDT Office of Planning Statewide Shapefile Vector Automated (ZIP)
parcels_hi_higp Tax Map Key (TMK) parcels Hawaii Geospatial Portal (file hosted by DBEDT) Statewide Shapefile Vector Automated (ZIP)
slope_hi_hcpt 10 m DEMs (used to derive slope) HCPT — Act 132 Cesspool Prioritization Statewide GeoTIFF 10 m Automated (GitHub)
watertable_hi_hcpt Depth to water table HCPT — Act 132 Cesspool Prioritization Statewide GeoTIFF — Automated (GitHub)
streams_hi_hcpt Streams HCPT — Act 132 Cesspool Prioritization Statewide Shapefile Vector Automated (GitHub)
wells_hi_hcpt Wells (split during prepare into domestic and municipal layers) HCPT — Molokai Statewide Shapefile Vector Automated (GitHub)
annual_rainfall_hi_hcpt Mean annual rainfall HCPT — Act 132 Cesspool Prioritization Statewide GeoTIFF — Automated (GitHub)
soils_hi_hcpt Soils HCPT — Act 132 Cesspool Prioritization Statewide Shapefile Vector Automated (GitHub)
dem_hi_pacioos USGS 10 m DEM (all main islands) PacIOOS THREDDS NCSS 8 islands GeoTIFF (converted from NetCDF) 10 m Automated (THREDDS)
building_footprints_maui Building footprints — Maui Robert / Johann Maui — Vector Manual (collaborator)
building_footprints_kauai Building footprints — Kauai Robert Kauai — Vector Manual (collaborator)
building_footprints_oahu_cchgis Building footprints — Oahu City & County of Honolulu GIS Oahu Shapefile Vector Manual (ArcGIS Hub)
cesspools_inventory_hi_hcpt Cesspool inventory Aimee (HCPT) Statewide — Vector Manual (collaborator)
flood_zones_hi_fema Flood hazard areas — DFIRM statewide FEMA via Hawaii Statewide GIS Program Statewide Shapefile Vector Manual (portal)

Outputs

Source layers are saved to data/01_inputs/source/ and are organized by dataset ID. Each layer remains in the original provider format (Shapefile, GeoTIFF, NetCDF) at this stage. Reprojection, mosaicking, and GeoPackage conversion happen in Prepare Input Data.

  • Project directory: HiOSDS-TechSuitabilityAnalysis/data/01_inputs/source/
  • GitHub link: Not applicable. The source data folder is excluded from the repository due to storage limitations (see .gitignore).

Processing Steps

The download notebook (notebooks/00_download_input_layers.ipynb) follows four steps:

  1. Import modules and helper functions. The notebook pulls download utilities from src/.
  2. Specify project and raw source data directory paths. Paths point at data/01_inputs/source/.
  3. Specify data source links per download method type. Layers are grouped by access pattern (zipped shapefile URLs, GitHub raw URLs, THREDDS NCSS queries, manual drops).
  4. Download all data or individual data layers. The matching helper runs for each method type. Manual layers are skipped automatically and listed in the notebook output.

Notes and Decisions

  • Manual inputs. Building footprints for Maui and Kauai, the cesspool inventory, and FEMA flood hazard areas arrive as manual file drops. These files must be placed in data/01_inputs/source/ before the prepare notebook can build the downstream layers.
  • Parcels dataset ID. The parcels layer is hosted by DBEDT but discoverable through the Hawaii Geospatial Portal. The project keeps the dataset ID parcels_hi_higp to match the existing source folder and notebook paths.

Resources