All tracks

ETL

5 modules, 10 hours. Work through them in order: each module lists the ones it needs first.

1. e1 Extract

   • Python
   • 2 hours

   Prerequisites: Group and aggregate (f2)

   Skills: Search API vs download API (asynchronous, with a DOI); paging, rate limits, retries; files with checksums

   • How do I get every open Tasmanian devil record, with a DOI to cite?

2. e2 Transform and load

   • Python
   • PostgreSQL with PostGIS
   • 2.5 hours

   Prerequisites: Extract (e1), Transactions (d4)

   Skills: Staging tables, types, COPY, upserts, geometry; GeoPandas to and from PostGIS

   • How do I load the download in one transaction, typed and checked, with each record matched to its species?

3. e3 Pipelines you can re-run

   • Python
   • PostgreSQL with PostGIS
   • 2 hours

   Prerequisites: Transform and load (e2)

   Skills: Run ids, deterministic outputs, checksums; critwire's own ingest → derive → build as the case study

   • Can I re-run the pipeline without doubling rows or changing its output?

4. e4 Quality and provenance

   • Python
   • PostgreSQL with PostGIS
   • 2 hours

   Prerequisites: Pipelines you can re-run (e3)

   Skills: Data quality checks written as SQL; license filters; provenance tables; citations; derived datasets

   • Which datasets must I credit, and under which license?
   • Which checks stop a bad load?

5. e5 Scheduling

   • Python
   • 1.5 hours

   Prerequisites: Quality and provenance (e4)

   Skills: cron, systemd timers, Windows Task Scheduler; logs, alerts, incremental loads

   • How do I refresh weekly, and know when a run broke?