The Forge · engineering, computing & technology
Data Engineering & Cloud
Moving data reliably at scale — pipelines, warehouses, and the cloud they run on.
Tables, joins, and transactions — the forty-year-old technology that still keeps the world's records.
Syllabus · 3 units · ~24 hours
Unit I — Asking Questions
SELECT, WHERE, ORDER BY · Aggregation and GROUP BY · Joins, one at a time
Unit II — Designing Tables
Keys and relationships · Normalization without dogma · Indexes and why queries get fast
Unit III — Trust
Transactions and ACID · Constraints · Backups and recovery
Batch and streaming pipelines that survive bad data, late data, and three a.m. failures.
Syllabus · 3 units · ~30 hours
Unit I — The Shape of the Problem
Sources, sinks, and schedules · Batch vs. streaming · Idempotency
Unit II — Building
Orchestration and DAGs · Transformations and data-quality checks · Backfills
Unit III — Operating
Monitoring and alerting · Schema drift · Late and duplicate data · The post-incident review
Virtual machines, object storage, and managed services — what the cloud sells and what it costs.
Syllabus · 3 units · ~20 hours
Unit I — What the Cloud Is
Datacenters, regions, and zones · Virtual machines and containers · The shared-responsibility model
Unit II — The Catalog
Object storage · Managed databases · Serverless functions · Load balancers and networks
Unit III — Bills and Judgment
Pricing models and the surprise invoice · When not to use the cloud · Vendor lock-in, soberly assessed
Clocks that disagree, networks that drop, and the algorithms that make many machines act as one.
Syllabus · 3 units · ~44 hours
Unit I — Why It's Hard
Partial failure · Unreliable networks and clocks · The CAP theorem, precisely stated
Unit II — Agreement
Replication strategies · Leader election · Consensus: Paxos and Raft
Unit III — At Scale
Partitioning and sharding · Eventual consistency · Transactions across machines · Designs that degrade gracefully