Introduction

This is a course, not a manual — and it is a sequel. By the end you will have built panoptes_etl, a real space-data ETL that pulls from CelesTrak, Space-Track, and TheSpaceDevs, normalizes the results, and writes the files the Panoptes eval harness reads. More importantly, you will understand why every piece is shaped the way it is — and specifically, how to build the simplest thing that already has the joints an orchestrator needs.

What we are building, and why

Panoptes measures how large language models make hard decisions. Its live scenario family, ca_geo, is collision avoidance in geostationary orbit: an operator deciding whether to maneuver, under time pressure, with uncertain attribution of who caused the conjunction. Those scenarios are only defensible if they are grounded in real orbital data — real satellites, real orbital elements, and above all real conjunctions. Supplying that ground truth is this ETL's job.

So the keystone is the conjunction: a real Conjunction Data Message is a collision-avoidance decision waiting to be posed to a model. Everything else — space objects, orbital elements, launches — exists to enrich and attribute conjunctions.

The tension this course is about

You could build this as a pile of one-off scripts. It would work, and it would never grow. You could instead build a full orchestrator — a scheduler, a daemon, a DAG engine — on day one, and never finish. This course takes the third path, the one worth teaching: build the simplest pipeline that already has the four seams an orchestrator needs, so that "add a scheduler," "add backfill," "add retries" all become later chapters that add a node, never a rewrite.

The four seams, installed one per arc:

1. A Source trait — every source is a uniform async fn. A new source is a new impl; nothing else changes.
2. Extract / Transform / Load as three separable stages, with a typed boundary between each.
3. A run ledger — an append-only record of what ran, when, at what watermark, success or failure.
4. Idempotent, content-addressed writes — so re-running is always safe.

Build those four, and the final arc's payoff writes itself: a DAG executor that drops the sources you already wrote into a graph and runs them — reusing every seam, and depending on none of them by name.

How the arcs are sequenced

Each source-arc introduces exactly one new real-world complication and installs one seam:

• Part I, Foundations — the ETL mental model, error handling for fallible I/O, and id-safety. Short, because this is a sequel: ownership, serde, traits, async/tokio, wiremock, and clap are assumed.
• Part II, Extraction (CelesTrak) — an open GET and the TLE text format. Installs the Source trait + the E/T/L split.
• Part III, Authenticated (Space-Track) — login, cookies, secrets, and rate limiting. Installs the run ledger.
• Part IV, Paginated (TheSpaceDevs) — cursor pagination and retry/backoff. Installs idempotent content-addressed writes.
• Part V, Orchestration — the payoff: a typed DAG executor over the seams, plus a capstone where you add a fourth source unassisted.
• Part VI, Retrieval (RAG) — embed the prose fields into a vector index; the orchestrator schedules the embedding node.
• Part VII, Wrap-Up — how this plugs into Panoptes, and the graduation path (DuckDB, a real scheduler) sketched against the seams.

A note on where you are starting from

If you finished the first Panoptes course, you already think in ownership, traits, and async. What is new here is not the language — it is the shape of ETL: partial failure, retries, idempotency, and the discipline of leaving seams for a future you cannot yet see. That discipline is the whole subject.

What stays out (on purpose)

We stop at "pipeline + seams + a real mini-orchestrator." We do not build a long-running scheduler daemon, a queryable run-history UI, or orbit propagation (sgp4), and we do not reach for a vector database. Each of those is named in the graduation path with the exact seam it snaps onto — because the point of the seams is that they make those additions cheap later, not that we spend this course on them.

The Architecture at a Glance

How the structs, enums, traits, and functions of the finished panoptes_etl connect. Four crates, built in dependency order — the arrow always points at etl-core:

etl-sources → etl-orchestrate → panoptes-etl, and all three → etl-core. etl-orchestrate depends on etl-core only: it schedules dyn Source and never names a concrete source, which is what lets your NASA capstone drop into the DAG with zero executor changes.

The pipeline

Each source extracts raw bytes, a transform parses them into typed Rows, and an idempotent sink appends JSONL that panoptes-gen reads. The Executor runs the pipelines in dependency order — retrying transient failures, gating on upstream failure, skipping sources still fresh — and the retrieval arc embeds the prose fields into a vector index.

%%{init:{'theme':'base','themeVariables':{'fontSize':'13px','lineColor':'#7c8b9a','textColor':'#1b2530','primaryColor':'#e9eef3','primaryBorderColor':'#9fb0bf','primaryTextColor':'#1b2530','clusterBkg':'#fbfcfd','clusterBorder':'#cbd5de'}}}%%
flowchart TB
  cli{{"panoptes-etl CLI · run · backfill · embed · status"}}:::bin
  ct[("CelesTrak · open TLE GET")]:::ext --> csrc["CelestrakSource"]:::src
  st[("Space-Track · login + cookie")]:::ext --> ssrc["SpaceTrackSource"]:::src
  sd[("TheSpaceDevs · paginated")]:::ext --> dsrc["SpaceDevsSource"]:::src
  bucket["TokenBucket · rate limit"]:::src -.-> ssrc
  retry["with_retry · backoff on transient EtlError"]:::core -.wraps.-> dsrc
  csrc --> raw["RawRecord"]:::core
  ssrc --> raw
  dsrc --> raw
  raw --> xf["Transform · parse TLE / CDM / launch JSON"]:::src
  xf --> row["Row · SpaceObject · Conjunction · Launch"]:::core
  row --> idem["IdempotentSink · content_id() dedupe"]:::core
  idem --> jsonl[("space_objects · conjunctions · launches · .jsonl")]:::store
  exec["Executor · Dag topological order · gate on failure · skip fresh"]:::orch
  exec -.drives.-> csrc
  exec -.drives.-> ssrc
  exec -.drives.-> dsrc
  ledger["Ledger · RunRecord · watermark"]:::core -.records.-> exec
  jsonl --> embed["EmbedSink · Embedder → VoyageEmbedder · prose only"]:::core
  embed --> vec[("embeddings.jsonl")]:::store
  vec --> idx["VectorIndex::top_k() · cosine"]:::core
  jsonl --> gen[("panoptes-gen reads the file contract")]:::ext
  cli -.-> exec

  classDef core fill:#e2efec,stroke:#1f8f80,color:#12231f;
  classDef src fill:#e6ecf5,stroke:#4a6fa5,color:#182436;
  classDef orch fill:#f6ead4,stroke:#b9791f,color:#3a2a10;
  classDef bin fill:#efe6f4,stroke:#7d5ba6,color:#2c1f36;
  classDef store fill:#eef1f5,stroke:#6b7a89,color:#1b2530;
  classDef ext fill:#f0f1f3,stroke:#9aa7b3,color:#3a444e,stroke-dasharray:4 3;

Teal = etl-core · slate = etl-sources · amber = etl-orchestrate · plum = panoptes-etl · cylinders = files · dashed = external API or downstream.

Type relationships

The E/T/L trait triad — Source, Transform, Sink — is the spine. Concrete sources implement Source; the domain types are the payload of the Row enum the sink writes; and Pipeline/Executor compose the traits as boxed trait objects, which is what lets the orchestrator stay ignorant of any concrete source. The keystone is Conjunction — a real conjunction data message is the ground truth of a ca_geo collision-avoidance vignette.

%%{init:{'theme':'neutral','themeVariables':{'fontSize':'13px','lineColor':'#7c8b9a'}}}%%
classDiagram
  namespace etl_core {
    class NoradId {
      <<newtype>>
      +u32 inner
    }
    class OrbitClass {
      <<enum>>
      LEO
      MEO
      GEO
      HEO
    }
    class OrbitalElements {
      +DateTime epoch
      +f64 inclination_deg
      +f64 eccentricity
      +f64 mean_motion_rev_per_day
      +orbit_class() OrbitClass
    }
    class SpaceObject {
      +NoradId norad_id
      +String name
      +String intl_designator
      +OrbitalElements elements
    }
    class Conjunction {
      +String id
      +DateTime tca
      +f64 miss_distance_km
      +f64 collision_probability
      +NoradId primary
      +NoradId secondary
    }
    class Launch {
      +String id
      +String name
      +DateTime net
      +String provider
      +String description
    }
    class Row {
      <<enum>>
      SpaceObject
      Conjunction
      Launch
    }
    class EtlError {
      <<enum>>
      Network
      RateLimited
      Auth
      Permanent
      Parse
      Io
    }
    class RawRecord {
      +String source
      +String payload
      +DateTime fetched_at
    }
    class Source {
      <<trait>>
      +name() str
      +extract() Vec~RawRecord~
    }
    class Transform {
      <<trait>>
      +transform(RawRecord) Vec~Row~
    }
    class Sink {
      <<trait>>
      +load(rows) usize
    }
    class Ledger {
      +append(RunRecord)
      +last_success(source) RunRecord
    }
    class RunRecord {
      +String source
      +Outcome outcome
      +DateTime watermark
      +usize rows_written
    }
    class Outcome {
      <<enum>>
      Success
      Failed
    }
    class Embedder {
      <<trait>>
      +embed(texts) vectors
    }
    class VectorIndex {
      +top_k(query, k) hits
    }
  }
  namespace etl_sources {
    class CelestrakSource
    class SpaceTrackSource
    class SpaceDevsSource
    class TokenBucket
    class VoyageEmbedder
  }
  namespace etl_orchestrate {
    class Dag {
      +add_task(id) TaskId
      +add_dep(task, dep)
      +topological_order() Vec~TaskId~
    }
    class Pipeline {
      +Box~Source~ source
      +Box~Transform~ transform
      +Box~Sink~ sink
    }
    class Executor {
      +run() RunReport
    }
  }
  SpaceObject *-- NoradId
  SpaceObject *-- OrbitalElements
  OrbitalElements ..> OrbitClass
  Conjunction *-- NoradId
  Row *-- SpaceObject
  Row *-- Conjunction
  Row *-- Launch
  RunRecord *-- Outcome
  CelestrakSource ..|> Source
  SpaceTrackSource ..|> Source
  SpaceDevsSource ..|> Source
  VoyageEmbedder ..|> Embedder
  Pipeline o-- Source
  Pipeline o-- Transform
  Pipeline o-- Sink
  Executor *-- Dag
  Executor o-- Pipeline
  Executor *-- Ledger

*-- composition · o-- aggregation (boxed trait object) · ..|> implements · «enum»/«trait»/«newtype» stereotypes · ~T~ = generic parameter.

Per-crate inventory

The full, verified code behind every box is in the Answer Key.

How to Use This Course

This is a sequel. It assumes you finished the first Panoptes course — or that you already think fluently in ownership, borrowing, serde, traits, async/tokio, wiremock, and clap. We do not re-teach the language here. What is new is the shape of ETL: partial failure, retries, idempotency, and the discipline of leaving seams for a future you cannot yet see. That is the whole subject, and it is what these chapters build.

The two chapter types

Every arc alternates between two kinds of chapter, and they ask different things of you.

Concept chapters are the lecture. Read them without touching the keyboard. They are deliberately verbose and they argue why before how — because in ETL the "why" (why this error is transient, why this write must be idempotent) is the part that bites you in production, not the syntax. Each concept chapter ends with a short Questions to lock list. If any of those are fuzzy, that is the signal to stop and re-read before moving to the build.

Build chapters are the pair-programming session. Here you write the code. A build chapter gives you the spec and the test names — never the implementation. You drive. Each one follows the same test-driven loop you already know:

1. Write the failing test. You write it, from the spec and the test name we give you — not by copying an answer.
2. Predict the failure. Say out loud (or note down) exactly what the compiler or the test runner will report.
3. Run it and check your prediction. The gap between prediction and reality is the lesson.
4. Write the minimal implementation. Just enough to make the test pass. No more.
5. Run again, watch it pass.
6. Commit.

The predict-then-run habit

The single most valuable thing you can do in a build chapter is write down your prediction before you run. A wrong prediction followed by the real message is the fastest way to learn what the types are actually enforcing. This matters more in an ETL than in a pure library, because many of the failures here are runtime failures — a mock returning a 429, a re-run that should write zero new rows — and predicting them trains the instinct you need when the real network misbehaves.

The concept-check quizzes

Each part ends with one graded quiz, carried only on that part's quiz page. These are not busywork — they target the exact misconceptions that cause bugs three arcs later: retrying a 404 forever, confusing two integer ids, forgetting which stage owns parsing. Answer honestly before revealing. Some questions are Tracing questions: a short program whose output — or whose refusal to compile — you must predict. Those are compiled by the real Rust compiler when the book builds, so the answer is not a matter of opinion.

Assumed toolkit — if you skipped the first course

This course will not stop to explain the following. If any of these is genuinely new to you, work the first Panoptes course (or an equivalent) before starting Part II, because every build chapter leans on them:

• Ownership, borrowing, and moves — why a function takes &self vs self, when a value is moved, why the borrow checker rejects aliasing a mutable borrow.
• serde — #[derive(Serialize, Deserialize)], container and field attributes (#[serde(tag = …)], rename_all, transparent), and serde_json.
• Traits and trait objects — defining a trait, impl Trait for T, dyn Trait, and object safety.
• async/await and tokio — async fn, .await, #[tokio::main]/#[tokio::test], and why async traits need async_trait.
• wiremock — starting a mock HTTP server in a test and pointing a client with an injectable base URL at it. Every source in this course is tested this way; no test hits a live network.
• clap — deriving a CLI with subcommands.
• The TDD loop itself — red, green, refactor, commit.

Concept: The ETL Mental Model & the Four Seams

Kind: Concept. No new crate — this chapter installs the architectural vocabulary the whole course leans on. Read it slowly; every later arc refers back to it by name.

The idea

ETL is three verbs: Extract data from somewhere you do not control, Transform it into a shape you do, and Load it into a place you can read later. That is the entire job description of panoptes_etl: pull orbital data from CelesTrak, Space-Track, and TheSpaceDevs; normalize it into typed rows; write the JSONL files the Panoptes eval harness reads.

You could build that as a pile of one-off scripts — a fetch_celestrak.rs, a fetch_spacetrack.rs, each with its own hard-coded URL and its own ad-hoc parsing. It would work today and it would never grow. The moment you want retries, or a schedule, or "only fetch what changed since yesterday," you are rewriting.

You could also swing the other way and build a full orchestrator on day one — a scheduler, a daemon, a DAG engine, a run database — and never ship anything.

This course takes the third path, and it is the whole thesis:

Build the simplest pipeline that already has the joints an orchestrator needs.

Simple now. But with the seams pre-cut, so that "add a scheduler," "add backfill," "add retries," and "add a fourth source" each become a later chapter that adds a node, never a rewrite. The seams are the point. Everything below is a description of the four we cut, why each one earns its place, which arc installs it, and where it finally gets spent in Parts V–VI.

Why "seam" is the right word

A seam is a place where the design lets you change one thing without disturbing the others. A tailor leaves seam allowance so a garment can be let out later; you are leaving design allowance so the pipeline can be let out later. A seam costs you a little discipline up front — a trait instead of a free function, a typed boundary instead of passing raw strings around — and pays you back the first time a requirement lands that would otherwise have been a rewrite.

The four seams below are not arbitrary. Each one corresponds to a specific real-world complication every ETL eventually hits: new source, stage failure isolation, knowing what already ran, and safe re-runs. Install the seam before the complication arrives and the complication becomes cheap.

Seam 1 — the Source trait: one shape for every source

The first complication is simply there is more than one source, and there will be more later. If every source has a bespoke signature, the orchestrator has to know each one by name. So we make every source wear the same uniform: an object-safe async trait.

// from crates/etl-core/src/pipeline.rs
#[async_trait]
pub trait Source: Send + Sync {
    fn name(&self) -> &str;
    async fn extract(&self) -> Result<Vec<RawRecord>, EtlError>;
}

This sketch needs async_trait, chrono, thiserror, and serde, so it is rust,ignore — the real thing lives in etl-core and is built in Part II. To run a version yourself, cargo new a lib crate and cargo add async-trait chrono thiserror serde --features serde/derive.

Two design choices carry the seam. Send + Sync means a Source can be moved to and shared across tokio tasks — the orchestrator will need that. And because the trait is object-safe, you can hold a Box<dyn Source>: the orchestrator can keep a list of sources and call extract() on each without knowing whether it is talking to CelesTrak or TheSpaceDevs. A new source is a new impl Source and nothing else in the system changes.

Installed: Part II (CelesTrak is the first impl Source). Spent: Part V, where the DAG executor schedules a Vec<Box<dyn Source>> and depends on none of them by name.

Seam 2 — E / T / L as three separable stages

The second complication is partial failure. A fetch can succeed while parsing fails; parsing can succeed while the disk write fails. If extract, transform, and load are tangled into one function, you cannot retry the fetch without re-parsing, or swap the sink without touching the parser. So we split them into three traits with a typed boundary between each:

// Extract produces RawRecord (bytes off the wire, unparsed).
// Transform turns &RawRecord into Vec<Row> (normalized, typed). Pure & sync.
// Load takes &[Row] and returns how many it wrote.

pub struct RawRecord {
    pub source: String,
    pub payload: String,
    pub fetched_at: DateTime<Utc>,
}

pub trait Transform: Send + Sync {
    fn transform(&self, raw: &RawRecord) -> Result<Vec<Row>, EtlError>;
}

#[async_trait]
pub trait Sink: Send + Sync {
    async fn load(&self, rows: &[Row]) -> Result<usize, EtlError>;
}

The boundary types are the seam. RawRecord is the contract between E and T: everything upstream deals in raw bytes, everything downstream deals in a Row. Notice that Transform is pure and synchronous — no network, no clock, no disk. That is deliberate: parsing is the part you most want to unit-test exhaustively, and a pure function is trivial to test. Source and Sink are async because they touch the outside world; Transform is not because it must not.

Row itself is a single tagged enum over every entity a source can emit, so one Sink can accept the output of any pipeline:

#[derive(Serialize, Deserialize)]
#[serde(tag = "kind", rename_all = "snake_case")]
pub enum Row {
    SpaceObject(SpaceObject),
    Conjunction(Conjunction),
    Launch(Launch),
}

Installed: Part II (the CelesTrak pipeline runs E → T → L end to end). Spent: Part V, where a Pipeline { source, transform, sink } becomes a single node the executor runs.

Seam 3 — the run ledger: knowing what already ran

The third complication is time and incrementality. A real ETL does not re-fetch the entire history of the universe on every run; it fetches what changed since last time. To do that it must remember what it did — when each source last ran, up to what watermark, and whether it succeeded. That memory is the run ledger: an append-only record, one line per run.

pub struct RunRecord {
    pub source: String,
    pub started_at: DateTime<Utc>,
    pub finished_at: DateTime<Utc>,
    pub outcome: Outcome,                     // Success | Failed(String)
    pub watermark: Option<DateTime<Utc>>,     // "I am caught up to here"
    pub rows_written: usize,
}

Append-only matters: you never mutate history, you only add to it, which means the ledger is itself an audit trail and is safe to read while it is being written. The watermark is what makes the next run incremental — it answers "where did I leave off?".

Installed: Part III (Space-Track, the first source with enough volume and auth cost to make re-fetching everything unacceptable). Spent: Part V, where the executor writes one RunRecord per node and reads the last watermark to skip work that is already up to date.

Seam 4 — idempotent, content-addressed writes: safe re-runs

The fourth complication is the ugliest: runs fail halfway and get retried. If a run writes 500 rows, dies, and is re-run, a naive sink writes those 500 rows again — now you have duplicates polluting the data the eval harness trusts. The fix is to make writes idempotent: running twice produces the same result as running once.

We get that by content-addressing each row — deriving its id from a hash of its own content rather than from an auto-incrementing counter or the wall clock:

// A row's identity is a hash of what it *is*, not when it arrived.
pub fn content_id(kind: &str, canonical: &str) -> String { /* sha256 hex */ }

// A Conjunction's id: content_id("conjunction", "{primary}-{secondary}-{tca}")

Because the id is a pure function of the content, the same conjunction always hashes to the same id. An idempotent sink keeps a set of ids it has already written and skips any it has seen, so a re-run writes zero new rows. This is what makes "just run it again" a safe response to any failure — which is exactly the property retries need.

Installed: Part IV (TheSpaceDevs, where pagination + retries make duplicate-on-retry a live hazard). Spent: Part IV directly (re-running the paginated fetch is safe), and again in Part VI, where embeddings are keyed by content_id("embed", text) so prose is never re-embedded — an expensive API call saved for free by the same seam.

The payoff, held in view

Here is why the discipline is worth it. When you reach Part V, the orchestrator is almost anticlimactic to write, because every joint it needs is already there:

• it holds sources as Box<dyn Source> — Seam 1;
• it runs each as a Pipeline of separable E/T/L stages — Seam 2;
• it records and reads watermarks to run incrementally — Seam 3;
• and it retries freely because re-runs are safe — Seam 4.

The DAG executor reuses all four and depends on none of them by name. etl-orchestrate will not even be allowed to mention a concrete source — the dependency graph forbids it, which is the proof that the seams actually decoupled the design rather than just looking like they did. That inversion is the destination. Everything in Parts I–IV is laying seam allowance so that Part V is an assembly, not a rewrite.

Questions to lock

1. Name the three verbs of ETL and, for each, say what type crosses the boundary out of it in this pipeline (RawRecord, Row, a count).
2. What are the four seams? For each, state the one real-world complication it exists to absorb.
3. Why is Transform a plain synchronous trait when Source and Sink are async? What does that buy you at test time?
4. What does "object-safe" let the orchestrator do with a Source, and why does it need that?
5. Why does content-addressing (id from a hash of the content) make re-runs safe, when an auto-increment id would not?
6. For each seam, which arc installs it and where does it get spent?

Concept: Error Handling for Fallible I/O

Kind: Concept. New crate: thiserror — this chapter shows it working before you build with it.

The idea

Every stage of an ETL touches something it does not control — a network, an auth server, a disk, a payload someone else formatted. So every stage is fallible, and the interesting question is never "did it fail?" but "is this failure worth retrying?"

That single distinction — transient vs permanent — is the load-bearing idea of this whole chapter, and it is the thing your error type exists to encode:

• A transient error might succeed if you try again after a wait: a dropped connection, a timeout, a 429 Too Many Requests. The world was briefly unavailable; it may be available in five seconds.
• A permanent error will fail identically every time: a malformed payload, bad credentials, a 404 for an endpoint that was renamed. Retrying it wastes time, hammers the server, and buries the real signal.

Get this wrong in the retrying direction and you build a machine that hits a 404 and retries it forever — a busy-loop against a resource that will never exist. Get it wrong in the other direction and you give up on a blip that a single retry would have fixed. The error type's job is to make the classification a property of the value, decided once, at the boundary where you actually know the cause.

A toy that mirrors the real thing: a coffee order

Before we touch orbital data, here is the exact same shape on a domain you cannot mistake for the real code — a barista filling a coffee order. Some ways an order fails are worth trying again; some are not.

We model every failure as one enum, and we use thiserror to derive the Display messages and the std::error::Error impl. Then we add two methods that answer the only questions the retry loop will ask: is this worth retrying, and did the failure come with a suggested wait?

use std::time::Duration;

use thiserror::Error;

/// Every failure a coffee order can hit, in one type.
#[derive(Debug, Error)]
pub enum BaristaError {
    /// The espresso machine jammed mid-pull. Transient — try again.
    #[error("machine jammed: {0}")]
    MachineJam(String),

    /// Out of beans; the kitchen is restocking. Transient, with a hint.
    #[error("restocking; try again in {retry_after_secs}s")]
    Restocking { retry_after_secs: u64 },

    /// The card was declined. Permanent — retrying charges nothing new.
    #[error("card declined: {0}")]
    CardDeclined(String),

    /// That drink is not on the menu. Permanent.
    #[error("not on the menu: {0}")]
    NotOnMenu(String),

    /// The handwritten order slip was illegible. Permanent.
    #[error("unreadable slip: {0}")]
    UnreadableSlip(String),
}

impl BaristaError {
    /// Whether retrying (after a wait) could plausibly succeed.
    pub fn is_transient(&self) -> bool {
        matches!(
            self,
            BaristaError::MachineJam(_) | BaristaError::Restocking { .. }
        )
    }

    /// The wait the kitchen asked for, when it gave one.
    pub fn retry_after(&self) -> Option<Duration> {
        match self {
            BaristaError::Restocking { retry_after_secs } => {
                Some(Duration::from_secs(*retry_after_secs))
            }
            _ => None,
        }
    }
}

thiserror is not available on the Rust playground, so these blocks are rust,ignore and have no play button. To run this yourself, cargo new a binary crate and cargo add thiserror. The #[error("…")] attribute is what generates the Display impl; {0} interpolates a tuple field and {retry_after_secs} a named struct field.

Drive it over one of each and print the classification:

fn main() {
    let failures = [
        BaristaError::MachineJam("group head stuck".into()),
        BaristaError::Restocking { retry_after_secs: 20 },
        BaristaError::CardDeclined("insufficient funds".into()),
        BaristaError::NotOnMenu("unicorn frappe".into()),
        BaristaError::UnreadableSlip("coffee smudge".into()),
    ];
    for f in &failures {
        println!(
            "{:>28}  transient={:<5}  retry_after={:?}",
            f.to_string(),
            f.is_transient(),
            f.retry_after()
        );
    }
}

   machine jammed: group head stuck  transient=true   retry_after=None
       restocking; try again in 20s  transient=true   retry_after=Some(20s)
  card declined: insufficient funds  transient=false  retry_after=None
     not on the menu: unicorn frappe  transient=false  retry_after=None
     unreadable slip: coffee smudge  transient=false  retry_after=None

Two things fall out of the design. First, is_transient is a matches! over exactly the two retryable variants — the classification is centralized, not scattered across every call site. Second, only Restocking carries a retry_after, because only it comes with a suggested wait; every other variant returns None, and the retry loop falls back to its own backoff. A transient error may or may not carry a hint; a permanent error never does.

The mapping is one-for-one

The BaristaError above is not an analogy — it is the same type as the one you will build, with the words swapped. Here is the correspondence, variant for variant, against etl-core's real EtlError:

• MachineJam(String) ↔ Network(String) — a connection reset or timeout. Transient.
• Restocking { retry_after_secs } ↔ RateLimited { retry_after_secs } — the server said slow down (429). Transient, with a hint.
• CardDeclined(String) ↔ Auth(String) — bad or missing credentials (401/403). Permanent.
• NotOnMenu(String) ↔ Permanent(String) — a 404, a 400, a renamed endpoint. Permanent.
• UnreadableSlip(String) ↔ Parse(String) — a payload that would not parse into the domain model. Permanent.

EtlError also carries one variant the toy omits: Io(#[from] std::io::Error), for a local read/write failure while touching a sink. The #[from] gives you a free From<std::io::Error> conversion so ? lifts an I/O error straight into EtlError. And the two methods are identical: EtlError::is_transient() is matches!(self, Network(_) | RateLimited { .. }), and EtlError::retry_after() returns Some(Duration) only for RateLimited. The domain is coffee vs conjunctions; the shape is the same shape. This classification is precisely what the Part IV retry loop consumes: with_retry retries only while is_transient() is true, and honors retry_after() when it is Some.

The trap: one transient bucket for every non-2xx

Here is the mistake the extraction arc walks straight into, and it is worth seeing now because it is seductive. When you first wire up an HTTP source, the tempting shortcut is: "the request succeeded if the status was 2xx; otherwise it failed, so treat every non-2xx as a transient network error and let the retry loop sort it out."

That code compiles, passes the happy-path test, and is wrong in a way that only shows up in production. It classifies a 404 as transient — so the retry loop hammers a nonexistent endpoint, backing off longer and longer, forever, before finally giving up much later than it should. It classifies a 401 as transient — so a bad API key triggers a retry storm instead of failing fast with "fix your credentials." The one bucket that should be transient, 429, gets lumped in with everything else and loses its Retry-After hint.

The correct classification reads the status by class, with 429 pulled out first:

/// Classify an HTTP status the way the retry loop must: not "2xx vs the rest".
fn classify(status: u16) -> &'static str {
    match status {
        200..=299 => "success",
        429 => "transient: honor Retry-After, then retry",
        400..=499 => "permanent: do NOT retry",
        500..=599 => "transient: back off and retry",
        _ => "unknown",
    }
}

fn main() {
    for status in [200, 404, 429, 400, 503, 401] {
        println!("{status} -> {}", classify(status));
    }
}

200 -> success
404 -> permanent: do NOT retry
429 -> transient: honor Retry-After, then retry
400 -> permanent: do NOT retry
503 -> transient: back off and retry
401 -> permanent: do NOT retry

The rule, in three lines:

• 429 → transient, with backoff. The server explicitly told you to slow down and usually said for how long (Retry-After). This becomes EtlError::RateLimited.
• Other 4xx → permanent. 400, 401, 403, 404 are all your request being wrong: bad input, bad credentials, wrong URL. Retrying an unchanged request gets the identical rejection. These become EtlError::Auth (401/403) or EtlError::Permanent (the rest).
• 5xx → transient. The server broke on its end; the same request may well succeed once it recovers. This becomes EtlError::Network.

Why an enum, and not just strings or anyhow

You might ask why the source layer needs a typed EtlError at all when anyhow exists. The answer is exactly the retry decision. anyhow::Error is opaque — you cannot ask it "were you a 429?" without stringly-typed sniffing. A thiserror enum makes the classification a method on the type, checkable at compile time and impossible to forget a case of. The binary at the top of the stack can still use anyhow for its main (and does); but the library stages that a retry loop reasons about need a type that can answer questions about itself. That is the division the first course drew, and it holds here.

Questions to lock

1. State the transient-vs-permanent distinction in one sentence, and give one example error on each side.
2. In EtlError, which two variants are transient? Which single variant carries a retry_after, and why only that one?
3. What does #[from] on Io(#[from] std::io::Error) generate, and how does it change what ? can do?
4. Why is classifying every non-2xx status as one transient bucket a bug? What specifically goes wrong for a 404? For a 401?
5. In the correct classify, why must the 429 arm come before the 400..=499 arm?
6. Why does the source layer use a typed thiserror enum instead of anyhow for its errors?

Concept: Newtypes & Id-Safety

Kind: Concept. No new crate — serde is assumed from the first course; this chapter is about a pattern, not a dependency.

The idea

An ETL is, at bottom, a machine for moving ids around. A conjunction references a primary object and a secondary object, each by its NORAD catalog number. A launch has an id; a payload count is a number; a launch sequence is a number. Almost everything is "some integer that identifies something." And there lies a quiet, expensive class of bug: all those integers have the same type, so the compiler cannot stop you from putting one where another belongs.

Pass the secondary's id where the primary's was expected and the code compiles, the tests on the happy path pass, and you have silently mislabeled a collision-avoidance decision. Nothing crashes. The data is just wrong, and it stays wrong until someone downstream notices two satellites attributed backwards.

The newtype pattern closes this off. Wrap the raw integer in a one-field struct that means this specific kind of id and no other, and the compiler starts rejecting the mix-up at the type level — for free, with zero runtime cost, because the wrapper compiles away entirely.

The bug, first — bare integers that compile and lie

Here is the failure mode with no newtypes. Two different ids, both u32, so the compiler sees them as interchangeable:

/// Both ids are just `u32`, so the compiler sees no difference between them.
fn ship_from_bin(bin: u32) {
    println!("dispatching a picker to bin {bin:04}");
}

fn main() {
    let bin_id: u32 = 42;
    let pallet_id: u32 = 7;
    // A copy-paste slip. This compiles cleanly and ships to the wrong place.
    ship_from_bin(pallet_id);
    let _ = bin_id;
}

dispatching a picker to bin 0007

That program is a bug that runs perfectly. ship_from_bin wanted a bin and got a pallet, and nothing objected. In panoptes_etl the equivalent slip is swapping a conjunction's primary and secondary NORAD ids — same shape, higher stakes.

The toy: a BinId newtype

Now give the id a type of its own. This mirrors etl-core's real NoradId on a domain you cannot confuse with it — a warehouse bin location. The newtype wraps a u32 and carries three behaviors: a padded Display, a validating FromStr, and a transparent serde form so the wire stays clean.

use std::fmt;
use std::str::FromStr;

use serde::{Deserialize, Serialize};

/// A warehouse bin location — a newtype over `u32`, not a bare integer.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)]
#[serde(transparent)]
pub struct BinId(pub u32);

impl fmt::Display for BinId {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        // Bin labels are printed zero-padded to four digits.
        write!(f, "{:04}", self.0)
    }
}

impl FromStr for BinId {
    type Err = String;

    fn from_str(s: &str) -> Result<Self, Self::Err> {
        s.trim()
            .parse::<u32>()
            .map(BinId)
            .map_err(|_| format!("invalid bin id: {s:?}"))
    }
}

fn main() {
    // Display: padded label.
    println!("label:      {}", BinId(42));
    // FromStr: trims and parses.
    println!("parsed:     {}", " 1337 ".parse::<BinId>().unwrap());
    // Bad input is rejected, not silently coerced.
    println!("rejected:   {:?}", "aisle-7".parse::<BinId>());
    // serde(transparent): the wire form is the bare integer, no wrapper object.
    println!("as json:    {}", serde_json::to_string(&BinId(42)).unwrap());
    let back: BinId = serde_json::from_str("42").unwrap();
    println!("from json:  {back}");
}

This block uses serde and serde_json, so it is marked rust,ignore. To run it, cargo new a binary crate and cargo add serde --features derive plus cargo add serde_json.

label:      0042
parsed:     1337
rejected:   Err("invalid bin id: \"aisle-7\"")
as json:    42
from json:  0042

Read the four behaviors off the output:

• Display is a presentation choice, not the storage. BinId(42) stores the integer 42 and displays 0042. The padding lives only in fmt; the value is untouched.
• FromStr is a validation boundary. Parsing " 1337 " trims and succeeds; parsing "aisle-7" returns Err. An unparseable string can never become a BinId — the type is only constructible from something that already validated. (Real ids will use .parse::<BinId>()? at the extract boundary, so bad input becomes an error there rather than a wrong value later.)
• #[serde(transparent)] keeps the wire clean. Without it, BinId(42) would serialize as a wrapper — {"0": 42} or a newtype-struct form — leaking the wrapper into JSON. With it, the id serializes as the bare integer 42 and round-trips straight back. The type safety is a compile-time-only fiction; the bytes on disk are just numbers.

Now the bug cannot compile

Give the two ids distinct newtypes and the copy-paste slip from the top of the chapter becomes a compile error instead of a silent mislabeling:

struct BinId(u32);
struct PalletId(u32);

/// Ships whatever is in a given bin. It wants a *bin*, specifically.
fn ship_from_bin(_bin: BinId) {}

fn main() {
    let pallet = PalletId(7);
    ship_from_bin(pallet); // wrong id kind
}

That fails to compile with E0308: mismatched types — "expected BinId, found PalletId". The exact bug that ran perfectly with bare u32s is now caught before the program even builds. That is the entire payoff of the pattern: a class of mix-up bug converted from a runtime data-corruption into a compile error, at no runtime cost.

The mapping is one-for-one

BinId is NoradId with the domain swapped. Against etl-core's real ids.rs:

• BinId(pub u32) ↔ NoradId(pub u32) — a one-field tuple struct over u32, with the same derive set (Debug, Clone, Copy, PartialEq, Eq, Hash, …, Serialize, Deserialize).
• #[serde(transparent)] ↔ #[serde(transparent)] — identical; a NoradId serializes as the bare integer 25544, not a wrapper object.
• Display padding to four digits ({:04}) ↔ padding to five ({:05}) — NORAD ids are conventionally shown zero-padded to five, so NoradId(733) displays 00733 while storing 733.
• FromStr trimming and parsing, Err on junk ↔ identical, except the real error type is EtlError::Parse (from the previous chapter) rather than a bare String — so a bad id flows into the same error taxonomy the retry loop understands, and it is correctly permanent (a malformed id will never parse no matter how many times you retry).

The domain is warehouse bins vs tracked satellites; the shape is the same shape. When you build NoradId in Part I's companion build work, you are re-deriving exactly the toy above with the padding width and error type changed.

Questions to lock

1. Show a two-line example where two bare-u32 ids let a wrong-argument bug compile. What does the newtype version do instead, and with what error code?
2. Where does a BinId do its validation, and what does that guarantee about every BinId value in the program?
3. What does #[serde(transparent)] change about the serialized form, and why do you want it for an id?
4. NoradId(733) stores 733 but displays 00733. Where does the zero-padding live, and does it affect the stored value or the serialized bytes?
5. Why does a newtype cost nothing at runtime? What is its memory layout compared to the bare integer?
6. In the real NoradId, FromStr returns EtlError::Parse rather than a String. Using the previous chapter, is a parse failure transient or permanent, and why does that classification make sense for a malformed id?

Concept-Check: Foundations

These questions target the three ideas Part I installed: the four seams of the pipeline, the transient-vs-permanent error split that drives retries, and newtype id-safety. The Tracing questions are compiled and run by the real Rust compiler when this book builds — predict the output (or the refusal to compile) before you reveal. Answer honestly; a wrong answer with a good explanation teaches more than a lucky guess.

Concept: The Source Trait & the E/T/L Boundary

Kind: Concept. Seam installed this arc: the Source trait + the Extract / Transform / Load split.

The idea

An ETL job does three things, and the whole discipline of this course is keeping them three separate things:

1. Extract (E) — talk to the outside world and come back with raw bytes. This is where the network lives: HTTP, timeouts, 429s, auth. It is async, it is fallible, and it knows nothing about what the bytes mean.
2. Transform (T) — turn those raw bytes into your normalized domain types. This is pure: no network, no clock, no files. Same input, same output, every time. It is where parsing — and therefore validation — happens.
3. Load (L) — write the normalized rows somewhere durable. For us that is a JSONL file; later it could be a database. It knows nothing about where the rows came from.

Draw the boundaries as types and each stage becomes independently testable, independently swappable, and — the payoff four arcs from now — independently schedulable. The orchestrator will hold a Source, a Transform, and a Sink as trait objects and run them in order, never knowing it is talking to CelesTrak versus Space-Track. That only works if the three stages never leak into each other.

The two typed boundaries are the load-bearing part. Between E and T sits a RawRecord (bytes plus provenance). Between T and L sits a Row (a normalized domain value). Everything upstream of a boundary can change without touching anything downstream, because the boundary is a type, not a convention.

   external API           RawRecord            Row              file
        │                     │                 │                │
        ▼        E            ▼        T         ▼       L        ▼
     Source ───────────►  (raw bytes) ──────► (domain) ──────► Sink
     async, fallible       pure parse          durable write

The three traits, exactly as etl-core declares them

Here are the real signatures you will implement. Read them as the contract for the whole pipeline:

// crates/etl-core/src/pipeline.rs
// (needs async-trait + chrono + serde; not playground-safe — see the toy below
//  for a runnable version of the same shape)

use async_trait::async_trait;

/// Raw bytes fetched from a source, before any parsing — the output of E.
#[derive(Debug, Clone, PartialEq)]
pub struct RawRecord {
    pub source: String,
    pub payload: String,
    pub fetched_at: DateTime<Utc>,
}

/// A normalized domain row — the typed boundary between T and L.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
#[serde(tag = "kind", rename_all = "snake_case")]
pub enum Row {
    SpaceObject(SpaceObject),
    Conjunction(Conjunction),
    Launch(Launch),
}

/// **E** — extract raw records from an external source.
#[async_trait]
pub trait Source: Send + Sync {
    fn name(&self) -> &str;
    async fn extract(&self) -> Result<Vec<RawRecord>, EtlError>;
}

/// **T** — parse raw records into normalized rows. Pure and synchronous.
pub trait Transform: Send + Sync {
    fn transform(&self, raw: &RawRecord) -> Result<Vec<Row>, EtlError>;
}

/// **L** — persist normalized rows to a sink; returns the count written.
#[async_trait]
pub trait Sink: Send + Sync {
    async fn load(&self, rows: &[Row]) -> Result<usize, EtlError>;
}

Four details in those signatures carry the whole design; notice each one:

• Source::extract is async; Transform::transform is not. The network lives in E and only in E. T is a plain synchronous function because parsing bytes needs no runtime — which is exactly why it is trivial to unit-test with a hand-written RawRecord and no server at all.
• Every method returns Result<_, EtlError>. One error type flows through the whole pipeline. The classification you built in Part I (is_transient()) is what the later retry loop reads off these results.
• All three traits are Send + Sync. That is the price of admission to being scheduled across threads on a tokio runtime. The orchestrator will move these across .await points and hold them behind Arc; without Send + Sync that would not compile.
• Row is one enum over every entity, tagged by kind. This is what lets a single Sink accept the output of any pipeline. We will come back to why the tag matters.

A runnable toy: the receipt fetcher

The trait objects above pull in async-trait, chrono, and serde, so they will not run on the playground. Here is the same shape on a domain that cannot be mistaken for the answer key — a corner shop's receipts — reduced to synchronous std so it runs anywhere. Fetch raw text → parse to typed rows → store; three traits, two typed boundaries:

// The raw bytes a source hands off — the output of E, before any parsing.
#[derive(Debug, Clone)]
struct RawReceipt {
    store: String,
    payload: String, // one "ITEM<TAB>CENTS" line per purchase
}

// The normalized row — the typed boundary between T and L.
#[derive(Debug, Clone, PartialEq)]
struct LineItem {
    item: String,
    cents: u32,
}

// E — pull raw text from somewhere external.
trait Fetch {
    fn fetch(&self) -> RawReceipt;
}

// T — parse raw text into normalized rows. Pure and total over its input.
trait Parse {
    fn parse(&self, raw: &RawReceipt) -> Vec<LineItem>;
}

// L — persist normalized rows; report how many landed.
trait Store {
    fn store(&mut self, rows: &[LineItem]) -> usize;
}

struct StubShop;
impl Fetch for StubShop {
    fn fetch(&self) -> RawReceipt {
        RawReceipt {
            store: "corner-shop".into(),
            payload: "coffee\t450\nbagel\t325\n".into(),
        }
    }
}

struct ReceiptParser;
impl Parse for ReceiptParser {
    fn parse(&self, raw: &RawReceipt) -> Vec<LineItem> {
        raw.payload
            .lines()
            .filter(|l| !l.is_empty())
            .map(|l| {
                let (item, cents) = l.split_once('\t').expect("well-formed line");
                LineItem { item: item.to_string(), cents: cents.parse().expect("numeric cents") }
            })
            .collect()
    }
}

struct MemStore {
    rows: Vec<LineItem>,
}
impl Store for MemStore {
    fn store(&mut self, rows: &[LineItem]) -> usize {
        self.rows.extend_from_slice(rows);
        rows.len()
    }
}

fn main() {
    // The pipeline names only the three seams, never the concrete shop.
    let source = StubShop;
    let transform = ReceiptParser;
    let mut sink = MemStore { rows: vec![] };

    let raw = source.fetch();          // E
    let rows = transform.parse(&raw);  // T
    let n = sink.store(&rows);         // L

    println!("fetched from {}", raw.store);
    println!("parsed {} rows", rows.len());
    println!("stored {n} rows");
    println!("first: {:?}", sink.rows[0]);
}

fetched from corner-shop
parsed 2 rows
stored 2 rows
first: LineItem { item: "coffee", cents: 450 }

main names Fetch, Parse, and Store — never StubShop by knowing what a corner shop is. Swap in a SupermarketApi that implements Fetch and the parse/store code does not change one character. That is the seam, in miniature.

Why Row is one tagged enum

Look again at the Sink signature: async fn load(&self, rows: &[Row]) -> …. One method, one row type — yet CelesTrak emits space objects, Space-Track emits conjunctions, and TheSpaceDevs emits launches. If each source had its own row type, you would need a different Sink per source, and the orchestrator could not treat them uniformly.

So Row is a single enum over all of them, and the #[serde(tag = "kind", …)] attribute makes each variant serialize with a discriminant field on the wire:

use serde::{Serialize, Deserialize};
#[derive(Serialize, Deserialize, Debug, PartialEq)]
#[serde(tag = "kind", rename_all = "snake_case")]
enum Row {
    SpaceObject { norad_id: u32 },
    Launch { name: String },
}

fn main() {
    let r = Row::Launch { name: "Starlink G-99".into() };
    let json = serde_json::to_string(&r).unwrap();
    println!("{json}");
    // Round-trips: the tag is how the reader knows which variant to build.
    let back: Row = serde_json::from_str(&json).unwrap();
    println!("{}", back == r);
}

{"kind":"launch","name":"Starlink G-99"}
true

The "kind":"launch" field is not decoration. It is how the downstream reader — panoptes-gen — knows which variant to reconstruct from a line of the file without out-of-band knowledge of which source wrote it. Untagged, {"name": "Starlink G-99"} would be ambiguous the moment two variants shared a field shape. The tag is the contract that lets one file hold rows from every pipeline.

Object safety: why the orchestrator can schedule a dyn Source

The orchestrator (Part V) does not know about CelestrakSource. It holds a Box<dyn Source> — a source with its concrete type erased behind a vtable — and calls .extract() through that pointer. For that to compile, Source must be dyn-compatible (the current name for "object-safe"): every method must be callable through a vtable.

The real Source::extract is async, which normally is not dyn-compatible — but #[async_trait] rewrites it into a method returning a boxed future, a single concrete signature a vtable can hold. That is the whole reason the trait carries the attribute. Here is the boxed-source pattern working (same shape as a_source_can_be_boxed_as_dyn in the real test suite):

// deps: async-trait = "0.1", tokio = { version = "1", features = ["full"] }
use async_trait::async_trait;

#[derive(Debug)]
struct RawReceipt { store: String, payload: String }

#[async_trait]
trait Fetch: Send + Sync {
    fn name(&self) -> &str;
    async fn fetch(&self) -> RawReceipt;
}

struct StubShop;

#[async_trait]
impl Fetch for StubShop {
    fn name(&self) -> &str { "corner-shop" }
    async fn fetch(&self) -> RawReceipt {
        RawReceipt { store: self.name().to_string(), payload: "coffee\t450\n".into() }
    }
}

#[tokio::main]
async fn main() {
    // The orchestrator holds exactly this: a source with its type erased.
    let source: Box<dyn Fetch> = Box::new(StubShop);
    let raw = source.fetch().await;
    println!("{} -> {:?}", source.name(), raw.store);
}

corner-shop -> "corner-shop"

The trap: a generic method silently makes the trait un-schedulable

Object safety is easy to break by accident, and the error only shows up at the Box<dyn …> line — far from the method that caused it. Add one generic method and the whole trait can no longer be a trait object:

trait Source {
    // The generic `T` is the poison: each call could pick a different T,
    // so there is no single function pointer to put in a vtable.
    fn extract_as<T: From<String>>(&self) -> T;
}

fn main() {
    // The error lands HERE, not on the method above.
    let _boxed: Box<dyn Source> = todo!();
}

error[E0038]: the trait `Source` is not dyn compatible
note: for a trait to be dyn compatible it needs to allow building a vtable

That E0038 is the compiler telling you the orchestrator would never be able to hold this trait. The fix is to keep trait methods monomorphic — take &self and concrete arguments, return concrete types — which is exactly why Source, Transform, and Sink have the plain, generic-free signatures they do. Schedulability is a property you design in from the first line, by keeping the trait dyn-compatible.

The one-for-one mapping

Everything in the receipt toy maps exactly onto what you are about to build, with the domain swapped and the network made real. RawReceipt becomes RawRecord { source, payload, fetched_at }; LineItem becomes the tagged Row enum; Fetch::fetch (sync) becomes Source::extract (async, returning Result<Vec<RawRecord>, EtlError>); Parse::parse becomes Transform::transform (&RawRecord → Vec<Row>, still pure); Store::store becomes Sink::load (async, appends JSONL, returns the count). The concrete StubShop becomes CelestrakSource { base_url, group }, ReceiptParser becomes CelestrakTransform (3-line TLE blocks → Row::SpaceObject), and MemStore becomes JsonlSink { path }. And where the toy's main named only the three traits, the orchestrator names only Box<dyn Source>, dyn Transform, and dyn Sink. The shape is identical; only the domain and the I/O change — learn the shape here and the CelesTrak build is filling in a template you already understand.

Questions to lock

1. Which stage is async and which is pure, and why is that split what makes the transform trivially unit-testable?
2. What are the two typed boundaries in the pipeline, and what type sits on each?
3. Why is Row a single enum tagged by kind rather than one row type per source? What downstream problem does the tag solve?
4. What does "dyn-compatible" mean, and which one change to a trait's method would make the orchestrator unable to hold a Box<dyn Source>?
5. Where does validation happen in E/T/L, and why is a successful Transform a proof of well-formedness rather than a step you run afterward?

The graded quiz for this arc is at the end of the part, on the Concept-Check: Extraction page.

Concept: Parsing the TLE Format

Kind: Concept. This is the T stage's hardest job: turning a 1960s fixed-width text format into typed orbital elements, correctly.

The idea

CelesTrak hands you satellite orbits as two-line element sets (TLEs) — a fixed-width text format designed for punch cards and never changed since. It looks like this (the canonical ISS example that every TLE tutorial uses, and that the real tle.rs test suite uses):

ISS (ZARYA)
1 25544U 98067A   08264.51782528 -.00002182  00000-0 -11606-4 0  2927
2 25544  51.6416 247.4627 0006703 130.5360 325.0288 15.72125391563537

A block is three lines: a name, then line 1 and line 2 of the actual elements. Nothing in it is delimited — there are no commas, no tabs, no key=value. A field is defined purely by which columns it occupies. The inclination is "whatever characters sit in columns 9 through 16 of line 2." Parse it by splitting on whitespace and you will be wrong the moment a field is blank or a sign is missing; you must slice by column.

Three things about this format bite everyone, and this chapter is about all three: column indexing (the format is specified 1-based; Rust slices are 0-based), the mod-10 checksum (how each line proves it was not corrupted in transit), and a pair of encoding quirks (a 2-digit year that pivots across a century, and an eccentricity with its decimal point left out).

A toy fixed-width record

Before the real thing, the same shape on a domain that cannot be the answer key — a ground-station log line. It has the two features that make TLEs tricky: fields pinned to columns, and an implied-decimal number. The spec is written the way real fixed-width specs are, 1-based and inclusive:

cols 1-3   station code    e.g. "KWJ"
cols 5-8   range, km       e.g. " 412"
cols 10-15 elevation       implied leading "0." → "123456" means 0.123456

// A toy fixed-width record that mirrors TLE's shape.
const REC: &str = "KWJ 412  123456";
//                  123456789012345   <- 1-based column ruler

/// Translate a 1-based inclusive column span [a, b] to a Rust byte slice.
/// The rule that removes the off-by-one for good: `line[a-1 .. b]`.
fn cols(line: &str, a: usize, b: usize) -> &str {
    &line[a - 1..b]
}

fn main() {
    let station = cols(REC, 1, 3);
    let range = cols(REC, 5, 8).trim();
    let elev_digits = cols(REC, 10, 15);
    let elevation: f64 = format!("0.{elev_digits}").parse().unwrap();

    println!("station:   {station:?}");
    println!("range:     {range:?}");
    println!("elevation: {elevation}");
}

station:   "KWJ"
range:     "412"
elevation: 0.123456

Two moves in that toy are exactly the moves the real parser makes: cols(line, a, b) maps a 1-based inclusive spec span onto line[a-1..b], and the implied decimal is recovered with format!("0.{digits}"). Hold onto both.

The trap: 0-based slice vs 1-based columns

Here is the single most common TLE-parsing bug, and it is silent. The spec says a field is in "columns 1–3." You reach for a Rust range and write line[1..3] because the numbers match. But Rust ranges are 0-based and end-exclusive, while the spec is 1-based and end-inclusive — so line[1..3] grabs the wrong two characters and, crucially, does not panic:

const REC: &str = "KWJ 412  123456";

fn main() {
    // Spec: station code is columns 1-3 (1-based, inclusive) => "KWJ".
    let correct = &REC[0..3]; // a-1 .. b  =>  1-1 .. 3
    println!("correct REC[0..3]: {correct:?}");

    // THE TRAP: reading "columns 1-3" as the Rust range 1..3.
    let wrong = &REC[1..3];
    println!("naive   REC[1..3]: {wrong:?}  <- off by one, no panic");
}

correct REC[0..3]: "KWJ"
naive   REC[1..3]: "WJ"  <- off by one, no panic

It compiles, it runs, it returns a plausible-looking string — and every downstream number is shifted by one column. That is why the real parser routes every field through a single helper. Look at tle.rs:

fn slice(line: &str, start: usize, end: usize) -> Result<&str, EtlError> {
    line.get(start..end)
        .ok_or_else(|| EtlError::Parse(format!("line too short for cols {start}..{end}: {line:?}")))
}

Note two defenses at once. First, the real code writes the spans already converted to 0-based half-open form — the ISS inclination, spec columns 9–16, is fetched as slice(line2, 8, 16). Converting once, at the call site, and never re-deriving it is how you kill the off-by-one. Second, it uses line.get(start..end) (which returns Option) rather than line[start..end] (which panics): a truncated line becomes an EtlError::Parse, not a crash.

The checksum: catching corruption

Each TLE line ends in a single mod-10 checksum digit in column 69. The rule is deliberately crude, because it was designed to survive teletype transmission: walk columns 1–68, add each digit's value, count each minus sign as 1, count everything else (letters, spaces, periods, plus signs) as 0, and take the total mod 10. Here is the real tle_checksum, verbatim:

/// mod-10 checksum: digits sum, each '-' counts 1, all else 0.
fn tle_checksum(line: &str) -> u8 {
    let sum: u32 = line
        .chars()
        .take(68) // columns 1..=68; the 69th char is the checksum itself
        .map(|c| match c {
            '0'..='9' => c.to_digit(10).unwrap(),
            '-' => 1,
            _ => 0,
        })
        .sum();
    (sum % 10) as u8
}

fn main() {
    // The canonical ISS TLE. Column 69 (the last char) is the checksum the
    // line claims: 7 on each line.
    let l1 = "1 25544U 98067A   08264.51782528 -.00002182  00000-0 -11606-4 0  2927";
    let l2 = "2 25544  51.6416 247.4627 0006703 130.5360 325.0288 15.72125391563537";

    println!("line 1 computed checksum: {}", tle_checksum(l1));
    println!("line 1 claims:            {}", l1.chars().nth(68).unwrap());
    println!("line 2 computed checksum: {}", tle_checksum(l2));
    println!("line 2 claims:            {}", l2.chars().nth(68).unwrap());

    // Flip one digit and the checksum no longer matches — corruption caught.
    let corrupt = l1.replacen("25544", "25545", 1);
    println!("after 1-digit edit:       {}", tle_checksum(&corrupt));
}

line 1 computed checksum: 7
line 1 claims:            7
line 2 computed checksum: 7
line 2 claims:            7
after 1-digit edit:       8

Two things to internalize. The .take(68) is not arbitrary: it stops before the checksum digit, because you cannot include the checksum in the sum you are checking it against. And '-' => 1 is the rule everyone forgets — minus signs contribute, plus signs and periods do not. The last line shows the payoff: change one character in the payload and the computed checksum (8) no longer equals what the line claims (7), so the corruption is caught before you ever try to parse a bad number.

The parser calls this through verify_checksum, which reads the claimed digit from column 69 and compares:

fn verify_checksum(line: &str) -> Result<(), EtlError> {
    let expected = line
        .chars()
        .nth(68) // column 69, 0-based index 68
        .and_then(|c| c.to_digit(10))
        .ok_or_else(|| EtlError::Parse(format!("missing checksum digit in {line:?}")))?
        as u8;
    let got = tle_checksum(line);
    if got != expected {
        return Err(EtlError::Parse(format!(
            "checksum mismatch in {line:?}: computed {got}, line says {expected}"
        )));
    }
    Ok(())
}

A bad checksum is an EtlError::Parse — which, from Part I, is permanent, not transient. Retrying a corrupt line will never fix it, so the retry loop (Part IV) must not waste attempts on it. The error classification you built earlier is exactly what makes "reject and move on" the correct behavior here.

The two encoding quirks

Epoch year pivot

The epoch is stored as a 2-digit year in columns 19–20 plus a fractional day-of-year in columns 21–32. Two digits cannot span a century, so TLEs use a fixed pivot: 57–99 mean 1957–1999, and 00–56 mean 2000–2056 (1957 is chosen because that is Sputnik — nothing was in orbit before it). The real decode_epoch encodes exactly that:

fn main() {
    for yy in [57, 99, 0, 8, 26, 56] {
        // The TLE century pivot, verbatim from decode_epoch:
        let year = if yy < 57 { 2000 + yy } else { 1900 + yy };
        println!("{yy:02} -> {year}");
    }
}

57 -> 1957
99 -> 1999
00 -> 2000
08 -> 2008
26 -> 2026
56 -> 2056

So the ISS line's 08 decodes to 2008, not 1908 — and a satellite launched in 2026 carries 26, decoding to 2026. Get the pivot backwards and every recent satellite jumps to the early 1900s, decades before spaceflight existed.

Implied-decimal eccentricity

Eccentricity is always between 0 and 1, so the format saves a character by omitting the leading 0.. Columns 27–33 of line 2 hold 0006703, which means 0.0006703. The parser puts the decimal back exactly the way the toy did:

fn main() {
    // Columns 27-33 of the ISS line 2, as raw characters:
    let raw = "0006703";
    let eccentricity: f64 = format!("0.{raw}").parse().unwrap();
    println!("eccentricity = {eccentricity}");
}

eccentricity = 0.0006703

Read those seven characters as a bare number and you get 6703.0 — an "eccentricity" seven million times too large, which would classify the ISS as a wildly elliptical comet. The format!("0.{}") is small but load-bearing.

Putting it together

parse_tle(line1, line2) runs the whole gauntlet in order: verify both checksums, confirm both lines carry the same NORAD id (a mismatch means the 3-line block was misaligned by an upstream truncation — something checksums alone cannot catch), then slice each field by its 0-based-converted column span and parse it. The worked ISS result, from the real parses_iss_tle_to_elements test:

inclination_deg        ≈ 51.6416   (cols 9-16  of line 2)
eccentricity           ≈ 0.0006703 (cols 27-33, decimal implied)
mean_motion_rev_per_day≈ 15.72125  (cols 53-63 of line 2)
orbit_class            = LEO        (derived: period ≈ 91.6 min < 128)

Every one of those numbers depends on getting the column span right, the checksum passing first, and the eccentricity's decimal restored. Miss any of the three traps and you get a plausible-looking OrbitalElements that is quietly, dangerously wrong — the worst kind of parse bug, because nothing crashes.

Questions to lock

1. The spec says a field is in "columns 9–16." What is the correct Rust slice range, and why does line[9..16] give the wrong answer without panicking?
2. In the mod-10 checksum, what does a - contribute, and what do +, ., and letters contribute? Which of these does the common broken implementation get wrong?
3. Why does tle_checksum take only the first 68 characters instead of all 69?
4. The ISS line's epoch year field is 08. What year is that, and what year would 57 be? State the pivot rule.
5. Columns 27–33 of line 2 read 0006703. What eccentricity does that encode, and what would you get by parsing it as a bare number?
6. Why is a bad-checksum line an EtlError::Parse (permanent) rather than an EtlError::Network (transient), and what does the retry loop do differently because of that?

The graded quiz for this arc is at the end of the part, on the Concept-Check: Extraction page.

Build: Workspace + Domain Model + the Source Trait

Maps to: Phase 0 / Task 1.1 (the etl-core seams). Kind: Build — spec and test names only. You write the implementation.

Objective

Turn the project into a Cargo workspace and build the etl-core crate that every other crate will depend on: the domain model (OrbitalElements, SpaceObject, and the Conjunction keystone), and the three pipeline traits (Source, Transform, Sink) plus their boundary types (RawRecord, Row). This is the crate the dependency arrow always points toward — it names nothing else in the workspace.

You already sketched EtlError and NoradId in Part I; this chapter is where they get a home, alongside the domain and the traits.

Scaffold

Convert to a workspace. Replace the top-level Cargo.toml with a virtual manifest (no [package], just [workspace]). It lists members and — the piece that pays off across six crates — a [workspace.dependencies] table every crate inherits from, so a version is pinned in exactly one place:

# Cargo.toml (workspace root)
[workspace]
resolver = "2"
members = ["crates/etl-core"]   # more crates join as later arcs add them

[workspace.dependencies]
serde = { version = "1", features = ["derive"] }
serde_json = "1"
chrono = { version = "0.4", features = ["serde"] }
thiserror = "2"
async-trait = "0.1"
tokio = { version = "1", features = ["full"] }
sha2 = "0.10"
# dev
pretty_assertions = "1"

Create the crate: crates/etl-core/Cargo.toml, crates/etl-core/src/lib.rs, and the modules error.rs, ids.rs, domain.rs, pipeline.rs.

crates/etl-core/Cargo.toml — inherit each dependency with { workspace = true }, and know why each is here:

Add tokio = { workspace = true, features = ["test-util"] } under [dev-dependencies] so async tests can use paused time later.

Expected result: cargo test -p etl-core → all domain and pipeline tests green.

The spec (givens)

Domain: OrbitClass (in domain.rs)

A coarse regime enum. Derives: Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize. serde: #[serde(rename_all = "SCREAMING_SNAKE_CASE")]. Variants: Leo, Meo, Geo, Heo (which serialize as "LEO", "MEO", "GEO", "HEO").

Domain: OrbitalElements (in domain.rs)

The Keplerian mean elements a TLE carries. Derives: Debug, Clone, Copy, PartialEq, Serialize, Deserialize. Fields (all f64 except the first):

epoch: DateTime<Utc>
inclination_deg, raan_deg, eccentricity,
arg_perigee_deg, mean_anomaly_deg, mean_motion_rev_per_day

Derived quantities are methods, not stored fields, so the stored form stays canonical. Constants and formulas (given — this is physics, not the lesson):

EARTH_MU_KM3_S2 = 398_600.4418   // Earth's gravitational parameter, km^3/s^2
EARTH_RADIUS_KM = 6378.137       // equatorial radius

period_minutes()      = 1440.0 / mean_motion_rev_per_day
semi_major_axis_km()  = cbrt(MU / n^2), where n = mean_motion * 2π / 86_400  (rad/s)
apogee_km()           = a * (1 + ecc) - R_earth
perigee_km()          = a * (1 - ecc) - R_earth

orbit_class() classifies in this order (the order matters — eccentricity is checked first):

ecc > 0.25                      -> HEO
else period_minutes < 128       -> LEO
else period_minutes in [1430, 1450]  -> GEO
else                            -> MEO

Domain: SpaceObject, Conjunction, Launch (in domain.rs)

All three derive Debug, Clone, PartialEq, Serialize, Deserialize.

SpaceObject { norad_id: NoradId, name: String, intl_designator: String,
              operator: Option<String>, elements: OrbitalElements }
   // intl_designator is the raw TLE field, e.g. "98067A" (not hyphenated)
   // convenience: orbit_class(&self) -> OrbitClass delegates to elements

Conjunction { id: String, tca: DateTime<Utc>, miss_distance_km: f64,
              collision_probability: f64, primary: NoradId, secondary: NoradId }
   // the keystone entity; `id` is content-addressed at load time (Part IV)

Launch { id: String, name: String, net: DateTime<Utc>, provider: String,
         payloads: Vec<String>, description: String }
   // `description` is prose — the field the RAG arc later embeds

The three traits + boundary types (in pipeline.rs)

Copy these signatures exactly; they are the contract the rest of the course builds against.

RawRecord { source: String, payload: String, fetched_at: DateTime<Utc> }
   derives: Debug, Clone, PartialEq

Row  (derives: Debug, Clone, PartialEq, Serialize, Deserialize)
   #[serde(tag = "kind", rename_all = "snake_case")]
   variants: SpaceObject(SpaceObject), Conjunction(Conjunction), Launch(Launch)

#[async_trait]
trait Source: Send + Sync {
    fn name(&self) -> &str;
    async fn extract(&self) -> Result<Vec<RawRecord>, EtlError>;
}

trait Transform: Send + Sync {
    fn transform(&self, raw: &RawRecord) -> Result<Vec<Row>, EtlError>;
}

#[async_trait]
trait Sink: Send + Sync {
    async fn load(&self, rows: &[Row]) -> Result<usize, EtlError>;
}

lib.rs declares the modules and re-exports the public names (EtlError, NoradId, the domain types, RawRecord, Row, Source, Transform, Sink) so downstream crates write use etl_core::Source;, not use etl_core::pipeline::Source;.

Test names to hit

In domain.rs:

• iss_is_low_earth_orbit — ISS-like elements (mean motion ≈ 15.5): assert period_minutes() ≈ 92.90, orbit_class() == Leo, and perigee_km() in 300.0..500.0.
• geostationary_is_classified_geo — GEO-like elements (mean motion ≈ 1.0027): assert period_minutes() ≈ 1436.1 and orbit_class() == Geo.
• orbit_class_serializes_screaming — serde_json::to_string(&OrbitClass::Geo) == "\"GEO\"".
• space_object_roundtrips_through_json — serialize a SpaceObject, deserialize it, assert equal.

In pipeline.rs:

• row_tags_by_kind — serialize a Row::Launch, assert the JSON contains "kind":"launch", then round-trip it back to an equal Row.
• a_source_can_be_boxed_as_dyn — define a Stub impl of Source inside the test, bind it as Box<dyn Source>, .extract().await it, and assert on the result. This test is the point: it proves the trait is dyn-compatible, which is the property the orchestrator depends on four arcs from now.

The build loop (you drive)

1. Workspace first. Convert the root Cargo.toml, create crates/etl-core/, and get an empty cargo build -p etl-core to succeed before writing any logic.
2. OrbitClass and OrbitalElements. Write orbit_class_serializes_screaming and iss_is_low_earth_orbit first. Predict the period for mean motion 15.5 before you run — 1440 / 15.5 — so a green test confirms your formula, not your luck.
3. Predict the classifier ordering. Before implementing orbit_class(), ask: an object with ecc = 0.3 and a 100-minute period — LEO or HEO? The branch order in the spec answers it. Write the code to match your prediction, then check.
4. SpaceObject + the roundtrip test.
5. The traits. Add RawRecord, Row, and the three trait definitions. Write row_tags_by_kind and a_source_can_be_boxed_as_dyn. If a_source_can_be_boxed_as_dyn will not compile, re-read the object-safety section of the concept page — the fix is in the trait, not the test.
6. Run green, commit.

Done when

cargo test -p etl-core is green across all six tests above (plus your Part I error/ids tests). You now have the crate every other arc imports — the domain, the seams, and the proof (a_source_can_be_boxed_as_dyn) that those seams can be scheduled. Next you make Source and Transform real against CelesTrak.

Build: The CelesTrak Source — Extract + Transform

Maps to: Tasks 1.1 (TLE parser) + 1.2 (CelesTrak source). Kind: Build — spec and test names only. No implementation.

Objective

Create the etl-sources crate and make the first real pipeline stages: the TLE parser (the heart of T), a small HTTP status classifier, the CelestrakSource (E — an open GET returning TLE text), and the CelestrakTransform (T — 3-line TLE blocks into Row::SpaceObject). Every test hits a wiremock server, never the live network.

By the end, CelestrakSource implements Source and CelestrakTransform implements Transform — the two traits you defined in etl-core, now with a real API behind them.

Scaffold

Create the crate: crates/etl-sources/Cargo.toml, src/lib.rs, src/tle.rs, src/http.rs, src/celestrak.rs. Add "crates/etl-sources" to the workspace members.

crates/etl-sources/Cargo.toml — depends only on etl-core plus the machinery a real HTTP source forces on us:

[dependencies]
etl-core = { path = "../etl-core" }
serde = { workspace = true }
serde_json = { workspace = true }
chrono = { workspace = true }        # DateTime for the epoch decode + fetched_at
async-trait = { workspace = true }   # Source::extract is async
reqwest = { workspace = true }       # the actual HTTP GET
tokio = { workspace = true }

[dev-dependencies]
pretty_assertions = { workspace = true }
tokio = { workspace = true, features = ["test-util"] }
wiremock = { workspace = true }      # the mock server every source test hits

reqwest, wiremock, and chrono are not on the Rust playground; build and test this in your workspace with cargo test -p etl-sources, exactly as the async arc of the first course did.

Expected result: cargo test -p etl-sources → all TLE, http, and celestrak tests green.

Part A — the TLE parser (tle.rs)

The concept page (Parsing the TLE Format) is the reference; this is the interface to build. Public functions:

pub fn tle_checksum(line: &str) -> u8
pub fn decode_epoch(year2: &str, day_frac: &str) -> Result<DateTime<Utc>, EtlError>
pub fn tle_norad(line1: &str) -> Result<u32, EtlError>
pub fn tle_intl_designator(line1: &str) -> Result<String, EtlError>
pub fn parse_tle(line1: &str, line2: &str) -> Result<OrbitalElements, EtlError>

Note parse_tle takes two lines (the name is handled by the transform, not here) and returns bare OrbitalElements.

The column map (given — the TLE wire format). These are already converted from the spec's 1-based inclusive columns to Rust's 0-based half-open slices, so you write them once and never re-derive:

Rules to encode (all covered on the concept page):

• tle_checksum: sum over the first 68 chars; digit → its value, '-' → 1, everything else → 0; result mod 10.
• Epoch year pivot: yy < 57 ⇒ 2000 + yy, else 1900 + yy. Day-of-year is 1-based; the fractional part is a fraction of a day.
• Eccentricity: parse format!("0.{}", cols_26_33.trim()).
• parse_tle order: verify both checksums first → confirm both lines carry the same NORAD id (a mismatch is a misaligned block that checksums cannot catch) → then slice and parse each field.
• Slice with line.get(a..b) (returns Option), never line[a..b] (panics): a short line must become EtlError::Parse, not a crash.

Test names to hit (use the canonical ISS block as the fixture — both its checksums are 7):

L1 = "1 25544U 98067A   08264.51782528 -.00002182  00000-0 -11606-4 0  2927"
L2 = "2 25544  51.6416 247.4627 0006703 130.5360 325.0288 15.72125391563537"

• checksum_matches_known_lines — tle_checksum(L1) == 7 and tle_checksum(L2) == 7.
• rejects_bad_checksum — flip the last digit of L2; parse_tle(L1, &bad) returns Err(EtlError::Parse(_)).
• decodes_2008_epoch — decode_epoch("08", "264.51782528") decodes to year 2008, month 9, day 20, hour 12.
• parses_iss_tle_to_elements — parse_tle(L1, L2): inclination ≈ 51.6416, eccentricity ≈ 0.0006703, mean motion ≈ 15.72125391, orbit_class() == Leo.
• reads_norad_and_designator — tle_norad(L1) == 25544, tle_intl_designator(L1) == "98067A".
• rejects_misaligned_line_pair — build a line 2 for a different catalog number with a corrected checksum, so only the cross-line id mismatch (not a checksum) can reject it; assert Err(EtlError::Parse(_)).

Part B — the HTTP status classifier (http.rs)

Every source funnels non-success responses through one function so the transient/permanent decision is made once. Public functions:

pub fn retry_after_secs(headers: &HeaderMap) -> u64   // parse Retry-After (secs); absent/unparseable => 1
pub fn classify_status(status: StatusCode, headers: &HeaderMap) -> EtlError

Mapping (given):

429            -> EtlError::RateLimited { retry_after_secs }   (transient, carries backoff)
401 | 403      -> EtlError::Auth(...)                          (permanent)
other 4xx      -> EtlError::Permanent(...)                     (permanent — retrying won't help)
5xx and else   -> EtlError::Network(...)                       (transient)

Test names to hit: maps_429_to_rate_limited_with_backoff, maps_401_403_to_permanent_auth, maps_404_to_permanent_not_network, maps_5xx_to_transient_network. Each asserts both the variant and (via is_transient()) its retry classification.

Part C — CelestrakSource (E) + CelestrakTransform (T) (celestrak.rs)

CelestrakSource — fields base_url: String, group: String; constructor new(base_url, group). It implements Source:

• name(&self) returns "celestrak".
• extract builds the URL {base}/NORAD/elements/gp.php?GROUP={group}&FORMAT=tle (trim a trailing / off base_url), does a reqwest::get, maps any transport error to EtlError::Network, and on a non-success status returns classify_status(status, headers). On success it reads the body text into one RawRecord { source: "celestrak", payload, fetched_at: Utc::now() } and returns a one-element Vec.

CelestrakTransform — a unit struct implementing Transform:

• Split raw.payload into lines, trim_end each, drop empties.
• Walk the lines in chunks(3). A chunk that is not exactly 3 lines is a trailing partial block → EtlError::Parse.
• For each (name, l1, l2): parse_tle(l1, l2)? for the elements, tle_norad(l1)? and tle_intl_designator(l1)? for the ids, operator: None, name trimmed. Push a Row::SpaceObject(...).

Test names to hit:

• extract_returns_raw_tle_text — the core E test. Start a MockServer; mount a mock matching method("GET"), path("/NORAD/elements/gp.php"), query_param("GROUP", "geo"), query_param("FORMAT", "tle"), responding 200 with the ISS block as a body string. Point CelestrakSource::new(server.uri(), "geo") at it; assert one RawRecord, source == "celestrak", payload contains "25544".
• transform_parses_tle_blocks_to_space_objects — the core T test. Hand CelestrakTransform a RawRecord whose payload is the ISS block; assert one row, a Row::SpaceObject with norad_id == NoradId(25544), name == "ISS (ZARYA)", intl_designator == "98067A".
• server_error_is_transient_network — mock returns 500; extract().await is Err(EtlError::Network(_)) and is_transient().
• rate_limit_is_retryable_with_backoff — mock returns 429 with header retry-after: 42; the error is EtlError::RateLimited { retry_after_secs: 42 } and is_transient().
• not_found_is_permanent_not_retryable — mock returns 404; the error is EtlError::Permanent(_) and !is_transient().

The build loop (you drive)

1. TLE checksum first. Write checksum_matches_known_lines, predict both values are 7, implement tle_checksum, run green.
2. rejects_bad_checksum, then decode_epoch. Predict decodes_2008_epoch's year before running — is 08 → 1908 or 2008? The pivot answers it.
3. parse_tle. With checksum + epoch green, assemble the field slices. Run parses_iss_tle_to_elements; if a number is off by a factor of a thousand, look at the eccentricity's implied decimal; if off by one column, look at the slice bounds.
4. classify_status. Small and pure — four tests, four branches.
5. extract_returns_raw_tle_text. Stand up the mock, point the source at server.uri(), assert on the RawRecord. Predict what happens if you drop the FORMAT=tle matcher (silent 404), then confirm.
6. transform_parses_tle_blocks_to_space_objects, then the three error-taxonomy tests.
7. Run green, commit.

Done when

cargo test -p etl-sources is green. CelestrakSource and CelestrakTransform now satisfy the Source and Transform traits from etl-core — the first concrete E and T. All that is missing is L: a place to write the rows.

Build: The JSONL Sink — Load

Maps to: Task 1.3. Kind: Build — spec and test names only. No implementation.

Objective

Implement JsonlSink — the L stage, and the file contract the rest of the world reads. It appends normalized Row values to a file as JSON Lines: one JSON object per line, appended (never truncated) so a run only ever adds to what came before. This is the exact format panoptes-gen consumes downstream, which is why "append-only, one row per line" is a contract and not an implementation detail.

Scaffold

Create: crates/etl-core/src/sink_jsonl.rs; declare the module and re-export JsonlSink from lib.rs.

Dependencies: none new. serde/serde_json and tokio were declared when you built etl-core. The async I/O uses tokio::fs and tokio::io::AsyncWriteExt, both under tokio's full features.

Expected result: cargo test -p etl-core sink → the sink tests pass.

The spec (givens)

JsonlSink — one field, path: PathBuf; constructor new(path: impl Into<PathBuf>). It implements the Sink trait from pipeline.rs:

#[async_trait]
impl Sink for JsonlSink {
    async fn load(&self, rows: &[Row]) -> Result<usize, EtlError> { ... }
}

Behavior load must have:

• Open path with create + append (tokio::fs::OpenOptions::new().create(true).append(true)). Append is the whole point: a second load must not erase the first.
• Serialize each Row with serde_json::to_string. A serialization failure maps to EtlError::Parse(...). (In practice Row always serializes, but the boundary stays honest.)
• Write one line per row — each JSON object followed by a '\n'. Build the whole batch into one buffer and do a single write_all, so a batch is one write, not N.
• Return the count of rows written (rows.len()).
• The ? on file operations relies on EtlError's #[from] std::io::Error (Part I) — an I/O failure surfaces as EtlError::Io for free.

Test names to hit

• writes_one_row_per_line_and_roundtrips — write a temp path (use std::env::temp_dir() plus the process id so parallel test runs do not collide, and remove_file first for a clean slate). load two rows; assert the returned count is 2; read the file back and assert it has exactly 2 lines; serde_json::from_str the first line and assert it equals the first row you wrote. This proves both halves of the contract at once: one row per line, and a clean round-trip through serde.
• append_accumulates_across_loads — call load twice on the same path with the same 2 rows; read the file and assert it now has 4 lines. This is the append guarantee: the second call added to the file rather than replacing it.

The build loop (you drive)

1. Write writes_one_row_per_line_and_roundtrips first. You need sample rows — reuse a couple of Row::Launch values. Predict the returned count and the line count before you run.
2. Implement load with create+append and the single buffered write_all.
3. Run, check the round-trip assertion — if the deserialized row differs, your serialize/deserialize derives on the domain types disagree (revisit the roundtrips test from the domain build).
4. Write append_accumulates_across_loads. Predict: if you accidentally used .truncate(true) or .write(true) without .append(true), how many lines would this test see? Then confirm append gives 4.
5. Run green, commit.

Done when

cargo test -p etl-core sink is green. You now have all three stages — E (CelestrakSource), T (CelestrakTransform), L (JsonlSink) — each satisfying its trait from etl-core. A complete pipeline is now just "call extract, feed each RawRecord through transform, hand the rows to load." The next arcs make that wiring incremental, authenticated, retried, and eventually scheduled — but the seams are all in place as of this chapter.

Concept-Check: Extraction

You have installed two seams this arc: the Source / Transform / Sink traits, and the Extract / Transform / Load split, with the CelesTrak source and the JSONL sink as the first concrete implementations. This check pins down the parts that are easy to get almost right — the TLE checksum arithmetic, why Row carries a tag, which stage owns parsing, and the object-safety property that lets the orchestrator schedule a dyn Source later.

Two of these are Tracing questions: read the program, decide whether it compiles, and if it does, predict its exact output before revealing the answer. Predicting is the point.

Concept: Session Auth & Secrets from the Environment

Kind: Concept. Seam introduced this arc: an authenticated source — a login handshake that yields a session cookie, plus the discipline of keeping secrets out of the URL and off disk.

The idea

CelesTrak was an open GET: no key, no login, anyone can pull the TLE catalog. Space-Track is not. Before it will answer a single query it wants you to log in — POST your username and password to a login endpoint, and in return it hands you a session cookie. Every subsequent request must carry that cookie, or the server treats you as a stranger and refuses.

That is the shape of a huge fraction of real-world APIs, and it forces two questions this chapter answers:

1. How does the login handshake work as code? You POST credentials once, pull the cookie off the response, and forward it by hand on the query that follows. Two requests, one session.
2. Where do the credentials come from — and where must they never go? They come from the environment. They must never be hard-coded in the source, never written to disk, and — the one people get wrong — never placed in a URL.

The cookie mechanic is mechanical; the secrets discipline is the part that separates a toy from something you would run against a real account. We take both in turn, on a toy you cannot mistake for the answer key, then map it onto the shipped SpaceTrackSource.

A runnable toy: the members-only clubhouse

Here is the whole session dance on a domain that is obviously not orbital data — a members-only clubhouse. The "server" knows the correct password and, on a good login, mints a session cookie. The lounge only serves callers who present that cookie. Everything is synchronous std, so it runs on the playground:

/// A members-only clubhouse. The server knows the correct password and, on a
/// successful login, hands back a short-lived session token — a cookie.
struct Clubhouse {
    password: String,
    valid_cookie: String,
}

impl Clubhouse {
    /// POST /login — check the password, mint a session cookie.
    fn login(&self, member: &str, pass: &str) -> Result<String, &'static str> {
        if pass == self.password {
            Ok(format!("session={}; member={member}", self.valid_cookie))
        } else {
            Err("401 bad credentials")
        }
    }

    /// GET /lounge — only serves callers who present the session cookie.
    fn lounge(&self, cookie: &str) -> Result<String, &'static str> {
        // The server reads the cookie back off the request, exactly as it
        // issued it — no re-sending of the password.
        if cookie.contains(&format!("session={}", self.valid_cookie)) {
            Ok("today's special: orbital-mechanics trivia night".into())
        } else {
            Err("403 no valid session")
        }
    }
}

fn main() {
    let server = Clubhouse {
        password: "hunter2".into(),
        valid_cookie: "abc123".into(),
    };

    // The secret comes from the environment, with a fallback so this toy runs
    // anywhere. In production there is no fallback — a missing var is an error.
    let secret = std::env::var("CLUB_PASSWORD").unwrap_or_else(|_| "hunter2".into());

    // 1. Log in once. The password crosses the wire in the POST body, never
    //    in the URL.
    let cookie = server.login("ada", &secret).expect("login should succeed");
    println!("got cookie:  {cookie}");

    // 2. Every later request forwards the cookie by hand — not the password.
    match server.lounge(&cookie) {
        Ok(data) => println!("lounge says: {data}"),
        Err(e) => println!("denied:      {e}"),
    }

    // 3. A request with no session is refused.
    match server.lounge("session=stolen-guess") {
        Ok(data) => println!("lounge says: {data}"),
        Err(e) => println!("no cookie:   {e}"),
    }

    // The anti-pattern, shown but not used: the secret in the URL query string.
    // This string would land in access logs, proxy caches, and browser history.
    let leaky_url = format!("https://club.example/lounge?password={secret}");
    let safe_url = "https://club.example/lounge".to_string();
    println!("leaky url:   {leaky_url}");
    println!("safe url:    {safe_url}  (+ Cookie header)");
}

got cookie:  session=abc123; member=ada
lounge says: today's special: orbital-mechanics trivia night
no cookie:   403 no valid session
leaky url:   https://club.example/lounge?password=hunter2
safe url:    https://club.example/lounge  (+ Cookie header)

Read the session lifecycle off those three calls:

• The password crosses the wire exactly once, in the login call. After that, only the cookie travels. If the cookie leaks, an attacker gets a bounded session; the password — the reusable, un-rotatable secret — stayed put.
• The cookie is forwarded by hand. The client holds the string returned from login and passes it into lounge. Nothing is automatic: no cookie jar, no framework magic. This is deliberately explicit because the real source does exactly this, one line at a time.
• No cookie means no service. The third call presents a bogus session and gets 403. The server does not fall back to re-authenticating; a missing or wrong cookie is simply a refusal.

Secrets come from the environment — and never touch the URL

Look again at where secret came from: std::env::var("CLUB_PASSWORD"). That is the whole rule for credentials. The environment is a per-process channel that is not committed to git, not baked into the binary, and not written to any file the program controls. A deployment sets the variable; the code reads it; nobody's password ends up in a commit.

Reading an environment variable is safe and ordinary. Note, though, that in Rust 2024 writing one (std::env::set_var) is an unsafe operation — it can race other threads reading the environment — a small, pointed reminder that the environment is process-global state you read from, not a scratchpad you scribble into. Your source only ever reads.

Now the part people get wrong. The two printed URLs are identical except that the leaky one carries the password as a query parameter. Both would "work." Only one is safe:

The mapping onto the shipped SpaceTrackSource

The clubhouse is SpaceTrackSource with the domain swapped and the I/O made real. Against the shipped crates/etl-sources/src/spacetrack.rs:

• Clubhouse::password + the env read ↔ Credentials, a two-field struct (user, pass) built either by Credentials::new("u", "p") or by Credentials::from_env(), which reads SPACETRACK_USER and SPACETRACK_PASS and returns EtlError::Auth("… not set") when either is missing. Missing credentials are an auth error, classified permanent — no amount of retrying conjures a password.
• login ↔ a real reqwest POST to {base}/ajaxauth/login, sending the credentials as form fields (identity, password) in the request body — not the URL.
• pulling the cookie off the response ↔ reading the SET_COOKIE response header, keeping only the first ;-separated segment (the name=value pair, dropping Path, HttpOnly, and friends).
• forwarding the cookie to lounge ↔ attaching that string as the COOKIE request header on the follow-up GET to the CDM query endpoint.

Here is the shipped handshake, trimmed to the two requests and the cookie in between. It uses reqwest, async-trait, and chrono, so it is rust,ignore — build and test it in the workspace with cargo test -p etl-sources, never on the playground:

async fn extract(&self) -> Result<Vec<RawRecord>, EtlError> {
    let base = self.base_url.trim_end_matches('/');
    let client = reqwest::Client::new();

    // 1. Log in. Credentials ride in the form body, never the URL.
    self.limiter.acquire().await; // (rate limiting — next chapter)
    let login = client
        .post(format!("{base}/ajaxauth/login"))
        .form(&[
            ("identity", self.creds.user.as_str()),
            ("password", self.creds.pass.as_str()),
        ])
        .send()
        .await
        .map_err(|e| EtlError::Network(e.to_string()))?;
    if !login.status().is_success() {
        // Non-2xx at login funnels through the SAME classifier as any request:
        // 401/403 -> Auth (permanent); 5xx/429 -> transient, still retryable.
        return Err(crate::http::classify_status(login.status(), login.headers()));
    }

    // 2. Pull the session cookie off the response, keep only "name=value".
    let cookie = login
        .headers()
        .get(reqwest::header::SET_COOKIE)
        .and_then(|v| v.to_str().ok())
        .and_then(|v| v.split(';').next())
        .ok_or_else(|| EtlError::Auth("login returned no session cookie".into()))?
        .to_string();

    // 3. Query, carrying the cookie by hand in the COOKIE header.
    self.limiter.acquire().await;
    let resp = client
        .get(format!(
            "{base}/basicspacedata/query/class/cdm_public/orderby/TCA/format/json"
        ))
        .header(reqwest::header::COOKIE, cookie)
        .send()
        .await
        .map_err(|e| EtlError::Network(e.to_string()))?;
    if !resp.status().is_success() {
        return Err(crate::http::classify_status(resp.status(), resp.headers()));
    }
    let payload = resp.text().await.map_err(|e| EtlError::Network(e.to_string()))?;
    Ok(vec![RawRecord { source: self.name().to_string(), payload, fetched_at: Utc::now() }])
}

The shape is the toy's, line for line: log in → keep the cookie → forward it → read the body. Only the transport and the domain changed. The self.limiter.acquire().await calls belong to the next chapter — for now, read them as "wait my turn before each request."

Transient vs permanent — at login, not just at query

Here is the detail that pays off two arcs later, and it is worth pausing on. Both the login request and the query request funnel their non-2xx responses through the same classify_status from Part II. That is not just code reuse — it is the point.

A login can fail two very different ways:

• 401 / 403 — the credentials are wrong. This is EtlError::Auth, and it is permanent. Retrying sends the identical wrong password and gets the identical rejection; the retry loop must give up immediately and tell you to fix your credentials.
• 503 / 429 — Space-Track's login endpoint is briefly overloaded or throttling you. This is EtlError::Network or EtlError::RateLimited, and it is transient. The credentials are fine; the server was momentarily unavailable. The retry loop should wait and try the whole handshake again.

If login errors were all bucketed as "auth failed," a transient 503 at the login step would abort a run that a single retry would have completed. Because login goes through classify_status, the transient/permanent decision you built once in Part II is made correctly here too — a failed login is classified by the same rules as a failed query.

Questions to lock

1. Walk the session handshake in three steps: what is sent in the login request, what comes back, and what the client attaches to the follow-up request. Which secret crosses the wire, and how many times?
2. Where do credentials come from in the shipped source, and what are the three places a secret must never go?
3. Why is a secret in a URL query string worse than the same secret in a request header or body? Name two places the query-string version leaks to.
4. Credentials::from_env returns EtlError::Auth when a variable is missing. Is that transient or permanent, and why is that the right call?
5. A login attempt comes back 503. Should the run give up or retry, and which function makes that decision correctly for both the login and the query steps?

The graded quiz for this arc is at the end of the part, on the Concept-Check: Authentication page.

Concept: Rate Limiting with a Token Bucket

Kind: Concept. Seam introduced this arc: a client-side rate limiter — a token bucket every request must pass through, and the paused-clock testing technique that makes it deterministic.

The idea

Space-Track will ban a client that hammers it. The etiquette is explicit: keep your request rate low, and do not burst without limit. So before every request — login included — the source must ask permission from a rate limiter that only lets requests through at a controlled pace.

The classic algorithm for this is a token bucket, and it is worth building by hand because the whole idea fits in a paragraph:

• The bucket holds up to capacity tokens and starts full.
• It refills continuously at refill_per_sec tokens per second, never exceeding capacity.
• To make a request you must spend one token. If the bucket has one, you take it and go immediately. If it is empty, you wait exactly long enough for one token to refill, then go.

Two behaviors fall straight out of that. A burst is fine up to capacity: a full bucket serves that many requests back-to-back with no waiting. After the burst, you are throttled to the steady refill rate — one request every 1 / refill_per_sec seconds. The bucket smooths a bursty caller down to a rate the server tolerates, while still allowing short spikes.

Why hand-rolled, not governor

There is a perfectly good crate (governor) that does this. We build it by hand anyway, for the same reason we hand-rolled the TLE parser: the token bucket is the lesson. Writing the refill arithmetic yourself — tokens = min(capacity, tokens + elapsed × rate) — is how you actually understand what a rate limiter guarantees and what it does not. It is thirty lines. Reach for governor in production if you like; here, the thirty lines are the point.

A runnable toy: the bucket with a logical clock

The real limiter sleeps on tokio::time, which does not run on the playground. So here is the same arithmetic with the clock made explicit — we pass in "the current logical time" instead of sleeping, which also makes every step visible. capacity tokens, refilling at refill_per_sec:

/// A token bucket driven by a *logical* clock we pass in — no real sleeping, so
/// the arithmetic is visible and deterministic. `capacity` tokens sit in the
/// bucket; it refills continuously at `refill_per_sec`, never past `capacity`.
struct Bucket {
    tokens: f64,
    capacity: f64,
    refill_per_sec: f64,
    last: f64, // logical seconds at which we last refilled
}

impl Bucket {
    fn new(capacity: f64, refill_per_sec: f64) -> Self {
        Self { tokens: capacity, capacity, refill_per_sec, last: 0.0 }
    }

    /// Try to take one token at logical time `now`.
    /// `Ok(())` if one was free (and consumed); `Err(wait)` = seconds until one is.
    fn acquire(&mut self, now: f64) -> Result<(), f64> {
        let elapsed = now - self.last;
        self.tokens = (self.tokens + elapsed * self.refill_per_sec).min(self.capacity);
        self.last = now;
        if self.tokens >= 1.0 {
            self.tokens -= 1.0;
            Ok(())
        } else {
            Err((1.0 - self.tokens) / self.refill_per_sec)
        }
    }
}

fn main() {
    // capacity 3, refills half a token per second (one every 2s).
    let mut b = Bucket::new(3.0, 0.5);

    // A full bucket serves a burst of 3 at t=0 with no waiting.
    for i in 1..=3 {
        println!("t=0  req {i}: {:?}", b.acquire(0.0));
    }
    // The 4th request at t=0 finds the bucket empty; it must wait.
    println!("t=0  req 4: {:?}", b.acquire(0.0));
    // After 2 logical seconds exactly one token has refilled.
    println!("t=2  req 5: {:?}", b.acquire(2.0));
}

t=0  req 1: Ok(())
t=0  req 2: Ok(())
t=0  req 3: Ok(())
t=0  req 4: Err(2.0)
t=2  req 5: Ok(())

Every line of that output is the algorithm showing its work:

• Requests 1–3 at t=0 all pass. The bucket started with 3 tokens; the burst spends them with no wait. That is the capacity guarantee.
• Request 4 at t=0 fails with Err(2.0). The bucket is empty and no time has passed, so zero tokens have refilled. The wait is (1.0 − 0.0) / 0.5 = 2.0 seconds — the time to refill one token at half a token per second.
• Request 5 at t=2 passes. Two logical seconds elapsed, refilling 2 × 0.5 = 1.0 token — exactly enough. The .min(self.capacity) never comes into play here, but it is the guard that stops a long idle period from letting the bucket overflow past 3 and permitting an unbounded burst later.

The real TokenBucket::acquire does not return a wait for you to handle — it sleeps that duration itself and loops, so from the caller's side acquire().await simply returns once a token is yours. Same arithmetic; the loop hides the waiting.

The shipped limiter, and the paused-clock test

The real thing (in crates/etl-sources/src/limiter.rs) is the same math wrapped for concurrency and real time: the state lives behind an Arc<Mutex<…>> so a cloned bucket shared across tasks stays consistent, and the wait is a real tokio::time::sleep. It is rust,ignore — tokio async is not playground-safe:

// deps: tokio = { version = "1", features = ["full"] }
pub async fn acquire(&self) {
    loop {
        let wait = {
            let mut s = self.state.lock().await;
            let now = Instant::now();
            let elapsed = now.duration_since(s.last).as_secs_f64();
            s.tokens = (s.tokens + elapsed * self.refill_per_sec).min(self.capacity);
            s.last = now;
            if s.tokens >= 1.0 {
                s.tokens -= 1.0;
                return; // a token was free — go now
            }
            // Not enough yet: compute how long one token takes to refill.
            Duration::from_secs_f64((1.0 - s.tokens) / self.refill_per_sec)
        }; // lock released here, BEFORE we sleep
        tokio::time::sleep(wait).await;
    }
}

Now the problem that technique solves. This limiter is about waiting. A test for "the bucket blocks when empty, then refills after one second" would seem to require the test to actually sleep a second — which makes the suite slow and, worse, flaky: on a loaded CI box the real elapsed time drifts, and an assertion like "took at least 900 ms" fails at random.

The fix is #[tokio::test(start_paused = true)]. Under a paused clock, tokio's timer is virtual. When every task in the test is parked on a sleep, tokio does not sit and wait for the wall clock — it auto-advances the virtual clock straight to the next timer's deadline. A one-second sleep therefore completes instantly in real time, yet Instant::now() inside the test reports that a full virtual second elapsed. The waiting is real to the code and free to the clock:

#[tokio::test(start_paused = true)]
async fn bucket_blocks_when_empty_then_refills() {
    let bucket = TokenBucket::new(1, 1.0);
    let start = Instant::now();
    bucket.acquire().await; // consumes the one starting token
    bucket.acquire().await; // empty: must wait ~1s for a refill
    // Virtual time advanced ~1s, but the test ran in microseconds.
    assert!(start.elapsed() >= Duration::from_millis(900));
}

The mapping onto the shipped code

• Bucket fields ↔ TokenBucket: capacity and refill_per_sec are the same; the toy's tokens + last live inside the real bucket's Arc<Mutex<State>> so clones share one bucket.
• Bucket::new(3.0, 0.5) ↔ TokenBucket::new(30, 0.5) — the Space-Track default: 30 tokens of burst, refilling one every two seconds. The real constructor takes capacity: u32 and converts to f64 internally.
• the toy's Err(wait) you handle ↔ the real acquire sleeps wait itself and loops, so the caller just .awaits.
• passing now explicitly ↔ Instant::now() under a paused clock in tests, real wall-clock time in production.

Every request in SpaceTrackSource::extract — the login POST and the CDM GET — begins with self.limiter.acquire().await. A token is spent per request, login included, which is exactly the etiquette Space-Track asks for.

Questions to lock

1. State the token-bucket rule in three parts (capacity, refill, spend-or-wait). What two behaviors — one about bursts, one about steady state — does it produce?
2. In the toy, request 4 returned Err(2.0). Where does the 2.0 come from, arithmetically?
3. What does .min(self.capacity) prevent? Give a scenario where dropping it would let a caller exceed the intended rate.
4. What does #[tokio::test(start_paused = true)] change about time, and why does that make a timing test both fast and deterministic rather than flaky?
5. Why must the shipped acquire drop the mutex guard before it sleeps? What goes wrong for other tasks if it does not?

The graded quiz for this arc is at the end of the part, on the Concept-Check: Authentication page.

Concept: The Run Ledger

Kind: Concept. Seam introduced this arc: the run ledger — an append-only record of every pipeline run, and the single artifact the orchestrator's incrementality, retry, and backfill all read from later.

The idea

So far a pipeline run is amnesiac: it fetches, transforms, writes, and forgets. Run it again tomorrow and it has no idea it ran today — no idea whether today succeeded, how far it got, or whether it should even bother. Every capability the orchestrator adds two arcs from now needs the pipeline to remember its own history:

• Incrementality — "only fetch what is new" — needs to know the watermark the last successful run reached.
• Retry / resume — "did the last run finish?" — needs to know the outcome of the last run.
• Backfill — "fill the gaps" — needs to know which runs are missing or failed.

All three read from one artifact: a run ledger. Each time a source runs, you append one record — when it started, when it finished, whether it succeeded, the high-water mark it reached, and how many rows it wrote. The ledger is the pipeline's memory, and this chapter builds it before anything consumes it, because you cannot be incremental until you can remember.

Two properties make the ledger trustworthy, and they are the whole lesson:

1. It is append-only. A run adds a line; it never rewrites or deletes an earlier one. The file is a complete, ordered history — an audit log, not a mutable "current state" you can corrupt with a bad write.
2. A failed run does not advance the watermark. The watermark that matters is the one from the last successful run. A run that blew up halfway must not be allowed to claim it made progress, or the next incremental run would skip data the failed run never actually fetched.

A runnable toy: the backup log

Here is both properties on a domain that is plainly not orbital data — a nightly backup job. Each run records the outcome and the watermark (here, a day-of-year, for legibility). The log is a Vec standing in for the append-only file; last_success scans it and returns the most recent successful run, skipping failures. Pure std, runs on the playground:

/// How a run ended. A failed run carries a reason; a successful one does not.
#[derive(Debug, Clone, PartialEq)]
enum Outcome {
    Success,
    Failed(String),
}

/// One line of the ledger: a single run of a single job. `watermark` is the
/// high-water mark the run reached — here, a day-of-year for legibility.
#[derive(Debug, Clone)]
struct Run {
    job: String,
    outcome: Outcome,
    watermark: u32,
}

/// An append-only log. In the real crate this is JSONL on disk; here it is a
/// `Vec` so the example runs anywhere. The rule is the same: `append` only ever
/// adds; it never rewrites a prior entry.
struct Ledger {
    runs: Vec<Run>,
}

impl Ledger {
    fn new() -> Self {
        Self { runs: vec![] }
    }

    fn append(&mut self, run: Run) {
        self.runs.push(run);
    }

    /// The most recent *successful* run for a job — failures are skipped.
    fn last_success(&self, job: &str) -> Option<&Run> {
        self.runs
            .iter()
            .filter(|r| r.job == job && r.outcome == Outcome::Success)
            .last()
    }
}

fn main() {
    let mut ledger = Ledger::new();

    // A good run advances the watermark to day 200.
    ledger.append(Run { job: "backup".into(), outcome: Outcome::Success, watermark: 200 });

    // The next run fails at day 201. It is recorded — but it must NOT count
    // as the last success, so it must not advance the watermark.
    ledger.append(Run {
        job: "backup".into(),
        outcome: Outcome::Failed("disk full".into()),
        watermark: 201,
    });

    let wm = ledger.last_success("backup").map(|r| r.watermark);
    println!("after a failure, watermark = {wm:?}");

    // A later good run at day 202 does advance it.
    ledger.append(Run { job: "backup".into(), outcome: Outcome::Success, watermark: 202 });
    let wm = ledger.last_success("backup").map(|r| r.watermark);
    println!("after a success, watermark = {wm:?}");

    // A job that never ran has no watermark at all.
    println!("unknown job            = {:?}", ledger.last_success("archive").map(|r| r.watermark));
}

after a failure, watermark = Some(200)
after a success, watermark = Some(202)
unknown job            = None

The three lines of output are the ledger's whole contract:

• After the failed run, the watermark is still 200 — not 201. The failure was appended (the history is complete and honest), but last_success filters it out, so the failed run's optimistic watermark never counts. This is the second property in action: a failure records itself without claiming progress.
• After the next success, the watermark advances to 202. Because last_success takes the last matching record, successive good runs move the watermark forward. Incrementality is "start from last_success().watermark."
• An unknown job returns None — the correct answer for "we have never successfully run this," which the orchestrator reads as "there is no watermark yet; do a full pull."

The append-only discipline, enforced by the type

Notice the toy's Ledger exposes exactly two operations: append and read (last_success). There is no update, no delete, no set. That is deliberate — the append-only guarantee is baked into the interface, not left to a convention you might violate later. The history can only grow.

The Outcome comparison inside last_success — r.outcome == Outcome::Success — is why Outcome derives PartialEq. Drop that derive and the filter stops compiling, which is a useful thing to have seen once:

// last_success must ask each record: was this a success?
enum Outcome { Success, Failed }

fn is_success(o: &Outcome) -> bool {
    *o == Outcome::Success // compares two Outcomes — needs PartialEq
}

fn main() {
    println!("{}", is_success(&Outcome::Success));
}

That fails to compile with error[E0369]: binary operation == cannot be applied to type Outcome — the compiler even suggests #[derive(PartialEq)]. The lesson is small but concrete: the ledger's failure-filtering rule depends on being able to compare outcomes, so Outcome must be PartialEq, exactly as it is in the shipped type.

The mapping onto the shipped Ledger

The toy is crates/etl-core/src/ledger.rs with the storage made durable and the fields fleshed out. The shipped code is rust,ignore — it uses chrono, serde, and tokio::fs:

#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
#[serde(tag = "outcome", rename_all = "snake_case")]
pub enum Outcome {
    Success,
    Failed { reason: String },
}

#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct RunRecord {
    pub source: String,
    pub started_at: DateTime<Utc>,
    pub finished_at: DateTime<Utc>,
    pub outcome: Outcome,
    pub watermark: Option<DateTime<Utc>>,
    pub rows_written: usize,
}

impl Ledger {
    pub async fn append(&self, rec: &RunRecord) -> Result<(), EtlError> {
        let mut file = OpenOptions::new()
            .create(true)
            .append(true) // <- append, never truncate: history only grows
            .open(&self.path)
            .await?;
        let line = serde_json::to_string(rec).map_err(|e| EtlError::Parse(e.to_string()))?;
        file.write_all(line.as_bytes()).await?;
        file.write_all(b"\n").await?; // one record per line — JSONL
        Ok(())
    }

    pub async fn last_success(&self, source: &str) -> Result<Option<RunRecord>, EtlError> {
        // ... read the file, or Ok(None) if it does not exist yet ...
        let mut last = None;
        for line in text.lines().filter(|l| !l.trim().is_empty()) {
            let rec: RunRecord = serde_json::from_str(line)?;
            if rec.source == source && rec.outcome == Outcome::Success {
                last = Some(rec); // last write for this source wins
            }
        }
        Ok(last)
    }
}

Field for field: the toy's job ↔ source; outcome: Outcome is the same idea, now #[serde(tag = "outcome")] so each JSONL line is self-describing (just like Row is tagged by kind); watermark: u32 ↔ watermark: Option<DateTime<Utc>> (a real timestamp, None before the first success); and the shipped record adds started_at, finished_at, and rows_written for the audit trail. The Vec becomes an append-only JSONL file opened with .create(true).append(true) — the same file contract the JSONL sink uses, applied to run history instead of rows. And last_success is identical in spirit: scan every line, keep the last one whose source matches and whose outcome is Success.

Questions to lock

1. Name the three orchestrator features the ledger seeds, and say which field of RunRecord each one reads.
2. Why must a failed run be recorded but not advance the watermark? What breaks in the next run if a failure were allowed to advance it?
3. What does last_success return for a source that has never succeeded, and how should the orchestrator interpret that?
4. Why is the ledger append-only? What would .truncate(true) instead of .append(true) silently destroy?
5. Why does Outcome need to derive PartialEq? Which line of last_success depends on it, and what error do you get without it?

The graded quiz for this arc is at the end of the part, on the Concept-Check: Authentication page.

Build: The Space-Track Source

Maps to: Tasks 2.1 (token-bucket limiter) + 2.2 (session auth + the Space-Track source). Kind: Build — spec and test names only. No implementation.

Objective

Build the first authenticated, rate-limited source. Two files:

1. limiter.rs — the hand-rolled TokenBucket from the rate-limiting concept: burst up to capacity, refill at a fixed rate, acquire sleeps until a token is free.
2. spacetrack.rs — Credentials (from new or the environment), SpaceTrackSource (E — log in, keep the cookie, run the CDM query, rate-limited), and SpaceTrackTransform (T — Space-Track CDM JSON into Row::Conjunction).

By the end, SpaceTrackSource implements Source and SpaceTrackTransform implements Transform — the same two traits CelesTrak implemented, now with a login handshake and a limiter in front. Every test hits a wiremock server or runs under paused tokio time; nothing touches the live network or a real Space-Track account.

Scaffold

Create: crates/etl-sources/src/limiter.rs and crates/etl-sources/src/spacetrack.rs; declare both modules in lib.rs.

Dependencies: none new — reqwest, async-trait, chrono, serde/serde_json, tokio, and the dev-only wiremock + pretty_assertions were all declared when you created etl-sources for CelesTrak. The paused-clock tests need tokio's test-util feature, which the crate's [dev-dependencies] already enables (tokio = { workspace = true, features = ["test-util"] }).

reqwest, wiremock, chrono, and tokio time are not on the Rust playground. Build and test in the workspace with cargo test -p etl-sources.

Expected result: cargo test -p etl-sources → all limiter and spacetrack tests green (alongside the CelesTrak tests from Part II).

Part A — the token bucket (limiter.rs)

The rate-limiting concept page is the reference; this is the interface to build.

TokenBucket — a #[derive(Clone)] struct with fields capacity: f64, refill_per_sec: f64, and state: Arc<Mutex<State>>, where State { tokens: f64, last: Instant }. The Arc<Mutex<…>> is what lets a cloned bucket, shared across tasks, refer to one shared pool of tokens.

• new(capacity: u32, refill_per_sec: f64) -> Self — start the bucket full (tokens = capacity) with last = Instant::now(). (The u32 capacity is stored as f64.)
• async fn acquire(&self) — the loop from the concept page: lock the state, refill by elapsed × refill_per_sec capped at capacity, and if at least one token is available, spend it and return; otherwise compute the wait for one token to refill, drop the lock, tokio::time::sleep(wait).await, and loop.

The rule that matters (from the concept page): compute the wait inside the lock, then release the guard before the .await. Holding a MutexGuard across tokio::time::sleep would block every other caller for the whole sleep.

Test names to hit (both #[tokio::test(start_paused = true)] so virtual time fast-forwards through the waits):

• bucket_allows_burst_up_to_capacity — TokenBucket::new(3, 0.5); acquire().await three times; assert start.elapsed() is under ~50 ms. A full bucket serves the burst with no real waiting.
• bucket_blocks_when_empty_then_refills — TokenBucket::new(1, 1.0); acquire().await twice. The second call finds the bucket empty and must wait ~1 s for a refill; assert start.elapsed() >= 900 ms. Under the paused clock this passes in microseconds of wall time.

Part B — Credentials + SpaceTrackSource (E) (spacetrack.rs)

Credentials — a struct with public user: String and pass: String.

• new(user: impl Into<String>, pass: impl Into<String>) -> Self.
• from_env() -> Result<Self, EtlError> — read SPACETRACK_USER and SPACETRACK_PASS; if either is missing, return EtlError::Auth("… not set"). (Missing credentials are permanent — the retry loop must not retry them.)

SpaceTrackSource — fields base_url: String, creds: Credentials, limiter: TokenBucket; constructor new(base_url, creds, limiter). It implements Source:

• name(&self) returns "spacetrack".
• extract performs the handshake from the concept page, spending a token before each request:
  1. self.limiter.acquire().await, then POST {base}/ajaxauth/login with the credentials as form fields identity and password (trim a trailing / off base_url). A transport error maps to EtlError::Network.
  2. On a non-success login status, return classify_status(status, headers) — so 401/403 become EtlError::Auth (permanent) while 5xx/429 stay transient.
  3. Read the SET_COOKIE header, keep only the first ;-separated segment; if absent, EtlError::Auth("login returned no session cookie").
  4. self.limiter.acquire().await, then GET {base}/basicspacedata/query/class/cdm_public/orderby/TCA/format/json with that string in the COOKIE header. Non-success status → classify_status.
  5. On success, read the body text into one RawRecord { source: "spacetrack", payload, fetched_at: Utc::now() }; return a one-element Vec.

Test names to hit (wiremock):

• login_then_query_uses_cookie — the core E test. Mount two mocks: a POST /ajaxauth/login returning 200 with set-cookie: chocolatechip=abc123; Path=/, and a GET on the CDM query path guarded by header_exists("cookie"), returning 200 with a CDM JSON body, marked .expect(1). Point SpaceTrackSource::new(server.uri(), Credentials::new("u", "p"), bucket()) at it; assert one RawRecord, and that its payload contains a known NORAD id. The .expect(1) verifies on drop that the cookie-guarded query was actually hit — proof the cookie was forwarded.
• login_failure_is_auth_error — mount only a POST /ajaxauth/login returning 401; assert extract().await is Err(EtlError::Auth(_)).

Part C — SpaceTrackTransform (T) (spacetrack.rs)

SpaceTrackTransform — a unit struct implementing Transform. Space-Track serializes a CDM (Conjunction Data Message) as JSON where every number arrives as a string. Deserialize into a private RawCdm with #[serde(rename = "…")] on each field:

TCA       -> tca:        String   (RFC-3339-ish, e.g. "2026-07-20T12:00:00.000000")
MIN_RNG   -> min_rng_m:  String   (miss distance in METERS, as a string)
PC        -> pc:         Option<String>   (collision probability; may be absent/empty)
SAT_1_ID  -> sat1:       String
SAT_2_ID  -> sat2:       String

transform parses the payload as Vec<RawCdm> (a parse failure → EtlError::Parse), then for each CDM builds a Conjunction:

• tca: parse with NaiveDateTime::parse_from_str(&c.tca, "%Y-%m-%dT%H:%M:%S%.f") then .and_utc(); a bad value → EtlError::Parse.
• miss_distance_km: parse min_rng_m as f64 and divide by 1000.0 (meters → kilometers); unparseable → EtlError::Parse.
• collision_probability: if pc is present and non-empty, parse it as f64; otherwise default to 0.0.
• primary / secondary: c.sat1.parse() and c.sat2.parse() into NoradId — the newtype's FromStr returns EtlError::Parse on junk, so ? lifts it straight into the taxonomy.
• id: format!("{}_{}_{}", primary.0, secondary.0, tca.format("%Y%m%dT%H%M%S")). (This is a readable composite key; Part IV replaces it with a content-addressed hash.)

Push each as Row::Conjunction(...).

Test name to hit:

• transform_parses_cdm_to_conjunction — hand SpaceTrackTransform a RawRecord whose payload is a one-element CDM array (MIN_RNG "550.5", PC "0.0001", sats "25544" / "48274"). Assert one row, a Row::Conjunction with primary == NoradId(25544), secondary == NoradId(48274), miss_distance_km ≈ 0.5505 (note the ÷1000), and collision_probability ≈ 0.0001.

The build loop (you drive)

1. Token bucket first. Write bucket_allows_burst_up_to_capacity, predict "no wait," implement new + the acquire loop, run green.
2. bucket_blocks_when_empty_then_refills. Predict the virtual elapsed time before running; confirm the paused clock makes it instant in real time. If it actually hangs, you are almost certainly holding the lock across the .await — release the guard before sleeping.
3. Credentials::from_env. Small and pure; a missing var is EtlError::Auth.
4. login_then_query_uses_cookie. Stand up both mocks; predict what happens if you drop the COOKIE header (silent 404 via the header_exists guard), then confirm. Get the happy path green.
5. login_failure_is_auth_error. One mock returning 401; assert EtlError::Auth.
6. transform_parses_cdm_to_conjunction. Build the RawCdm structs, wire the meters→km and string→number conversions. If the miss distance is 1000× too big, you skipped the divide.
7. Run green, commit.

Done when

cargo test -p etl-sources is green. You now have a second concrete Source/Transform pair — authenticated, rate-limited, emitting the Conjunction keystone — behind the exact same traits as CelesTrak. The orchestrator will not be able to tell them apart, which is the entire point. Next you build the ledger that lets a run remember it happened.

Build: The Ledger + the Conjunction Keystone Entity

Maps to: Task 2.3 (the run ledger) + the Conjunction domain type (defined in Part I's domain build, first emitted for real this arc). Kind: Build — spec and test names only. No implementation.

Objective

Build the run ledger — the append-only record of every pipeline run that Part V's orchestrator reads for incrementality, retry, and backfill — and pin down the Conjunction keystone entity that the Space-Track transform you just built emits. The ledger is new code (ledger.rs); Conjunction already exists in domain.rs from Part I, but this is the arc where it stops being a definition and starts being data, so we revisit it as the entity it was designed to be.

By the end, cargo test -p etl-core covers the ledger's three behaviors: append-and-read-back, failures are ignored by last_success, and the watermark advances across successful runs.

Part A — the Conjunction keystone (recap, domain.rs)

Conjunction is the reason the whole project exists: a predicted close approach between two tracked objects. It is the "keystone" because it is the entity that references other entities — a conjunction is meaningless without the two NoradIds it points at. Its shipped shape (from Part I):

Conjunction {
    id: String,                    // composite key now; content-addressed in Part IV
    tca: DateTime<Utc>,            // time of closest approach
    miss_distance_km: f64,         // closest separation, in kilometers
    collision_probability: f64,    // 0.0 when the source omits it
    primary: NoradId,              // the two objects, by catalog number...
    secondary: NoradId,            // ...and this is where id-safety earns its keep
}

Why the keystone makes the newtype pay off. primary and secondary are both NoradId, not bare u32. If they were plain integers, swapping the two arguments when constructing a Conjunction would compile silently and mislabel a collision-avoidance record — the exact bug the newtype chapter warned about. Because both are NoradId, the values cannot be confused with any other kind of id (a launch id, a payload count); the ordering of the two NoradIds within a conjunction is then a discipline the transform gets right by construction. The keystone entity is where Part I's id-safety seam earns its keep.

You do not write new code in this part — Conjunction was built and tested in domain.rs already. The point is to recognize what the Space-Track transform produced: the first real Conjunction values, keyed by the composite id the transform formats.

Part B — the run ledger (ledger.rs)

The ledger concept page is the reference; this is the interface to build.

Create: crates/etl-core/src/ledger.rs; declare the module and re-export Ledger, RunRecord, and Outcome from lib.rs.

Dependencies: none new. The append-only file uses tokio::fs::OpenOptions and tokio::io::AsyncWriteExt (both under tokio's full features), and serde/serde_json for the JSONL encoding — all already declared in etl-core.

Outcome — how a run ended. Derives Debug, Clone, PartialEq, Serialize, Deserialize; tagged #[serde(tag = "outcome", rename_all = "snake_case")] so each variant is self-describing on the wire (the same tagging idea as Row's kind):

Outcome::Success
Outcome::Failed { reason: String }

PartialEq is not optional here — last_success compares rec.outcome == Outcome::Success, and without the derive that comparison will not compile (error[E0369]).

RunRecord — one line of the ledger. Derives Debug, Clone, PartialEq, Serialize, Deserialize:

RunRecord {
    source: String,                    // which source ran
    started_at: DateTime<Utc>,
    finished_at: DateTime<Utc>,
    outcome: Outcome,
    watermark: Option<DateTime<Utc>>,  // high-water mark reached; None before first success
    rows_written: usize,
}

Ledger — one field, path: PathBuf; constructor new(path: impl Into<PathBuf>). Two async methods:

• append(&self, rec: &RunRecord) -> Result<(), EtlError> — open path with create + append (OpenOptions::new().create(true).append(true)), serialize the record with serde_json::to_string (failure → EtlError::Parse), and write the line followed by a '\n'. One record per line — JSONL. The ? on the file ops relies on EtlError's #[from] std::io::Error from Part I.
• last_success(&self, source: &str) -> Result<Option<RunRecord>, EtlError> — read the file to a string; if it does not exist yet, return Ok(None) (match on ErrorKind::NotFound), not an error. Parse each non-empty line into a RunRecord; keep the last one whose source matches and whose outcome == Outcome::Success. Return that, or None.

Test names to hit

• append_then_last_success_reads_back — append one successful RunRecord for "celestrak"; assert last_success("celestrak") returns Some(rec) equal to what you wrote, and last_success("spacetrack") returns None. Proves the round-trip and the source filter. (Use a temp path — std::env::temp_dir() plus the process id — and remove_file first for a clean slate, exactly as the JSONL sink tests do.)
• last_success_ignores_failures — append a Success at watermark 2026-07-18…, then a Failed { reason } at watermark 2026-07-19…. Assert last_success returns the run whose watermark is still 2026-07-18…. This is the load-bearing test: the failed run was recorded but must not advance the watermark.
• watermark_advances_across_successful_runs — append two Success runs at 2026-07-18… then 2026-07-19…; assert last_success returns the 2026-07-19… watermark. Successive successes move the mark forward.

The build loop (you drive)

1. Outcome and RunRecord first. Define the types with their derives and serde tags. If a comparison later fails to compile, you dropped PartialEq on Outcome.
2. append + append_then_last_success_reads_back. Write the round-trip test, implement append (create+append, one line) and a first last_success. Predict the returned record and the None for the absent source before running.
3. last_success_ignores_failures. Predict the watermark (see the callout), then confirm your filter includes the outcome == Success clause.
4. watermark_advances_across_successful_runs. Two successes; assert the later watermark wins because last_success takes the last match.
5. Run green, commit.

Done when

cargo test -p etl-core is green across the three ledger tests (alongside the domain, pipeline, and sink tests from earlier arcs). The pipeline can now remember its own runs — the memory that Part IV's retry loop and Part V's incremental executor will both read from. Every seam the orchestrator needs is now in place: the E/T/L traits, the file contract, the error taxonomy, and — as of this chapter — the run ledger.

Concept-Check: Authentication

This arc added the machinery a private API demands: a login handshake that yields a session cookie, a token-bucket limiter in front of every request, and the run ledger that lets a pipeline remember what it did. This check pins down the parts that are easy to get almost right — how the cookie is forwarded, why a 503 at login is not the same as a 401, the token-bucket arithmetic and the paused clock that tests it, and the one rule that makes the ledger correct: a failed run never advances the watermark.

Three of these are Tracing questions: read the program, decide whether it compiles, and if it does, predict its exact output before revealing the answer. Predicting is the point — a wrong prediction with a good explanation teaches more than a lucky guess.

Concept: Pagination & Following next

Kind: Concept. The new problem this arc: the data no longer fits in one response — you must follow a cursor to the end, and defend against a cursor that never ends.

The idea

CelesTrak handed you every space object in one GET; the whole payload arrived in a single response. TheSpaceDevs (the Launch Library 2 API) does not work that way. There are thousands of launches, so the API returns them a page at a time — a hundred rows plus a pointer to the next hundred:

GET /2.2.0/launch/?limit=100&mode=detailed
{
  "next": "https://…/2.2.0/launch/?limit=100&offset=100&mode=detailed",
  "results": [ {…}, {…}, … 100 launches … ]
}

The response carries its own continuation. next is a full URL for the following page; results is this page's rows. To collect everything you follow the cursor: fetch, take the rows, read next, fetch that, repeat — until next comes back null. That null is the only thing that ends the loop.

This is cursor pagination (the server hands you an opaque "where to go next"), as opposed to offset pagination (you compute ?offset= yourself). The distinction rarely matters to the caller: either way, Extract for this source is not one request but a loop of requests, and the loop's termination condition lives in the response body, not in your code.

A runnable toy: draining pages by their cursor

The real extract pulls in reqwest, async-trait, and chrono, so it will not run on the playground. Here is the same shape on a domain that cannot be mistaken for the answer key — an in-memory "API" that is just a map from URL to page. Fetch a page, take its items, follow its next, stop when next is None:

use std::collections::HashMap;
use std::collections::HashSet;

// A page: some items, plus an optional cursor to the NEXT page.
#[derive(Clone)]
struct Page {
    items: Vec<String>,
    next: Option<String>,
}

// Stand-in for "the API": a fixed set of pages keyed by URL.
fn fetch(pages: &HashMap<String, Page>, url: &str) -> Page {
    pages[url].clone()
}

// Follow the `next` cursor to the end, guarding against a cyclic `next`.
fn drain(pages: &HashMap<String, Page>, start: &str) -> Result<Vec<String>, String> {
    let mut all = Vec::new();
    let mut visited = HashSet::new();
    let mut next = Some(start.to_string());
    while let Some(url) = next {
        // The cycle guard: refuse to fetch a URL we have already fetched.
        if !visited.insert(url.clone()) {
            return Err(format!("pagination cycle at {url}"));
        }
        let page = fetch(pages, &url);
        all.extend(page.items);
        next = page.next; // follow the cursor; None ends the loop
    }
    Ok(all)
}

fn main() {
    let mut pages = HashMap::new();
    pages.insert("/p1".to_string(), Page { items: vec!["a".into(), "b".into()], next: Some("/p2".into()) });
    pages.insert("/p2".to_string(), Page { items: vec!["c".into()], next: None });
    println!("{:?}", drain(&pages, "/p1"));

    // A buggy server whose last page points back at the first — an infinite loop
    // without the guard.
    let mut cyclic = HashMap::new();
    cyclic.insert("/p1".to_string(), Page { items: vec!["a".into()], next: Some("/p2".into()) });
    cyclic.insert("/p2".to_string(), Page { items: vec!["b".into()], next: Some("/p1".into()) });
    println!("{:?}", drain(&cyclic, "/p1"));
}

Ok(["a", "b", "c"])
Err("pagination cycle at /p1")

Two moves in that toy are exactly the moves the real source makes. The while let Some(url) = next loop is the follow-the-cursor pattern: the loop variable is refilled from each page's own next, and a None (the API's null) is what drains it. And visited.insert(url.clone()) is the cycle guard, which the next section is entirely about.

The trap: a cursor that never terminates

Following next to the end assumes the chain reaches an end. A correct server guarantees it: each next points strictly forward, and the last page's next is null. But your Extract loop is at the mercy of the server, and a buggy or malicious server can hand you a next that points back at a page you already fetched — or at itself. Now "follow next until it is null" never terminates. Your process spins forever, hammering the API, filling memory with duplicate rows, and never returning.

This is not paranoia; it is the difference between a loop that is correct on good input and one that is safe on any input. The Extract stage talks to the outside world, so it must assume the outside world can be wrong.

The defense is a visited set: remember every URL you have fetched, and refuse to fetch one twice. The shipped SpaceDevsSource::extract does exactly this — one HashSet and one line:

// crates/etl-sources/src/spacedevs.rs
// (real code — needs reqwest + async-trait + chrono; not playground-safe.
//  The toy above is the runnable version of this loop.)
let mut next = Some(format!("{base}/2.2.0/launch/?limit=100&mode=detailed"));
let mut pages = Vec::new();
// Guard against a cyclic or self-referential `next` from a buggy server.
let mut visited = std::collections::HashSet::new();

while let Some(url) = next {
    if !visited.insert(url.clone()) {
        return Err(EtlError::Parse(format!("pagination cycle at {url}")));
    }
    let body = /* fetch url (with retry — see the next concept) */;
    let page: Page = serde_json::from_str(&body)?;
    next = page.next;
    pages.push(RawRecord { source: self.name().to_string(), payload: body, fetched_at: Utc::now() });
}
Ok(pages)

HashSet::insert returns false when the value was already present — so if !visited.insert(url.clone()) reads as "if this URL was already seen, stop." One page produces one RawRecord; the transform parses each page's results later. The loop cannot run more than once per distinct URL, so it is guaranteed to terminate on any input the server can produce.

Why one RawRecord per page

Notice the loop pushes a separate RawRecord for each page rather than concatenating all the bodies into one. This keeps the E/T boundary clean: each RawRecord is exactly one server response, with its own fetched_at provenance, and the Transform parses one page's JSON at a time. If a later page is malformed, the parse error names that page; the good pages before it are untouched raw records. Extract's job is to bring back every page's bytes faithfully — deciding what those bytes mean is still the transform's job, one page at a time.

The one-for-one mapping

Everything in the toy maps onto the real source with the domain swapped and the I/O made real. The toy's HashMap<String, Page> fetch becomes a real reqwest::get against {base_url}/2.2.0/launch/?limit=100&mode=detailed; the toy's Page { items, next } becomes the deserialized Page { results, next }; the toy's all.extend(page.items) becomes pages.push(RawRecord { … }) (one raw record per page, parsed later); and the visited HashSet<String> is identical in both — the cycle guard does not change one character when the network becomes real. The shape is the loop; only the fetch inside it becomes async and fallible.

Questions to lock

1. What ends the follow-next loop under normal operation — and where does that termination signal live, in your code or in the server's response?
2. What can a buggy server do to a naive "follow next until null" loop, and what single data structure defends against it?
3. HashSet::insert returns a bool. What does false mean, and how does if !visited.insert(url) use that to detect a cycle?
4. Why is a detected pagination cycle classified as EtlError::Parse (permanent) rather than a transient error — and what does that classification save the retry loop from doing?
5. Why does the source emit one RawRecord per page instead of concatenating every page's body into one record?

The graded quiz for this arc is at the end of the part, on the Concept-Check: Pagination page.

Concept: Retry with Backoff & 429 / Retry-After

Kind: Concept. Seam spent this arc: the transient/permanent classification from Part I becomes a real retry loop.

The idea

A paginated extract makes many requests, and over many requests a transient failure is not a possibility, it is a certainty: a connection resets, a load balancer hiccups, the API returns 429 Too Many Requests because you are pulling a hundred pages in a row. The wrong response is to let the whole run die on the first blip. The right response is to retry — but only the failures that retrying could actually fix, and only after waiting, so you do not amplify the problem by hammering a struggling server.

Two rules make a retry loop correct rather than dangerous:

1. Retry only transient errors. Retrying a 404 or a checksum failure will produce the identical failure every time — it just wastes attempts and delays the inevitable error. This is precisely the is_transient() distinction you built in Part I: Network and RateLimited are worth retrying; Parse, Auth, and Permanent are not.
2. Back off between attempts, with a cap. Wait longer after each successive failure so a momentarily overloaded server gets breathing room — but cap the wait so you do not sleep for an hour. This is capped exponential backoff: base, 2×base, 4×base, … clamped to max_delay. And when the server tells you how long to wait (a Retry-After header on a 429), that number wins over anything you would compute.

A runnable toy: watching the backoff decision

The real with_retry needs tokio to actually sleep, so it is not playground-safe. But the decision it makes on each failure is pure arithmetic, and we can watch it. This toy mirrors the shipped logic exactly — retry only transient, Retry-After wins, otherwise capped exponential — and instead of sleeping, it records the delay each attempt would wait:

use std::time::Duration;

// A tiny error taxonomy mirroring EtlError's transient/permanent split.
#[derive(Debug)]
enum Err_ {
    Network,                    // transient
    RateLimited { after: u64 }, // transient, carries a server backoff
    Permanent,                  // never worth retrying
}

impl Err_ {
    fn is_transient(&self) -> bool {
        matches!(self, Err_::Network | Err_::RateLimited { .. })
    }
    fn retry_after(&self) -> Option<Duration> {
        match self {
            Err_::RateLimited { after } => Some(Duration::from_secs(*after)),
            _ => None,
        }
    }
}

struct Policy {
    max_attempts: u32,
    base: Duration,
    cap: Duration,
}

// The exact backoff decision `with_retry` makes, pulled out so we can watch it:
// a server-supplied Retry-After wins; otherwise capped exponential.
fn backoff(policy: &Policy, err: &Err_, attempt: u32) -> Duration {
    err.retry_after().unwrap_or_else(|| {
        let factor = 2u32.saturating_pow(attempt - 1);
        (policy.base * factor).min(policy.cap)
    })
}

// Run `op`, collecting the delays a real loop would sleep between attempts.
fn run(
    policy: &Policy,
    mut op: impl FnMut(u32) -> Result<&'static str, Err_>,
) -> (Result<&'static str, String>, Vec<Duration>) {
    let mut delays = Vec::new();
    let mut attempt = 0u32;
    loop {
        match op(attempt) {
            Ok(v) => return (Ok(v), delays),
            Err(e) => {
                attempt += 1;
                if attempt >= policy.max_attempts || !e.is_transient() {
                    return (Err(format!("{e:?}")), delays);
                }
                delays.push(backoff(policy, &e, attempt));
            }
        }
    }
}

fn main() {
    let policy = Policy {
        max_attempts: 5,
        base: Duration::from_millis(200),
        cap: Duration::from_secs(10),
    };

    // Transient until the 4th call: watch the delays double.
    let (r, d) = run(&policy, |n| if n < 3 { Err(Err_::Network) } else { Ok("ok") });
    println!("transient x3 -> {r:?}, delays = {d:?}");

    // A permanent error stops immediately — no delays at all.
    let (r, d) = run(&policy, |_| Err::<&str, _>(Err_::Permanent));
    println!("permanent    -> {r:?}, delays = {d:?}");

    // Retry-After (5s) wins over the computed 200ms backoff.
    let (r, d) = run(&policy, |n| if n == 0 { Err(Err_::RateLimited { after: 5 }) } else { Ok("ok") });
    println!("rate-limited -> {r:?}, delays = {d:?}");
}

transient x3 -> Ok("ok"), delays = [200ms, 400ms, 800ms]
permanent    -> Err("Permanent"), delays = []
rate-limited -> Ok("ok"), delays = [5s]

Read the three lines against the three rules. The transient run doubles the wait each time — 200ms, 400ms, 800ms — classic exponential backoff (and had it kept failing, the fourth delay would have been 1600ms, then it would clamp to the 10s cap). The permanent run does zero waiting and zero extra attempts: is_transient() returned false, so the loop returned the error on the first failure. And the rate-limited run waited 5s — the server's Retry-After, not the 200ms the exponential formula would have produced.

The real thing, verbatim

The shipped with_retry lives in etl-core (retry.rs) and is the exact same loop, made async so it can actually sleep:

// crates/etl-core/src/retry.rs  (needs tokio; not playground-safe)
pub async fn with_retry<F, Fut, T>(policy: RetryPolicy, mut op: F) -> Result<T, EtlError>
where
    F: FnMut() -> Fut,
    Fut: Future<Output = Result<T, EtlError>>,
{
    let mut attempt: u32 = 0;
    loop {
        match op().await {
            Ok(value) => return Ok(value),
            Err(err) => {
                attempt += 1;
                if attempt >= policy.max_attempts || !err.is_transient() {
                    return Err(err);
                }
                // A server-requested backoff always wins; otherwise exponential.
                let backoff = err.retry_after().unwrap_or_else(|| {
                    let factor = 2u32.saturating_pow(attempt - 1);
                    (policy.base_delay * factor).min(policy.max_delay)
                });
                sleep(backoff).await;
            }
        }
    }
}

Line for line, it is the toy: increment attempt, bail on attempt >= max_attempts || !is_transient(), compute retry_after().unwrap_or_else(exponential), sleep. op is an FnMut returning a fresh Future each call — it must be re-runnable, because retrying means calling it again. RetryPolicy defaults to max_attempts: 4, base_delay: 200ms, max_delay: 10s.

Deterministic tests via paused time

There is an obvious problem testing this: a test for "backs off 5 seconds on Retry-After" cannot actually sleep 5 real seconds — CI would crawl and the assertion would be flaky. The fix is tokio's paused clock. Under #[tokio::test(start_paused = true)], time does not advance on its own; a sleep completes instantly as far as the wall clock is concerned, but tokio::time::Instant::now() still advances by the slept duration. So you get to assert on elapsed virtual time without waiting:

// crates/etl-core/src/retry.rs — the shipped test, verbatim
#[tokio::test(start_paused = true)]
async fn honors_retry_after() {
    let calls = Arc::new(AtomicU32::new(0));
    let c = calls.clone();
    let start = tokio::time::Instant::now();
    let out = with_retry(policy(), || {
        let c = c.clone();
        async move {
            let n = c.fetch_add(1, Ordering::SeqCst);
            if n == 0 {
                Err(EtlError::RateLimited { retry_after_secs: 5 })
            } else {
                Ok(())
            }
        }
    })
    .await;
    assert!(out.is_ok());
    // The 5s Retry-After beats the 100ms base backoff.
    assert!(start.elapsed() >= Duration::from_secs(5));
}

The test runs in microseconds yet proves the loop waited five (virtual) seconds. That is the whole point of paused time: a timing-dependent behavior becomes a deterministic, fast assertion. The sibling tests retries_transient_then_succeeds (fails twice, then succeeds — three calls) and gives_up_on_permanent_immediately (one call, no retry) use the same paused-clock harness with an AtomicU32 counting how many times op actually ran.

Where the classification comes from

with_retry never inspects an HTTP status; it only reads is_transient() and retry_after() off the EtlError it is handed. The mapping from status code to error variant happens before the retry loop ever sees it, in classify_status (from Part II's http.rs): 429 → RateLimited { retry_after_secs }, 401|403 → Auth, other 4xx → Permanent, 5xx → Network. And RateLimited is where retry_after_secs is populated — parsed from the Retry-After header. So the chain is: response status → classify_status → typed EtlError → with_retry reads is_transient()/retry_after(). Each piece decides one thing, once.

Questions to lock

1. What are the two conditions that make with_retry stop and return the error instead of retrying?
2. Trace the backoff sequence for base = 200ms, cap = 1000ms across attempts 1–5. Where does the cap first bite?
3. When a 429 carries Retry-After: 5, what does with_retry sleep — the header value or the computed exponential backoff — and why is that the right call?
4. Why can a test for "waits 5 seconds" run in microseconds under #[tokio::test(start_paused = true)]? What does the paused clock do to sleep versus to Instant::now()?
5. with_retry never looks at an HTTP status code. What does it look at instead, and which earlier function is responsible for turning a status into that?

The graded quiz for this arc is at the end of the part, on the Concept-Check: Pagination page.

Concept: Idempotent, Content-Addressed Writes

Kind: Concept. Seam installed this arc: content_id / content_key and the IdempotentSink — the precondition for every retry, backfill, and scheduled re-run to come.

The idea

The retry loop from the last concept has a consequence you have to design for: the same work can run more than once. A request that times out after the server already processed it gets retried; a scheduled backfill re-fetches a range you already have; a crashed run gets restarted from the top. If your Load stage blindly appends everything it is handed, every one of those produces duplicate rows in your output. Re-running the pipeline should be safe — running it twice should leave the same data as running it once. That property has a name: idempotency.

The way you get it is content addressing: give every row a stable id derived from its own content, then refuse to write a row whose id you have already written. Two rows with identical content get identical ids and collapse to one; any change to the content yields a different id and a distinct row. The id is not a database sequence or a random UUID — it is a hash of the row's canonical bytes, so it is reproducible across runs, across processes, across machines.

The real code uses SHA-256:

// crates/etl-core/src/content.rs  (needs the `sha2` crate; not playground-safe)
/// A deterministic id for a value: `sha256(kind ‖ 0x00 ‖ canonical)`, hex.
pub fn content_id(kind: &str, canonical: &str) -> String {
    let mut hasher = Sha256::new();
    hasher.update(kind.as_bytes());
    hasher.update([0u8]);
    hasher.update(canonical.as_bytes());
    format!("{:x}", hasher.finalize())
}

Two design choices in three lines. The kind tag ("row", "conjunction", "embed", …) is hashed in first, so two different entity types can never collide on an id even if their canonical strings happen to coincide. The 0x00 separator between kind and canonical stops the boundary from being ambiguous — without it, kind="ab",canonical="c" and kind="a",canonical="bc" would hash the same bytes. Small, deliberate, and load-bearing.

A runnable toy: content ids and idempotent load

SHA-256 needs the sha2 crate, which is not playground-safe, so this toy substitutes a tiny hand-rolled digest (FNV-1a). The hash function is not the lesson — the properties are: same content in, same id out; different content, different id; and a seen-set that makes a second load of the same rows a no-op.

use std::collections::HashSet;

// A toy stand-in for sha256: FNV-1a as a short hex digest. The real code uses
// sha2::Sha256; the only property that matters here is the one both share —
// same bytes in, same digest out; different bytes, a different digest.
fn digest(bytes: &[u8]) -> String {
    let mut h: u64 = 0xcbf29ce484222325;
    for &b in bytes {
        h ^= b as u64;
        h = h.wrapping_mul(0x100000001b3);
    }
    format!("{h:016x}")
}

// content_id(kind, canonical) = digest(kind ‖ 0x00 ‖ canonical). The kind tag
// plus a separator keep two entity types from colliding on equal canonical text.
fn content_id(kind: &str, canonical: &str) -> String {
    let mut buf = Vec::new();
    buf.extend_from_slice(kind.as_bytes());
    buf.push(0);
    buf.extend_from_slice(canonical.as_bytes());
    digest(&buf)
}

// A row whose canonical form CAN fail (mirrors serde_json refusing a NaN float).
// content_key is therefore fallible — a Result, never a silent default.
struct Row {
    name: String,
    alt_km: f64,
}

fn canonical(row: &Row) -> Result<String, String> {
    if row.alt_km.is_nan() {
        return Err(format!("cannot canonicalize {}: NaN altitude", row.name));
    }
    Ok(format!("{}|{}", row.name, row.alt_km))
}

fn content_key(row: &Row) -> Result<String, String> {
    Ok(content_id("row", &canonical(row)?))
}

fn main() {
    // Content-addressing: equal content -> equal id; any change -> a new id.
    println!("iss       = {}", content_id("row", "ISS|420"));
    println!("iss again = {}", content_id("row", "ISS|420")); // identical
    println!("hst       = {}", content_id("row", "HST|540")); // different content
    // The kind tag matters: same canonical text, different entity type -> new id.
    println!("as launch = {}", content_id("launch", "ISS|420"));

    // Idempotent load: forward only keys not seen this run.
    let rows = vec![
        Row { name: "ISS".into(), alt_km: 420.0 },
        Row { name: "HST".into(), alt_km: 540.0 },
        Row { name: "ISS".into(), alt_km: 420.0 }, // a duplicate of the first
    ];
    let mut seen = HashSet::new();
    let mut written = 0;
    for r in &rows {
        if seen.insert(content_key(r).unwrap()) {
            written += 1;
        }
    }
    println!("wrote {written} of {} rows", rows.len());
}

iss       = 471231c61412846c
iss again = 471231c61412846c
hst       = 895a0f4eeaadb2eb
as launch = 14c1e913350a5b5d
wrote 2 of 3 rows

The first two lines are byte-identical: same content, same id — that is what makes dedup possible. hst differs because its content differs, and as launch differs from iss even though the canonical string "ISS|420" is the same, because the kind tag went into the hash. And three input rows, one of them a duplicate, produced two writes. Feed those same rows in again and the seen-set already holds every key: zero new writes. That is idempotency.

The real seam: Row::content_key and IdempotentSink

On the real Row, the canonical form is its JSON, and the dedupe key is the content id of that JSON:

// crates/etl-core/src/content.rs
impl Row {
    pub fn content_key(&self) -> Result<String, EtlError> {
        // serde_json is deterministic for our structs (fields in declaration
        // order), so equal rows hash equal.
        let canonical = serde_json::to_string(self).map_err(|e| EtlError::Parse(e.to_string()))?;
        Ok(content_id("row", &canonical))
    }
}

IdempotentSink is a decorator: it wraps any Sink, keeps a set of keys it has already forwarded, and passes only the fresh rows to the inner sink:

// crates/etl-core/src/content.rs
#[async_trait]
impl<S: Sink> Sink for IdempotentSink<S> {
    async fn load(&self, rows: &[Row]) -> Result<usize, EtlError> {
        let mut fresh = Vec::new();
        {
            let mut seen = self.seen.lock().await;
            for row in rows {
                if seen.insert(row.content_key()?) {
                    fresh.push(row.clone());
                }
            }
        }
        if fresh.is_empty() {
            return Ok(0);
        }
        self.inner.load(&fresh).await
    }
}

Because it wraps S: Sink and is a Sink, you can slip it in front of the JsonlSink without either the sink or the orchestrator knowing — the same decorator pattern as any middleware. Two constructors: IdempotentSink::new(inner) starts with an empty seen-set (dedup within one run), and IdempotentSink::with_seen(inner, keys) pre-loads keys already on disk so re-runs across processes stay idempotent too. The shipped test rerun_writes_zero_new_rows loads two rows (returns 2), then loads the identical two again and asserts the second call returns 0.

Why content_key is fallible — and the trap of "fixing" it

Look again at the signature: content_key(&self) -> Result<String, EtlError>. It would be so tempting to make it return a plain String — call sites get noisier when every key is a ?. But serialization can genuinely fail (a NaN float has no JSON representation), and the question is what to do when it does. The wrong answer, unwrap_or_default(), is a quiet catastrophe:

use std::collections::HashSet;

fn digest(bytes: &[u8]) -> String {
    let mut h: u64 = 0xcbf29ce484222325;
    for &b in bytes {
        h ^= b as u64;
        h = h.wrapping_mul(0x100000001b3);
    }
    format!("{h:016x}")
}

fn content_id(kind: &str, canonical: &str) -> String {
    digest(format!("{kind}\0{canonical}").as_bytes())
}

struct Row {
    name: String,
    alt_km: f64,
}

fn canonical(row: &Row) -> Result<String, String> {
    if row.alt_km.is_nan() {
        return Err(format!("cannot canonicalize {}: NaN altitude", row.name));
    }
    Ok(format!("{}|{}", row.name, row.alt_km))
}

// THE BUG: swallow the error into a default. Every un-canonicalizable row now
// shares the SAME empty key — so the idempotent sink treats them as duplicates.
fn content_key_bad(row: &Row) -> String {
    canonical(row)
        .map(|c| content_id("row", &c))
        .unwrap_or_default()
}

fn main() {
    let a = Row { name: "Alpha".into(), alt_km: f64::NAN };
    let b = Row { name: "Beta".into(), alt_km: f64::NAN };

    // Two genuinely distinct rows...
    println!("bad key A = {:?}", content_key_bad(&a));
    println!("bad key B = {:?}", content_key_bad(&b));

    let mut seen = HashSet::new();
    let mut written = 0;
    for r in [&a, &b] {
        if seen.insert(content_key_bad(r)) {
            written += 1;
        }
    }
    // ...collapse to one empty key: the second is silently dropped as a "dup".
    println!("wrote {written} of 2 distinct rows (should be 2)");
}

bad key A = ""
bad key B = ""
wrote 1 of 2 distinct rows (should be 2)

Two distinct rows, both un-serializable, both mapped to the empty string, and the idempotent sink concluded they were the same row and dropped the second. That is data loss disguised as deduplication — and it is silent, which makes it the worst kind. Returning a Result forces the error to the surface (? propagates it as EtlError::Parse), where a bad row is rejected loudly instead of being quietly merged into another. The shipped code's comment says it outright: fallible on purpose, never unwrap_or_default.

Why this is the linchpin for later arcs

Idempotent, content-addressed writes are what make everything downstream safe to repeat. The retry loop can re-run a request without fear of doubling rows. The orchestrator (Part V) can restart a crashed pipeline from the top and only the genuinely new rows land. The RAG arc (Part VI) keys embeddings by content_id("embed", text) so an expensive embedding is computed once and never recomputed for text it has already seen. None of that works without a stable, content-derived id — which is why this small hashing seam is installed here, before the arcs that spend it.

Questions to lock

1. What does "idempotent" mean for a Load stage, and why does adding a retry loop force you to care about it?
2. In content_id(kind, canonical), what two collisions do the kind tag and the 0x00 separator each prevent?
3. IdempotentSink wraps any S: Sink and is itself a Sink. What pattern is that, and what does it let you do without the inner sink's cooperation?
4. Why does Row::content_key return Result<String, EtlError> instead of String? Describe the exact failure that unwrap_or_default() would cause.
5. What is the difference between IdempotentSink::new and IdempotentSink::with_seen, and which one keeps re-runs idempotent across separate processes?

The graded quiz for this arc is at the end of the part, on the Concept-Check: Pagination page.

Build: The TheSpaceDevs Source — Pagination

Maps to: Task 3.3. Kind: Build — spec and test names only. No implementation.

Objective

Build SpaceDevsSource (E — a paginated GET that follows the next cursor to the end) and SpaceDevsTransform (T — deeply nested Launch Library 2 JSON into Row::Launch, including the prose description that the RAG arc will later embed). Every test hits a wiremock server, never the live network.

This is the third concrete Source/Transform pair. It is also the first source whose Extract is a loop of requests rather than a single one, and the first to lean on the with_retry middleware built in the companion page.

Scaffold

Create: crates/etl-sources/src/spacedevs.rs; add pub mod spacedevs; and re-exports to crates/etl-sources/src/lib.rs.

Dependencies: no manifest changes — reqwest, async-trait, chrono, serde, serde_json, tokio, and (dev) wiremock + pretty_assertions were all declared when you created etl-sources in Part II.

reqwest, wiremock, and chrono are not on the Rust playground; build and test this in your workspace with cargo test -p etl-sources spacedevs.

Expected result: cargo test -p etl-sources spacedevs → all four tests green.

The spec (givens)

SpaceDevsSource — fields base_url: String and policy: RetryPolicy; constructor new(base_url: impl Into<String>) that stores the base URL and RetryPolicy::default(). It implements Source:

• name(&self) returns "spacedevs".
• extract:
  • Trim a trailing / off base_url. Seed the cursor with {base}/2.2.0/launch/?limit=100&mode=detailed.
  • Keep a HashSet<String> of visited URLs. Before fetching each URL, insert it; if insert returns false (already seen), return EtlError::Parse("pagination cycle at {url}").
  • Fetch each page through with_retry(self.policy, …): inside the closure, reqwest::Client::get(&url).send() (transport error → EtlError::Network); on a non-success status return classify_status(status, headers); on success read the body text (resp.text() error → EtlError::Network).
  • Deserialize the body into a Page (parse error → EtlError::Parse). Set the cursor to page.next. Push one RawRecord { source: "spacedevs", payload: <body>, fetched_at: Utc::now() } per page.
  • Loop until the cursor is None; return the Vec<RawRecord>.

SpaceDevsTransform — a unit struct implementing Transform:

• Deserialize raw.payload into a Page (parse error → EtlError::Parse).
• For each launch in page.results: parse net as DateTime<Utc> (parse error → EtlError::Parse naming the bad value); derive payloads and description from the optional mission (a missing mission ⇒ empty payloads, empty description; a mission's absent name ⇒ empty payloads, absent description ⇒ String::new()); take provider from launch_service_provider.name (absent ⇒ String::default()). Push a Row::Launch(Launch { id, name, net, provider, payloads, description }).

The deserialization structs (given — the LL2 shape you model, ignoring the dozens of fields you do not need):

Page   { next: Option<String>, results: Vec<RawLaunch> }
RawLaunch { id: String, name: String, net: String,
            launch_service_provider: Option<Provider>, mission: Option<Mission> }
Provider  { name: String }
Mission   { name: Option<String>, description: Option<String> }

Note next is Option<String> — serde maps JSON null to None, which is exactly the loop's termination signal. And serde ignores every LL2 field you did not declare, so you model only the five you use.

Concepts exercised

• Cursor pagination: the follow-next loop and the visited-set cycle guard (Pagination & Following next).
• with_retry wrapping each page fetch (Retry with Backoff).
• classify_status mapping a non-2xx response onto the error taxonomy (Part II's http.rs).
• Deserializing deeply nested, mostly-ignored JSON into a flat Row::Launch.

Test names to hit

Use a helper that emits one page of JSON with a parameterized next (a null when you pass None), carrying one launch with a nested mission and launch_service_provider:

• follows_next_until_null — the core pagination test. Mount two mocks: path("/2.2.0/launch/") returns a page whose next points at {server.uri()}/page2; path("/page2") returns a page with next: null. Point SpaceDevsSource::new(server.uri()) at it; assert extract().await yields 2 RawRecords.
• rejects_pagination_cycle — the first page's next points back at itself ({server.uri()}/2.2.0/launch/). Assert extract().await is Err(EtlError::Parse(_)) — the cycle guard, not an infinite loop.
• http_error_is_network_error — the mock returns 500; assert extract().await is Err(EtlError::Network(_)) (5xx is transient).
• parses_nested_launch_json — the core T test, no server. Hand SpaceDevsTransform a RawRecord whose payload is one page; assert one Row::Launch with provider == "SpaceX", payloads == ["Starlink G-99"], and a description containing the mission prose.

The build loop (you drive)

1. Transform first — it needs no server. Write parses_nested_launch_json against a hand-written page JSON. Predict: what should payloads and description be when mission is null? Decide before implementing.
2. follows_next_until_null. Stand up two mocks; watch the loop consume both pages and stop on the null. Predict what happens if you forget to set the cursor from page.next (it stops after one page).
3. rejects_pagination_cycle. Point the first page's next at itself. Confirm the guard fires as EtlError::Parse rather than hanging the test.
4. http_error_is_network_error. A 500 mock; confirm the 5xx maps to transient Network.
5. Run green, commit.

Done when

cargo test -p etl-sources spacedevs is green. SpaceDevsSource and SpaceDevsTransform now satisfy Source and Transform from etl-core, with pagination, retry, and cycle-safety all exercised against a mock. Three sources now feed one uniform Row enum; the orchestrator in Part V will schedule all three without knowing which is which.

Build: Retry Middleware + the Idempotent Sink

Maps to: Tasks 3.1 + 3.2. Kind: Build — spec and test names only. No implementation.

Objective

Build the two etl-core seams the paginated source depends on: with_retry + RetryPolicy (retry transient failures with capped exponential backoff, honoring Retry-After) and the content-addressing machinery — content_id, Row::content_key, and the IdempotentSink decorator that makes re-running the pipeline write each row at most once.

Both live in etl-core (not etl-sources): they are seams every other crate consumes, so they belong in the crate the dependency arrow points toward.

Scaffold

Create: crates/etl-core/src/retry.rs and crates/etl-core/src/content.rs; add pub mod retry; / pub mod content; and re-exports to crates/etl-core/src/lib.rs:

pub use content::{content_id, IdempotentSink};
pub use retry::{with_retry, RetryPolicy};

Dependencies (etl-core): add sha2 (content ids) and ensure tokio (with time — for sleep and, in tests, test-util) and async-trait are present. serde_json is already there for Row's canonical form.

sha2 and tokio are not playground-safe; build and test in the workspace with cargo test -p etl-core retry and cargo test -p etl-core content.

Expected result: cargo test -p etl-core retry and cargo test -p etl-core content → all tests green.

Part A — with_retry + RetryPolicy (retry.rs)

RetryPolicy — #[derive(Clone, Copy, Debug)] with fields max_attempts: u32, base_delay: Duration, max_delay: Duration. Default = { max_attempts: 4, base_delay: 200ms, max_delay: 10s }.

with_retry — the signature and behavior:

pub async fn with_retry<F, Fut, T>(policy: RetryPolicy, op: F) -> Result<T, EtlError>
where F: FnMut() -> Fut, Fut: Future<Output = Result<T, EtlError>>

• Loop, calling op().await each iteration. On Ok, return it.
• On Err, increment attempt. If attempt >= policy.max_attempts or !err.is_transient(), return the error immediately.
• Otherwise compute the backoff — err.retry_after() if the error carries one (a 429's Retry-After), else capped exponential (base_delay * 2^(attempt-1)).min(max_delay) — sleep it, and loop.

op is FnMut returning a fresh Future each call because retrying means re-running it.

Test names to hit (all #[tokio::test(start_paused = true)], using an AtomicU32 to count op calls):

• retries_transient_then_succeeds — op returns Network on the first two calls, then Ok(42); assert the result is 42 and op ran exactly 3 times.
• gives_up_on_permanent_immediately — op always returns Parse; assert the error is Parse and op ran exactly 1 time (no retry on a permanent error).
• honors_retry_after — op returns RateLimited { retry_after_secs: 5 } once, then Ok; capture tokio::time::Instant::now() before and assert start.elapsed() >= 5s (the server's Retry-After beats the small base backoff — provable in microseconds because the clock is paused).

Part B — content addressing + IdempotentSink (content.rs)

content_id — pub fn content_id(kind: &str, canonical: &str) -> String: SHA-256 of kind bytes, then a single 0x00 byte, then canonical bytes, formatted as lowercase hex. The kind tag prevents cross-entity id collisions; the 0x00 separator removes boundary ambiguity between kind and canonical.

Row::content_key — pub fn content_key(&self) -> Result<String, EtlError>: serialize self to JSON (serde_json::to_string, error → EtlError::Parse), then content_id("row", &canonical). Fallible on purpose — never unwrap_or_default, which would collapse distinct un-serializable rows onto one shared key and silently drop them as duplicates.

IdempotentSink<S: Sink> — fields inner: S and seen: Mutex<HashSet<String>>. Two constructors: new(inner) (empty seen-set) and with_seen(inner, keys: impl IntoIterator<Item = String>) (pre-loaded, for cross-process re-runs). It implements Sink: under the lock, insert each row's content_key()? into seen; the rows whose key was newly inserted are "fresh." If none are fresh, return Ok(0); otherwise forward only the fresh rows to inner.load and return its count.

Test names to hit:

• same_content_hashes_stable_distinct_content_differs — content_id("row","x") == content_id("row","x"); != content_id("row","y"); and content_id("a","x") != content_id("b","x") (the kind tag matters).
• content_key_is_stable_and_distinguishes_rows — two equal Row::Launches have equal keys; a row with a changed field has a different key.
• rerun_writes_zero_new_rows — wrap a counting sink; load two rows returns 2; loading the identical two again returns 0.
• distinct_rows_all_written — three distinct rows through the sink return 3.

Concepts exercised

• Capped exponential backoff, transient-only retry, Retry-After precedence (Retry with Backoff).
• Deterministic timing tests via tokio's paused clock.
• Content-addressed ids and the decorator-Sink idempotency pattern (Idempotent, Content-Addressed Writes).

The build loop (you drive)

1. content_id first — pure and tiny. Write same_content_hashes_stable_distinct_content_differs, implement, run green.
2. content_key. Write content_key_is_stable_and_distinguishes_rows. Predict: why must the return type be Result, not String?
3. IdempotentSink. Write rerun_writes_zero_new_rows against a counting stub sink; implement the decorator; then distinct_rows_all_written.
4. with_retry. Write gives_up_on_permanent_immediately (proves the is_transient gate) before retries_transient_then_succeeds. Add honors_retry_after last, under start_paused = true.
5. Run green, commit.

Done when

cargo test -p etl-core retry content is green. with_retry and IdempotentSink are now available to every source — SpaceDevsSource already wraps each page fetch in with_retry, and any pipeline can slip an IdempotentSink in front of its JsonlSink to make re-runs safe. These are the seams the orchestration and retrieval arcs will spend.

Concept-Check: Pagination

You installed three seams this arc: cursor pagination with a cycle guard, with_retry (capped exponential backoff, transient-only, Retry-After wins), and content-addressed idempotent writes. This check pins down the parts that are easy to get almost right — where the follow-next loop terminates and how it defends against a cursor that never does, exactly which failures a retry loop should and should not retry (and how long it waits), and why the content key is fallible.

Three of these are Tracing questions: read the program, decide whether it compiles, and if it does, predict its exact output before revealing the answer. Predicting is the point.

Concept: Modeling a Pipeline as a Typed DAG

Kind: Concept. Seam installed this arc: the task graph — pipelines become nodes, dependencies become edges, and execution order becomes a computed property of the graph rather than a hand-written sequence.

The idea

You now have three concrete pipelines — CelesTrak, Space-Track, TheSpaceDevs — each a Source + Transform + Sink. Left to yourself, you would run them by writing them out in an order: extract this, then that, then the third. That works right up until one pipeline needs another's output first, or you add a fourth, or you want an embedding step that must run after the launches land. The hand-written order becomes a thing you maintain by hand, and hand-maintained orderings rot.

The fix is to stop writing the order down and start computing it. Model the pipelines as a directed acyclic graph (DAG):

• Each pipeline is a node (a TaskId).
• "B needs A's output first" is a directed edge — a prerequisite.
• "Acyclic" is the promise that the prerequisites never loop back on themselves. If they did, there would be no valid order to run them in, and that is a bug you want caught before anything runs, not a deadlock you discover at 3am.

Given that graph, a topological sort produces a linear run order in which every node appears after all of its prerequisites. You declare the dependencies; the algorithm derives the order. Add a fourth pipeline with one add_dep call and the order recomputes itself.

        ┌─────────┐
        │  sift   │            a topological order is any linearization
        └────┬────┘            where every arrow points "backwards" in
          ┌──┴───┐             the list — a prerequisite is always to
          ▼      ▼             the left of the task that needs it:
     ┌────────┐ ┌────────┐
     │ grease │ │ whisk  │        sift → grease → whisk → bake
     └────┬───┘ └───┬────┘
          └────┬────┘
               ▼
          ┌────────┐
          │  bake  │
          └────────┘

Kahn's algorithm, and why the shipped code uses a sorted ready-set

The topological sort in etl-orchestrate is Kahn's algorithm. The mental model is small:

1. Compute each node's indegree — how many prerequisites it has that have not run yet.
2. Every node with indegree 0 is ready (nothing is blocking it). Put those in a ready-set.
3. Pop a ready node, append it to the output order, and decrement the indegree of everything that depended on it. Any of those that just hit 0 is now ready — add it.
4. Repeat until the ready-set is empty.

There is a design decision hiding in step 3: which ready node do you pop when several are ready at once? In the diamond above, once sift runs, both grease and whisk become ready simultaneously. A plain queue or a HashSet would pop them in whatever order the hash happened to land — which can differ run to run, and differ between machines. That is a nightmare to test and a nightmare to debug: a pipeline order that changes for no visible reason.

The shipped code makes the tie-break deterministic by keeping the ready-set in a BTreeSet, which is always sorted. When several tasks are ready, the alphabetically-first one runs next. Same graph, same order, every single time — on your laptop, in CI, on a colleague's machine.

This example is pure std — HashMap, HashSet, BTreeSet, Vec — so it runs on the Rust playground. It is the same algorithm the shipped Dag::topological_order uses, on a cake-baking domain so it cannot be mistaken for the answer key.

use std::collections::{BTreeSet, HashMap, HashSet};

/// A tiny task graph over string ids. Edges record prerequisites:
/// `add_dep("frost", "bake")` means "frost depends on bake" — bake runs first.
#[derive(Default)]
struct Dag {
    nodes: HashSet<String>,
    prereqs: HashMap<String, Vec<String>>,
}

impl Dag {
    fn add_task(&mut self, id: &str) {
        self.nodes.insert(id.to_string());
        self.prereqs.entry(id.to_string()).or_default();
    }

    fn add_dep(&mut self, task: &str, depends_on: &str) {
        self.nodes.insert(task.to_string());
        self.nodes.insert(depends_on.to_string());
        self.prereqs.entry(depends_on.to_string()).or_default();
        self.prereqs
            .entry(task.to_string())
            .or_default()
            .push(depends_on.to_string());
    }

    /// Kahn's algorithm with a *sorted* ready-set (BTreeSet), so ties break
    /// alphabetically and the order is identical on every run.
    fn topological_order(&self) -> Result<Vec<String>, String> {
        // indegree = number of prerequisites still unmet.
        let mut indegree: HashMap<String, usize> = self
            .nodes
            .iter()
            .map(|n| (n.clone(), self.prereqs.get(n).map_or(0, |p| p.len())))
            .collect();

        // Reverse index: prerequisite -> the tasks waiting on it.
        let mut dependents: HashMap<String, Vec<String>> = HashMap::new();
        for (task, prereqs) in &self.prereqs {
            for p in prereqs {
                dependents.entry(p.clone()).or_default().push(task.clone());
            }
        }

        // Everything with no prerequisites is ready now.
        let mut ready: BTreeSet<String> = indegree
            .iter()
            .filter(|(_, d)| **d == 0)
            .map(|(t, _)| t.clone())
            .collect();

        let mut order = Vec::with_capacity(self.nodes.len());
        while let Some(next) = ready.iter().next().cloned() {
            ready.remove(&next);
            if let Some(children) = dependents.get(&next) {
                for child in children {
                    let d = indegree.get_mut(child).expect("child in indegree");
                    *d -= 1;
                    if *d == 0 {
                        ready.insert(child.clone());
                    }
                }
            }
            order.push(next);
        }

        // Anything left unordered sits in a cycle: its indegree never hit zero.
        if order.len() != self.nodes.len() {
            return Err(format!(
                "cycle detected: ordered {} of {} tasks",
                order.len(),
                self.nodes.len()
            ));
        }
        Ok(order)
    }
}

fn main() {
    // A diamond: `sift` feeds both `whisk` and `grease`; `bake` needs both.
    let mut dag = Dag::default();
    dag.add_task("sift");
    dag.add_task("whisk");
    dag.add_task("grease");
    dag.add_task("bake");
    dag.add_dep("whisk", "sift");
    dag.add_dep("grease", "sift");
    dag.add_dep("bake", "whisk");
    dag.add_dep("bake", "grease");

    let order = dag.topological_order().unwrap();
    println!("{order:?}");

    // Run it again — the sorted ready-set means the answer never wobbles.
    let again = dag.topological_order().unwrap();
    println!("stable: {}", order == again);
}

["sift", "grease", "whisk", "bake"]
stable: true

Read the output carefully. sift is first because it is the only node with no prerequisites. Then grease beats whisk — not because it was added first (it was not), but because "grease" < "whisk" alphabetically and the BTreeSet always hands back its smallest element. bake is last because its indegree only reaches 0 after both grease and whisk have run. And the second call produces the byte-identical order: determinism here is not luck, it is the data structure.

Cycle detection falls out for free

Notice that topological_order never explicitly searches for a cycle. It does not need to. Kahn's algorithm can only ever emit a node once that node's indegree has reached 0 — once all its prerequisites have already been emitted. If some nodes depend on each other in a loop, none of them can ever reach indegree 0, so none of them are ever emitted. The loop runs dry, and the output order is shorter than the node count.

That length mismatch is the cycle detector: if order.len() != self.nodes.len(). The shipped code turns it into an EtlError::Parse (a cycle is, in effect, a malformed pipeline definition); the toy returns a String error, but the check is identical:

use std::collections::{BTreeSet, HashMap, HashSet};
#[derive(Default)]
struct Dag { nodes: HashSet<String>, prereqs: HashMap<String, Vec<String>> }
impl Dag {
    fn add_dep(&mut self, task: &str, depends_on: &str) {
        self.nodes.insert(task.to_string());
        self.nodes.insert(depends_on.to_string());
        self.prereqs.entry(depends_on.to_string()).or_default();
        self.prereqs.entry(task.to_string()).or_default().push(depends_on.to_string());
    }
    fn topological_order(&self) -> Result<Vec<String>, String> {
        let mut indegree: HashMap<String, usize> =
            self.nodes.iter().map(|n| (n.clone(), self.prereqs.get(n).map_or(0, |p| p.len()))).collect();
        let mut dependents: HashMap<String, Vec<String>> = HashMap::new();
        for (task, prereqs) in &self.prereqs {
            for p in prereqs { dependents.entry(p.clone()).or_default().push(task.clone()); }
        }
        let mut ready: BTreeSet<String> =
            indegree.iter().filter(|(_, d)| **d == 0).map(|(t, _)| t.clone()).collect();
        let mut order = Vec::new();
        while let Some(next) = ready.iter().next().cloned() {
            ready.remove(&next);
            if let Some(children) = dependents.get(&next) {
                for child in children {
                    let d = indegree.get_mut(child).expect("child in indegree");
                    *d -= 1;
                    if *d == 0 { ready.insert(child.clone()); }
                }
            }
            order.push(next);
        }
        if order.len() != self.nodes.len() {
            return Err(format!("cycle detected: ordered {} of {} tasks", order.len(), self.nodes.len()));
        }
        Ok(order)
    }
}
fn main() {
    // a depends on b, and b depends on a: neither can ever be "ready".
    let mut dag = Dag::default();
    dag.add_dep("a", "b");
    dag.add_dep("b", "a");
    match dag.topological_order() {
        Ok(order) => println!("ordered: {order:?}"),
        Err(e) => println!("rejected: {e}"),
    }
}

rejected: cycle detected: ordered 0 of 2 tasks

Zero of two: a waits on b, b waits on a, the ready-set starts empty, the while loop never executes, and the count check fires immediately. A malformed pipeline graph is rejected before a single extract is attempted — the "acyclic" in DAG is enforced, not assumed.

Why this is the seam that unlocks the arc

The three earlier arcs each installed a seam — the Source/Transform/Sink traits, the run ledger, the retry middleware. The DAG is the seam that lets those pieces be composed without a human choosing the order. Once "what depends on what" is data in a graph rather than lines in a main function, three things become free that were previously manual:

• Order is computed, so adding a node cannot silently run it too early.
• Cycles are rejected up front, so an impossible pipeline never half-runs.
• Determinism means a run order you can write a test against — which is exactly what the next build page does.

The executor builds directly on top of this: it takes the topological_order() and walks it, but with two twists a bare sort does not have — it can skip a node that is still fresh, and it can gate a node whose prerequisite failed. Both of those need the order first. This is the foundation they stand on.

Questions to lock

1. What do nodes and edges represent in the pipeline DAG, and what does "acyclic" guarantee that makes a topological order possible at all?
2. In Kahn's algorithm, what is a node's indegree, and what event lets a node move into the ready-set?
3. Why does the shipped code use a BTreeSet for the ready-set instead of a plain queue or HashSet — what property would be lost otherwise?
4. How does topological_order detect a cycle without ever explicitly searching for one? What single comparison is the whole detector?
5. The code stores prereqs (task → its prerequisites) but builds a dependents index inside the sort. Why does Kahn's algorithm need the reverse mapping?

The graded quiz for this arc is at the end of the part, on the Concept-Check: Orchestration page.

Concept: Dependency Inversion — the Executor Knows Only the Trait

Kind: Concept. The property this arc proves: the orchestrator schedules dyn Source / dyn Transform / dyn Sink and depends on etl-core only. It has never heard of CelesTrak. The crate boundary is the proof, and cargo tree is how you read it.

The idea

Here is the arrangement you would reach for if you were not being careful. The executor needs to run the CelesTrak pipeline, so it imports CelestrakSource. It needs Space-Track too, so it imports SpaceTrackSource. Now etl-orchestrate depends on etl-sources, and every time you add a source you edit the orchestrator. The scheduler — the piece that should be the most stable code in the system — churns every time a leaf changes. That is a dependency pointing the wrong way: the high-level policy (how to schedule) depends on the low-level detail (which sources exist).

Dependency inversion flips it. Both the executor and the concrete sources depend on an abstraction — the Source trait, which lives in etl-core — and neither depends on the other. The executor holds a Box<dyn Source> and calls .extract() through the vtable. It does not know, and cannot know, whether the box contains a CelestrakSource or a NASA source you will write next week. The arrows both point inward, at the trait:

   BEFORE (dependency points down at the detail):

     etl-orchestrate ──depends on──▶ etl-sources ──▶ CelestrakSource
        (policy)                       (detail)
     every new source edits the orchestrator ✗


   AFTER (both depend on the abstraction in the middle):

     etl-orchestrate ──▶ Source trait (etl-core) ◀── etl-sources
        (policy)            (abstraction)              (detail)
     a new source touches neither the trait nor the orchestrator ✓

The executor's own doc comment states the contract outright: it "schedules dyn Source / dyn Transform / dyn Sink and has no knowledge of any concrete source — that separation is enforced by the crate boundary, not by convention."

The crate boundary is the enforcement

"By convention" would mean: we all agree not to import concrete sources into the orchestrator, and we hope nobody does. That is not a guarantee — it is a coding-review habit that fails the first time someone is in a hurry.

The Panoptes workspace makes it a compile-time guarantee instead. etl-orchestrate's Cargo.toml simply does not list etl-sources as a dependency:

[package]
name = "etl-orchestrate"
edition = "2024"

# Depends on etl-core ONLY. It schedules `dyn Source` and must never know a
# concrete source exists — that constraint is the dependency-inversion lesson.
[dependencies]
etl-core = { path = "../etl-core" }
chrono = { workspace = true }

Because the dependency is not declared, CelestrakSource is not in scope inside etl-orchestrate. If you tried to write use etl_sources::CelestrakSource; in executor.rs, it would not compile — unresolved import, missing crate. The wrong dependency is not discouraged; it is impossible. The layering is load-bearing structure, checked by cargo on every build.

Reading the proof with cargo tree

You do not have to take the manifest's word for it. cargo tree -p etl-orchestrate prints the crate's entire dependency closure — everything it pulls in, transitively. If etl-sources were anywhere in that tree, it would appear. Here is the real output, trimmed to the two direct dependencies and their headline sub-deps:

$ cargo tree -p etl-orchestrate
etl-orchestrate v0.1.0
├── chrono v0.4.45
│   └── ... (time math for the freshness window)
└── etl-core v0.1.0
    ├── async-trait v0.1.91 (proc-macro)
    ├── chrono v0.4.45
    ├── serde v1.0.229
    ├── serde_json v1.0.150
    ├── sha2 v0.10.9
    ├── thiserror v2.0.19
    └── tokio v1.53.0
[dev-dependencies]
├── async-trait
├── pretty_assertions
└── tokio

Two direct dependencies: chrono (for the freshness math) and etl-core (for the traits, Ledger, RunRecord, with_retry). etl-sources is not present — not directly, not transitively. The orchestrator's world is the abstraction layer and nothing below it. That absence is the whole lesson, and it is a one-command check you can put in CI: grep the tree for etl-sources and fail the build if it ever shows up.

A runnable toy: a scheduler that never learns a concrete type

The trait objects in the real code pull in async-trait and tokio, so they will not run on the playground. Here is the same shape reduced to synchronous std — a job scheduler that owns a list of type-erased jobs and runs them in order, never naming Backup or Report:

// The scheduler's whole world: one trait. It never learns a concrete type.
trait Job {
    fn name(&self) -> &str;
    fn run(&self) -> usize; // returns "units of work done"
}

// A scheduler that owns a list of type-erased jobs and runs them in order.
// Nothing here mentions Backup, Report, or any concrete job — only `dyn Job`.
struct Scheduler {
    jobs: Vec<Box<dyn Job>>,
}

impl Scheduler {
    fn new() -> Self {
        Scheduler { jobs: Vec::new() }
    }
    fn add(&mut self, job: Box<dyn Job>) {
        self.jobs.push(job);
    }
    fn run_all(&self) -> usize {
        let mut total = 0;
        for job in &self.jobs {
            let n = job.run();
            println!("{:<8} did {n} units", job.name());
            total += n;
        }
        total
    }
}

// Concrete jobs — the "leaf" layer. These would live in a *different* crate
// that depends on the scheduler's, never the other way around.
struct Backup;
impl Job for Backup {
    fn name(&self) -> &str { "backup" }
    fn run(&self) -> usize { 12 }
}

struct Report;
impl Job for Report {
    fn name(&self) -> &str { "report" }
    fn run(&self) -> usize { 3 }
}

fn main() {
    let mut sched = Scheduler::new();
    // A fourth, fifth, hundredth job type would need zero scheduler changes:
    // if it implements Job, `add` accepts it.
    sched.add(Box::new(Backup));
    sched.add(Box::new(Report));
    let total = sched.run_all();
    println!("total: {total}");
}

backup   did 12 units
report   did 3 units
total: 15

Scheduler names Job and nothing else. Backup and Report sit at the leaf layer — in the real workspace they are a different crate, etl-sources, and the dependency goes leaf → trait, never scheduler → leaf. Swap Report for a NasaSource and Scheduler does not change one character. That is the payoff you will feel in the capstone: a fourth source drops into the DAG with zero orchestrator edits, precisely because the orchestrator was built to know only the trait.

Why this is the real payoff of the whole course

Every arc has been quietly setting this up. Part II defined the traits and made them dyn-compatible. Parts III and IV built three concrete implementations behind those traits. This arc collects the reward: because the executor depends only on the abstraction, it schedules all three sources uniformly, and it will schedule the fourth — the NASA source you write in the capstone — without knowing it exists. The system got a new capability by adding a leaf, not by editing the core. That is the difference between a codebase that ossifies as it grows and one that stays soft where it needs to. The crate graph, not good intentions, is what keeps it that way.

Questions to lock

1. In the "before" arrangement, which way does the dependency arrow point, and why does that make the orchestrator churn every time a source is added?
2. What is the abstraction that both the executor and the concrete sources depend on, and in which crate does it live?
3. Why is "we agree not to import concrete sources into the orchestrator" weaker than what the workspace actually does — and what makes the wrong import impossible rather than merely discouraged?
4. What would cargo tree -p etl-orchestrate show if the layering were broken, and what does its absence from the tree prove?
5. Which crate is the only one allowed to name a concrete source, and why is concentrating that knowledge at the top of the graph the right call?

The graded quiz for this arc is at the end of the part, on the Concept-Check: Orchestration page.

Build: The Task Graph + Topological Execution

Maps to: Task 4.1. Kind: Build — spec and test names only. No implementation.

Objective

Create the etl-orchestrate crate and its first module: a typed task graph, Dag, with a deterministic topological sort. This is the pure-data foundation the executor stands on — no async, no I/O, no traits from etl-core beyond the error type. By the end, topological_order() turns a set of tasks and dependencies into a stable run order, and rejects any graph that contains a cycle.

Scaffold

Create the crate: crates/etl-orchestrate/Cargo.toml, src/lib.rs, src/graph.rs. Add "crates/etl-orchestrate" to the workspace members.

crates/etl-orchestrate/Cargo.toml — this is the crate where the dependency-inversion lesson is enforced by the manifest, so the dependency list is deliberately short:

[dependencies]
etl-core = { path = "../etl-core" }   # for EtlError only, in this module
chrono = { workspace = true }         # the executor module needs it next

[dev-dependencies]
pretty_assertions = { workspace = true }
tokio = { workspace = true }
async-trait = { workspace = true }

There is no etl-sources dependency, and there never will be. That absence is the point of the whole arc — see Concept: Dependency Inversion. Prove it after this build with cargo tree -p etl-orchestrate.

Expected result: cargo test -p etl-orchestrate graph → the three graph tests green.

The spec (givens)

• TaskId(pub String) — a newtype over the task's name. Derive Clone, PartialEq, Eq, Hash, PartialOrd, Ord — the Ord is what lets the ready-set sort. A TaskId::new(impl Into<String>) constructor.
• Dag — holds nodes: HashSet<TaskId> and prereqs: HashMap<TaskId, Vec<TaskId>> (task → its prerequisites). Derive Default.
  • add_task(&mut self, id: impl Into<String>) -> TaskId — register a node with an empty prereq list, return its id.
  • add_dep(&mut self, task: &TaskId, depends_on: &TaskId) — record that task depends on depends_on. Must insert both ids as nodes (so an edge can introduce a node that was never add_task-ed) and push depends_on onto task's prereq list.
  • prerequisites(&self, task: &TaskId) -> &[TaskId] — the direct prereqs, or an empty slice for an unknown task.
  • topological_order(&self) -> Result<Vec<TaskId>, EtlError> — Kahn's algorithm with a BTreeSet ready-set so ties break deterministically (alphabetically). A cycle (ordered count ≠ node count) returns EtlError::Parse("cycle detected: …").

The algorithm is spelled out step by step, with a runnable std-only mirror, on Concept: Modeling a Pipeline as a Typed DAG. This build is that concept made real over TaskId and EtlError.

Concepts exercised

• Kahn's topological sort; indegree bookkeeping and the reverse dependents index.
• Deterministic iteration with BTreeSet (vs the run-to-run wobble of HashSet).
• Cycle detection as a length-mismatch invariant, not a separate search.
• A newtype (TaskId) whose derived Ord is what makes the sort stable.

The build loop (you drive)

1. TaskId + Dag skeleton. Define the newtype with its derives and the two fields; write add_task / add_dep / prerequisites. Nothing to run yet.
2. linear_chain_orders_correctly. Three tasks a → b → c; assert the order is exactly ["a", "b", "c"]. Implement enough of topological_order to pass it.
3. diamond_orders_deps_before_dependents. a feeds b and c; d needs both. Predict first: with a sorted ready-set, do you get ["a", "b", "c", "d"] or ["a", "c", "b", "d"]? Decide from the tie-break rule before running.
4. cycle_is_rejected. a depends on b and b depends on a; assert topological_order() is Err(EtlError::Parse(_)). Confirm the length check is what fires.
5. Run green, prove the boundary: cargo tree -p etl-orchestrate | grep etl-sources returns nothing. Commit.

Done when

cargo test -p etl-orchestrate graph is green and cargo tree -p etl-orchestrate shows only etl-core and chrono — no concrete-source crate anywhere in the closure. You now have a deterministic, cycle-safe run order. The executor is next: it walks this order and adds the two behaviors a bare sort lacks — freshness-skip and failure-gating.

Build: Retries, Watermarks & Incremental Runs

Maps to: Task 4.2. Kind: Build — spec and test names only. No implementation.

Objective

Build the executor: the piece that owns the DAG, the pipelines, and the ledger, and runs everything in topological order. It adds the three behaviors that turn a bare topological sort into a real scheduler:

1. Retries — each pipeline's extract goes through with_retry, so a transient failure is retried per the policy before it counts as a failure.
2. Freshness-skip (incremental) — a source whose last successful run in the ledger is newer than its refresh window is skipped, not re-fetched.
3. Failure-gating — a node whose prerequisite failed (or was itself blocked) does not run at all. The DAG edges gate execution, not merely order it.

This is where etl-orchestrate finally touches the etl-core traits — dyn Source, dyn Transform, dyn Sink, Ledger, with_retry — and still names no concrete source.

Scaffold

Create: crates/etl-orchestrate/src/executor.rs. Modify: src/lib.rs to pub mod executor; and re-export Executor, Pipeline, RunReport.

Dependencies: none new — etl-core and chrono were declared when you created the crate; async-trait, tokio, and pretty_assertions are already dev-dependencies for the stub sources the tests need.

Expected result: cargo test -p etl-orchestrate executor → the executor tests green (#[tokio::test], stub sources, no network).

The spec (givens)

Pipeline — one extract → transform → load chain, over trait objects:

• Fields: source: Box<dyn Source>, transform: Box<dyn Transform>, sink: Box<dyn Sink>, policy: RetryPolicy, refresh_after: Option<Duration>.
• Pipeline::new(source, transform, sink) — defaults policy to RetryPolicy::default() and refresh_after to None. Callers set refresh_after afterward when they want incremental behavior.
• run_once(&self) -> Result<usize, EtlError> — extract through with_retry(self.policy, || self.source.extract()), transform every raw record (flat-mapping each into rows), then sink.load(&rows).await once. Returns the row count the sink reports.
• A private is_fresh(&self, ledger, id) -> Result<bool, EtlError> — false if refresh_after is None or the ledger has no prior success for this id; otherwise Utc::now() - last.finished_at < refresh.

RunReport — the outcome of a whole DAG run. Derive Debug, Default. Three Vecs:

• records: Vec<RunRecord> — one per node that actually ran (success or failure).
• skipped: Vec<TaskId> — nodes skipped because they were still fresh.
• blocked: Vec<TaskId> — nodes not run because a prerequisite failed or was itself blocked.

Executor — owns dag: Dag, pipelines: HashMap<TaskId, Pipeline>, ledger: Ledger.

• new(ledger); add_pipeline(&mut self, id: &str, pipeline) -> TaskId (registers the task in the DAG and stores the pipeline); add_dep(&mut self, task, depends_on) (delegates to the DAG).
• run(&self) -> Result<RunReport, EtlError> — the heart of the arc:
  1. Compute dag.topological_order()? (a cycle short-circuits the whole run with EtlError::Parse).
  2. Walk the order, carrying a bad: HashSet<TaskId> of failed-or-blocked ids.
  3. For each node: if any prerequisite is in bad, add this node to bad and to report.blocked, and continue — so the block propagates transitively down the graph.
  4. Else if is_fresh, push to report.skipped and continue.
  5. Else run run_once. On Ok, record Outcome::Success with a watermark: Some(finished_at) and the row count. On Err, add the id to bad and record Outcome::Failed { reason }.
  6. Append the RunRecord to the ledger and to report.records.

Concepts exercised

• Scheduling over trait objects (Box<dyn Source/Transform/Sink>) — the dependency-inversion payoff in code.
• The retry middleware (with_retry + RetryPolicy) wired into the extract step.
• Watermark-driven incremental runs: reading last_success().finished_at from the ledger.
• Transitive failure propagation via a bad set threaded through the topological walk.

The build loop (you drive)

1. runs_pipelines_in_dependency_order. Two stub sources a and b, add_dep(b, a); each stub pushes its name to a shared log on extract. Assert the log is ["a", "b"] and report.records.len() == 2.
2. records_a_run_per_node. Two independent stubs; assert one RunRecord per node and every outcome Success.
3. failed_upstream_blocks_dependent. A FailingSource at a, a stub at b, add_dep(b, a). Predict first: how many records, how many blocked, and how many times is b's source extracted? Assert records.len() == 1, report.blocked == [b], and b's extract call-count is 0.
4. incremental_skips_up_to_date_source. Seed the ledger with a very recent success for the source; set refresh_after = Some(Duration::hours(6)). Assert report.skipped.len() == 1, report.records empty, and the source's extract call-count is 0.
5. Run green, commit.

Done when

cargo test -p etl-orchestrate is green across graph and executor. You now have a scheduler that retries transient failures, skips fresh sources, and refuses to run a node whose upstream is broken — all over trait objects, with no concrete source in sight. The only thing missing is a front door: the CLI that names the real sources and calls run().

Build: The panoptes-etl CLI

Maps to: Task 4.3. Kind: Build — spec and test names only. No implementation.

Objective

Build the panoptes-etl binary: the front door. This is the only crate in the workspace that names a concrete source — it imports CelestrakSource, SpaceTrackSource, SpaceDevsSource from etl-sources, wires each into a Pipeline, hands the pipelines to the Executor, and exposes it all as a clap subcommand CLI: run, backfill, embed, status. Everything below it — the orchestrator, the traits — stays source-agnostic; the binary is where the abstract machinery meets the concrete world.

Scaffold

Create the crate: crates/panoptes-etl/Cargo.toml, src/main.rs, src/cli.rs. Add "crates/panoptes-etl" to the workspace members.

Cargo.toml — this crate depends on all three layers plus the CLI machinery:

[dependencies]
etl-core = { path = "../etl-core" }
etl-orchestrate = { path = "../etl-orchestrate" }
etl-sources = { path = "../etl-sources" }   # the ONLY crate that may name these
clap = { workspace = true, features = ["derive"] }
tokio = { workspace = true }
anyhow = { workspace = true }
chrono = { workspace = true }
serde_json = { workspace = true }

Split the crate in two: cli.rs holds the testable parts (the clap types and the pure status-report renderer); main.rs holds the #[tokio::main] wiring that names concrete sources and cannot be unit-tested without the network. Test the seam, not the socket.

Expected result: cargo test -p panoptes-etl → the CLI-parse and status tests green; panoptes-etl --help lists the four subcommands.

The spec (givens)

cli.rs — the parseable surface:

• Cli (derive Parser): a #[command(subcommand)] command: Command, plus two global flags — --out-dir: PathBuf (default "data") and --ledger: PathBuf (default "data/runs.jsonl").
• Command (derive Subcommand, Debug, PartialEq, Eq): four variants —
  • Run — run every pipeline once, skipping sources still fresh.
  • Backfill — re-run every pipeline, ignoring freshness.
  • Embed — embed prose fields of already-loaded launches (Part VI).
  • Status — show the last successful watermark per source.
• pub const SOURCES: &[&str] = &["celestrak", "spacetrack", "spacedevs"]; — the known sources, in DAG order.
• status_report(ledger: &Ledger, sources: &[&str]) -> Result<String, EtlError> — for each source, look up ledger.last_success(source); render either "{source:<12} last success {watermark} ({rows} rows)" or "{source:<12} never run"; join with newlines. Pure over its ledger argument, so it is trivially testable.

main.rs — the wiring (given, not built here):

• #[tokio::main] async fn main() -> anyhow::Result<()>: parse the Cli, create the out-dir and Ledger, then match cli.command.
• Run / Backfill both build the executor and call run(); the difference is a single boolean — incremental = cli.command == Command::Run — which sets a Duration::hours(6) refresh window (or None for backfill). Print each record, each skipped, each blocked node.
• build_executor is where the three CelestrakSource / SpaceTrackSource / SpaceDevsSource are constructed with their base URLs and wired into Pipelines with an idempotent JSONL sink. Space-Track is added only when its credentials are present in the environment.

Concepts exercised

• clap derive with a #[command(subcommand)] enum and global flags that may appear before the subcommand.
• Separating the testable CLI surface (cli.rs) from the un-testable I/O wiring (main.rs).
• The single-boolean-toggle pattern: run vs backfill differ only in the freshness window.
• The binary as the sole owner of concrete-source knowledge — the top of the dependency-inversion story.

The build loop (you drive)

1. cli_parses_run_subcommand. Cli::try_parse_from(["panoptes-etl", "run"]); assert cli.command == Command::Run. Define Cli / Command to pass it.
2. cli_parses_flags_before_subcommand. ["panoptes-etl", "--out-dir", "/tmp/x", "status"]; assert the subcommand is Status and out_dir parsed. Predict: does clap accept a global flag before the subcommand by default?
3. status_reports_last_watermark. Seed a temp ledger with one success for celestrak (42 rows, a known watermark); call status_report(&ledger, SOURCES); assert the string contains celestrak, the watermark date, "42 rows", and that an unseeded source (spacedevs) reads "never run".
4. Wire main.rs — construct the three sources in build_executor, dispatch each subcommand. Run panoptes-etl --help and confirm all four verbs appear.
5. Run green, commit.

Done when

cargo test -p panoptes-etl is green, panoptes-etl --help lists run, backfill, embed, and status, and panoptes-etl status against a fresh ledger prints never run for all three sources. The pipeline is now a single command with discoverable verbs — and the capstone will prove it absorbs a fourth source without the orchestrator noticing.

Capstone: Add the NASA Source (You Drive)

Maps to: Task 4.4. Kind: Capstone — specification only. No walkthrough, no test names handed to you, no build loop. You have built three sources; this is the one you build alone.

The claim to prove

The entire arc has been an argument that the Source trait absorbs a new data source with zero orchestrator changes. This capstone is where you cash that claim. You will add a fourth source — NASA — as a new leaf in etl-sources, wire it into the CLI exactly like the other three, and watch the DAG schedule it without a single edit to etl-orchestrate. If you find yourself opening executor.rs or graph.rs, stop: the design says you should not have to, and if you do, something is being done at the wrong layer.

What you are building

A new source that reads from NASA's open API at api.nasa.gov, normalizes its responses into Rows, and runs as a pipeline in the DAG alongside CelesTrak, Space-Track, and TheSpaceDevs.

You choose the endpoint and the mapping. NASA publishes many datasets behind one API key — APOD (astronomy picture of the day), NeoWs (near-Earth objects), DONKI (space-weather notifications), and more. Pick one whose shape appeals to you. The point of the capstone is not a specific endpoint; it is that any of them slots in through the same trait.

Hard requirements (the givens)

• New file: crates/etl-sources/src/nasa.rs, exported from that crate's lib.rs. This is the only new source code the capstone requires.
• API key from the environment, passed as a query parameter. Read NASA_API_KEY from the environment (as the other credentialed sources read theirs) and attach it to the request as a query param — e.g. ?api_key=…. It is a query parameter, not a header, for this API. Never hard-code the key; never commit it.
• A NasaSource implementing Source. name() returns a stable string like "nasa"; extract() does the HTTP GET, maps transport errors and non-success statuses through the same classify_status helper the other sources use, and returns RawRecord(s).
• A transform implementing Transform. Parse the JSON payload into one or more of the existing Row variants, or add a new Row variant in etl-core if your chosen endpoint genuinely needs one. You decide the Row mapping — that decision is the substance of the capstone.
• Tested against wiremock, never the live API. Follow the pattern from the CelesTrak build: a mock server, a query_param("api_key", …) matcher, a canned body, assertions on the parsed RawRecord / Row. Watch for the silent-404 trap when your matcher chain is wrong.
• Wired into the CLI like the other three. Add a NasaSource pipeline in build_executor in panoptes-etl/src/main.rs, with an idempotent JSONL sink and the same freshness window.

The extension point

build_executor in crates/panoptes-etl/src/main.rs already marks exactly where your pipeline goes. The comment is there waiting for you:

// CAPSTONE (you drive, unassisted): add your NASA pipeline here.
// A `NasaSource` reads its key from `NASA_API_KEY` and passes it as a query
// parameter; wire it with a transform and a `JsonlSink` exactly like the
// three above. The Source trait is all the orchestrator needs — nothing in
// this file's structure has to change to admit a fourth source.

Adding your pipeline is three or four lines mirroring the CelesTrak block right above it: construct the source, wrap it in a Pipeline with a transform and an idempotent sink, set refresh_after, and add_pipeline("nasa", …).

The property to observe (this is the grade)

When you are done, take stock of what you did not touch:

• etl-orchestrate/src/graph.rs — unchanged.
• etl-orchestrate/src/executor.rs — unchanged.
• etl-orchestrate/Cargo.toml — still depends on etl-core only. Re-run cargo tree -p etl-orchestrate and confirm etl-sources is still absent, even though a fourth source now runs through the executor.

The orchestrator scheduled a source that did not exist when the orchestrator was written, because it only ever knew the trait. That is dependency inversion delivering — not as a slogan, but as a diff you can measure by its emptiness. A new capability arrived by adding a leaf, not by editing the core.

Self-checks (you write the tests)

No test names are given — deciding what to assert is part of the exercise. But you are done when, at minimum:

1. cargo test -p etl-sources nasa is green: the extract test hits a mock (asserting your api_key query param is sent), and the transform test turns a canned payload into the Row(s) you designed.
2. panoptes-etl status lists your new source (extend SOURCES if you want it in the status report), and panoptes-etl run executes it in DAG order with the others.
3. git diff --stat shows changes in etl-sources and panoptes-etl — and nothing in etl-orchestrate.

Concept-Check: Orchestration

This arc installed the seam that composes everything you built before it: the pipelines became nodes in a typed DAG, execution order became a computed property (Kahn's sort with a deterministic, sorted ready-set), and the executor learned two things a bare sort cannot do — skip a still-fresh source, and gate a node whose upstream failed. Above it all sits the dependency-inversion property that is the real payoff of the course: the executor depends on etl-core only and schedules dyn Source, so a fourth source drops in with zero orchestrator changes.

This check pins down the parts that are easy to get almost right — why the ready-set is a BTreeSet and not a queue, how cycle detection falls out of a length comparison, the difference between a blocked node and a failed one, and what cargo tree proves about the crate boundary.

Two of these are Tracing questions: read the program, decide whether it compiles, and if it does, predict its exact output before revealing the answer. Predicting is the point.

Concept: Structured vs Semantic Retrieval

Kind: Concept. Seam installed this arc: the retrieval split — SQL/filter for the numbers, embeddings for the prose — and a brute-force cosine index.

The idea

Four arcs in, your pipeline has produced a pile of normalized Rows: space objects with orbital elements, conjunctions with miss distances and probabilities, launches with dates and providers. Now something downstream wants to retrieve from that pile. There are two fundamentally different kinds of question it can ask, and conflating them is the single most common mistake in a first RAG system.

1. Structured questions are about the numeric and categorical fields. "Conjunctions with a collision probability above 0.001." "Launches by SpaceX after July." "The 20 objects with the lowest perigee." These are exact predicates over values that already have a precise meaning. You answer them by filtering and sorting — SQL WHERE/ORDER BY, or the Rust equivalent, iter().filter(...).sort_by(...).

2. Semantic questions are about meaning in prose. "Launches that sound like rideshare missions." "Anything describing a broadband constellation." The words in a description field carry meaning that no WHERE clause can match, because "rideshare" and "batch of smallsats for multiple customers" are close in meaning but share no keywords. You answer these by embedding the prose into vectors and ranking by similarity.

The whole discipline of this arc is a single rule: use the right mechanism for each kind of field, and never cross the streams. You do not embed the numbers, and you do not try to WHERE-clause the prose.

Why you must not semantic-search the numbers

It is tempting — especially once embeddings feel like magic — to just embed everything: turn each whole row into text, embed that, and let cosine similarity handle every query. Resist it. Embedding the numbers is wrong on every axis that matters:

• It is lossy where you can least afford it. A collision probability of 0.001 versus 0.01 is a tenfold difference that decides whether an operator moves a satellite. Rounded into a many-dimensional vector alongside a text blob, that distinction dissolves. Numbers have exact meaning; embeddings approximate. You would be throwing away precision you already had for free.
• It answers the wrong question. Cosine similarity ranks by closeness, but "probability above 0.001" is a threshold, not a ranking. There is no query vector whose nearest neighbors are exactly the rows over a cutoff. The operation you need — a filter — is not the operation an index performs.
• It is expensive and non-deterministic for a job a comparison operator does instantly and exactly. Embedding costs an API call per text; a > costs nothing and never flakes.

The numeric entities — SpaceObject, Conjunction, and the numeric fields of Launch — are for structured retrieval. The prose — and in this project that means exactly one field, a launch's description — is for semantic retrieval. That is why, when you look at the shipped code, Row::prose() returns Some for a non-empty launch description and None for everything else:

// crates/etl-core/src/vector.rs — needs the Row enum from etl-core
impl Row {
    /// The prose worth embedding, if any. Only text fields — never the numbers.
    pub fn prose(&self) -> Option<String> {
        match self {
            Row::Launch(l) if !l.description.trim().is_empty() => Some(l.description.clone()),
            _ => None,
        }
    }
}

A SpaceObject returns None. A Conjunction returns None. A Launch with an empty description returns None. That Option is the retrieval split, expressed as a method: it is the codebase's single point of truth for "what, in this row, is meaning-bearing prose rather than a number." Everything the embed stage does keys off it.

A prose field is not a vector

Before you can rank prose by meaning, you have to turn it into a vector. This is not a formality — it is a type-level fact, and the compiler will remind you. cosine takes two &[f32]; a string is not a slice of floats:

fn cosine(a: &[f32], b: &[f32]) -> f32 {
    a.iter().zip(b).map(|(x, y)| x * y).sum()
}

fn main() {
    // Raw prose is text, not a vector. You cannot rank meaning without first
    // turning each side into an embedding.
    let query = ["broadband satellites"];
    let doc = ["a launch of broadband sats"];
    let _ = cosine(&query, &doc);
}

error[E0308]: arguments to this function are incorrect
   |             ^^^^^^ ------  ---- expected `&[f32]`, found `&[&str; 1]`

That E0308 is the type system stating the whole reason the embed stage exists: there is no path from text to a similarity score that skips the embedder. Semantic retrieval always has two halves — embed the corpus once (the load target you build next chapter), and embed the query the same way at search time, then compare vectors. The two "sides" the toy below feeds to cosine are both already vectors precisely because both were embedded first.

A runnable toy: the two mechanisms side by side

Here is the split on a domain that cannot be mistaken for the answer key — a bookstore. Each book has a numeric price_cents (structured: filter it) and a prose blurb that has already been embedded into blurb_vec (semantic: cosine-rank it). Watch the two queries use two completely different operations over the same data:

// Cosine similarity; 0.0 if either side is the zero vector. Same shape as
// etl-core's `cosine` — the whole semantic index is built on this one line.
fn cosine(a: &[f32], b: &[f32]) -> f32 {
    let dot: f32 = a.iter().zip(b).map(|(x, y)| x * y).sum();
    let na = a.iter().map(|x| x * x).sum::<f32>().sqrt();
    let nb = b.iter().map(|x| x * x).sum::<f32>().sqrt();
    if na == 0.0 || nb == 0.0 { 0.0 } else { dot / (na * nb) }
}

struct Book {
    title: &'static str,
    price_cents: u32,    // a number: filter/sort it, never embed it
    blurb_vec: Vec<f32>, // the prose blurb, already embedded
}

fn main() {
    let catalog = vec![
        Book { title: "Orbital Mechanics",   price_cents: 4200, blurb_vec: vec![0.9, 0.1, 0.0] },
        Book { title: "The Quiet Garden",    price_cents: 1500, blurb_vec: vec![0.0, 0.2, 0.9] },
        Book { title: "Rockets for Kids",    price_cents: 1200, blurb_vec: vec![0.7, 0.3, 0.2] },
    ];

    // STRUCTURED: "books under $20" is a numeric predicate. Exact, cheap,
    // no embeddings — asking a vector index this would be absurd.
    println!("under $20:");
    for b in catalog.iter().filter(|b| b.price_cents < 2000) {
        println!("  {} (${}.{:02})", b.title, b.price_cents / 100, b.price_cents % 100);
    }

    // SEMANTIC: "something like a spaceflight book" is a meaning query. We
    // embed the QUERY the same way the blurbs were embedded, then rank by
    // cosine. The numbers (price) play no part here.
    let query = vec![0.9, 0.1, 0.05]; // pretend-embedding of "spaceflight"
    let mut ranked: Vec<(&str, f32)> = catalog
        .iter()
        .map(|b| (b.title, cosine(&query, &b.blurb_vec)))
        .collect();
    ranked.sort_by(|a, b| b.1.total_cmp(&a.1));
    println!("closest in meaning:");
    for (title, score) in ranked.iter().take(2) {
        println!("  {title} ({score:.3})");
    }
}

under $20:
  The Quiet Garden ($15.00)
  Rockets for Kids ($12.00)
closest in meaning:
  Orbital Mechanics (0.998)
  Rockets for Kids (0.938)

Look at what each query touched. The "under $20" query read price_cents and never looked at a vector; a cheap paperback about a garden and a cheap kids' book both qualify because price is all that matters. The "closest in meaning" query read blurb_vec and never looked at price; the two spaceflight-ish books rise to the top, the $15 garden book sinks, and the $42 price of the top hit is irrelevant to the ranking. Same catalog, two orthogonal mechanisms. That is the split you are building, with the domain swapped and the vectors made real.

Brute-force cosine is the correct tool here — no vector database

Read the total workload honestly: this project's corpus is thousands of launch descriptions, not millions of web pages. top_k over the whole index is a linear scan — compute cosine against every row, sort, take k. At a few thousand rows that is sub-millisecond, runs in a plain Vec<EmbeddedRow>, needs no extra process, no index build, no ANN approximation, and returns the exact nearest neighbors every time.

A vector database (FAISS, Qdrant, pgvector, a hosted service) buys you approximate nearest-neighbor search that stays fast at millions-to-billions of vectors — and charges you an operational dependency, an index that must be rebuilt as data changes, and approximate results for that speed. At this corpus size you would be paying all of that cost to make a fast, exact operation slightly faster and less correct. Brute force is not the cheap placeholder here; it is the right engineering answer for the scale. The shipped VectorIndex is a Vec and a sort, on purpose.

The one-for-one mapping

Everything in the bookstore toy maps onto what you build this arc. Book.price_cents is the numeric side — Conjunction.collision_probability, Launch.net, an orbital perigee — retrieved by filter/sort (structured). Book.blurb_vec is Launch.description after it passes through Row::prose() and the embedder — retrieved by cosine + top_k (semantic). The toy's cosine is etl_core::cosine verbatim. The toy's ranked.sort_by(...).take(2) is VectorIndex::top_k. And the toy's hard separation — price never enters the meaning query, vectors never enter the price query — is the discipline that keeps Row::prose() returning None for everything that is not a launch description. Learn the split here; the build pages are filling in a template you already understand.

Questions to lock

1. Give one structured question and one semantic question over the space corpus, and name the mechanism each one uses.
2. State three concrete reasons embedding a collision_probability (or any number) is the wrong tool.
3. Which single field in this project is meaning-bearing prose, and what does Row::prose() return for a SpaceObject or a Conjunction?
4. Semantic search has two halves. What has to happen to the incoming query before you can compute a similarity against the index, and why must it use the same model?
5. Why is a brute-force cosine scan the correct choice at this corpus size rather than a stopgap — and what specific evidence would justify adding a vector database later?

The graded quiz for this arc is at the end of the part, on the Concept-Check: Retrieval page.

Concept: The Embedder Trait & Content-Addressed Embeddings

Kind: Concept. Seam installed this arc: the Embedder trait — the embedding boundary — plus the content-addressed cache that makes embedding idempotent.

The idea

An embedder turns text into vectors. That is the entire job of the interface. But how you draw the boundary around that job decides whether the rest of your system is testable, swappable, and — the point of this chapter — cheap to re-run. The shipped Embedder trait is deliberately tiny:

// crates/etl-core/src/embed.rs
// (needs async-trait; not playground-safe — see the toy below for the same shape)
use async_trait::async_trait;

/// Turns text into vectors. One vector per input, same order.
#[async_trait]
pub trait Embedder: Send + Sync {
    async fn embed(&self, texts: &[String]) -> Result<Vec<Vec<f32>>, EtlError>;
}

If that shape looks familiar, it should: it is the same move as the first course's ModelClient. One async method, Send + Sync so it can be scheduled across threads, Result<_, EtlError> so failures flow through the one pipeline error type, and a batch in / batch out signature (&[String] → Vec<Vec<f32>>, one vector per input, same order). The trait names what an embedder does; it says nothing about who — Voyage, OpenAI, a local model, or a mock. That silence is the seam.

Embeddings are the definitive expensive, idempotent, content-addressed workload

You met content-addressing in Part IV, for idempotent writes. Embeddings are the workload that makes it matter, because embedding has three properties that line up exactly:

• Expensive. Every embed call is a network round trip to a paid API. Embedding the same launch description twice is money spent twice for a byte-identical result.
• Idempotent by nature. A given model embedding a given string yields the same vector every time. The output is a pure function of the input — which is the precondition for caching to be correct, not just fast.
• Content-addressed for free. Because the output depends only on the text, the text (hashed) is the cache key. content_id("embed", text) — the same sha256(kind ‖ 0x00 ‖ canonical) you built for rows — gives a stable id for a piece of prose. Same text, same id, every run, across processes.

Put those together and the rule writes itself: the same text must never be embedded twice. Not within a run, not across runs, not across machines that share the index file. The content id is how you enforce it — you embed a text only if you have not already seen its id.

A runnable toy: mock the embedder, cache by content

The real trait needs async-trait, so it will not run on the playground. Here is the same shape on a domain that cannot be mistaken for the answer key, reduced to synchronous std so it runs anywhere: a MockEmbedder that counts how many texts it was asked to embed, and a seen set keyed by content id that guarantees a repeated text is embedded exactly once.

use std::collections::HashSet;

/// Turns text into vectors. One vector per input, same order. Mirrors the real
/// `Embedder` trait — swappable and, crucially, mockable.
trait Embed {
    fn embed(&self, texts: &[String]) -> Vec<Vec<f32>>;
}

/// A test double: deterministic, counts how many texts it was asked to embed,
/// and never touches the network. This is what wiremock/mocks buy you.
struct MockEmbedder {
    calls: std::cell::Cell<usize>,
}
impl Embed for MockEmbedder {
    fn embed(&self, texts: &[String]) -> Vec<Vec<f32>> {
        self.calls.set(self.calls.get() + texts.len());
        texts.iter().map(|t| vec![t.len() as f32, 1.0]).collect()
    }
}

/// A content id for a piece of text. The real code hashes with sha256; here the
/// text itself is the key — the POINT is that identical text yields an identical
/// id, so a `seen` set can skip it.
fn content_id(text: &str) -> String {
    format!("embed\u{0}{text}")
}

fn main() {
    let embedder = MockEmbedder { calls: std::cell::Cell::new(0) };
    let mut seen: HashSet<String> = HashSet::new();

    // Two loads. The second repeats a text already embedded in the first.
    let batch1 = vec!["a launch of broadband sats".to_string(), "a resupply mission".to_string()];
    let batch2 = vec!["a resupply mission".to_string(), "a crewed test flight".to_string()];

    for batch in [batch1, batch2] {
        // Only embed texts whose content id we have not seen before.
        let fresh: Vec<String> = batch
            .into_iter()
            .filter(|t| seen.insert(content_id(t)))
            .collect();
        let vectors = embedder.embed(&fresh);
        println!("this load embedded {} fresh text(s)", vectors.len());
    }

    // 4 texts arrived across two loads, but one was a repeat.
    println!("total texts actually embedded: {}", embedder.calls.get());
}

this load embedded 2 fresh text(s)
this load embedded 1 fresh text(s)
total texts actually embedded: 3

Four texts arrived across the two loads. One — "a resupply mission" — appeared in both. The seen set caught it the second time (HashSet::insert returns false when the key is already present), so the mock was asked to embed only 3 distinct texts. That is the content-addressed cache in miniature: the expensive operation runs once per distinct content, no matter how many times the content shows up or how many separate loads it is spread across.

From "same run" to "across runs" — seeding the seen set

An in-memory seen set dedupes within a single process. But an ETL job runs on a schedule: today's run must not re-embed prose that yesterday's run already paid for. The fix is the same one the idempotent sink uses — seed the cache from what is already on disk. The shipped EmbedSink has two constructors for exactly this:

// crates/etl-core/src/vector.rs — needs etl-core (Embedder, content_id, ...)
impl<E: Embedder> EmbedSink<E> {
    /// Fresh: nothing embedded yet.
    pub fn new(embedder: E, path: impl Into<PathBuf>) -> Self { /* ... */ }

    /// Pre-load already-embedded ids so re-runs across processes never re-embed
    /// the same prose (embedding is expensive — this is the whole point).
    pub fn with_seen(
        embedder: E,
        path: impl Into<PathBuf>,
        ids: impl IntoIterator<Item = String>,
    ) -> Self { /* ... */ }
}

At startup the binary reads the existing embeddings.jsonl, collects every EmbeddedRow.id, and hands those ids to with_seen. The seen set therefore begins each run already populated with everything ever embedded — so the very first thing a re-run does is recognize that all its prose is old and make zero API calls. The content id is the bridge that makes an in-memory set behave like a durable, cross-run cache: the ids on disk and the ids computed this run are the same function of the same text, so they match.

The one-for-one mapping

The toy's Embed trait is etl_core::Embedder with the async stripped off for the playground. MockEmbedder is the test-suite's CountingEmbedder — deterministic vectors, a call counter. The toy's content_id is the real content_id("embed", text) (sha256 instead of a format string, same contract: same text → same id). The toy's seen HashSet is EmbedSink's Mutex<HashSet<String>>, and the toy's "filter to fresh, then embed" loop is the body of EmbedSink::load. The real production embedder behind the trait is VoyageEmbedder — bearer-authenticated, wiremock-tested, never hitting the live API in a test. Learn the boundary and the cache here; the build page is filling in a template you already understand.

Questions to lock

1. What is the exact signature of Embedder::embed, and what does each part (async, Send + Sync, &[String] → Vec<Vec<f32>>, Result<_, EtlError>) buy you?
2. Embeddings are called the "definitive" content-addressed workload. Name the three properties that make content-addressed caching both correct and worth it here.
3. HashSet::insert returns a bool. How does that one return value implement "embed each distinct text at most once"?
4. Why can you assert "embedded exactly N times" against a mock but not against the real API, and which shipped test relies on this?
5. An in-memory seen set only dedupes within a run. What does EmbedSink::with_seen seed it from, and why does the content id make an in-memory set behave like a durable cross-run cache?

The graded quiz for this arc is at the end of the part, on the Concept-Check: Retrieval page.

Build: The Embed Load Target

Maps to: Task 5.1 + 5.2 (the load half). Kind: Build — spec and test names only. No implementation.

Objective

Implement two things that form the write side of semantic retrieval: VoyageEmbedder, a concrete Embedder backed by a real HTTP API but tested entirely against a mock; and EmbedSink, a Sink that embeds prose only, content-addressed so the same text is never embedded twice. Together they are how launch descriptions become durable, deduplicated vectors on disk.

Scaffold

Create: crates/etl-core/src/vector.rs (the EmbedSink, plus Row::prose); crates/etl-sources/src/voyage.rs (the VoyageEmbedder). The Embedder trait itself lives in crates/etl-core/src/embed.rs (Task 5.1). Declare the modules and re-export EmbedSink, EmbeddedRow, VectorIndex, cosine from etl-core's lib.rs, and VoyageEmbedder from etl-sources's lib.rs.

Dependencies: none new. reqwest, serde/serde_json, tokio, async-trait, sha2 and the dev-only wiremock were all declared in earlier arcs. The embedder reuses the HTTP-error classifier (crate::http::classify_status) from the authenticated arc.

Expected result: cargo test -p etl-sources voyage → the Voyage tests pass; cargo test -p etl-core embed (or vector) → the sink tests pass.

The spec (givens) — VoyageEmbedder

• Fields: base_url, api_key, model (all String; base_url injectable so tests point at a mock). Constructor new(base_url, api_key) defaulting model to "voyage-3"; plus from_env(base_url) that reads VOYAGE_API_KEY and maps a missing key to EtlError::Auth.
• Implements Embedder. embed(&self, texts: &[String]):
  • Empty input is a no-op: if texts is empty, return Ok(vec![]) without making a request. (This is a real correctness rule — an empty batch must not cost a round trip, and some APIs reject an empty input.)
  • POST {base_url}/v1/embeddings (trim a trailing / off base_url), bearer auth with the key, JSON body {"input": <texts>, "model": <model>}.
  • On a non-success status, return classify_status(status, headers) — the same transient/permanent split the retry layer reads.
  • Parse {"data": [{"embedding": [...]}, ...]} and return the embeddings in order. A parse failure maps to EtlError::Parse.

The spec (givens) — Row::prose + EmbedSink

• Row::prose(&self) -> Option<String> — returns Some(description) only for a Row::Launch whose description is non-empty after trim; None for SpaceObject, Conjunction, and empty-description launches. This method is the structured-vs-semantic split; the sink embeds exactly what it returns.
• EmbeddedRow { id: String, text: String, vector: Vec<f32> } — one embedded prose field; Serialize/Deserialize, so it round-trips through JSONL.
• EmbedSink<E: Embedder> — fields: the embedder, a path: PathBuf, and a seen: Mutex<HashSet<String>>. Two constructors: new(embedder, path) (empty seen) and with_seen(embedder, path, ids) (seed seen from ids already on disk).
• impl Sink for EmbedSink — load(&self, rows):
  1. For each row, take row.prose(); skip None. Compute content_id("embed", &text). Insert the id into seen; keep the text only if the insert was new (dedupe within and across loads).
  2. If nothing is fresh, return Ok(0) — no embed call.
  3. Embed the fresh texts in one batch. Guard the length: if the embedder returns a vector count that differs from the text count, return EtlError::Parse — never silently misalign texts and vectors.
  4. Append one EmbeddedRow per (id, text, vector) as JSONL (create + append, single buffered write_all, same file discipline as JsonlSink). Return the count written.

Test names to hit

VoyageEmbedder (wiremock):

• embeds_batch_of_texts_with_bearer_auth — mount a mock matching POST /v1/embeddings and header("authorization", "Bearer test-key"), returning two embeddings. Call embed with two texts; assert the parsed vectors equal the mocked ones. The header matcher is an assertion that your client actually sent bearer auth.
• empty_input_makes_no_request — construct an embedder pointing at an unroutable address, mount no mock, and assert embed(&[]) returns an empty Vec — proving the empty-input short-circuit never calls out.

EmbedSink:

• embed_sink_only_touches_prose — load a Launch (has prose) and a Conjunction (no prose) through a CountingEmbedder; assert the return is 1 and the embedder was called for exactly 1 text. Proves the split: numbers are never embedded.
• cached_text_not_re_embedded — load two launches with identical descriptions; assert 1 written and 1 embed call (dedupe within a load). Load the same rows again; assert 0 written and the call count unchanged (dedupe across loads). This is the content-addressed cache, made a test.

The build loop (you drive)

1. Build the Embedder trait first (Task 5.1, embed.rs) — one async method; it is three lines but it is the seam everything else names.
2. Write embeds_batch_of_texts_with_bearer_auth, then implement VoyageEmbedder against it. Add empty_input_makes_no_request and confirm the short-circuit.
3. Write embed_sink_only_touches_prose with a CountingEmbedder mock (deterministic vectors, an atomic call counter). Implement Row::prose and the prose-filtering half of load.
4. Write cached_text_not_re_embedded. Predict both loads' results, then implement the seen/content_id dedupe. Watch the lock scope (see the trap).
5. Run green, commit.

Done when

cargo test -p etl-sources voyage and cargo test -p etl-core embed both pass. You now have the write side of retrieval: prose becomes deduplicated EmbeddedRow JSONL, at most one embed call per distinct text, and a real HTTP client proven correct without spending a cent. The next build page reads those vectors back — the search side — and wires the whole embed step into the orchestrator.

Build: The Brute-Force Retriever + Wiring it as a DAG Node

Maps to: Task 5.2 (the search half) + 5.3. Kind: Build — spec and test names only. No implementation.

Objective

Implement the read side of semantic retrieval — cosine, EmbeddedRow's index, and VectorIndex::top_k — and then wire the embed step into the pipeline as something the orchestrator schedules: the embed subcommand today, a DAG node downstream of spacedevs in a fuller build. This is where the vectors written last chapter become answers to queries.

Scaffold

Create/extend: crates/etl-core/src/vector.rs (add cosine, VectorIndex, top_k, VectorIndex::from_jsonl). Modify: crates/panoptes-etl/src/cli.rs (add the Embed subcommand) and crates/panoptes-etl/src/main.rs (the embed_launches wiring).

Dependencies: none new. serde/serde_json, tokio, clap are all present from earlier arcs.

Expected result: cargo test -p etl-core cosine / top_k → the index tests pass; cargo test -p panoptes-etl → the CLI parses the new subcommand.

The spec (givens) — the index

• pub fn cosine(a: &[f32], b: &[f32]) -> f32 — dot product over the L2 norms; return 0.0 if either norm is zero (never divide by zero). Identical vectors score 1.0; orthogonal vectors score 0.0.
• VectorIndex { pub rows: Vec<EmbeddedRow> } — new(rows), Default, and from_jsonl(text) (one EmbeddedRow per non-blank line; a parse failure maps to EtlError::Parse). This is the whole "database": a Vec, in memory.
• top_k(&self, query: &[f32], k: usize) -> Vec<(String, f32)> — score every row's vector against query with cosine, sort descending by score (use f32::total_cmp — f32 is not Ord), truncate to k, return (id, score) pairs. Highest similarity first.

The spec (givens) — wiring the step

• Embed subcommand — add an Embed variant to the clap Command enum with a doc comment ("Embed the prose fields of already-loaded launches into a vector index"), and a match arm in main that calls embed_launches(&cli.out_dir) and prints the count of new prose fields embedded.
• embed_launches(out_dir) — read out_dir/launches.jsonl into Vec<Row> (error out if empty — "run panoptes-etl run first"); read the existing out_dir/embeddings.jsonl for already-embedded ids; build VoyageEmbedder::from_env("https://api.voyageai.com"); construct EmbedSink::with_seen(embedder, index_path, seen_ids); load the rows and return the count.
• read_embedded_ids(path) — parse each line as EmbeddedRow, collect the ids; a missing file yields an empty Vec (a first run has nothing embedded yet).

Test names to hit

• cosine_of_identical_is_one_orthogonal_is_zero — assert cosine of a vector with itself is 1.0 (within a small epsilon) and of two orthogonal vectors is ~0.0. Pins the two anchor points of the metric.
• top_k_ranks_by_similarity — build a VectorIndex of three rows (one near the query, one opposite, one in between); top_k(query, 2); assert the result has length 2, the first id is the nearest row, and scores are non-increasing (hits[0].1 >= hits[1].1).
• embed_node_runs_after_sources (integration-style) / cli_parses_embed_subcommand — assert Cli::try_parse_from(["panoptes-etl", "embed"]) yields Command::Embed, and (fuller build) that the executor schedules the embed node only after spacedevs succeeds.

The build loop (you drive)

1. Write cosine_of_identical_is_one_orthogonal_is_zero, implement cosine with the zero-norm guard.
2. Write top_k_ranks_by_similarity. Predict the ranking, then implement top_k with total_cmp + truncate.
3. Add the Embed subcommand and cli_parses_embed_subcommand; wire embed_launches / read_embedded_ids in main.
4. Run green, commit. Optionally: promote the embed step to a real DAG node downstream of spacedevs and prove ordering.

Done when

cargo test -p etl-core covers cosine/top_k and cargo test -p panoptes-etl covers the subcommand. You can now run panoptes-etl run to populate launches.jsonl, then panoptes-etl embed to turn its prose into a deduplicated vector index, and VectorIndex::top_k answers "what launches are most like this query vector?" — the semantic half of retrieval, standing beside the structured filters, exactly as the split promised. That completes the four seams and the RAG arc: the pipeline now extracts, transforms, loads, schedules, and retrieves.

Concept-Check: Retrieval

You have installed the fourth and final seam this arc: the retrieval split — structured filters for the numbers, embeddings for the prose — behind the Embedder trait and a brute-force cosine VectorIndex. This check pins down the parts that are easy to get almost right: which field is meaning-bearing prose (and why the numbers stay out of the index), why embedding is the definitive content-addressed workload, how the seen/content_id cache guarantees a text is embedded at most once, and the exact behavior of cosine and top_k.

Three of these are Tracing questions: read the program, decide whether it compiles, and if it does, predict its exact output before revealing the answer. Predicting is the point — the cosine and top_k traces are the same arithmetic the shipped index runs.

Where This Plugs Into Panoptes

Kind: Wrap-up.

You have built the pipeline, the seams, and a real mini-orchestrator over them. This chapter is the one that makes all of it count: it shows where the files you write are read, and why the whole ETL was worth building. The ETL was never the point. Grounding Panoptes' scenarios in real orbital data was the point — and that grounding is a handful of files on disk.

The connection is a file contract, not a call

The single most important design decision in this course is one you never had to write code for: panoptes_etl and Panoptes are joined by files on disk, not by a shared database or an in-process crate dependency. The ETL runs, writes normalized JSONL to an output directory, and stops. Later — minutes or days later — panoptes-gen reads those files when it builds vignettes. Neither crate imports the other. Neither holds a lock the other waits on.

This mirrors Panoptes' own existing scenarios/ directory: a producer writes, a consumer reads, and the contract between them is a serialization format nobody can silently break, because breaking it means a serde deserialize error at the boundary — loud, immediate, and impossible to miss.

The contract has exactly one shape. Every entity the ETL emits is a variant of one enum:

#[derive(Serialize, Deserialize)]
#[serde(tag = "kind", rename_all = "snake_case")]
pub enum Row {
    SpaceObject(SpaceObject),
    Conjunction(Conjunction),
    Launch(Launch),
}

Because Row is internally tagged by kind, every line of every output file is self-describing. A consumer can read a mixed stream and dispatch on kind without a schema handshake. That is the whole file contract, in one derive.

What each file carries

The binary writes four artifacts into the output directory, plus the ledger. Each has one reader and one job:

Note what is not a top-level file: OrbitalElements never stands alone. It lives inside each SpaceObject as the elements field, and the derived quantities — period, apogee, perigee, orbit class — are computed on demand rather than stored, so the file stays canonical. When panoptes-gen needs to know that an object is in GEO, it calls orbit_class(); the file carries the seven Keplerian numbers, and the classification is a function of them.

The keystone: a Conjunction is a ca_geo scenario

Everything in this course orbited one entity. Here is the payoff for making it the keystone.

Panoptes' one live scenario family, ca_geo, is collision avoidance in geostationary orbit: an operator deciding whether to maneuver, under time pressure, with uncertain attribution of who caused the conjunction. The family has four parameters — attribution_confidence, time_pressure (Hours/Days), reversibility, and info_request. A Conjunction row is the factual spine those parameters dress:

pub struct Conjunction {
    pub id: String,               // content-addressed at load time
    pub tca: DateTime<Utc>,       // time of closest approach
    pub miss_distance_km: f64,
    pub collision_probability: f64,
    pub primary: NoradId,
    pub secondary: NoradId,
}

Read that against the scenario grid and the mapping is almost mechanical:

• tca is where time_pressure comes from. A TCA a few hours out is the Hours cell; days out is the Days cell. The pressure is not invented — it is the clock on a real event.
• primary / secondary are the join keys back into space_objects.jsonl. Resolve the secondary and its operator field, and you have the raw material for attribution_confidence: a named commercial operator reads differently from an unattributed debris fragment. This is precisely why the other entities exist — to enrich and attribute conjunctions.
• miss_distance_km and collision_probability are the stakes. They are what makes the decision hard rather than obvious, and they are measured, not stipulated.

What the ETL does not supply is reversibility and info_request. Those are decision framings Panoptes overlays on the factual event — knobs on the vignette, not facts about the sky. That division is exactly right: the ETL owns what is true, and Panoptes owns how the question is posed. A real Conjunction Data Message is a collision-avoidance decision waiting to be asked; the ETL's job was to deliver it real, and the keystone is where "real" lives.

Retrieval feeds Panoptes two ways

The RAG arc produced embeddings.jsonl and a brute-force retriever. Only prose was ever embedded — Row::prose() returns text for a Launch description and None for everything numeric, because you never semantic-search the numbers. That retriever hands Panoptes two distinct capabilities, and the split matters:

1. Grounding a vignette's prose. When panoptes-gen writes the narrative around a conjunction — the mission context, the operator's history, the launch that put the object up there — it can retrieve the most relevant real descriptions and weave them in. The vignette reads as factual because it is built from facts pulled by cosine similarity over real launch prose.

2. As a retrieval eval treatment arm. Retrieval can instead be the thing under test: one arm of the experiment gives the model retrieved context, another withholds it, and Panoptes measures whether grounding changes the decision. Here retrieval is not a writing aid — it is an independent variable.

Both consume the same embeddings.jsonl through the same top_k interface. The choice between "grounding" and "treatment arm" is Panoptes' to make, not the ETL's — augment-and-generate was always out of scope here. The ETL's contract ends at the vector file. What Panoptes does with a retrieved fact is a research-design decision, and keeping that decision on Panoptes' side of the file boundary is the same discipline as everything else in this course: produce the artifact, expose it cleanly, and let the consumer decide what it means.

What you actually shipped

The harness reads real satellites, real elements, and real conjunctions from four files, and it can regenerate scenarios from scratch whenever the sky changes. That is the deliverable. The seams — the Source trait, the E/T/L split, the ledger, content-addressing — were the discipline that let you build it without painting yourself into a corner. The next chapter is about the corners you deliberately left open.

The Graduation Path

Kind: Wrap-up.

Everything in this chapter is design, not code. Nothing here was built in the course — and that is the entire point. This course climbed to "pipeline + seams + a real mini-orchestrator" and stopped there on purpose. A production data platform has four more things this one does not: a real storage-and-search engine, a production rate limiter, orbit propagation, and a scheduler that runs on its own. This chapter names each one, points at the exact seam it snaps onto, and explains why leaving it out was a design decision rather than an omission.

The claim the whole course has been making is testable right here: if the four seams are real, each of these four additions is a node you add, not a rewrite you suffer. Let's check.

First, the four seams — because they are the argument

Every graduation below snaps onto one of these. Keep them in view:

1. The Source trait — extraction behind one async fn. New source, new impl.
2. The E/T/L split — extract, transform, load as three swappable stages joined by typed boundaries (RawRecord → Row → Sink).
3. The run ledger — an append-only RunRecord per run, carrying the watermark.
4. Idempotent, content-addressed writes — a sha256 of the row's canonical form gives a stable id, so re-running is always safe.

A seam is only real if you can replace what sits behind it without touching what sits in front. That is what "additive, not a rewrite" means, and it is what each of the following tests.

DuckDB + VSS over Parquet — snaps onto the Sink / VectorIndex boundary

What it replaces. Two things at once: the JsonlSink load target, and the brute-force VectorIndex. Today the ETL writes newline-delimited JSON and searches vectors by scanning every row with cosine. The graduation writes columnar Parquet and searches with DuckDB's VSS (vector similarity search) extension — indexed nearest-neighbour instead of a linear scan, plus real SQL over the numeric entities.

The seam it snaps onto. Loading was never println!-to-a-file; it was always a trait:

#[async_trait]
pub trait Sink: Send + Sync {
    async fn load(&self, rows: &[Row]) -> Result<usize, EtlError>;
}

A DuckDbSink is just another impl Sink. Nothing upstream of the sink — no source, no transform, no orchestrator — knows or cares whether the bytes land as JSONL or Parquet. On the retrieval side, VectorIndex already exposes a narrow read interface — from_jsonl to build, top_k(query, k) to search. A DuckDB-backed index that keeps the same top_k shape drops in behind that boundary the same way. The EmbedSink that pays for embeddings stays exactly as it is; only where the vectors land and how they are searched changes.

Why it was deferred. The corpus is a few thousand prose fields, not millions. At that scale brute-force cosine is not a compromise — it is the correct tool, and reaching for a vector database would teach the wrong instinct: that you need heavy machinery before you have the data volume that justifies it. Building the honest version first, behind a trait, is what makes the upgrade cheap when the volume finally arrives.

governor — snaps onto the TokenBucket constructor argument

What it replaces. The hand-rolled TokenBucket limiter — TokenBucket::new(capacity, refill_per_sec), with an acquire() that sleeps until a token is free. governor is the production-grade equivalent: battle-tested, more limiting algorithms, no rate-limiter bugs of your own to maintain.

The seam it snaps onto. The limiter was never hard-wired into the source. It is a constructor parameter:

SpaceTrackSource::new(SPACETRACK_BASE, creds, TokenBucket::new(30, 0.5))

Anything that gates a call before the fetch can take that slot. Wrap governor to satisfy the same "await before you fetch" shape and pass it where TokenBucket goes. The source's extract logic does not move a line. This is the smallest graduation of the four precisely because the injection point was designed in from the start.

Why it was deferred. Rate limiting is the Part III lesson. You cannot understand what governor does for you until you have felt the problem it solves — Space-Track banning an impolite client — and built the token bucket that fixes it by hand. Hand-rolling the thing that is the lesson, and reaching for the library for the thing that is not, is the rule the whole stack follows.

sgp4 orbit propagation — snaps onto the parsed OrbitalElements

What it adds. sgp4 takes a set of orbital elements at an epoch and propagates them forward to compute where an object will be at a future time. With it, you could compute conjunctions yourself — screen every pair of objects for close approaches — instead of ingesting conjunctions that arrive pre-computed.

The seam it snaps onto. This one is different: it is not the Source/Transform boundary. Propagation consumes the domain model the transform already produces. Your OrbitalElements struct is the input sgp4 wants:

pub struct OrbitalElements {
    pub epoch: DateTime<Utc>,
    pub inclination_deg: f64,
    pub raan_deg: f64,
    pub eccentricity: f64,
    pub arg_perigee_deg: f64,
    pub mean_anomaly_deg: f64,
    pub mean_motion_rev_per_day: f64,
}

Parsing a TLE into these seven numbers is ETL. Turning these numbers into a future position is computation on top of the parsed model — a downstream consumer of the file contract, not a new stage inside a pipeline. That distinction is why it lives on the graduation shelf rather than in the transform arc.

Why it was deferred. Propagation is a rabbit hole — reference frames, drag models, numerical stability — and none of it is the ETL lesson. Parsing and normalizing the elements is. And critically, the course did not need it: real conjunctions arrive pre-computed from Space-Track / SOCRATES, so the keystone entity was available without propagating anything. Building your own conjunction screening is a genuinely new capability, not a missing piece of this one.

A real scheduler / daemon — snaps onto the Executor and the ledger watermark

What it adds. Today the pipeline runs when you type panoptes-etl run. The graduation is a long-running process that drives the Executor on a cron or interval — waking up, running the DAG, sleeping, repeating — with no human at the keyboard. This is the option-C ceiling: the thing that finally makes the project look like Airflow.

The seams it snaps onto — two of them. The executor already exposes the one method a scheduler needs to call:

let report = executor.run().await?;   // runs the whole DAG once

A daemon is a loop around that call plus a clock. But the loop would be blind without the ledger. Incremental runs already read it: Ledger::last_success finds the last good run, and Pipeline::is_fresh skips a source whose last success is inside its refresh_after window. Every RunRecord carries a watermark, and the executor stamps it on success:

pub struct RunRecord {
    pub source: String,
    pub finished_at: DateTime<Utc>,
    pub outcome: Outcome,
    pub watermark: Option<DateTime<Utc>>,
    pub rows_written: usize,
}

The scheduler reads that watermark to decide what to fetch and when to fetch it. Retries with backoff are already in the executor, reused from Part IV. So the daemon adds a driver on top of finished machinery — a tokio timer that calls run() — without reaching inside any pipeline. Airflow's value proposition, stripped to its core, is a good ledger plus a scheduler that reads it; you built the ledger, and the scheduler is the loop it was always waiting for.

Why it was deferred. Two reasons, both honest. First, a background scheduler is genuinely painful to test-drive: time, concurrency, and flakiness fight the clean one-test-per-chapter rhythm this course depends on, and a shaky capstone would undercut the discipline the whole book teaches. Second, it earns its own course. The right place for a scheduler is a sequel that can give concurrency and time the careful treatment they deserve — not a rushed final chapter here.

The discipline was the subject

Look back at the four graduations. Each one replaces or extends something behind a seam, and in every case the thing in front of the seam does not move:

• DuckDB slots behind Sink and VectorIndex — sources and orchestrator untouched.
• governor slots into a constructor argument — extract untouched.
• sgp4 consumes OrbitalElements — the transform untouched.
• The scheduler loops over Executor::run and reads the ledger — the pipelines untouched.

That is the proof the seams were real. You did not build these four things, and because you built the simplest pipeline that already had the joints an orchestrator needs, you do not have to rewrite anything to add them later. Leaving a seam for a future you cannot yet see — and having that future arrive as a new node instead of a demolition — was the whole subject of the course. The graduation shelf is not a list of what is missing. It is the evidence that the discipline paid off.

Appendix: Task-to-Chapter Map

Each build chapter maps to one task in the Answer Key. The reference implementation is panoptes_etl; every task's code compiles and its tests pass (65 in total).

The invariants

These held throughout the build and are worth re-checking against your own copy:

1. One seam per arc. Every arc ships a working pipeline and installs exactly one seam an orchestrator later needs.
2. No build chapter uses machinery no earlier chapter introduced. If a build needs a crate or API with no lead-in, a concept chapter precedes it.
3. etl-orchestrate depends on etl-core only. The executor schedules dyn Source and cannot name a concrete source; cargo tree -p etl-orchestrate proves it. Only panoptes-etl (the binary) names concrete sources.
4. Every live API is mocked with wiremock; no test hits the network. Time- based tests (limiter, retry) use paused tokio time, so they are deterministic.
5. The keystone is Conjunction. A conjunction data message is a raw collision-avoidance decision — the ground truth a Panoptes ca_geo vignette is built from.

Appendix: The Answer Key (Full Task Plan)

This is the complete, verified reference implementation — the code you check your own work against. It is the panoptes_etl workspace exactly as it ships: it compiles, passes 65 tests, and is clippy-clean.

Read it as a reference, not a script. Every file is reproduced in full, including its #[cfg(test)] module, because the tests are part of the answer — they pin the behavior each build chapter asks you to reach. The listings here are reference-only (they are not compiled by the book build), but they are the real, shipped source.

The sections follow the course arcs in dependency order: Foundations → Extraction → Authenticated → Paginated → Orchestration → Retrieval. Each file has a stable anchor (id="task-…") so a build chapter can deep-link straight to its answer.

The dependency arrow always points toward etl-core. etl-orchestrate depends on etl-core only and schedules dyn Source; only the panoptes-etl binary names concrete sources. That constraint is enforced by the crate boundaries below, not by convention.

Part I — Foundations

The workspace skeleton and the two Part I prerequisites — the error taxonomy and the NoradId newtype — that every later file leans on.

Cargo.toml — Workspace manifest (Part I)

The virtual workspace: four member crates and one shared dependency table so versions never drift between crates.

[workspace]
resolver = "2"
members = ["crates/etl-core", "crates/etl-sources", "crates/etl-orchestrate", "crates/panoptes-etl"]

[workspace.dependencies]
serde = { version = "1", features = ["derive"] }
serde_json = "1"
chrono = { version = "0.4", features = ["serde"] }
thiserror = "2"
anyhow = "1"
async-trait = "0.1"
tokio = { version = "1", features = ["full"] }
reqwest = { version = "0.12", features = ["json"] }
clap = { version = "4", features = ["derive"] }
sha2 = "0.10"
itertools = "0.13"
# dev
wiremock = "0.6"
pretty_assertions = "1"

crates/etl-core/Cargo.toml — the core crate manifest (Part I)

etl-core depends on nothing else in the workspace. It pulls the test-util feature of tokio only as a dev-dependency, for the paused-time tests.

[package]
name = "etl-core"
version = "0.1.0"
edition = "2024"

[dependencies]
serde = { workspace = true }
serde_json = { workspace = true }
chrono = { workspace = true }
thiserror = { workspace = true }
async-trait = { workspace = true }
sha2 = { workspace = true }
tokio = { workspace = true }

[dev-dependencies]
pretty_assertions = { workspace = true }
tokio = { workspace = true, features = ["test-util"] }

crates/etl-core/src/lib.rs — the core crate root (Part I)

The module list and the flat re-export surface every other crate imports from. It names modules built across every later arc (ledger, retry, content, embed, vector) — this is the final root; you grow it arc by arc.

//! `etl-core` — the seams every other crate depends on.
//!
//! Domain model, the `Source` / `Transform` / `Sink` traits, the error
//! taxonomy, and id newtypes. This crate depends on nothing else in the
//! workspace; the dependency arrow always points *toward* it.

pub mod content;
pub mod domain;
pub mod embed;
pub mod error;
pub mod ids;
pub mod ledger;
pub mod pipeline;
pub mod retry;
pub mod sink_jsonl;
pub mod vector;

pub use content::{content_id, IdempotentSink};
pub use domain::{Conjunction, Launch, OrbitClass, OrbitalElements, SpaceObject};
pub use embed::Embedder;
pub use error::EtlError;
pub use ids::NoradId;
pub use ledger::{Ledger, Outcome, RunRecord};
pub use pipeline::{RawRecord, Row, Sink, Source, Transform};
pub use retry::{with_retry, RetryPolicy};
pub use sink_jsonl::JsonlSink;
pub use vector::{cosine, EmbedSink, EmbeddedRow, VectorIndex};

crates/etl-core/src/error.rs — the error taxonomy (Part I)

The single EtlError every stage returns. The load-bearing distinction is transient vs permanent: is_transient() decides what the retry layer will ever retry, and retry_after() surfaces a server-supplied backoff.

use std::time::Duration;

use thiserror::Error;

/// The single error type every ETL stage returns.
///
/// The load-bearing distinction is **transient vs permanent**: a transient
/// error (a dropped connection, a rate-limit response) is worth retrying with
/// backoff; a permanent one (a malformed payload, bad credentials) will fail
/// identically every time and must surface immediately.
#[derive(Debug, Error)]
pub enum EtlError {
    /// A network-level failure — connection reset, timeout, 5xx. Transient.
    #[error("network error: {0}")]
    Network(String),

    /// The server asked us to slow down. Transient, with a hint. `429`.
    #[error("rate limited; retry after {retry_after_secs}s")]
    RateLimited { retry_after_secs: u64 },

    /// Authentication failed — bad or missing credentials. Permanent. `401`/`403`.
    #[error("auth error: {0}")]
    Auth(String),

    /// A non-retryable client error — a `404`, a `400`, a renamed endpoint.
    /// Permanent: it will fail identically on every retry.
    #[error("permanent error: {0}")]
    Permanent(String),

    /// A payload could not be parsed into the domain model. Permanent.
    #[error("parse error: {0}")]
    Parse(String),

    /// A local I/O failure while reading or writing a sink.
    #[error("io error: {0}")]
    Io(#[from] std::io::Error),
}

impl EtlError {
    /// Whether retrying this error (after backoff) could plausibly succeed.
    pub fn is_transient(&self) -> bool {
        matches!(self, EtlError::Network(_) | EtlError::RateLimited { .. })
    }

    /// The server-requested backoff, when the error carries one.
    pub fn retry_after(&self) -> Option<Duration> {
        match self {
            EtlError::RateLimited { retry_after_secs } => {
                Some(Duration::from_secs(*retry_after_secs))
            }
            _ => None,
        }
    }
}

#[cfg(test)]
mod tests {
    use super::*;
    use pretty_assertions::assert_eq;

    #[test]
    fn network_and_rate_limit_are_transient() {
        assert!(EtlError::Network("reset".into()).is_transient());
        assert!(EtlError::RateLimited { retry_after_secs: 30 }.is_transient());
    }

    #[test]
    fn parse_auth_and_permanent_are_not_transient() {
        assert!(!EtlError::Parse("bad line".into()).is_transient());
        assert!(!EtlError::Auth("401".into()).is_transient());
        assert!(!EtlError::Permanent("404".into()).is_transient());
    }

    #[test]
    fn retry_after_only_on_rate_limit() {
        assert_eq!(
            EtlError::RateLimited { retry_after_secs: 12 }.retry_after(),
            Some(Duration::from_secs(12))
        );
        assert_eq!(EtlError::Network("x".into()).retry_after(), None);
    }
}

Tests: network_and_rate_limit_are_transient, parse_auth_and_permanent_are_not_transient, retry_after_only_on_rate_limit.

crates/etl-core/src/ids.rs — the NoradId newtype (Part I)

A u32 NORAD catalog number wrapped so it can't be confused with any other integer id. #[serde(transparent)] keeps the wire form a bare integer; Display zero-pads to five digits; FromStr returns an EtlError::Parse.

use std::fmt;
use std::str::FromStr;

use serde::{Deserialize, Serialize};

use crate::error::EtlError;

/// A NORAD catalog number — the canonical id for a tracked space object.
///
/// A newtype rather than a bare `u32` so it cannot be confused with any other
/// numeric id (a launch sequence, a payload count). The `#[serde(transparent)]`
/// makes it serialize as the plain integer, so the wire format stays clean.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord, Serialize, Deserialize)]
#[serde(transparent)]
pub struct NoradId(pub u32);

impl fmt::Display for NoradId {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        // NORAD ids are conventionally shown zero-padded to five digits.
        write!(f, "{:05}", self.0)
    }
}

impl FromStr for NoradId {
    type Err = EtlError;

    fn from_str(s: &str) -> Result<Self, Self::Err> {
        s.trim()
            .parse::<u32>()
            .map(NoradId)
            .map_err(|_| EtlError::Parse(format!("invalid NORAD id: {s:?}")))
    }
}

#[cfg(test)]
mod tests {
    use super::*;
    use pretty_assertions::assert_eq;

    #[test]
    fn display_is_zero_padded_to_five() {
        assert_eq!(NoradId(25544).to_string(), "25544");
        assert_eq!(NoradId(733).to_string(), "00733");
    }

    #[test]
    fn parses_from_trimmed_digits() {
        assert_eq!("25544".parse::<NoradId>().unwrap(), NoradId(25544));
        assert_eq!(" 00733 ".parse::<NoradId>().unwrap(), NoradId(733));
    }

    #[test]
    fn rejects_non_numeric() {
        assert!("ISS".parse::<NoradId>().is_err());
    }

    #[test]
    fn serializes_as_bare_integer() {
        assert_eq!(serde_json::to_string(&NoradId(25544)).unwrap(), "25544");
        assert_eq!(
            serde_json::from_str::<NoradId>("25544").unwrap(),
            NoradId(25544)
        );
    }
}

Tests: display_is_zero_padded_to_five, parses_from_trimmed_digits, rejects_non_numeric, serializes_as_bare_integer.

Part II — The Extraction Arc (CelesTrak)

The domain model, the three E/T/L traits, the JSONL sink, and the first real source — CelesTrak — with its TLE parser and HTTP-status classifier.

crates/etl-core/src/domain.rs — the domain model (Part II)

OrbitalElements, the derived orbit-class classifier, and the three stored entities (SpaceObject, Conjunction, Launch). Derived quantities (period, apogee, class) are computed on demand so the stored form stays canonical.

use chrono::{DateTime, Utc};
use serde::{Deserialize, Serialize};

use crate::ids::NoradId;

/// Standard gravitational parameter of Earth, km^3 / s^2.
const EARTH_MU_KM3_S2: f64 = 398600.4418;
/// Equatorial radius of Earth, km.
const EARTH_RADIUS_KM: f64 = 6378.137;

/// A coarse orbit-regime classification, derived from the elements.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "SCREAMING_SNAKE_CASE")]
pub enum OrbitClass {
    /// Low Earth orbit.
    Leo,
    /// Medium Earth orbit.
    Meo,
    /// Geostationary / geosynchronous.
    Geo,
    /// Highly elliptical orbit.
    Heo,
}

/// The Keplerian mean elements of an orbit at a given epoch.
///
/// These are the numbers a TLE carries. The derived quantities (period,
/// apogee, perigee, class) are computed on demand rather than stored, so the
/// stored form stays canonical.
#[derive(Debug, Clone, Copy, PartialEq, Serialize, Deserialize)]
pub struct OrbitalElements {
    pub epoch: DateTime<Utc>,
    pub inclination_deg: f64,
    pub raan_deg: f64,
    pub eccentricity: f64,
    pub arg_perigee_deg: f64,
    pub mean_anomaly_deg: f64,
    pub mean_motion_rev_per_day: f64,
}

impl OrbitalElements {
    /// Orbital period in minutes, straight from mean motion.
    pub fn period_minutes(&self) -> f64 {
        1440.0 / self.mean_motion_rev_per_day
    }

    /// Semi-major axis in km, from mean motion via Kepler's third law.
    pub fn semi_major_axis_km(&self) -> f64 {
        let n_rad_s = self.mean_motion_rev_per_day * 2.0 * std::f64::consts::PI / 86_400.0;
        (EARTH_MU_KM3_S2 / (n_rad_s * n_rad_s)).cbrt()
    }

    /// Apogee altitude above the equator, km.
    pub fn apogee_km(&self) -> f64 {
        self.semi_major_axis_km() * (1.0 + self.eccentricity) - EARTH_RADIUS_KM
    }

    /// Perigee altitude above the equator, km.
    pub fn perigee_km(&self) -> f64 {
        self.semi_major_axis_km() * (1.0 - self.eccentricity) - EARTH_RADIUS_KM
    }

    /// A coarse regime classification from period and eccentricity.
    pub fn orbit_class(&self) -> OrbitClass {
        let period = self.period_minutes();
        if self.eccentricity > 0.25 {
            OrbitClass::Heo
        } else if period < 128.0 {
            OrbitClass::Leo
        } else if (1430.0..=1450.0).contains(&period) {
            OrbitClass::Geo
        } else {
            OrbitClass::Meo
        }
    }
}

/// A tracked space object with its most recent elements.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct SpaceObject {
    pub norad_id: NoradId,
    pub name: String,
    /// The international designator as it appears in the TLE (e.g. `"98067A"`),
    /// i.e. the raw 2-digit-year field — not the hyphenated COSPAR form.
    pub intl_designator: String,
    pub operator: Option<String>,
    pub elements: OrbitalElements,
}

impl SpaceObject {
    pub fn orbit_class(&self) -> OrbitClass {
        self.elements.orbit_class()
    }
}

/// A conjunction between two objects — the keystone entity, a raw
/// collision-avoidance decision. `id` is content-addressed at load time.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct Conjunction {
    pub id: String,
    pub tca: DateTime<Utc>,
    pub miss_distance_km: f64,
    pub collision_probability: f64,
    pub primary: NoradId,
    pub secondary: NoradId,
}

/// A launch, sourced from TheSpaceDevs. `description` is a prose field —
/// the kind of text the RAG arc later embeds.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct Launch {
    pub id: String,
    pub name: String,
    pub net: DateTime<Utc>,
    pub provider: String,
    pub payloads: Vec<String>,
    pub description: String,
}

#[cfg(test)]
mod tests {
    use super::*;
    use pretty_assertions::assert_eq;

    fn iss_elements() -> OrbitalElements {
        OrbitalElements {
            epoch: "2026-07-01T00:00:00Z".parse().unwrap(),
            inclination_deg: 51.64,
            raan_deg: 250.0,
            eccentricity: 0.0007,
            arg_perigee_deg: 130.0,
            mean_anomaly_deg: 230.0,
            mean_motion_rev_per_day: 15.5,
        }
    }

    fn geo_elements() -> OrbitalElements {
        OrbitalElements {
            epoch: "2026-07-01T00:00:00Z".parse().unwrap(),
            inclination_deg: 0.05,
            raan_deg: 95.0,
            eccentricity: 0.0002,
            arg_perigee_deg: 270.0,
            mean_anomaly_deg: 90.0,
            mean_motion_rev_per_day: 1.0027,
        }
    }

    #[test]
    fn iss_is_low_earth_orbit() {
        let e = iss_elements();
        // 1440 / 15.5 = 92.9 min.
        assert!((e.period_minutes() - 92.90).abs() < 0.01);
        assert_eq!(e.orbit_class(), OrbitClass::Leo);
        // ISS perigee sits a few hundred km up.
        assert!((300.0..500.0).contains(&e.perigee_km()));
    }

    #[test]
    fn geostationary_is_classified_geo() {
        let e = geo_elements();
        assert!((e.period_minutes() - 1436.1).abs() < 0.5);
        assert_eq!(e.orbit_class(), OrbitClass::Geo);
    }

    #[test]
    fn orbit_class_serializes_screaming() {
        assert_eq!(
            serde_json::to_string(&OrbitClass::Geo).unwrap(),
            "\"GEO\""
        );
    }

    #[test]
    fn space_object_roundtrips_through_json() {
        let obj = SpaceObject {
            norad_id: NoradId(25544),
            name: "ISS (ZARYA)".into(),
            intl_designator: "98067A".into(),
            operator: Some("NASA".into()),
            elements: iss_elements(),
        };
        let json = serde_json::to_string(&obj).unwrap();
        let back: SpaceObject = serde_json::from_str(&json).unwrap();
        assert_eq!(back, obj);
    }
}

Tests: iss_is_low_earth_orbit, geostationary_is_classified_geo, orbit_class_serializes_screaming, space_object_roundtrips_through_json.

crates/etl-core/src/pipeline.rs — the E/T/L traits (Part II)

RawRecord (the output of E), the tagged Row enum (the T→L boundary), and the three object-safe traits Source / Transform / Sink. Source and Sink are async via async_trait; Transform is pure and synchronous.

use async_trait::async_trait;
use chrono::{DateTime, Utc};
use serde::{Deserialize, Serialize};

use crate::domain::{Conjunction, Launch, SpaceObject};
use crate::error::EtlError;

/// Raw bytes fetched from a source, before any parsing — the output of E.
#[derive(Debug, Clone, PartialEq)]
pub struct RawRecord {
    pub source: String,
    pub payload: String,
    pub fetched_at: DateTime<Utc>,
}

/// A normalized domain row — the typed boundary between T and L.
///
/// A single enum over every entity a source can emit lets one `Sink` accept
/// the output of any pipeline without knowing which source produced it.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
#[serde(tag = "kind", rename_all = "snake_case")]
pub enum Row {
    SpaceObject(SpaceObject),
    Conjunction(Conjunction),
    Launch(Launch),
}

/// **E** — extract raw records from an external source.
///
/// `Send + Sync` and object-safe (via `async_trait`) so the orchestrator can
/// schedule a `dyn Source` without knowing the concrete type.
#[async_trait]
pub trait Source: Send + Sync {
    fn name(&self) -> &str;
    async fn extract(&self) -> Result<Vec<RawRecord>, EtlError>;
}

/// **T** — parse raw records into normalized rows. Pure and synchronous.
pub trait Transform: Send + Sync {
    fn transform(&self, raw: &RawRecord) -> Result<Vec<Row>, EtlError>;
}

/// **L** — persist normalized rows to a sink; returns the count written.
#[async_trait]
pub trait Sink: Send + Sync {
    async fn load(&self, rows: &[Row]) -> Result<usize, EtlError>;
}

#[cfg(test)]
mod tests {
    use super::*;
    use crate::domain::{Launch, OrbitalElements, SpaceObject};
    use crate::ids::NoradId;
    use pretty_assertions::assert_eq;

    #[test]
    fn row_tags_by_kind() {
        let launch = Row::Launch(Launch {
            id: "ll2-42".into(),
            name: "Starlink G-99".into(),
            net: "2026-07-10T12:00:00Z".parse().unwrap(),
            provider: "SpaceX".into(),
            payloads: vec!["Starlink".into()],
            description: "A batch of broadband satellites.".into(),
        });
        let json = serde_json::to_string(&launch).unwrap();
        assert!(json.contains("\"kind\":\"launch\""));
        let back: Row = serde_json::from_str(&json).unwrap();
        assert_eq!(back, launch);
    }

    #[tokio::test]
    async fn a_source_can_be_boxed_as_dyn() {
        struct Stub;
        #[async_trait]
        impl Source for Stub {
            fn name(&self) -> &str {
                "stub"
            }
            async fn extract(&self) -> Result<Vec<RawRecord>, EtlError> {
                Ok(vec![RawRecord {
                    source: "stub".into(),
                    payload: "hi".into(),
                    fetched_at: "2026-07-01T00:00:00Z".parse().unwrap(),
                }])
            }
        }
        let source: Box<dyn Source> = Box::new(Stub);
        let raw = source.extract().await.unwrap();
        assert_eq!(source.name(), "stub");
        assert_eq!(raw.len(), 1);
    }

    // Silences unused-import warnings for types exercised only in other tests.
    #[allow(dead_code)]
    fn _touch(_: SpaceObject, _: OrbitalElements, _: NoradId) {}
}

Tests: row_tags_by_kind, a_source_can_be_boxed_as_dyn.

crates/etl-core/src/sink_jsonl.rs — the JSONL sink (Part II)

The L contract panoptes-gen reads: newline-delimited Row values, appended (never truncated) so a run never clobbers prior output.

use std::path::PathBuf;

use async_trait::async_trait;
use tokio::fs::OpenOptions;
use tokio::io::AsyncWriteExt;

use crate::error::EtlError;
use crate::pipeline::{Row, Sink};

/// Appends normalized rows to a file as JSON Lines — one JSON object per line.
///
/// This is the load contract `panoptes-gen` reads: newline-delimited `Row`
/// values, appended so a run never truncates prior output.
pub struct JsonlSink {
    pub path: PathBuf,
}

impl JsonlSink {
    pub fn new(path: impl Into<PathBuf>) -> Self {
        Self { path: path.into() }
    }
}

#[async_trait]
impl Sink for JsonlSink {
    async fn load(&self, rows: &[Row]) -> Result<usize, EtlError> {
        let mut file = OpenOptions::new()
            .create(true)
            .append(true)
            .open(&self.path)
            .await?;
        let mut buf = String::new();
        for row in rows {
            let line = serde_json::to_string(row)
                .map_err(|e| EtlError::Parse(e.to_string()))?;
            buf.push_str(&line);
            buf.push('\n');
        }
        file.write_all(buf.as_bytes()).await?;
        Ok(rows.len())
    }
}

#[cfg(test)]
mod tests {
    use super::*;
    use crate::domain::Launch;
    use pretty_assertions::assert_eq;

    fn sample_rows() -> Vec<Row> {
        vec![
            Row::Launch(Launch {
                id: "a".into(),
                name: "One".into(),
                net: "2026-07-10T12:00:00Z".parse().unwrap(),
                provider: "SpaceX".into(),
                payloads: vec![],
                description: "first".into(),
            }),
            Row::Launch(Launch {
                id: "b".into(),
                name: "Two".into(),
                net: "2026-07-11T12:00:00Z".parse().unwrap(),
                provider: "RocketLab".into(),
                payloads: vec![],
                description: "second".into(),
            }),
        ]
    }

    #[tokio::test]
    async fn writes_one_row_per_line_and_roundtrips() {
        let dir = std::env::temp_dir().join(format!("etl-jsonl-{}", std::process::id()));
        tokio::fs::create_dir_all(&dir).await.unwrap();
        let path = dir.join("rows.jsonl");
        let _ = tokio::fs::remove_file(&path).await;

        let sink = JsonlSink::new(&path);
        let rows = sample_rows();
        let n = sink.load(&rows).await.unwrap();
        assert_eq!(n, 2);

        let text = tokio::fs::read_to_string(&path).await.unwrap();
        let lines: Vec<&str> = text.lines().collect();
        assert_eq!(lines.len(), 2);
        let back: Row = serde_json::from_str(lines[0]).unwrap();
        assert_eq!(back, rows[0]);
    }

    #[tokio::test]
    async fn append_accumulates_across_loads() {
        let dir = std::env::temp_dir().join(format!("etl-jsonl-app-{}", std::process::id()));
        tokio::fs::create_dir_all(&dir).await.unwrap();
        let path = dir.join("rows.jsonl");
        let _ = tokio::fs::remove_file(&path).await;

        let sink = JsonlSink::new(&path);
        sink.load(&sample_rows()).await.unwrap();
        sink.load(&sample_rows()).await.unwrap();

        let text = tokio::fs::read_to_string(&path).await.unwrap();
        assert_eq!(text.lines().count(), 4);
    }
}

Tests: writes_one_row_per_line_and_roundtrips, append_accumulates_across_loads.

crates/etl-sources/Cargo.toml — the sources crate manifest (Part II)

Concrete sources. Depends on etl-core and reqwest; mocks every live API with wiremock as a dev-dependency.

[package]
name = "etl-sources"
version = "0.1.0"
edition = "2024"

[dependencies]
etl-core = { path = "../etl-core" }
serde = { workspace = true }
serde_json = { workspace = true }
chrono = { workspace = true }
async-trait = { workspace = true }
reqwest = { workspace = true }
tokio = { workspace = true }

[dev-dependencies]
pretty_assertions = { workspace = true }
tokio = { workspace = true, features = ["test-util"] }
wiremock = { workspace = true }

crates/etl-sources/src/lib.rs — the sources crate root (Part II)

The module list and re-exports for every concrete source and its support machinery. Like the core root, this is the final form; you grow it arc by arc.

//! `etl-sources` — concrete `Source`/`Transform` implementations and the
//! machinery each real API forces on us (TLE parsing, rate limiting, retry,
//! pagination). Depends only on `etl-core`.

pub mod celestrak;
pub mod http;
pub mod limiter;
pub mod spacedevs;
pub mod spacetrack;
pub mod tle;
pub mod voyage;

pub use celestrak::{CelestrakSource, CelestrakTransform};
pub use limiter::TokenBucket;
pub use spacedevs::{SpaceDevsSource, SpaceDevsTransform};
pub use spacetrack::{Credentials, SpaceTrackSource, SpaceTrackTransform};
pub use voyage::VoyageEmbedder;

crates/etl-sources/src/http.rs — the HTTP-status classifier (Part II)

The single place that maps a non-success HTTP status onto the EtlError taxonomy — so the transient/permanent decision is made once, correctly, and every source funnels through it.

//! Mapping HTTP responses onto the [`EtlError`] taxonomy.
//!
//! This is the single place that decides which statuses are worth retrying.
//! Every source funnels non-success responses through [`classify_status`] so
//! the transient/permanent distinction is made once, correctly.

use etl_core::EtlError;
use reqwest::header::HeaderMap;
use reqwest::StatusCode;

/// Parse a `Retry-After` header value, in seconds. Absent/unparseable ⇒ 1s.
///
/// Only the delta-seconds form is handled; the HTTP-date form
/// (`Retry-After: Wed, 21 Oct 2026 07:28:00 GMT`) falls back to 1s. Real
/// APIs here use seconds; parsing the date form is left as a hardening step.
pub fn retry_after_secs(headers: &HeaderMap) -> u64 {
    headers
        .get(reqwest::header::RETRY_AFTER)
        .and_then(|v| v.to_str().ok())
        .and_then(|s| s.trim().parse().ok())
        .unwrap_or(1)
}

/// Map a non-success status onto the error taxonomy.
///
/// - `429` ⇒ [`EtlError::RateLimited`] (transient, carries the backoff)
/// - `401` / `403` ⇒ [`EtlError::Auth`] (permanent)
/// - other `4xx` ⇒ [`EtlError::Permanent`] (permanent — retrying won't help)
/// - `5xx` and anything else ⇒ [`EtlError::Network`] (transient)
pub fn classify_status(status: StatusCode, headers: &HeaderMap) -> EtlError {
    match status.as_u16() {
        429 => EtlError::RateLimited {
            retry_after_secs: retry_after_secs(headers),
        },
        401 | 403 => EtlError::Auth(format!("http {status}")),
        s if (400..500).contains(&s) => EtlError::Permanent(format!("http {status}")),
        _ => EtlError::Network(format!("http {status}")),
    }
}

#[cfg(test)]
mod tests {
    use super::*;

    fn headers_with_retry_after(v: &str) -> HeaderMap {
        let mut h = HeaderMap::new();
        h.insert(reqwest::header::RETRY_AFTER, v.parse().unwrap());
        h
    }

    #[test]
    fn maps_429_to_rate_limited_with_backoff() {
        let err = classify_status(StatusCode::TOO_MANY_REQUESTS, &headers_with_retry_after("30"));
        assert!(matches!(err, EtlError::RateLimited { retry_after_secs: 30 }));
        assert!(err.is_transient());
    }

    #[test]
    fn maps_401_403_to_permanent_auth() {
        assert!(matches!(
            classify_status(StatusCode::UNAUTHORIZED, &HeaderMap::new()),
            EtlError::Auth(_)
        ));
        assert!(!classify_status(StatusCode::FORBIDDEN, &HeaderMap::new()).is_transient());
    }

    #[test]
    fn maps_404_to_permanent_not_network() {
        let err = classify_status(StatusCode::NOT_FOUND, &HeaderMap::new());
        assert!(matches!(err, EtlError::Permanent(_)));
        assert!(!err.is_transient());
    }

    #[test]
    fn maps_5xx_to_transient_network() {
        let err = classify_status(StatusCode::INTERNAL_SERVER_ERROR, &HeaderMap::new());
        assert!(matches!(err, EtlError::Network(_)));
        assert!(err.is_transient());
    }
}

Tests: maps_429_to_rate_limited_with_backoff, maps_401_403_to_permanent_auth, maps_404_to_permanent_not_network, maps_5xx_to_transient_network.

crates/etl-sources/src/tle.rs — TLE parser (Part II)

A hand-rolled parser for the fixed-width two-line element format: mod-10 checksum (minus counts as 1), the 2-digit-year epoch pivot, byte-slice column extraction, and the cross-line catalog-number check that catches misalignment a checksum can't.

//! A hand-rolled parser for the two-line element (TLE) set format.
//!
//! TLE is a fixed-width legacy text format: fields live at exact column
//! positions, and each line ends in a mod-10 checksum digit. Parsing it by
//! column — and validating the checksum — is the extraction lesson.

use chrono::{DateTime, Duration, TimeZone, Utc};
use etl_core::{EtlError, OrbitalElements};

/// The mod-10 checksum of a TLE line: sum the digits in columns 1–68, count
/// each minus sign as 1 and everything else as 0, take the result mod 10.
pub fn tle_checksum(line: &str) -> u8 {
    let sum: u32 = line
        .chars()
        .take(68)
        .map(|c| match c {
            '0'..='9' => c.to_digit(10).unwrap(),
            '-' => 1,
            _ => 0,
        })
        .sum();
    (sum % 10) as u8
}

/// Decode a TLE epoch (2-digit year + fractional day-of-year) to UTC.
///
/// Years 57–99 are 1957–1999; 00–56 are 2000–2056 (the TLE convention).
pub fn decode_epoch(year2: &str, day_frac: &str) -> Result<DateTime<Utc>, EtlError> {
    let yy: i32 = year2
        .trim()
        .parse()
        .map_err(|_| EtlError::Parse(format!("bad epoch year: {year2:?}")))?;
    let year = if yy < 57 { 2000 + yy } else { 1900 + yy };
    let doy: f64 = day_frac
        .trim()
        .parse()
        .map_err(|_| EtlError::Parse(format!("bad epoch day: {day_frac:?}")))?;

    let start = Utc
        .with_ymd_and_hms(year, 1, 1, 0, 0, 0)
        .single()
        .ok_or_else(|| EtlError::Parse(format!("bad epoch year: {year}")))?;
    // Day-of-year is 1-based; the fractional part is a fraction of a day.
    let seconds = (doy - 1.0) * 86_400.0;
    Ok(start + Duration::milliseconds((seconds * 1000.0).round() as i64))
}

/// The NORAD catalog number carried in columns 3–7 of either line.
pub fn tle_norad(line1: &str) -> Result<u32, EtlError> {
    slice(line1, 2, 7)?
        .trim()
        .parse()
        .map_err(|_| EtlError::Parse(format!("bad NORAD id in {line1:?}")))
}

/// The international designator, columns 10–17 of line 1.
pub fn tle_intl_designator(line1: &str) -> Result<String, EtlError> {
    Ok(slice(line1, 9, 17)?.trim().to_string())
}

/// Parse a name + two element lines into [`OrbitalElements`], validating both
/// checksums first.
pub fn parse_tle(line1: &str, line2: &str) -> Result<OrbitalElements, EtlError> {
    verify_checksum(line1)?;
    verify_checksum(line2)?;

    // Both lines carry the catalog number; a mismatch means the 3-line block
    // was misaligned (an upstream truncation) — checksums can't catch that.
    if slice(line1, 2, 7)?.trim() != slice(line2, 2, 7)?.trim() {
        return Err(EtlError::Parse(format!(
            "TLE line mismatch: line 1 is {:?}, line 2 is {:?}",
            slice(line1, 2, 7)?,
            slice(line2, 2, 7)?
        )));
    }

    let epoch = decode_epoch(slice(line1, 18, 20)?, slice(line1, 20, 32)?)?;

    let inclination_deg = field(line2, 8, 16)?;
    let raan_deg = field(line2, 17, 25)?;
    let eccentricity = format!("0.{}", slice(line2, 26, 33)?.trim())
        .parse::<f64>()
        .map_err(|_| EtlError::Parse(format!("bad eccentricity in {line2:?}")))?;
    let arg_perigee_deg = field(line2, 34, 42)?;
    let mean_anomaly_deg = field(line2, 43, 51)?;
    let mean_motion_rev_per_day = field(line2, 52, 63)?;

    Ok(OrbitalElements {
        epoch,
        inclination_deg,
        raan_deg,
        eccentricity,
        arg_perigee_deg,
        mean_anomaly_deg,
        mean_motion_rev_per_day,
    })
}

fn verify_checksum(line: &str) -> Result<(), EtlError> {
    let expected = line
        .chars()
        .nth(68)
        .and_then(|c| c.to_digit(10))
        .ok_or_else(|| EtlError::Parse(format!("missing checksum digit in {line:?}")))?
        as u8;
    let got = tle_checksum(line);
    if got != expected {
        return Err(EtlError::Parse(format!(
            "checksum mismatch in {line:?}: computed {got}, line says {expected}"
        )));
    }
    Ok(())
}

fn slice(line: &str, start: usize, end: usize) -> Result<&str, EtlError> {
    line.get(start..end)
        .ok_or_else(|| EtlError::Parse(format!("line too short for cols {start}..{end}: {line:?}")))
}

fn field(line: &str, start: usize, end: usize) -> Result<f64, EtlError> {
    slice(line, start, end)?
        .trim()
        .parse()
        .map_err(|_| EtlError::Parse(format!("bad number at cols {start}..{end} in {line:?}")))
}

#[cfg(test)]
mod tests {
    use super::*;
    use chrono::{Datelike, Timelike};
    use etl_core::OrbitClass;
    use pretty_assertions::assert_eq;

    // The canonical ISS TLE (Wikipedia's worked example). Both checksums are 7.
    const L1: &str = "1 25544U 98067A   08264.51782528 -.00002182  00000-0 -11606-4 0  2927";
    const L2: &str = "2 25544  51.6416 247.4627 0006703 130.5360 325.0288 15.72125391563537";

    #[test]
    fn checksum_matches_known_lines() {
        assert_eq!(tle_checksum(L1), 7);
        assert_eq!(tle_checksum(L2), 7);
    }

    #[test]
    fn rejects_bad_checksum() {
        // Flip the last digit; verification must fail as a parse error.
        let bad = format!("{}9", &L2[..68]);
        let err = parse_tle(L1, &bad).unwrap_err();
        assert!(matches!(err, EtlError::Parse(_)));
    }

    #[test]
    fn decodes_2008_epoch() {
        let e = decode_epoch("08", "264.51782528").unwrap();
        assert_eq!(e.year(), 2008);
        assert_eq!(e.month(), 9);
        assert_eq!(e.day(), 20);
        assert_eq!(e.hour(), 12);
    }

    #[test]
    fn parses_iss_tle_to_elements() {
        let e = parse_tle(L1, L2).unwrap();
        assert!((e.inclination_deg - 51.6416).abs() < 1e-4);
        assert!((e.eccentricity - 0.0006703).abs() < 1e-9);
        assert!((e.mean_motion_rev_per_day - 15.72125391).abs() < 1e-6);
        assert_eq!(e.orbit_class(), OrbitClass::Leo);
    }

    #[test]
    fn reads_norad_and_designator() {
        assert_eq!(tle_norad(L1).unwrap(), 25544);
        assert_eq!(tle_intl_designator(L1).unwrap(), "98067A");
    }

    #[test]
    fn rejects_misaligned_line_pair() {
        // A line 2 for a *different* catalog number, with a corrected checksum,
        // so only the cross-line mismatch (not a checksum) can reject it.
        let other_body = &L2.replacen("25544", "12345", 1)[..68];
        let mismatched = format!("{other_body}{}", tle_checksum(other_body));
        let err = parse_tle(L1, &mismatched).unwrap_err();
        assert!(matches!(err, EtlError::Parse(_)));
    }
}

Tests: checksum_matches_known_lines, rejects_bad_checksum, decodes_2008_epoch, parses_iss_tle_to_elements, reads_norad_and_designator, rejects_misaligned_line_pair.

crates/etl-sources/src/celestrak.rs — the CelesTrak source (Part II)

The first real E/T split: CelestrakSource::extract is an open HTTP GET returning TLE text; CelestrakTransform chunks the text into 3-line blocks and parses each into a Row::SpaceObject. Wiremock covers success, 500, 429, and 404.

//! The CelesTrak source: an open HTTP GET returning TLE text (Extract), plus
//! the transform that parses 3-line TLE blocks into `Row::SpaceObject`.

use async_trait::async_trait;
use chrono::Utc;
use etl_core::{EtlError, RawRecord, Row, SpaceObject, Source, Transform};

use crate::http::classify_status;
use crate::tle::{parse_tle, tle_intl_designator, tle_norad};
use etl_core::NoradId;

/// Fetches a named group of elements from CelesTrak in TLE format.
pub struct CelestrakSource {
    pub base_url: String,
    pub group: String,
}

impl CelestrakSource {
    pub fn new(base_url: impl Into<String>, group: impl Into<String>) -> Self {
        Self {
            base_url: base_url.into(),
            group: group.into(),
        }
    }
}

#[async_trait]
impl Source for CelestrakSource {
    fn name(&self) -> &str {
        "celestrak"
    }

    async fn extract(&self) -> Result<Vec<RawRecord>, EtlError> {
        let url = format!(
            "{}/NORAD/elements/gp.php?GROUP={}&FORMAT=tle",
            self.base_url.trim_end_matches('/'),
            self.group
        );
        let resp = reqwest::get(&url)
            .await
            .map_err(|e| EtlError::Network(e.to_string()))?;
        if !resp.status().is_success() {
            return Err(classify_status(resp.status(), resp.headers()));
        }
        let payload = resp
            .text()
            .await
            .map_err(|e| EtlError::Network(e.to_string()))?;
        Ok(vec![RawRecord {
            source: self.name().to_string(),
            payload,
            fetched_at: Utc::now(),
        }])
    }
}

/// Parses CelesTrak TLE text into normalized space objects.
pub struct CelestrakTransform;

impl Transform for CelestrakTransform {
    fn transform(&self, raw: &RawRecord) -> Result<Vec<Row>, EtlError> {
        let lines: Vec<&str> = raw
            .payload
            .lines()
            .map(|l| l.trim_end())
            .filter(|l| !l.is_empty())
            .collect();

        let mut rows = Vec::new();
        for block in lines.chunks(3) {
            if block.len() != 3 {
                return Err(EtlError::Parse(format!(
                    "trailing partial TLE block: {block:?}"
                )));
            }
            let (name, l1, l2) = (block[0], block[1], block[2]);
            let elements = parse_tle(l1, l2)?;
            rows.push(Row::SpaceObject(SpaceObject {
                norad_id: NoradId(tle_norad(l1)?),
                name: name.trim().to_string(),
                intl_designator: tle_intl_designator(l1)?,
                operator: None,
                elements,
            }));
        }
        Ok(rows)
    }
}

#[cfg(test)]
mod tests {
    use super::*;
    use pretty_assertions::assert_eq;
    use wiremock::matchers::{method, path, query_param};
    use wiremock::{Mock, MockServer, ResponseTemplate};

    const BODY: &str = "ISS (ZARYA)\n\
        1 25544U 98067A   08264.51782528 -.00002182  00000-0 -11606-4 0  2927\n\
        2 25544  51.6416 247.4627 0006703 130.5360 325.0288 15.72125391563537\n";

    #[tokio::test]
    async fn extract_returns_raw_tle_text() {
        let server = MockServer::start().await;
        Mock::given(method("GET"))
            .and(path("/NORAD/elements/gp.php"))
            .and(query_param("GROUP", "geo"))
            .and(query_param("FORMAT", "tle"))
            .respond_with(ResponseTemplate::new(200).set_body_string(BODY))
            .mount(&server)
            .await;

        let src = CelestrakSource::new(server.uri(), "geo");
        let raw = src.extract().await.unwrap();
        assert_eq!(raw.len(), 1);
        assert_eq!(raw[0].source, "celestrak");
        assert!(raw[0].payload.contains("25544"));
    }

    #[tokio::test]
    async fn server_error_is_transient_network() {
        let server = MockServer::start().await;
        Mock::given(method("GET"))
            .respond_with(ResponseTemplate::new(500))
            .mount(&server)
            .await;
        let src = CelestrakSource::new(server.uri(), "geo");
        let err = src.extract().await.unwrap_err();
        assert!(matches!(err, EtlError::Network(_)));
        assert!(err.is_transient());
    }

    #[tokio::test]
    async fn rate_limit_is_retryable_with_backoff() {
        let server = MockServer::start().await;
        Mock::given(method("GET"))
            .respond_with(ResponseTemplate::new(429).insert_header("retry-after", "42"))
            .mount(&server)
            .await;
        let src = CelestrakSource::new(server.uri(), "geo");
        let err = src.extract().await.unwrap_err();
        assert!(matches!(err, EtlError::RateLimited { retry_after_secs: 42 }));
        assert!(err.is_transient());
    }

    #[tokio::test]
    async fn not_found_is_permanent_not_retryable() {
        let server = MockServer::start().await;
        Mock::given(method("GET"))
            .respond_with(ResponseTemplate::new(404))
            .mount(&server)
            .await;
        let src = CelestrakSource::new(server.uri(), "nonesuch");
        let err = src.extract().await.unwrap_err();
        assert!(matches!(err, EtlError::Permanent(_)));
        assert!(!err.is_transient());
    }

    #[test]
    fn transform_parses_tle_blocks_to_space_objects() {
        let raw = RawRecord {
            source: "celestrak".into(),
            payload: BODY.into(),
            fetched_at: Utc::now(),
        };
        let rows = CelestrakTransform.transform(&raw).unwrap();
        assert_eq!(rows.len(), 1);
        match &rows[0] {
            Row::SpaceObject(o) => {
                assert_eq!(o.norad_id, NoradId(25544));
                assert_eq!(o.name, "ISS (ZARYA)");
                assert_eq!(o.intl_designator, "98067A");
            }
            other => panic!("expected SpaceObject, got {other:?}"),
        }
    }
}

Tests: extract_returns_raw_tle_text, server_error_is_transient_network, rate_limit_is_retryable_with_backoff, not_found_is_permanent_not_retryable, transform_parses_tle_blocks_to_space_objects.

Part III — The Authenticated Arc (Space-Track)

Cookie-session auth, a hand-rolled token bucket, and the run ledger — the append-only artifact the orchestrator later reads for watermarks and retries.

crates/etl-core/src/ledger.rs — the run ledger (Part III)

An append-only JSONL log of every run. last_success reads back the most recent successful RunRecord for a source — the seed of incremental pulls, retries, and backfill.

//! The run ledger — an append-only record of every pipeline run.
//!
//! This one artifact is the seed of everything an orchestrator adds later:
//! incremental pulls read the last watermark, retries read the last outcome,
//! backfill reads the gaps. It mirrors the first course's append-only response
//! log.

use std::path::PathBuf;

use chrono::{DateTime, Utc};
use serde::{Deserialize, Serialize};
use tokio::fs::OpenOptions;
use tokio::io::AsyncWriteExt;

use crate::error::EtlError;

/// How a run ended.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
#[serde(tag = "outcome", rename_all = "snake_case")]
pub enum Outcome {
    Success,
    Failed { reason: String },
}

/// One line in the ledger: a single run of a single source.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct RunRecord {
    pub source: String,
    pub started_at: DateTime<Utc>,
    pub finished_at: DateTime<Utc>,
    pub outcome: Outcome,
    /// The high-water mark reached. The taught incremental path skips a source
    /// when its last success is recent (see `Executor::is_fresh`, which reads
    /// `finished_at`); consuming this watermark to bound a query — "fetch only
    /// rows after it" — is the graduation step covered in Part VII.
    pub watermark: Option<DateTime<Utc>>,
    pub rows_written: usize,
}

/// Append-only JSONL ledger.
pub struct Ledger {
    pub path: PathBuf,
}

impl Ledger {
    pub fn new(path: impl Into<PathBuf>) -> Self {
        Self { path: path.into() }
    }

    /// Append one run record.
    pub async fn append(&self, rec: &RunRecord) -> Result<(), EtlError> {
        let mut file = OpenOptions::new()
            .create(true)
            .append(true)
            .open(&self.path)
            .await?;
        let line = serde_json::to_string(rec).map_err(|e| EtlError::Parse(e.to_string()))?;
        file.write_all(line.as_bytes()).await?;
        file.write_all(b"\n").await?;
        Ok(())
    }

    /// The most recent *successful* run for a source, if any.
    pub async fn last_success(&self, source: &str) -> Result<Option<RunRecord>, EtlError> {
        let text = match tokio::fs::read_to_string(&self.path).await {
            Ok(t) => t,
            Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(None),
            Err(e) => return Err(e.into()),
        };
        let mut last = None;
        for line in text.lines().filter(|l| !l.trim().is_empty()) {
            let rec: RunRecord =
                serde_json::from_str(line).map_err(|e| EtlError::Parse(e.to_string()))?;
            if rec.source == source && rec.outcome == Outcome::Success {
                last = Some(rec);
            }
        }
        Ok(last)
    }
}

#[cfg(test)]
mod tests {
    use super::*;
    use pretty_assertions::assert_eq;

    fn rec(source: &str, watermark: &str, outcome: Outcome) -> RunRecord {
        RunRecord {
            source: source.into(),
            started_at: "2026-07-19T00:00:00Z".parse().unwrap(),
            finished_at: "2026-07-19T00:01:00Z".parse().unwrap(),
            outcome,
            watermark: Some(watermark.parse().unwrap()),
            rows_written: 10,
        }
    }

    fn temp_ledger(tag: &str) -> Ledger {
        let dir = std::env::temp_dir().join(format!("etl-ledger-{tag}-{}", std::process::id()));
        std::fs::create_dir_all(&dir).unwrap();
        let path = dir.join("runs.jsonl");
        let _ = std::fs::remove_file(&path);
        Ledger::new(path)
    }

    #[tokio::test]
    async fn append_then_last_success_reads_back() {
        let ledger = temp_ledger("read");
        let r = rec("celestrak", "2026-07-19T00:00:00Z", Outcome::Success);
        ledger.append(&r).await.unwrap();
        assert_eq!(ledger.last_success("celestrak").await.unwrap(), Some(r));
        assert_eq!(ledger.last_success("spacetrack").await.unwrap(), None);
    }

    #[tokio::test]
    async fn last_success_ignores_failures() {
        let ledger = temp_ledger("fail");
        ledger
            .append(&rec("celestrak", "2026-07-18T00:00:00Z", Outcome::Success))
            .await
            .unwrap();
        ledger
            .append(&rec(
                "celestrak",
                "2026-07-19T00:00:00Z",
                Outcome::Failed { reason: "boom".into() },
            ))
            .await
            .unwrap();
        let last = ledger.last_success("celestrak").await.unwrap().unwrap();
        // The failed run must not advance the watermark.
        assert_eq!(last.watermark, Some("2026-07-18T00:00:00Z".parse().unwrap()));
    }

    #[tokio::test]
    async fn watermark_advances_across_successful_runs() {
        let ledger = temp_ledger("advance");
        ledger
            .append(&rec("celestrak", "2026-07-18T00:00:00Z", Outcome::Success))
            .await
            .unwrap();
        ledger
            .append(&rec("celestrak", "2026-07-19T00:00:00Z", Outcome::Success))
            .await
            .unwrap();
        let last = ledger.last_success("celestrak").await.unwrap().unwrap();
        assert_eq!(last.watermark, Some("2026-07-19T00:00:00Z".parse().unwrap()));
    }
}

Tests: append_then_last_success_reads_back, last_success_ignores_failures, watermark_advances_across_successful_runs.

crates/etl-sources/src/limiter.rs — the token-bucket rate limiter (Part III)

A classic token bucket built by hand: up to capacity tokens, refilling continuously at refill_per_sec; acquire sleeps until a token is free. Tested under paused tokio time so the timing is deterministic.

//! A hand-rolled token-bucket rate limiter.
//!
//! Space-Track bans clients that hammer it, so every request must pass through
//! a bucket that refills at a fixed rate. Building it by hand — rather than
//! reaching for `governor` — is the rate-limiting lesson.

use std::sync::Arc;

use tokio::sync::Mutex;
use tokio::time::{Duration, Instant};

/// A classic token bucket: up to `capacity` tokens, refilling continuously at
/// `refill_per_sec`. `acquire` returns as soon as a token is available,
/// sleeping (and letting the bucket refill) when it is empty.
#[derive(Clone)]
pub struct TokenBucket {
    capacity: f64,
    refill_per_sec: f64,
    state: Arc<Mutex<State>>,
}

struct State {
    tokens: f64,
    last: Instant,
}

impl TokenBucket {
    pub fn new(capacity: u32, refill_per_sec: f64) -> Self {
        Self {
            capacity: capacity as f64,
            refill_per_sec,
            state: Arc::new(Mutex::new(State {
                tokens: capacity as f64,
                last: Instant::now(),
            })),
        }
    }

    /// Wait until a token is free, then consume it.
    pub async fn acquire(&self) {
        loop {
            let wait = {
                let mut s = self.state.lock().await;
                let now = Instant::now();
                let elapsed = now.duration_since(s.last).as_secs_f64();
                s.tokens = (s.tokens + elapsed * self.refill_per_sec).min(self.capacity);
                s.last = now;
                if s.tokens >= 1.0 {
                    s.tokens -= 1.0;
                    return;
                }
                // Not enough yet: sleep for the time it takes to refill one token.
                Duration::from_secs_f64((1.0 - s.tokens) / self.refill_per_sec)
            };
            tokio::time::sleep(wait).await;
        }
    }
}

#[cfg(test)]
mod tests {
    use super::*;

    #[tokio::test(start_paused = true)]
    async fn bucket_allows_burst_up_to_capacity() {
        let bucket = TokenBucket::new(3, 0.5);
        let start = Instant::now();
        for _ in 0..3 {
            bucket.acquire().await;
        }
        // A full bucket serves the burst with no waiting.
        assert!(start.elapsed() < Duration::from_millis(50));
    }

    #[tokio::test(start_paused = true)]
    async fn bucket_blocks_when_empty_then_refills() {
        let bucket = TokenBucket::new(1, 1.0);
        let start = Instant::now();
        bucket.acquire().await; // consumes the one starting token
        bucket.acquire().await; // must wait ~1s for a refill
        assert!(start.elapsed() >= Duration::from_millis(900));
    }
}

Tests: bucket_allows_burst_up_to_capacity, bucket_blocks_when_empty_then_refills.

crates/etl-sources/src/spacetrack.rs — the Space-Track source (Part III)

Cookie-session auth (log in, carry the Set-Cookie by hand) plus rate limiting on every request, then the transform of CDM JSON — where numbers arrive as strings — into Row::Conjunction, the keystone entity. Credentials come from the environment, never the URL.

//! The Space-Track source: cookie-session auth + rate limiting (Extract), plus
//! the transform from CDM JSON into `Row::Conjunction` — the keystone entity.

use async_trait::async_trait;
use chrono::{NaiveDateTime, Utc};
use etl_core::{Conjunction, EtlError, NoradId, RawRecord, Row, Source, Transform};
use serde::Deserialize;

use crate::limiter::TokenBucket;

/// Space-Track login credentials. Kept out of the URL and off disk.
pub struct Credentials {
    pub user: String,
    pub pass: String,
}

impl Credentials {
    pub fn new(user: impl Into<String>, pass: impl Into<String>) -> Self {
        Self {
            user: user.into(),
            pass: pass.into(),
        }
    }

    /// Read credentials from the environment. Missing vars are an auth error.
    pub fn from_env() -> Result<Self, EtlError> {
        let user = std::env::var("SPACETRACK_USER")
            .map_err(|_| EtlError::Auth("SPACETRACK_USER not set".into()))?;
        let pass = std::env::var("SPACETRACK_PASS")
            .map_err(|_| EtlError::Auth("SPACETRACK_PASS not set".into()))?;
        Ok(Self { user, pass })
    }
}

/// Fetches public conjunction data messages, authenticated and rate-limited.
pub struct SpaceTrackSource {
    pub base_url: String,
    pub creds: Credentials,
    pub limiter: TokenBucket,
}

impl SpaceTrackSource {
    pub fn new(base_url: impl Into<String>, creds: Credentials, limiter: TokenBucket) -> Self {
        Self {
            base_url: base_url.into(),
            creds,
            limiter,
        }
    }
}

#[async_trait]
impl Source for SpaceTrackSource {
    fn name(&self) -> &str {
        "spacetrack"
    }

    async fn extract(&self) -> Result<Vec<RawRecord>, EtlError> {
        let base = self.base_url.trim_end_matches('/');
        let client = reqwest::Client::new();

        // 1. Log in — a token is spent on every request, login included.
        self.limiter.acquire().await;
        let login = client
            .post(format!("{base}/ajaxauth/login"))
            .form(&[
                ("identity", self.creds.user.as_str()),
                ("password", self.creds.pass.as_str()),
            ])
            .send()
            .await
            .map_err(|e| EtlError::Network(e.to_string()))?;
        if !login.status().is_success() {
            // 401/403 -> Auth (permanent); 5xx/429 during login stay transient
            // so the executor still retries them. classify_status does both.
            return Err(crate::http::classify_status(login.status(), login.headers()));
        }
        let cookie = login
            .headers()
            .get(reqwest::header::SET_COOKIE)
            .and_then(|v| v.to_str().ok())
            .and_then(|v| v.split(';').next())
            .ok_or_else(|| EtlError::Auth("login returned no session cookie".into()))?
            .to_string();

        // 2. Query the CDM class, carrying the session cookie by hand.
        self.limiter.acquire().await;
        let resp = client
            .get(format!(
                "{base}/basicspacedata/query/class/cdm_public/orderby/TCA/format/json"
            ))
            .header(reqwest::header::COOKIE, cookie)
            .send()
            .await
            .map_err(|e| EtlError::Network(e.to_string()))?;
        if !resp.status().is_success() {
            return Err(crate::http::classify_status(resp.status(), resp.headers()));
        }
        let payload = resp.text().await.map_err(|e| EtlError::Network(e.to_string()))?;
        Ok(vec![RawRecord {
            source: self.name().to_string(),
            payload,
            fetched_at: Utc::now(),
        }])
    }
}

/// One CDM as Space-Track serializes it — numbers arrive as strings.
#[derive(Deserialize)]
struct RawCdm {
    #[serde(rename = "TCA")]
    tca: String,
    #[serde(rename = "MIN_RNG")]
    min_rng_m: String,
    #[serde(rename = "PC")]
    pc: Option<String>,
    #[serde(rename = "SAT_1_ID")]
    sat1: String,
    #[serde(rename = "SAT_2_ID")]
    sat2: String,
}

/// Parses Space-Track CDM JSON into conjunctions.
pub struct SpaceTrackTransform;

impl Transform for SpaceTrackTransform {
    fn transform(&self, raw: &RawRecord) -> Result<Vec<Row>, EtlError> {
        let cdms: Vec<RawCdm> =
            serde_json::from_str(&raw.payload).map_err(|e| EtlError::Parse(e.to_string()))?;
        let mut rows = Vec::with_capacity(cdms.len());
        for c in cdms {
            let tca = NaiveDateTime::parse_from_str(&c.tca, "%Y-%m-%dT%H:%M:%S%.f")
                .map_err(|e| EtlError::Parse(format!("bad TCA {:?}: {e}", c.tca)))?
                .and_utc();
            let miss_distance_km = c
                .min_rng_m
                .trim()
                .parse::<f64>()
                .map_err(|_| EtlError::Parse(format!("bad MIN_RNG {:?}", c.min_rng_m)))?
                / 1000.0;
            let collision_probability = match c.pc.as_deref() {
                Some(s) if !s.trim().is_empty() => s
                    .trim()
                    .parse::<f64>()
                    .map_err(|_| EtlError::Parse(format!("bad PC {:?}", c.pc)))?,
                _ => 0.0,
            };
            let primary: NoradId = c.sat1.parse()?;
            let secondary: NoradId = c.sat2.parse()?;
            rows.push(Row::Conjunction(Conjunction {
                id: format!("{}_{}_{}", primary.0, secondary.0, tca.format("%Y%m%dT%H%M%S")),
                tca,
                miss_distance_km,
                collision_probability,
                primary,
                secondary,
            }));
        }
        Ok(rows)
    }
}

#[cfg(test)]
mod tests {
    use super::*;
    use chrono::Utc;
    use pretty_assertions::assert_eq;
    use wiremock::matchers::{header_exists, method, path};
    use wiremock::{Mock, MockServer, ResponseTemplate};

    const CDM_JSON: &str = r#"[
        {"TCA":"2026-07-20T12:00:00.000000","MIN_RNG":"550.5","PC":"0.0001","SAT_1_ID":"25544","SAT_2_ID":"48274"}
    ]"#;

    fn bucket() -> TokenBucket {
        TokenBucket::new(30, 0.5)
    }

    #[tokio::test]
    async fn login_then_query_uses_cookie() {
        let server = MockServer::start().await;
        Mock::given(method("POST"))
            .and(path("/ajaxauth/login"))
            .respond_with(
                ResponseTemplate::new(200)
                    .insert_header("set-cookie", "chocolatechip=abc123; Path=/"),
            )
            .mount(&server)
            .await;
        Mock::given(method("GET"))
            .and(path("/basicspacedata/query/class/cdm_public/orderby/TCA/format/json"))
            .and(header_exists("cookie"))
            .respond_with(ResponseTemplate::new(200).set_body_string(CDM_JSON))
            .expect(1)
            .mount(&server)
            .await;

        let src = SpaceTrackSource::new(server.uri(), Credentials::new("u", "p"), bucket());
        let raw = src.extract().await.unwrap();
        assert_eq!(raw.len(), 1);
        assert!(raw[0].payload.contains("25544"));
        // .expect(1) verifies on drop that the cookie-guarded query was hit.
    }

    #[tokio::test]
    async fn login_failure_is_auth_error() {
        let server = MockServer::start().await;
        Mock::given(method("POST"))
            .and(path("/ajaxauth/login"))
            .respond_with(ResponseTemplate::new(401))
            .mount(&server)
            .await;
        let src = SpaceTrackSource::new(server.uri(), Credentials::new("u", "bad"), bucket());
        let err = src.extract().await.unwrap_err();
        assert!(matches!(err, EtlError::Auth(_)));
    }

    #[test]
    fn transform_parses_cdm_to_conjunction() {
        let raw = RawRecord {
            source: "spacetrack".into(),
            payload: CDM_JSON.into(),
            fetched_at: Utc::now(),
        };
        let rows = SpaceTrackTransform.transform(&raw).unwrap();
        assert_eq!(rows.len(), 1);
        match &rows[0] {
            Row::Conjunction(c) => {
                assert_eq!(c.primary, NoradId(25544));
                assert_eq!(c.secondary, NoradId(48274));
                assert!((c.miss_distance_km - 0.5505).abs() < 1e-9);
                assert!((c.collision_probability - 0.0001).abs() < 1e-9);
            }
            other => panic!("expected Conjunction, got {other:?}"),
        }
    }
}

Tests: login_then_query_uses_cookie, login_failure_is_auth_error, transform_parses_cdm_to_conjunction.

Part IV — The Paginated Arc (TheSpaceDevs)

Retry with capped exponential backoff, content-addressed idempotent writes, and a cursor-paginated source that follows next to the end.

crates/etl-core/src/retry.rs — retry with backoff (Part IV)

with_retry runs an operation, retrying only transient failures up to max_attempts, backing off exponentially — but a server-supplied Retry-After always wins. Tested under paused time.

//! Retry with capped exponential backoff — retrying only what is worth
//! retrying.
//!
//! The policy leans entirely on `EtlError::is_transient()`: permanent errors
//! return immediately, transient ones back off, and a server-supplied
//! `Retry-After` always wins over the computed backoff.

use std::future::Future;

use tokio::time::{Duration, sleep};

use crate::error::EtlError;

#[derive(Clone, Copy, Debug)]
pub struct RetryPolicy {
    pub max_attempts: u32,
    pub base_delay: Duration,
    pub max_delay: Duration,
}

impl Default for RetryPolicy {
    fn default() -> Self {
        Self {
            max_attempts: 4,
            base_delay: Duration::from_millis(200),
            max_delay: Duration::from_secs(10),
        }
    }
}

/// Run `op`, retrying transient failures up to `policy.max_attempts` times.
pub async fn with_retry<F, Fut, T>(policy: RetryPolicy, mut op: F) -> Result<T, EtlError>
where
    F: FnMut() -> Fut,
    Fut: Future<Output = Result<T, EtlError>>,
{
    let mut attempt: u32 = 0;
    loop {
        match op().await {
            Ok(value) => return Ok(value),
            Err(err) => {
                attempt += 1;
                if attempt >= policy.max_attempts || !err.is_transient() {
                    return Err(err);
                }
                // A server-requested backoff always wins; otherwise exponential.
                let backoff = err.retry_after().unwrap_or_else(|| {
                    let factor = 2u32.saturating_pow(attempt - 1);
                    (policy.base_delay * factor).min(policy.max_delay)
                });
                sleep(backoff).await;
            }
        }
    }
}

#[cfg(test)]
mod tests {
    use super::*;
    use std::sync::atomic::{AtomicU32, Ordering};
    use std::sync::Arc;

    fn policy() -> RetryPolicy {
        RetryPolicy {
            max_attempts: 4,
            base_delay: Duration::from_millis(100),
            max_delay: Duration::from_secs(10),
        }
    }

    #[tokio::test(start_paused = true)]
    async fn retries_transient_then_succeeds() {
        let calls = Arc::new(AtomicU32::new(0));
        let c = calls.clone();
        let out = with_retry(policy(), || {
            let c = c.clone();
            async move {
                let n = c.fetch_add(1, Ordering::SeqCst);
                if n < 2 {
                    Err(EtlError::Network("reset".into()))
                } else {
                    Ok(42)
                }
            }
        })
        .await
        .unwrap();
        assert_eq!(out, 42);
        assert_eq!(calls.load(Ordering::SeqCst), 3);
    }

    #[tokio::test(start_paused = true)]
    async fn gives_up_on_permanent_immediately() {
        let calls = Arc::new(AtomicU32::new(0));
        let c = calls.clone();
        let err = with_retry(policy(), || {
            let c = c.clone();
            async move {
                c.fetch_add(1, Ordering::SeqCst);
                Err::<(), _>(EtlError::Parse("bad".into()))
            }
        })
        .await
        .unwrap_err();
        assert!(matches!(err, EtlError::Parse(_)));
        assert_eq!(calls.load(Ordering::SeqCst), 1);
    }

    #[tokio::test(start_paused = true)]
    async fn honors_retry_after() {
        let calls = Arc::new(AtomicU32::new(0));
        let c = calls.clone();
        let start = tokio::time::Instant::now();
        let out = with_retry(policy(), || {
            let c = c.clone();
            async move {
                let n = c.fetch_add(1, Ordering::SeqCst);
                if n == 0 {
                    Err(EtlError::RateLimited { retry_after_secs: 5 })
                } else {
                    Ok(())
                }
            }
        })
        .await;
        assert!(out.is_ok());
        // The 5s Retry-After beats the 100ms base backoff.
        assert!(start.elapsed() >= Duration::from_secs(5));
    }
}

Tests: retries_transient_then_succeeds, gives_up_on_permanent_immediately, honors_retry_after.

crates/etl-core/src/content.rs — content-addressing & the idempotent sink (Part IV)

content_id derives a stable sha256(kind ‖ 0x00 ‖ canonical) id; Row::content_key uses it as a dedupe key (fallible by design). IdempotentSink wraps any Sink and forwards only rows it has not seen — the precondition for any retry or backfill.

//! Content-addressing and idempotent writes.
//!
//! A stable id derived from a row's *content* means re-running a pipeline
//! writes each row at most once — the precondition for any retry, backfill, or
//! scheduling story. This is where the `sha2` seam stops being decorative.

use std::collections::HashSet;

use async_trait::async_trait;
use sha2::{Digest, Sha256};
use tokio::sync::Mutex;

use crate::error::EtlError;
use crate::pipeline::{Row, Sink};

/// A deterministic id for a value: `sha256(kind ‖ 0x00 ‖ canonical)`, hex.
///
/// The `kind` tag keeps ids from two different entity types from ever
/// colliding even if their canonical strings coincide.
pub fn content_id(kind: &str, canonical: &str) -> String {
    let mut hasher = Sha256::new();
    hasher.update(kind.as_bytes());
    hasher.update([0u8]);
    hasher.update(canonical.as_bytes());
    format!("{:x}", hasher.finalize())
}

impl Row {
    /// The dedupe key for this row — the content id of its canonical JSON.
    ///
    /// Fallible on purpose: a non-serializable field (e.g. a `NaN` float) must
    /// surface as an error, never silently collapse to a shared key that would
    /// make [`IdempotentSink`] discard distinct rows as duplicates.
    pub fn content_key(&self) -> Result<String, EtlError> {
        // serde_json is deterministic for our structs (fields in declaration
        // order), so equal rows hash equal.
        let canonical = serde_json::to_string(self).map_err(|e| EtlError::Parse(e.to_string()))?;
        Ok(content_id("row", &canonical))
    }
}

/// Wraps any `Sink`, forwarding only rows whose content it has not seen before.
///
/// The `seen` set is in-memory (per run). Seeding it from an existing sink file
/// at construction is the cross-run refinement noted in the course.
pub struct IdempotentSink<S: Sink> {
    inner: S,
    seen: Mutex<HashSet<String>>,
}

impl<S: Sink> IdempotentSink<S> {
    pub fn new(inner: S) -> Self {
        Self {
            inner,
            seen: Mutex::new(HashSet::new()),
        }
    }

    /// Pre-load already-written keys so re-runs across processes stay idempotent.
    pub fn with_seen(inner: S, keys: impl IntoIterator<Item = String>) -> Self {
        Self {
            inner,
            seen: Mutex::new(keys.into_iter().collect()),
        }
    }
}

#[async_trait]
impl<S: Sink> Sink for IdempotentSink<S> {
    async fn load(&self, rows: &[Row]) -> Result<usize, EtlError> {
        let mut fresh = Vec::new();
        {
            let mut seen = self.seen.lock().await;
            for row in rows {
                if seen.insert(row.content_key()?) {
                    fresh.push(row.clone());
                }
            }
        }
        if fresh.is_empty() {
            return Ok(0);
        }
        self.inner.load(&fresh).await
    }
}

#[cfg(test)]
mod tests {
    use super::*;
    use crate::domain::Launch;
    use pretty_assertions::assert_eq;
    use std::sync::atomic::{AtomicUsize, Ordering};

    fn launch(id: &str) -> Row {
        Row::Launch(Launch {
            id: id.into(),
            name: "L".into(),
            net: "2026-07-10T12:00:00Z".parse().unwrap(),
            provider: "SpaceX".into(),
            payloads: vec![],
            description: "d".into(),
        })
    }

    /// A counting sink that records how many rows actually reached it.
    struct CountingSink {
        total: AtomicUsize,
    }
    #[async_trait]
    impl Sink for CountingSink {
        async fn load(&self, rows: &[Row]) -> Result<usize, EtlError> {
            self.total.fetch_add(rows.len(), Ordering::SeqCst);
            Ok(rows.len())
        }
    }

    #[test]
    fn same_content_hashes_stable_distinct_content_differs() {
        assert_eq!(content_id("row", "x"), content_id("row", "x"));
        assert_ne!(content_id("row", "x"), content_id("row", "y"));
        assert_ne!(content_id("a", "x"), content_id("b", "x"));
    }

    #[tokio::test]
    async fn rerun_writes_zero_new_rows() {
        let sink = IdempotentSink::new(CountingSink {
            total: AtomicUsize::new(0),
        });
        let rows = vec![launch("a"), launch("b")];
        assert_eq!(sink.load(&rows).await.unwrap(), 2);
        // Same rows again: nothing new reaches the inner sink.
        assert_eq!(sink.load(&rows).await.unwrap(), 0);
    }

    #[test]
    fn content_key_is_stable_and_distinguishes_rows() {
        // Equal rows -> equal keys; a changed field -> a different key. This is
        // the contract IdempotentSink relies on, and it is fallible by design
        // (never unwrap_or_default, which would collapse distinct rows to one).
        let a = launch("a");
        let a2 = launch("a");
        let b = launch("b");
        assert_eq!(a.content_key().unwrap(), a2.content_key().unwrap());
        assert_ne!(a.content_key().unwrap(), b.content_key().unwrap());
    }

    #[tokio::test]
    async fn distinct_rows_all_written() {
        let sink = IdempotentSink::new(CountingSink {
            total: AtomicUsize::new(0),
        });
        let rows = vec![launch("a"), launch("b"), launch("c")];
        assert_eq!(sink.load(&rows).await.unwrap(), 3);
    }
}

Tests: same_content_hashes_stable_distinct_content_differs, rerun_writes_zero_new_rows, content_key_is_stable_and_distinguishes_rows, distinct_rows_all_written.

crates/etl-sources/src/spacedevs.rs — the TheSpaceDevs source (Part IV)

Cursor pagination: extract follows the next cursor to the end, wraps each page fetch in with_retry, and guards against a cyclic next with a visited set. The transform pulls launches out of deeply nested Launch Library 2 JSON.

//! The TheSpaceDevs source (Launch Library 2): cursor pagination over a
//! throttled REST API (Extract), plus the transform of deeply nested launch
//! JSON into `Row::Launch`.

use async_trait::async_trait;
use chrono::{DateTime, Utc};
use etl_core::{EtlError, Launch, RawRecord, Row, Source, Transform};
use serde::Deserialize;

use crate::http::classify_status;
use etl_core::{with_retry, RetryPolicy};

/// Fetches launches, following the `next` cursor to the end.
pub struct SpaceDevsSource {
    pub base_url: String,
    pub policy: RetryPolicy,
}

impl SpaceDevsSource {
    pub fn new(base_url: impl Into<String>) -> Self {
        Self {
            base_url: base_url.into(),
            policy: RetryPolicy::default(),
        }
    }
}

#[async_trait]
impl Source for SpaceDevsSource {
    fn name(&self) -> &str {
        "spacedevs"
    }

    async fn extract(&self) -> Result<Vec<RawRecord>, EtlError> {
        let base = self.base_url.trim_end_matches('/');
        let client = reqwest::Client::new();
        let mut next = Some(format!("{base}/2.2.0/launch/?limit=100&mode=detailed"));
        let mut pages = Vec::new();
        // Guard against a cyclic or self-referential `next` from a buggy server.
        let mut visited = std::collections::HashSet::new();

        while let Some(url) = next {
            if !visited.insert(url.clone()) {
                return Err(EtlError::Parse(format!("pagination cycle at {url}")));
            }
            let body = with_retry(self.policy, || {
                let client = &client;
                let url = url.clone();
                async move {
                    let resp = client
                        .get(&url)
                        .send()
                        .await
                        .map_err(|e| EtlError::Network(e.to_string()))?;
                    // 429 -> RateLimited, 4xx -> permanent, 5xx -> transient.
                    if !resp.status().is_success() {
                        return Err(classify_status(resp.status(), resp.headers()));
                    }
                    resp.text().await.map_err(|e| EtlError::Network(e.to_string()))
                }
            })
            .await?;

            let page: Page =
                serde_json::from_str(&body).map_err(|e| EtlError::Parse(e.to_string()))?;
            next = page.next;
            pages.push(RawRecord {
                source: self.name().to_string(),
                payload: body,
                fetched_at: Utc::now(),
            });
        }
        Ok(pages)
    }
}

#[derive(Deserialize)]
struct Page {
    next: Option<String>,
    #[allow(dead_code)]
    results: Vec<RawLaunch>,
}

#[derive(Deserialize)]
struct RawLaunch {
    id: String,
    name: String,
    net: String,
    launch_service_provider: Option<Provider>,
    mission: Option<Mission>,
}

#[derive(Deserialize)]
struct Provider {
    name: String,
}

#[derive(Deserialize)]
struct Mission {
    name: Option<String>,
    description: Option<String>,
}

/// Parses one Launch Library 2 page into launches.
pub struct SpaceDevsTransform;

impl Transform for SpaceDevsTransform {
    fn transform(&self, raw: &RawRecord) -> Result<Vec<Row>, EtlError> {
        let page: Page =
            serde_json::from_str(&raw.payload).map_err(|e| EtlError::Parse(e.to_string()))?;
        let mut rows = Vec::with_capacity(page.results.len());
        for l in page.results {
            let net = l
                .net
                .parse::<DateTime<Utc>>()
                .map_err(|e| EtlError::Parse(format!("bad net {:?}: {e}", l.net)))?;
            let (payloads, description) = match l.mission {
                Some(m) => (
                    m.name.into_iter().collect::<Vec<String>>(),
                    m.description.unwrap_or_default(),
                ),
                None => (vec![], String::new()),
            };
            rows.push(Row::Launch(Launch {
                id: l.id,
                name: l.name,
                net,
                provider: l.launch_service_provider.map(|p| p.name).unwrap_or_default(),
                payloads,
                description,
            }));
        }
        Ok(rows)
    }
}

#[cfg(test)]
mod tests {
    use super::*;
    use etl_core::Row;
    use pretty_assertions::assert_eq;
    use wiremock::matchers::{method, path};
    use wiremock::{Mock, MockServer, ResponseTemplate};

    fn page_json(next: Option<&str>) -> String {
        let next_field = match next {
            Some(u) => format!("\"{u}\""),
            None => "null".to_string(),
        };
        format!(
            r#"{{"next": {next_field}, "results": [
                {{"id":"ll2-1","name":"Falcon 9 | Starlink","net":"2026-07-10T12:00:00Z",
                  "launch_service_provider":{{"name":"SpaceX"}},
                  "mission":{{"name":"Starlink G-99","description":"A batch of broadband satellites."}}}}
            ]}}"#
        )
    }

    #[tokio::test]
    async fn follows_next_until_null() {
        let server = MockServer::start().await;
        let page2 = format!("{}/page2", server.uri());
        Mock::given(method("GET"))
            .and(path("/2.2.0/launch/"))
            .respond_with(ResponseTemplate::new(200).set_body_string(page_json(Some(&page2))))
            .mount(&server)
            .await;
        Mock::given(method("GET"))
            .and(path("/page2"))
            .respond_with(ResponseTemplate::new(200).set_body_string(page_json(None)))
            .mount(&server)
            .await;

        let src = SpaceDevsSource::new(server.uri());
        let pages = src.extract().await.unwrap();
        assert_eq!(pages.len(), 2);
    }

    #[tokio::test]
    async fn rejects_pagination_cycle() {
        let server = MockServer::start().await;
        // The first page's `next` points back at the first page — a cycle.
        let self_url = format!("{}/2.2.0/launch/", server.uri());
        Mock::given(method("GET"))
            .and(path("/2.2.0/launch/"))
            .respond_with(ResponseTemplate::new(200).set_body_string(page_json(Some(&self_url))))
            .mount(&server)
            .await;
        let src = SpaceDevsSource::new(server.uri());
        let err = src.extract().await.unwrap_err();
        assert!(matches!(err, EtlError::Parse(_)));
    }

    #[tokio::test]
    async fn http_error_is_network_error() {
        let server = MockServer::start().await;
        Mock::given(method("GET"))
            .respond_with(ResponseTemplate::new(500))
            .mount(&server)
            .await;
        let src = SpaceDevsSource::new(server.uri());
        let err = src.extract().await.unwrap_err();
        assert!(matches!(err, EtlError::Network(_)));
    }

    #[test]
    fn parses_nested_launch_json() {
        let raw = RawRecord {
            source: "spacedevs".into(),
            payload: page_json(None),
            fetched_at: Utc::now(),
        };
        let rows = SpaceDevsTransform.transform(&raw).unwrap();
        assert_eq!(rows.len(), 1);
        match &rows[0] {
            Row::Launch(l) => {
                assert_eq!(l.provider, "SpaceX");
                assert_eq!(l.payloads, vec!["Starlink G-99".to_string()]);
                assert!(l.description.contains("broadband"));
            }
            other => panic!("expected Launch, got {other:?}"),
        }
    }
}

Tests: follows_next_until_null, rejects_pagination_cycle, http_error_is_network_error, parses_nested_launch_json.

Part V — The Orchestration Arc

The typed task DAG, the executor that runs it (with retry, ledger recording, freshness gating, and block propagation), and the panoptes-etl binary that wires the concrete sources in — the only crate that names them.

crates/etl-orchestrate/Cargo.toml — the orchestrator manifest (Part V)

Depends on etl-core only — the dependency-inversion constraint made mechanical. cargo tree -p etl-orchestrate proves no concrete source is reachable from here.

[package]
name = "etl-orchestrate"
version = "0.1.0"
edition = "2024"

# Depends on etl-core ONLY. It schedules `dyn Source` and must never know a
# concrete source exists — that constraint is the dependency-inversion lesson.
[dependencies]
etl-core = { path = "../etl-core" }
chrono = { workspace = true }

[dev-dependencies]
pretty_assertions = { workspace = true }
tokio = { workspace = true }
async-trait = { workspace = true }

crates/etl-orchestrate/src/lib.rs — the orchestrator crate root (Part V)

//! `etl-orchestrate` — a typed task DAG and the executor that runs it.
//!
//! Depends on `etl-core` only. The executor schedules `dyn Source` /
//! `dyn Transform` / `dyn Sink` and has no knowledge of any concrete source —
//! that separation is enforced by the crate boundary, not by convention.

pub mod executor;
pub mod graph;

pub use executor::{Executor, Pipeline, RunReport};
pub use graph::{Dag, TaskId};

crates/etl-orchestrate/src/graph.rs — the task DAG (Part V)

A directed acyclic graph of TaskIds with topological_order via Kahn's algorithm over a sorted ready-set, so the order is deterministic and a remaining node signals a cycle.

//! A typed task DAG with deterministic topological execution order.

use std::collections::{BTreeSet, HashMap, HashSet};

use etl_core::EtlError;

/// The identifier of a task in the DAG.
#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
pub struct TaskId(pub String);

impl TaskId {
    pub fn new(s: impl Into<String>) -> Self {
        TaskId(s.into())
    }
}

/// A directed acyclic graph of tasks. Edges record prerequisites: `add_dep(b,
/// a)` means "b depends on a", so a runs before b.
#[derive(Default)]
pub struct Dag {
    nodes: HashSet<TaskId>,
    /// task -> its prerequisites.
    prereqs: HashMap<TaskId, Vec<TaskId>>,
}

impl Dag {
    pub fn new() -> Self {
        Self::default()
    }

    pub fn add_task(&mut self, id: impl Into<String>) -> TaskId {
        let t = TaskId(id.into());
        self.nodes.insert(t.clone());
        self.prereqs.entry(t.clone()).or_default();
        t
    }

    /// Record that `task` depends on `depends_on`.
    pub fn add_dep(&mut self, task: &TaskId, depends_on: &TaskId) {
        self.nodes.insert(task.clone());
        self.nodes.insert(depends_on.clone());
        self.prereqs.entry(depends_on.clone()).or_default();
        self.prereqs
            .entry(task.clone())
            .or_default()
            .push(depends_on.clone());
    }

    /// The direct prerequisites of a task (empty if none / unknown).
    pub fn prerequisites(&self, task: &TaskId) -> &[TaskId] {
        self.prereqs.get(task).map_or(&[], |v| v.as_slice())
    }

    /// Kahn's algorithm with a sorted ready-set, so the order is deterministic.
    /// A remaining node means a cycle.
    pub fn topological_order(&self) -> Result<Vec<TaskId>, EtlError> {
        let mut indegree: HashMap<TaskId, usize> = self
            .nodes
            .iter()
            .map(|n| (n.clone(), self.prereqs.get(n).map_or(0, |p| p.len())))
            .collect();

        let mut dependents: HashMap<TaskId, Vec<TaskId>> = HashMap::new();
        for (task, prereqs) in &self.prereqs {
            for p in prereqs {
                dependents.entry(p.clone()).or_default().push(task.clone());
            }
        }

        let mut ready: BTreeSet<TaskId> = indegree
            .iter()
            .filter(|(_, d)| **d == 0)
            .map(|(t, _)| t.clone())
            .collect();

        let mut order = Vec::with_capacity(self.nodes.len());
        while let Some(next) = ready.iter().next().cloned() {
            ready.remove(&next);
            if let Some(children) = dependents.get(&next) {
                for child in children {
                    let d = indegree.get_mut(child).expect("child in indegree");
                    *d -= 1;
                    if *d == 0 {
                        ready.insert(child.clone());
                    }
                }
            }
            order.push(next);
        }

        if order.len() != self.nodes.len() {
            return Err(EtlError::Parse(format!(
                "cycle detected: ordered {} of {} tasks",
                order.len(),
                self.nodes.len()
            )));
        }
        Ok(order)
    }
}

#[cfg(test)]
mod tests {
    use super::*;
    use pretty_assertions::assert_eq;

    fn ids(v: &[TaskId]) -> Vec<String> {
        v.iter().map(|t| t.0.clone()).collect()
    }

    #[test]
    fn linear_chain_orders_correctly() {
        let mut dag = Dag::new();
        let a = dag.add_task("a");
        let b = dag.add_task("b");
        let c = dag.add_task("c");
        dag.add_dep(&b, &a);
        dag.add_dep(&c, &b);
        assert_eq!(ids(&dag.topological_order().unwrap()), ["a", "b", "c"]);
    }

    #[test]
    fn diamond_orders_deps_before_dependents() {
        let mut dag = Dag::new();
        let a = dag.add_task("a");
        let b = dag.add_task("b");
        let c = dag.add_task("c");
        let d = dag.add_task("d");
        dag.add_dep(&b, &a);
        dag.add_dep(&c, &a);
        dag.add_dep(&d, &b);
        dag.add_dep(&d, &c);
        let order = ids(&dag.topological_order().unwrap());
        // a before everything; d after everything; b,c deterministic (sorted).
        assert_eq!(order, ["a", "b", "c", "d"]);
    }

    #[test]
    fn cycle_is_rejected() {
        let mut dag = Dag::new();
        let a = dag.add_task("a");
        let b = dag.add_task("b");
        dag.add_dep(&a, &b);
        dag.add_dep(&b, &a);
        let err = dag.topological_order().unwrap_err();
        assert!(matches!(err, EtlError::Parse(_)));
    }
}

Tests: linear_chain_orders_correctly, diamond_orders_deps_before_dependents, cycle_is_rejected.

crates/etl-orchestrate/src/executor.rs — the executor (Part V)

Pipeline bundles a boxed dyn Source/dyn Transform/dyn Sink with a retry policy and an optional freshness window. Executor::run walks the DAG in topological order, skips still-fresh sources, records every run in the ledger, and propagates a block downstream when a prerequisite fails. It never names a concrete source.

//! The executor: runs pipelines in topological order, retries transient
//! failures, records every run in the ledger, and skips sources still fresh
//! from a recent successful run.
//!
//! It is built entirely on `etl-core` traits — `dyn Source`, `dyn Transform`,
//! `dyn Sink` — and never names a concrete source.

use std::collections::HashMap;

use chrono::{Duration, Utc};
use etl_core::{
    with_retry, EtlError, Ledger, Outcome, RetryPolicy, RunRecord, Sink, Source, Transform,
};

use crate::graph::{Dag, TaskId};

/// One extract → transform → load chain.
pub struct Pipeline {
    pub source: Box<dyn Source>,
    pub transform: Box<dyn Transform>,
    pub sink: Box<dyn Sink>,
    pub policy: RetryPolicy,
    /// If set, skip the run when the last success is newer than this.
    pub refresh_after: Option<Duration>,
}

impl Pipeline {
    pub fn new(
        source: Box<dyn Source>,
        transform: Box<dyn Transform>,
        sink: Box<dyn Sink>,
    ) -> Self {
        Self {
            source,
            transform,
            sink,
            policy: RetryPolicy::default(),
            refresh_after: None,
        }
    }

    /// Extract (with retry) → transform every raw record → load once.
    pub async fn run_once(&self) -> Result<usize, EtlError> {
        let raws = with_retry(self.policy, || self.source.extract()).await?;
        let mut rows = Vec::new();
        for raw in &raws {
            rows.extend(self.transform.transform(raw)?);
        }
        self.sink.load(&rows).await
    }

    async fn is_fresh(&self, ledger: &Ledger, id: &TaskId) -> Result<bool, EtlError> {
        let Some(refresh) = self.refresh_after else {
            return Ok(false);
        };
        let Some(last) = ledger.last_success(&id.0).await? else {
            return Ok(false);
        };
        Ok(Utc::now() - last.finished_at < refresh)
    }
}

/// The outcome of a whole DAG run.
#[derive(Debug, Default)]
pub struct RunReport {
    pub records: Vec<RunRecord>,
    pub skipped: Vec<TaskId>,
    /// Nodes not run because a prerequisite failed (or was itself blocked).
    pub blocked: Vec<TaskId>,
}

/// Owns the DAG, the pipelines, and the ledger.
pub struct Executor {
    dag: Dag,
    pipelines: HashMap<TaskId, Pipeline>,
    ledger: Ledger,
}

impl Executor {
    pub fn new(ledger: Ledger) -> Self {
        Self {
            dag: Dag::new(),
            pipelines: HashMap::new(),
            ledger,
        }
    }

    pub fn add_pipeline(&mut self, id: &str, pipeline: Pipeline) -> TaskId {
        let t = self.dag.add_task(id);
        self.pipelines.insert(t.clone(), pipeline);
        t
    }

    pub fn add_dep(&mut self, task: &TaskId, depends_on: &TaskId) {
        self.dag.add_dep(task, depends_on);
    }

    /// Run every pipeline in dependency order.
    ///
    /// A node whose prerequisite failed (or was itself blocked) does not run —
    /// the DAG edges gate execution, not just order it. `bad` accumulates both
    /// failed and blocked ids so the block propagates down the graph.
    pub async fn run(&self) -> Result<RunReport, EtlError> {
        let order = self.dag.topological_order()?;
        let mut report = RunReport::default();
        let mut bad: std::collections::HashSet<TaskId> = std::collections::HashSet::new();
        for id in order {
            let Some(pipeline) = self.pipelines.get(&id) else {
                continue;
            };
            if self.dag.prerequisites(&id).iter().any(|p| bad.contains(p)) {
                bad.insert(id.clone());
                report.blocked.push(id);
                continue;
            }
            if pipeline.is_fresh(&self.ledger, &id).await? {
                report.skipped.push(id.clone());
                continue;
            }
            let started_at = Utc::now();
            let result = pipeline.run_once().await;
            let finished_at = Utc::now();
            let record = match result {
                Ok(rows_written) => RunRecord {
                    source: id.0.clone(),
                    started_at,
                    finished_at,
                    outcome: Outcome::Success,
                    watermark: Some(finished_at),
                    rows_written,
                },
                Err(e) => {
                    bad.insert(id.clone());
                    RunRecord {
                        source: id.0.clone(),
                        started_at,
                        finished_at,
                        outcome: Outcome::Failed { reason: e.to_string() },
                        watermark: None,
                        rows_written: 0,
                    }
                }
            };
            self.ledger.append(&record).await?;
            report.records.push(record);
        }
        Ok(report)
    }
}

#[cfg(test)]
mod tests {
    use super::*;
    use async_trait::async_trait;
    use etl_core::{RawRecord, Row};
    use pretty_assertions::assert_eq;
    use std::sync::atomic::{AtomicUsize, Ordering};
    use std::sync::{Arc, Mutex};

    /// Records the order in which sources are extracted, and counts calls.
    struct StubSource {
        name: String,
        log: Arc<Mutex<Vec<String>>>,
        calls: Arc<AtomicUsize>,
    }
    #[async_trait]
    impl Source for StubSource {
        fn name(&self) -> &str {
            &self.name
        }
        async fn extract(&self) -> Result<Vec<RawRecord>, EtlError> {
            self.calls.fetch_add(1, Ordering::SeqCst);
            self.log.lock().unwrap().push(self.name.clone());
            Ok(vec![RawRecord {
                source: self.name.clone(),
                payload: String::new(),
                fetched_at: Utc::now(),
            }])
        }
    }

    struct EmptyTransform;
    impl Transform for EmptyTransform {
        fn transform(&self, _raw: &RawRecord) -> Result<Vec<Row>, EtlError> {
            Ok(vec![])
        }
    }

    struct NullSink;
    #[async_trait]
    impl Sink for NullSink {
        async fn load(&self, rows: &[Row]) -> Result<usize, EtlError> {
            Ok(rows.len())
        }
    }

    fn temp_ledger(tag: &str) -> Ledger {
        let dir = std::env::temp_dir().join(format!("etl-exec-{tag}-{}", std::process::id()));
        std::fs::create_dir_all(&dir).unwrap();
        let path = dir.join("runs.jsonl");
        let _ = std::fs::remove_file(&path);
        Ledger::new(path)
    }

    fn stub_pipeline(name: &str, log: Arc<Mutex<Vec<String>>>, calls: Arc<AtomicUsize>) -> Pipeline {
        Pipeline::new(
            Box::new(StubSource {
                name: name.into(),
                log,
                calls,
            }),
            Box::new(EmptyTransform),
            Box::new(NullSink),
        )
    }

    #[tokio::test]
    async fn runs_pipelines_in_dependency_order() {
        let log = Arc::new(Mutex::new(Vec::new()));
        let mut exec = Executor::new(temp_ledger("order"));
        let a = exec.add_pipeline(
            "a",
            stub_pipeline("a", log.clone(), Arc::new(AtomicUsize::new(0))),
        );
        let b = exec.add_pipeline(
            "b",
            stub_pipeline("b", log.clone(), Arc::new(AtomicUsize::new(0))),
        );
        exec.add_dep(&b, &a);

        let report = exec.run().await.unwrap();
        assert_eq!(report.records.len(), 2);
        assert_eq!(*log.lock().unwrap(), vec!["a".to_string(), "b".to_string()]);
    }

    #[tokio::test]
    async fn records_a_run_per_node() {
        let mut exec = Executor::new(temp_ledger("perrun"));
        exec.add_pipeline(
            "a",
            stub_pipeline("a", Arc::new(Mutex::new(vec![])), Arc::new(AtomicUsize::new(0))),
        );
        exec.add_pipeline(
            "b",
            stub_pipeline("b", Arc::new(Mutex::new(vec![])), Arc::new(AtomicUsize::new(0))),
        );
        let report = exec.run().await.unwrap();
        assert_eq!(report.records.len(), 2);
        assert!(report.records.iter().all(|r| r.outcome == Outcome::Success));
    }

    /// A source that always fails permanently.
    struct FailingSource {
        calls: Arc<AtomicUsize>,
    }
    #[async_trait]
    impl Source for FailingSource {
        fn name(&self) -> &str {
            "failing"
        }
        async fn extract(&self) -> Result<Vec<RawRecord>, EtlError> {
            self.calls.fetch_add(1, Ordering::SeqCst);
            Err(EtlError::Parse("boom".into()))
        }
    }

    #[tokio::test]
    async fn failed_upstream_blocks_dependent() {
        let mut exec = Executor::new(temp_ledger("block"));
        let a = exec.add_pipeline(
            "a",
            Pipeline::new(
                Box::new(FailingSource { calls: Arc::new(AtomicUsize::new(0)) }),
                Box::new(EmptyTransform),
                Box::new(NullSink),
            ),
        );
        let downstream_calls = Arc::new(AtomicUsize::new(0));
        let b = exec.add_pipeline(
            "b",
            stub_pipeline("b", Arc::new(Mutex::new(vec![])), downstream_calls.clone()),
        );
        exec.add_dep(&b, &a);

        let report = exec.run().await.unwrap();
        // a ran and failed; b was blocked and never extracted.
        assert_eq!(report.records.len(), 1);
        assert!(matches!(report.records[0].outcome, Outcome::Failed { .. }));
        assert_eq!(report.blocked, vec![b]);
        assert_eq!(downstream_calls.load(Ordering::SeqCst), 0);
    }

    #[tokio::test]
    async fn incremental_skips_up_to_date_source() {
        let ledger = temp_ledger("incr");
        // Seed a very recent successful run for "celestrak".
        ledger
            .append(&RunRecord {
                source: "celestrak".into(),
                started_at: Utc::now(),
                finished_at: Utc::now(),
                outcome: Outcome::Success,
                watermark: Some(Utc::now()),
                rows_written: 5,
            })
            .await
            .unwrap();

        let calls = Arc::new(AtomicUsize::new(0));
        let mut pipeline =
            stub_pipeline("celestrak", Arc::new(Mutex::new(vec![])), calls.clone());
        pipeline.refresh_after = Some(Duration::hours(6));

        let mut exec = Executor::new(ledger);
        exec.add_pipeline("celestrak", pipeline);

        let report = exec.run().await.unwrap();
        assert_eq!(report.skipped.len(), 1);
        assert!(report.records.is_empty());
        // The source was never extracted.
        assert_eq!(calls.load(Ordering::SeqCst), 0);
    }
}

Tests: runs_pipelines_in_dependency_order, records_a_run_per_node, failed_upstream_blocks_dependent, incremental_skips_up_to_date_source.

crates/panoptes-etl/Cargo.toml — the binary manifest (Part V)

The binary — the only crate that names concrete sources and wires them into the DAG. It depends on all three library crates.

[package]
name = "panoptes-etl"
version = "0.1.0"
edition = "2024"

# The binary — the ONLY crate that names concrete sources and wires them into
# the DAG. Depends on all three library crates.
[dependencies]
etl-core = { path = "../etl-core" }
etl-sources = { path = "../etl-sources" }
etl-orchestrate = { path = "../etl-orchestrate" }
serde_json = { workspace = true }
clap = { workspace = true }
anyhow = { workspace = true }
tokio = { workspace = true }
chrono = { workspace = true }

[dev-dependencies]
pretty_assertions = { workspace = true }

crates/panoptes-etl/src/cli.rs — the CLI surface (Part V)

The clap Cli/Command definitions and the ledger-status report — the testable parts of the binary, kept out of main.rs so they can be unit-tested.

//! CLI surface and the ledger-status report — the testable parts of the binary.

use std::path::PathBuf;

use clap::{Parser, Subcommand};
use etl_core::{EtlError, Ledger};

/// The Panoptes space-data ETL.
#[derive(Parser, Debug)]
#[command(name = "panoptes-etl", version, about)]
pub struct Cli {
    #[command(subcommand)]
    pub command: Command,

    /// Directory for normalized JSONL output.
    #[arg(long, default_value = "data")]
    pub out_dir: PathBuf,

    /// Path to the run ledger.
    #[arg(long, default_value = "data/runs.jsonl")]
    pub ledger: PathBuf,
}

#[derive(Subcommand, Debug, PartialEq, Eq)]
pub enum Command {
    /// Run every pipeline once, skipping sources still fresh.
    Run,
    /// Re-run every pipeline, ignoring freshness (full backfill).
    Backfill,
    /// Embed the prose fields of already-loaded launches into a vector index.
    Embed,
    /// Show the last successful watermark per source.
    Status,
}

/// The sources this binary knows how to run, in DAG order.
pub const SOURCES: &[&str] = &["celestrak", "spacetrack", "spacedevs"];

/// Render a status report: the last successful watermark per known source.
pub async fn status_report(ledger: &Ledger, sources: &[&str]) -> Result<String, EtlError> {
    let mut lines = Vec::new();
    for source in sources {
        let line = match ledger.last_success(source).await? {
            Some(rec) => {
                let wm = rec
                    .watermark
                    .map(|w| w.to_rfc3339())
                    .unwrap_or_else(|| "-".into());
                format!("{source:<12} last success {} ({} rows)", wm, rec.rows_written)
            }
            None => format!("{source:<12} never run"),
        };
        lines.push(line);
    }
    Ok(lines.join("\n"))
}

#[cfg(test)]
mod tests {
    use super::*;
    use etl_core::{Outcome, RunRecord};
    use chrono::Utc;

    #[test]
    fn cli_parses_run_subcommand() {
        let cli = Cli::try_parse_from(["panoptes-etl", "run"]).unwrap();
        assert_eq!(cli.command, Command::Run);
    }

    #[test]
    fn cli_parses_flags_before_subcommand() {
        let cli =
            Cli::try_parse_from(["panoptes-etl", "--out-dir", "/tmp/x", "status"]).unwrap();
        assert_eq!(cli.command, Command::Status);
        assert_eq!(cli.out_dir, PathBuf::from("/tmp/x"));
    }

    #[tokio::test]
    async fn status_reports_last_watermark() {
        let dir = std::env::temp_dir().join(format!("etl-cli-{}", std::process::id()));
        std::fs::create_dir_all(&dir).unwrap();
        let path = dir.join("runs.jsonl");
        let _ = std::fs::remove_file(&path);
        let ledger = Ledger::new(&path);
        ledger
            .append(&RunRecord {
                source: "celestrak".into(),
                started_at: Utc::now(),
                finished_at: Utc::now(),
                outcome: Outcome::Success,
                watermark: Some("2026-07-19T00:00:00Z".parse().unwrap()),
                rows_written: 42,
            })
            .await
            .unwrap();

        let report = status_report(&ledger, SOURCES).await.unwrap();
        assert!(report.contains("celestrak"));
        assert!(report.contains("2026-07-19"));
        assert!(report.contains("42 rows"));
        assert!(report.contains("spacedevs"));
        assert!(report.contains("never run"));
    }
}

Tests: cli_parses_run_subcommand, cli_parses_flags_before_subcommand, status_reports_last_watermark.

crates/panoptes-etl/src/main.rs — the binary entry point (Part V)

Wires the concrete sources into the DAG and dispatches the subcommands. This is the only file in the workspace that names CelestrakSource, SpaceTrackSource, SpaceDevsSource, and VoyageEmbedder. The commented capstone block shows where your NASA source goes — nothing structural changes to admit a fourth source.

//! The `panoptes-etl` binary: wires the concrete sources into a DAG and runs
//! it. This is the only crate in the workspace that names a concrete source.

mod cli;

use std::path::Path;

use anyhow::Result;
use chrono::Duration;
use clap::Parser;
use std::path::PathBuf;

use etl_core::{EmbedSink, EmbeddedRow, IdempotentSink, JsonlSink, Ledger, Row, Sink};
use etl_orchestrate::{Executor, Pipeline};
use etl_sources::{
    CelestrakSource, CelestrakTransform, Credentials, SpaceDevsSource, SpaceDevsTransform,
    SpaceTrackSource, SpaceTrackTransform, TokenBucket, VoyageEmbedder,
};

use cli::{status_report, Cli, Command, SOURCES};

const CELESTRAK_BASE: &str = "https://celestrak.org";
const SPACETRACK_BASE: &str = "https://www.space-track.org";
const SPACEDEVS_BASE: &str = "https://ll.thespacedevs.com";

#[tokio::main]
async fn main() -> Result<()> {
    let cli = Cli::parse();
    tokio::fs::create_dir_all(&cli.out_dir).await?;
    let ledger = Ledger::new(&cli.ledger);

    match cli.command {
        Command::Status => {
            println!("{}", status_report(&ledger, SOURCES).await?);
        }
        Command::Run | Command::Backfill => {
            let incremental = cli.command == Command::Run;
            let executor = build_executor(&cli.out_dir, ledger, incremental).await?;
            let report = executor.run().await?;
            for r in &report.records {
                println!("{:<12} {:?} ({} rows)", r.source, r.outcome, r.rows_written);
            }
            for s in &report.skipped {
                println!("{:<12} skipped (fresh)", s.0);
            }
            for b in &report.blocked {
                println!("{:<12} blocked (upstream failed)", b.0);
            }
        }
        Command::Embed => {
            let n = embed_launches(&cli.out_dir).await?;
            println!("embedded {n} new prose fields");
        }
    }
    Ok(())
}

/// Read the rows already written to a JSONL file (empty if it doesn't exist).
async fn read_rows(path: &Path) -> Result<Vec<Row>> {
    match tokio::fs::read_to_string(path).await {
        Ok(text) => Ok(text
            .lines()
            .filter(|l| !l.trim().is_empty())
            .map(serde_json::from_str)
            .collect::<Result<_, _>>()?),
        Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(vec![]),
        Err(e) => Err(e.into()),
    }
}

/// A JSONL sink wrapped for idempotency, seeded with the content keys already
/// on disk so a re-run (or `backfill`) never writes a row twice.
async fn idempotent_sink(path: PathBuf) -> Result<IdempotentSink<JsonlSink>> {
    let mut keys = Vec::new();
    for row in read_rows(&path).await? {
        keys.push(row.content_key()?);
    }
    Ok(IdempotentSink::with_seen(JsonlSink::new(path), keys))
}

/// Embed the prose fields of already-loaded launches into a vector index.
/// The embed step consumes the sources' output — in a fuller build it would be
/// a DAG node depending on `spacedevs`. Seeded from the existing index so
/// already-embedded prose is never paid for twice.
async fn embed_launches(out_dir: &Path) -> Result<usize> {
    let rows = read_rows(&out_dir.join("launches.jsonl")).await?;
    if rows.is_empty() {
        anyhow::bail!("no launches to embed — run `panoptes-etl run` first");
    }

    let index_path = out_dir.join("embeddings.jsonl");
    let seen_ids = read_embedded_ids(&index_path).await?;

    let embedder = VoyageEmbedder::from_env("https://api.voyageai.com")?;
    let sink = EmbedSink::with_seen(embedder, index_path, seen_ids);
    Ok(sink.load(&rows).await?)
}

/// The ids already present in an embeddings JSONL file (empty if absent).
async fn read_embedded_ids(path: &Path) -> Result<Vec<String>> {
    match tokio::fs::read_to_string(path).await {
        Ok(text) => Ok(text
            .lines()
            .filter(|l| !l.trim().is_empty())
            .map(|l| serde_json::from_str::<EmbeddedRow>(l).map(|r| r.id))
            .collect::<Result<_, _>>()?),
        Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(vec![]),
        Err(e) => Err(e.into()),
    }
}

/// Assemble the DAG. `incremental` sets a freshness window so `run` skips
/// recently-succeeded sources; `backfill` passes `false` to force every source.
async fn build_executor(out_dir: &Path, ledger: Ledger, incremental: bool) -> Result<Executor> {
    let refresh = incremental.then(|| Duration::hours(6));
    let mut executor = Executor::new(ledger);

    let mut celestrak = Pipeline::new(
        Box::new(CelestrakSource::new(CELESTRAK_BASE, "geo")),
        Box::new(CelestrakTransform),
        Box::new(idempotent_sink(out_dir.join("space_objects.jsonl")).await?),
    );
    celestrak.refresh_after = refresh;
    executor.add_pipeline("celestrak", celestrak);

    // Space-Track needs credentials; include it only when they are present.
    match Credentials::from_env() {
        Ok(creds) => {
            let mut spacetrack = Pipeline::new(
                Box::new(SpaceTrackSource::new(
                    SPACETRACK_BASE,
                    creds,
                    TokenBucket::new(30, 0.5),
                )),
                Box::new(SpaceTrackTransform),
                Box::new(idempotent_sink(out_dir.join("conjunctions.jsonl")).await?),
            );
            spacetrack.refresh_after = refresh;
            executor.add_pipeline("spacetrack", spacetrack);
        }
        Err(_) => eprintln!("note: SPACETRACK_USER/PASS not set — skipping spacetrack"),
    }

    let mut spacedevs = Pipeline::new(
        Box::new(SpaceDevsSource::new(SPACEDEVS_BASE)),
        Box::new(SpaceDevsTransform),
        Box::new(idempotent_sink(out_dir.join("launches.jsonl")).await?),
    );
    spacedevs.refresh_after = refresh;
    executor.add_pipeline("spacedevs", spacedevs);

    // CAPSTONE (you drive, unassisted): add your NASA pipeline here.
    // A `NasaSource` reads its key from `NASA_API_KEY` and passes it as a query
    // parameter; wire it with a transform and a `JsonlSink` exactly like the
    // three above. The Source trait is all the orchestrator needs — nothing in
    // this file's structure has to change to admit a fourth source.

    Ok(executor)
}

Tests: none in this file — the binary's testable surface lives in cli.rs.

Capstone (Part V, you drive). The NASA source has no reference file: the commented block in build_executor above is the whole answer key for it. You write NasaSource + its transform in etl-sources, then add one Pipeline here — no structural change, because Source is all the orchestrator needs.

Part VI — The Retrieval Arc (RAG)

The embedder boundary, a concrete Voyage-backed embedder, the embed load target, and the brute-force cosine retriever.

crates/etl-core/src/embed.rs — the Embedder trait (Part VI)

The embedding boundary — one text-to-vectors method, provider-agnostic and mockable. Mirrors the first course's ModelClient.

//! The embedding boundary — a trait mirroring the first course's `ModelClient`.
//!
//! Embeddings are the definitive expensive, idempotent, content-addressed
//! workload: the same text must never be embedded twice. The trait keeps the
//! provider swappable and mockable.

use async_trait::async_trait;

use crate::error::EtlError;

/// Turns text into vectors. One vector per input, same order.
#[async_trait]
pub trait Embedder: Send + Sync {
    async fn embed(&self, texts: &[String]) -> Result<Vec<Vec<f32>>, EtlError>;
}

Tests: none — the trait is exercised by vector.rs (a counting stub) and voyage.rs (the concrete impl).

crates/etl-core/src/vector.rs — the embed sink & cosine index (Part VI)

cosine similarity, the EmbeddedRow record, a brute-force VectorIndex with top_k, Row::prose (only text fields are ever embedded), and EmbedSink — a load target that embeds prose and appends EmbeddedRow JSONL, never embedding the same text twice.

//! Brute-force semantic retrieval: an embed load target and a cosine index.
//!
//! The corpus is thousands of prose fields, not millions, so linear cosine
//! search is the correct tool — no vector database. Only *prose* is embedded;
//! the numeric entities are for structured (SQL/filter) retrieval instead.

use std::collections::HashSet;
use std::path::PathBuf;

use async_trait::async_trait;
use serde::{Deserialize, Serialize};
use tokio::fs::OpenOptions;
use tokio::io::AsyncWriteExt;
use tokio::sync::Mutex;

use crate::content::content_id;
use crate::embed::Embedder;
use crate::error::EtlError;
use crate::pipeline::{Row, Sink};

/// Cosine similarity of two vectors; 0.0 if either is the zero vector.
pub fn cosine(a: &[f32], b: &[f32]) -> f32 {
    let dot: f32 = a.iter().zip(b).map(|(x, y)| x * y).sum();
    let na = a.iter().map(|x| x * x).sum::<f32>().sqrt();
    let nb = b.iter().map(|x| x * x).sum::<f32>().sqrt();
    if na == 0.0 || nb == 0.0 {
        0.0
    } else {
        dot / (na * nb)
    }
}

/// One embedded prose field: a content-addressed id, its text, and its vector.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct EmbeddedRow {
    pub id: String,
    pub text: String,
    pub vector: Vec<f32>,
}

/// A brute-force cosine index over embedded rows.
#[derive(Default)]
pub struct VectorIndex {
    pub rows: Vec<EmbeddedRow>,
}

impl VectorIndex {
    pub fn new(rows: Vec<EmbeddedRow>) -> Self {
        Self { rows }
    }

    pub fn from_jsonl(text: &str) -> Result<Self, EtlError> {
        let mut rows = Vec::new();
        for line in text.lines().filter(|l| !l.trim().is_empty()) {
            rows.push(serde_json::from_str(line).map_err(|e| EtlError::Parse(e.to_string()))?);
        }
        Ok(Self { rows })
    }

    /// The `k` most similar rows to `query`, highest score first.
    pub fn top_k(&self, query: &[f32], k: usize) -> Vec<(String, f32)> {
        let mut scored: Vec<(String, f32)> = self
            .rows
            .iter()
            .map(|r| (r.id.clone(), cosine(query, &r.vector)))
            .collect();
        scored.sort_by(|a, b| b.1.total_cmp(&a.1));
        scored.truncate(k);
        scored
    }
}

impl Row {
    /// The prose worth embedding, if any. Only text fields — never the numbers.
    pub fn prose(&self) -> Option<String> {
        match self {
            Row::Launch(l) if !l.description.trim().is_empty() => Some(l.description.clone()),
            _ => None,
        }
    }
}

/// A load target that embeds prose fields and appends `EmbeddedRow` JSONL,
/// never embedding the same text twice.
pub struct EmbedSink<E: Embedder> {
    embedder: E,
    path: PathBuf,
    seen: Mutex<HashSet<String>>,
}

impl<E: Embedder> EmbedSink<E> {
    pub fn new(embedder: E, path: impl Into<PathBuf>) -> Self {
        Self {
            embedder,
            path: path.into(),
            seen: Mutex::new(HashSet::new()),
        }
    }

    /// Pre-load already-embedded ids so re-runs across processes never re-embed
    /// the same prose (embedding is expensive — this is the whole point).
    pub fn with_seen(embedder: E, path: impl Into<PathBuf>, ids: impl IntoIterator<Item = String>) -> Self {
        Self {
            embedder,
            path: path.into(),
            seen: Mutex::new(ids.into_iter().collect()),
        }
    }
}

#[async_trait]
impl<E: Embedder> Sink for EmbedSink<E> {
    async fn load(&self, rows: &[Row]) -> Result<usize, EtlError> {
        // Collect the fresh prose to embed, keyed by content id.
        let mut fresh: Vec<(String, String)> = Vec::new();
        {
            let mut seen = self.seen.lock().await;
            for row in rows {
                if let Some(text) = row.prose() {
                    let id = content_id("embed", &text);
                    if seen.insert(id.clone()) {
                        fresh.push((id, text));
                    }
                }
            }
        }
        if fresh.is_empty() {
            return Ok(0);
        }

        let texts: Vec<String> = fresh.iter().map(|(_, t)| t.clone()).collect();
        let vectors = self.embedder.embed(&texts).await?;
        if vectors.len() != texts.len() {
            return Err(EtlError::Parse(format!(
                "embedder returned {} vectors for {} texts",
                vectors.len(),
                texts.len()
            )));
        }

        let mut file = OpenOptions::new()
            .create(true)
            .append(true)
            .open(&self.path)
            .await?;
        let mut buf = String::new();
        for ((id, text), vector) in fresh.into_iter().zip(vectors) {
            let row = EmbeddedRow { id, text, vector };
            buf.push_str(&serde_json::to_string(&row).map_err(|e| EtlError::Parse(e.to_string()))?);
            buf.push('\n');
        }
        let written = buf.lines().count();
        file.write_all(buf.as_bytes()).await?;
        Ok(written)
    }
}

#[cfg(test)]
mod tests {
    use super::*;
    use crate::domain::{Conjunction, Launch};
    use crate::ids::NoradId;
    use std::sync::atomic::{AtomicUsize, Ordering};
    use std::sync::Arc;

    #[test]
    fn cosine_of_identical_is_one_orthogonal_is_zero() {
        assert!((cosine(&[1.0, 2.0, 3.0], &[1.0, 2.0, 3.0]) - 1.0).abs() < 1e-6);
        assert!(cosine(&[1.0, 0.0], &[0.0, 1.0]).abs() < 1e-6);
    }

    #[test]
    fn top_k_ranks_by_similarity() {
        let index = VectorIndex::new(vec![
            EmbeddedRow { id: "near".into(), text: "".into(), vector: vec![1.0, 0.1] },
            EmbeddedRow { id: "far".into(), text: "".into(), vector: vec![-1.0, 0.0] },
            EmbeddedRow { id: "mid".into(), text: "".into(), vector: vec![0.5, 0.5] },
        ]);
        let hits = index.top_k(&[1.0, 0.0], 2);
        assert_eq!(hits.len(), 2);
        assert_eq!(hits[0].0, "near");
        assert!(hits[0].1 >= hits[1].1);
    }

    /// Counts how many texts were embedded and returns fixed-width vectors.
    struct CountingEmbedder {
        calls: Arc<AtomicUsize>,
    }
    #[async_trait]
    impl Embedder for CountingEmbedder {
        async fn embed(&self, texts: &[String]) -> Result<Vec<Vec<f32>>, EtlError> {
            self.calls.fetch_add(texts.len(), Ordering::SeqCst);
            Ok(texts.iter().map(|t| vec![t.len() as f32, 1.0]).collect())
        }
    }

    fn launch(desc: &str) -> Row {
        Row::Launch(Launch {
            id: "l".into(),
            name: "L".into(),
            net: "2026-07-10T12:00:00Z".parse().unwrap(),
            provider: "SpaceX".into(),
            payloads: vec![],
            description: desc.into(),
        })
    }

    fn temp_path(tag: &str) -> PathBuf {
        let dir = std::env::temp_dir().join(format!("etl-embed-{tag}-{}", std::process::id()));
        std::fs::create_dir_all(&dir).unwrap();
        let path = dir.join("vectors.jsonl");
        let _ = std::fs::remove_file(&path);
        path
    }

    #[tokio::test]
    async fn embed_sink_only_touches_prose() {
        let calls = Arc::new(AtomicUsize::new(0));
        let sink = EmbedSink::new(
            CountingEmbedder { calls: calls.clone() },
            temp_path("prose"),
        );
        let rows = vec![
            launch("A batch of broadband satellites."),
            // A conjunction has no prose — it must not be embedded.
            Row::Conjunction(Conjunction {
                id: "c".into(),
                tca: "2026-07-20T12:00:00Z".parse().unwrap(),
                miss_distance_km: 0.5,
                collision_probability: 0.001,
                primary: NoradId(1),
                secondary: NoradId(2),
            }),
        ];
        let n = sink.load(&rows).await.unwrap();
        assert_eq!(n, 1);
        assert_eq!(calls.load(Ordering::SeqCst), 1);
    }

    #[tokio::test]
    async fn cached_text_not_re_embedded() {
        let calls = Arc::new(AtomicUsize::new(0));
        let sink = EmbedSink::new(
            CountingEmbedder { calls: calls.clone() },
            temp_path("cache"),
        );
        let rows = vec![launch("same text"), launch("same text")];
        let n = sink.load(&rows).await.unwrap();
        assert_eq!(n, 1); // identical prose embedded once
        assert_eq!(calls.load(Ordering::SeqCst), 1);
        // A second load of the same text embeds nothing new.
        assert_eq!(sink.load(&rows).await.unwrap(), 0);
        assert_eq!(calls.load(Ordering::SeqCst), 1);
    }
}

Tests: cosine_of_identical_is_one_orthogonal_is_zero, top_k_ranks_by_similarity, embed_sink_only_touches_prose, cached_text_not_re_embedded.

crates/etl-sources/src/voyage.rs — the Voyage embedder (Part VI)

A concrete Embedder backed by the Voyage AI embeddings API — injectable base URL, bearer key, request/response pair — structurally identical to the first course's ModelClient, and mocked with wiremock.

//! A concrete [`Embedder`] backed by the Voyage AI embeddings API.
//!
//! Structurally identical to the first course's Anthropic `ModelClient`: an
//! injectable base URL, a bearer key, a request/response pair — and it is
//! mocked with `wiremock` in tests, never hitting the live API.

use async_trait::async_trait;
use etl_core::{Embedder, EtlError};
use serde::{Deserialize, Serialize};

pub struct VoyageEmbedder {
    pub base_url: String,
    pub api_key: String,
    pub model: String,
}

impl VoyageEmbedder {
    pub fn new(base_url: impl Into<String>, api_key: impl Into<String>) -> Self {
        Self {
            base_url: base_url.into(),
            api_key: api_key.into(),
            model: "voyage-3".to_string(),
        }
    }

    /// Read the key from `VOYAGE_API_KEY`; missing ⇒ auth error.
    pub fn from_env(base_url: impl Into<String>) -> Result<Self, EtlError> {
        let key = std::env::var("VOYAGE_API_KEY")
            .map_err(|_| EtlError::Auth("VOYAGE_API_KEY not set".into()))?;
        Ok(Self::new(base_url, key))
    }
}

#[derive(Serialize)]
struct EmbedRequest<'a> {
    input: &'a [String],
    model: &'a str,
}

#[derive(Deserialize)]
struct EmbedResponse {
    data: Vec<EmbedData>,
}

#[derive(Deserialize)]
struct EmbedData {
    embedding: Vec<f32>,
}

#[async_trait]
impl Embedder for VoyageEmbedder {
    async fn embed(&self, texts: &[String]) -> Result<Vec<Vec<f32>>, EtlError> {
        if texts.is_empty() {
            return Ok(vec![]);
        }
        let resp = reqwest::Client::new()
            .post(format!("{}/v1/embeddings", self.base_url.trim_end_matches('/')))
            .bearer_auth(&self.api_key)
            .json(&EmbedRequest {
                input: texts,
                model: &self.model,
            })
            .send()
            .await
            .map_err(|e| EtlError::Network(e.to_string()))?;
        if !resp.status().is_success() {
            return Err(crate::http::classify_status(resp.status(), resp.headers()));
        }
        let parsed: EmbedResponse =
            resp.json().await.map_err(|e| EtlError::Parse(e.to_string()))?;
        Ok(parsed.data.into_iter().map(|d| d.embedding).collect())
    }
}

#[cfg(test)]
mod tests {
    use super::*;
    use pretty_assertions::assert_eq;
    use wiremock::matchers::{header, method, path};
    use wiremock::{Mock, MockServer, ResponseTemplate};

    #[tokio::test]
    async fn embeds_batch_of_texts_with_bearer_auth() {
        let server = MockServer::start().await;
        Mock::given(method("POST"))
            .and(path("/v1/embeddings"))
            .and(header("authorization", "Bearer test-key"))
            .respond_with(ResponseTemplate::new(200).set_body_string(
                r#"{"data":[{"embedding":[0.1,0.2]},{"embedding":[0.3,0.4]}]}"#,
            ))
            .expect(1)
            .mount(&server)
            .await;

        let embedder = VoyageEmbedder::new(server.uri(), "test-key");
        let vectors = embedder
            .embed(&["hello".to_string(), "world".to_string()])
            .await
            .unwrap();
        assert_eq!(vectors, vec![vec![0.1, 0.2], vec![0.3, 0.4]]);
    }

    #[tokio::test]
    async fn empty_input_makes_no_request() {
        // No mock mounted: if embed() called out, it would error.
        let embedder = VoyageEmbedder::new("http://127.0.0.1:1", "k");
        assert_eq!(embedder.embed(&[]).await.unwrap(), Vec::<Vec<f32>>::new());
    }
}

Tests: embeds_batch_of_texts_with_bearer_auth, empty_input_makes_no_request.

Verifying the whole thing

Run these from the workspace root. All three must pass exactly as shown.

1. The full test suite — 65 tests, all passing:

$ cargo test

The per-crate breakdown:

Every live API (CelesTrak, Space-Track, TheSpaceDevs, Voyage) is mocked with wiremock; no test touches the network. The time-based tests (limiter, retry) run under paused tokio time, so they are deterministic.

2. Clippy, clean across all targets:

$ cargo clippy --all-targets

Produces no warnings.

3. The dependency-inversion proof — etl-orchestrate depends on etl-core only:

$ cargo tree -p etl-orchestrate

Among the workspace crates, the only one that appears is etl-core (alongside chrono and their external transitive deps). etl-sources and panoptes-etl are absent — the orchestrator schedules dyn Source and cannot name a concrete source. Only the panoptes-etl binary names them. That is the dependency arrow pointing toward etl-core, enforced by the crate graph rather than by discipline.

If all three commands come back the way this page describes them, your build matches the reference exactly.

Appendix: Workspace Scaffold

The finished workspace, annotated with the chapter that creates each file. The dependency arrow always points toward etl-core; only the binary names concrete sources.

The tree

panoptes_etl/
├── Cargo.toml                         # workspace manifest        (Part II)
├── crates/
│   ├── etl-core/                      # the seams — depends on nothing in-workspace
│   │   ├── Cargo.toml
│   │   └── src/
│   │       ├── lib.rs                 # module wiring + re-exports
│   │       ├── error.rs               # EtlError taxonomy         (Part I)
│   │       ├── ids.rs                 # NoradId newtype           (Part I)
│   │       ├── domain.rs              # SpaceObject/OrbitalElements/Conjunction/Launch (Part I–III)
│   │       ├── pipeline.rs            # Source/Transform/Sink, RawRecord, Row (Part II)
│   │       ├── sink_jsonl.rs          # JsonlSink — the file contract (Part II)
│   │       ├── ledger.rs              # RunRecord/Outcome/Ledger  (Part III)
│   │       ├── retry.rs               # with_retry, RetryPolicy   (Part IV)
│   │       ├── content.rs             # content_id, IdempotentSink (Part IV)
│   │       ├── embed.rs               # Embedder trait            (Part VI)
│   │       └── vector.rs              # cosine, VectorIndex, EmbedSink (Part VI)
│   ├── etl-sources/                   # concrete sources — depends on etl-core
│   │   ├── Cargo.toml
│   │   └── src/
│   │       ├── lib.rs
│   │       ├── tle.rs                 # hand-rolled TLE parser    (Part II)
│   │       ├── celestrak.rs           # CelestrakSource/Transform (Part II)
│   │       ├── http.rs                # classify_status taxonomy funnel (Part II–III)
│   │       ├── limiter.rs             # TokenBucket               (Part III)
│   │       ├── spacetrack.rs          # SpaceTrackSource/Transform (Part III)
│   │       ├── spacedevs.rs           # SpaceDevsSource/Transform (Part IV)
│   │       └── voyage.rs              # VoyageEmbedder            (Part VI)
│   ├── etl-orchestrate/               # the DAG executor — depends on etl-core ONLY
│   │   ├── Cargo.toml
│   │   └── src/
│   │       ├── lib.rs
│   │       ├── graph.rs               # Dag, TaskId, topological_order (Part V)
│   │       └── executor.rs            # Pipeline, Executor, RunReport (Part V)
│   └── panoptes-etl/                  # the binary — the only crate naming concrete sources
│       ├── Cargo.toml
│       └── src/
│           ├── main.rs                # wiring + build_executor   (Part V)
│           └── cli.rs                 # clap surface + status     (Part V)
└── data/                             # gitignored ETL output (JSONL) + runs.jsonl ledger

The workspace manifest

[workspace]
resolver = "2"
members = ["crates/etl-core", "crates/etl-sources", "crates/etl-orchestrate", "crates/panoptes-etl"]

[workspace.dependencies]
serde = { version = "1", features = ["derive"] }  # derive on the domain model
serde_json = "1"                                   # JSONL wire format
chrono = { version = "0.4", features = ["serde"] } # epochs, TCA, watermarks
thiserror = "2"                                     # the EtlError taxonomy (libs)
anyhow = "1"                                        # the binary's error type
async-trait = "0.1"                                 # dyn-compatible async traits
tokio = { version = "1", features = ["full"] }      # runtime, fs, time
reqwest = { version = "0.12", features = ["json"] } # HTTP client
clap = { version = "4", features = ["derive"] }     # CLI
sha2 = "0.10"                                        # content-addressing
itertools = "0.13"                                   # iterator helpers
wiremock = "0.6"                                      # mock every live API (dev)
pretty_assertions = "1"                               # readable test diffs (dev)

Per-crate dependency matrix

The empty etl-orchestrate → reqwest / etl-sources cells are the point: the executor cannot reach a concrete source. cargo tree -p etl-orchestrate shows only etl-core (+ chrono/serde). That is the dependency-inversion seam, enforced by the build, not by convention.

Expected test progression

Cumulative workspace test count after each part (run cargo test):

Final workspace: 65 tests — etl-core 29, etl-sources 26, etl-orchestrate 7, panoptes-etl 3 — clippy clean.

Appendix: Reference Books

Where to go deeper on the ideas this course leans on. None is required reading — the course is self-contained — but each is the canonical source for a mechanism we used.

Rust language & libraries

• The Rust Programming Language ("the book") — ownership, traits, error handling. The baseline this sequel assumes.
• Rust for Rustaceans (Jon Gjengset) — trait objects and object safety (why dyn Source needs async-trait), and API design for library crates like etl-core.
• Rust Atomics and Locks (Mara Bos) — the concurrency model behind the TokenBucket's Mutex and the shared-state reasoning in the executor.
• Asynchronous Programming in Rust (the async book) — futures, await, and the runtime that tokio provides.
• The serde and tokio docs — the two libraries most load-bearing here; the serde data model explains the Row tagged-enum wire format.

Domain: space situational awareness

• CelesTrak documentation (Dr. T.S. Kelso) — the authoritative reference for the TLE format, the GP element API, and orbit classification.
• Space-Track.org API docs — the CDM (conjunction data message) schema and the query language, plus the rate-limit etiquette the TokenBucket respects.
• The Launch Library 2 API (TheSpaceDevs) — the paginated launch schema.
• Revisiting Spacetrack Report #3 (Vallado et al.) — the definitive treatment of TLE mean elements and SGP4, if you take the propagation graduation step.

Data engineering & orchestration

• The Dagster and Apache Airflow docs — the systems this ETL's four seams are designed to grow toward. Read them to see what "add a scheduler" actually entails once the ledger and DAG are in place.