Rust workspace · finished architecture
The space-data ETL that feeds the Panoptes harness — the complete four-crate project the sequel course builds. Four sources normalize into one Row enum written as JSONL; a DAG executor schedules the pipelines; a retrieval arc embeds the prose. Every arc installs one seam an orchestrator needs.
Everything hangs off one trio of traits. A Source fetches raw bytes — extract returns a RawRecord (the payload, plus where and when it came from); a Transform parses those bytes into normalized Rows; a Sink writes them. Row is a single enum — a SpaceObject (a satellite and its OrbitalElements), a Conjunction, or a Launch — so one sink accepts the output of any source without knowing which produced it. Each real source is just a Source + Transform pair: CelestrakSource fetches TLE text and parses its columns, checksum, and epoch; SpaceTrackSource logs in for a session cookie and pulls conjunction data messages, politely throttled by a hand-rolled TokenBucket; SpaceDevsSource walks a paginated API. The keystone is the Conjunction — a close approach between two tracked objects, which is a collision-avoidance decision waiting to be posed to a model, and the ground truth a Panoptes ca_geo vignette is built from.
What makes this an ETL and not a pile of scripts is four seams. EtlError splits every failure into transient (a dropped connection, a 429) or permanent (a 404, malformed data), and with_retry backs off and retries only the transient ones. NoradId is a newtype, so a catalog number can never be mistaken for any other integer. The Ledger is an append-only record of every run — which source, when, the high-water watermark it reached, success or failure — and it is what lets a re-run skip a source that is still fresh. content_id hashes a row's content into a stable id, so IdempotentSink recognizes a row it has already written and a backfill costs nothing. None of these is clever on its own; together they are the joints an orchestrator needs.
The payoff is the Executor. You describe the work as a Dag — a graph of tasks — and it runs them in topological order, retrying transient failures, gating a node when its upstream fails, and skipping sources the ledger says are current. Crucially, etl-orchestrate depends on etl-core alone: it schedules a dyn Source and cannot even name a concrete source, which is why adding the NASA capstone changes nothing about the executor. Last is retrieval: EmbedSink runs the prose fields — a launch's description, never the numbers — through an Embedder (the VoyageEmbedder), content-addressed so any text is embedded once, and a VectorIndex answers top_k by cosine similarity. The normalized JSONL and that vector index are the two artifacts Panoptes reads.
Each source extracts raw bytes, a transform parses them into typed Rows, and an idempotent sink appends JSONL that panoptes-gen reads. The Executor runs the pipelines in dependency order — retrying transient failures, gating on upstream failure, skipping sources still fresh — and the retrieval arc embeds the prose fields into a vector index.
The E/T/L trait triad — Source, Transform, Sink — is the spine. Concrete sources implement Source; the domain types are the payload of the Row enum the sink writes; and Pipeline/Executor compose the traits as boxed trait objects, which is what lets the orchestrator stay ignorant of any concrete source.
One seam per arc; the dependency arrow always points at etl-core.
The seams. Domain model, the E/T/L traits, and everything the orchestrator schedules against.
Concrete sources + the machinery each API forces. Depends only on core.
The DAG executor. Depends on core ONLY — proven by cargo tree.
The binary. The only crate that names concrete sources and wires them into the DAG.