logistics-rate-pipeline — measured results
Eight synthetic ocean-freight rate feeds → a concurrent Go ingestion service (bounded worker pool, retries, graceful shutdown, Prometheus/pprof) → a PySpark normalizer producing a rate matrix with named reject rules. All feeds are invented; the engineering is the point.
Code Live rate-matrix explorer Raw result files Frozen spec & decision log 
1. Worker-pool benchmark
Median wall time over 5 reps per worker count, 20,810 records per run, Go go1.27.0, ingestd 3663789.
| workers | median wall | speedup | records | peak RSS |
|---|---|---|---|---|
| 1 | 6.85 s | 1.00× | 20,810 | 24 MB |
| 2 | 3.77 s | 1.82× | 20,810 | 28 MB |
| 4 | 3.77 s | 1.82× | 20,810 | 29 MB |
| 8 | 3.77 s | 1.82× | 20,810 | 33 MB |
| 16 | 3.77 s | 1.82× | 20,810 | 33 MB |
Why it plateaus at 1.82× and not the 4× the plan assumed: the pool cannot beat the critical path. One feed (aurora) serves two sequential cursor pages at ~1.9 s each, so no worker count finishes under ≈ 3.8 s while the other seven feeds sum to ≈ 2.9 s. The number is committed as measured (decision D-40); the only fix is page-level parallelism, which cursor pagination forbids by contract.
Source: bench/results/20260917T070639Z-pool.md and the JSON beside it.
2. pprof-driven optimization — CSV parse, 20 000 rows
Allocation profile showed 55 % of bytes in ComputeHash (a growing strings.Builder, its final copy, and hex encoding). Fix: a sync.Pool-ed scratch buffer, sha256.Sum256, hex into a stack array, one string.
| metric | before | after | change |
|---|---|---|---|
| ns/op | 17,631,463 | 14,986,805 | ≈ −15 % |
| B/op | 16,066,655 | 8,233,314 | −49 % |
| allocs/op | 220,018 | 80,030 | −64 % |
Source: bench/results/20260917T071149Z-parse.md (raw go test -bench output and pprof top included).
3. Normalization funnel (PySpark 4.2.0)
Dedup is a window over (carrier, lane, container, validity) keeping the highest-priority source; rejects keep their raw values and the rule name so nothing is silently dropped.
Rejects by named rule (66 rows)
Stage durations (local[2], 8 GB laptop)
Source: bench/results/20260917T070639Z-normalize.json — run id fixture, as-of 2026-09-01.
4. Correctness checks
| check | result | where |
|---|---|---|
go test -race ./... | clean | CI, every push |
goroutine leaks (goleak in every pool test, live /debug/pprof/goroutineleak) | 0 | internal/pool, internal/server |
| Cancelled mid-run | no manifest, 0 promoted parts | TestCancelMidRunWritesNoManifestAndNoParts, tests/e2e/shutdown.sh |
| Spark rules | 25 tests incl. every rule pass/fail and an exact-count fixture | spark/tests |
Still pending
Container image sizes, the in-cluster shutdown test on kind, the EKS run, Databricks Free Edition and Redshift loads, and end-to-end latency — each lands here as a committed result file once run. This page is regenerated by scripts/render_results.py; nothing on it is typed by hand.