When a pipeline runs on schedule and is still wrong
A daily SQL pull, reshaped in Power Query, published on a fixed schedule. It ran without failing for months, and one day it was quietly wrong.
What happened
A morning extract pulled sales and stock movements out of 1C, Power Query reshaped it, and the report refreshed at 7am. That had run unattended for the better part of a year — no errors, no complaints, which I read as "working."
One month the supplier numbers looked slightly off. Not broken, just off by a margin small enough to miss in a scan. The cause: that day's export from 1C had finished forty minutes later than usual. The extract query ran at its scheduled time regardless, and read a table that was still half-written. Nothing failed. The job logged success. The number was wrong.
The job wasn't checking what mattered
The schedule assumed each step would be ready when the next one ran, not that it had actually finished. A completed run and a correct run were treated as the same event, and only the first one was being checked.
The other failure mode showed up once I started looking for it: re-running a step after fixing an error appended a new batch instead of replacing the old one. A number corrected by hand in March would quietly double by April, and it would read as growth.
A pipeline succeeding and a pipeline being right are two different events. Only one of them shows up in a log by default.
What changed
Nothing elaborate — I didn't reach for an orchestration tool for a job this size. Each step now writes one row to a small status table on completion: timestamp, row count, and whether it reconciled against the source total. The next step checks that row before it runs, and a re-run always overwrites the previous batch instead of appending to it.
It's two rules, not a rebuild. But they're the difference between a pipeline that runs and one I'd actually put a number from in front of someone.