Most data pipelines can be a little wrong for a little while. A metric drifts, someone notices next week, you patch the join.
Remediation pipelines are not like that. If you miss a customer, someone does not get the letter, the refund, or the account correction they are owed. If you include the wrong one, you have a different problem. The work I did on Wells Fargo’s highest-visibility programs was less about sparkle and more about making it difficult for the pipeline to lie.
A few habits that stuck:
Parity is a product. We treated upstream/downstream consistency as a first-class check, not a query you run when you get nervous. A custom parity framework on the high-throughput ETL (10M+ rows a day) existed to fail loudly when the two sides of a transformation stopped agreeing.
Validation has to outlive you. A test suite that only the author understands will not get run. The real-estate metrics work only mattered once the validation suite was something the rest of the department could adopt — same checks, same language, not a one-off notebook.
Logging is for auditors, not just for you. The letter-generation system (100k+ customer letters a month) needed error handling that a reviewer could follow months later. “It probably sent” is not a state.
I still like building product UI. I also like systems where the definition of done includes “we can prove what happened.” Those two instincts are the same job.