When should you fix data quality at the source instead of adding transformation step?

Naomi Teng
Updated 14 hours ago in

A common challenge in analytics is dealing with inconsistent formats, duplicate records, missing values, and mismatched definitions across systems. Transformation tools can standardize much of this, but repeated fixes can turn into a growing collection of exceptions and workarounds.
At some point, it becomes worth asking whether the pipeline is solving the problem or simply compensating for it.
For those working with ETL, Alteryx, SQL, or Python pipelines, what tells you that a transformation belongs in the pipeline versus being a problem that should be fixed in the source system?
How do you handle it when the source team cannot make changes?

  • 1
  • 18
  • 13 hours ago
 
14 hours ago

My rule is to fix data at the source when the issue reflects incorrect business logic or broken data contracts, and use transformations when the change is intentional and specific to a downstream use case.

For example, if an upstream system repeatedly sends invalid customer IDs or inconsistent status codes, fixing it in every ETL pipeline just creates multiple points of failure. The same defect can surface in a new dashboard, model, or downstream consumer that doesn’t have that workaround.

However, transformations are still necessary for legitimate differences in representation, schema mapping, and analytical requirements. Not every inconsistency is a source-level defect.

If the source team cannot make changes, I’d isolate the correction in a shared, version-controlled transformation layer rather than duplicating patches across pipelines. I’d also add data quality tests, monitor exception rates, and document the workaround with a clear owner.

One important distinction: a transformation can make bad data usable, but it doesn’t necessarily make the underlying system reliable. The goal should be to prevent recurring defects wherever possible, while keeping unavoidable corrections explicit, testable, and observable.

  • Liked by
Reply
Cancel
Loading more replies