Data engineering2 min read
Data contracts: stop upstream changes breaking everything downstream
A data contract is a promise between the team that produces data and the teams that depend on it. Here is how to put them in place without slowing anyone down.
Written by The metadatum team
Most pipeline failures don't start in the pipeline. They start upstream, when a source system renames a column, changes a type, or starts sending nulls where there never were any. Nothing tells the data team until a dashboard is wrong, and by then the people relying on it have stopped trusting the numbers.
What a data contract is
A data contract is an agreement, written down and enforced in code, about the shape and meaning of a dataset. It names the owner, the schema, the freshness guarantee, and what happens when any of those change.
| Part | What it covers | Enforced by |
|---|---|---|
| Schema | Columns, types, nullability | Validation at ingestion |
| Semantics | What each field means | Documentation and review |
| Freshness | How late data may arrive | Monitoring and alerts |
| Ownership | Who to call when it breaks | The contract itself |
How to put them in place
- Start with the three datasets that break most often, not all of them.
- Write the contract with the producing team, in their terms.
- Validate at the boundary, before anything lands in the platform.
- Fail loudly and early, with a message that names the owner.
A minimal schema check at ingestion looks like this:
from pydantic import BaseModel, Field
class Order(BaseModel):
order_id: str
customer_id: str
amount: float = Field(ge=0)
currency: str = Field(min_length=3, max_length=3)
def validate(rows: list[dict]) -> list[Order]:
return [Order(**row) for row in rows]
Keep the contract alive
A contract nobody updates is worse than none, because people believe it. Store contracts next to the code that produces the data, review changes like any other pull request, and version them, so a breaking change becomes a new version that consumers move to on their own schedule.
What changes
Upstream teams learn about breaking changes before they ship them, not after. Downstream teams stop writing defensive code for problems that should never reach them. And when something does break, everyone knows whose problem it is.
