Many data engineering teams already have a contract.yaml sitting in their repository. It has a schema, types, maybe even descriptions. The problem is that nobody enforces it. The contract lives in Git, while data keeps flowing: with nulls in a required column, negative order_id, or a birth date in the future. The contract is documentation, not a control mechanism. And that is exactly why it is useless.
A data contract without validation is technical debt in its purest form: it looks like a safeguard, but it is not. In this article we show why that happens and how to turn a contract into a mechanism that actually protects your pipeline.
Table of Contents
What a data contract is and what it lacks
A data contract is an agreement between a data producer and a data consumer: which columns, which types, which constraints, and which quality expectations. Formally speaking, it defines the structure and content of data at a specific stage of the pipeline. That sounds solid, until we notice that the contract itself is only a declaration.
In our article on the principles of data quality, we outline three pillars: define, detect, remediate. A contract addresses only the first one. Without detection and remediation, it remains a wish written in YAML. This is exactly the same problem that appears with data conformity: if a format is not enforced and monitored, incorrectly formatted values will eventually reach the system, and the consequences can be serious, including regulatory violations.
Examples of useless contracts
The most common patterns of a “dead” contract look familiar:
A contract without checks: the file says “
emailcannot be NULL,” but the pipeline has nonulls_counttest on that column. Nobody will notice when the rule is broken.A contract without an owner: when a check fails, there is no person or team accountable. The violation lands in a dashboard queue and stays there.
A contract validated only on the producer side: the producer “pushes through” data that formally passes schema checks but is business-useless. The consumer finds out after the fact.
A contract that covers everything at once: trying to validate record-by-record in real time without streaming and without an architecture change ends with checks being disabled after the first week.
How to detect that your contract is not working
Before you fix the contract, check whether it works at all. Three warning signs:
No result history: if you cannot see check results from the last few days, the contract is probably not being run.
Zero violations for months: with real data that is suspicious. It usually means thresholds are too loose or checks do not cover key columns.
A gap between producer and consumer: the consumer reports issues that the producer “does not see” in their checks. That is a sign validation works on only one side.
How to fix a contract: best practices
Fixing it does not require a new tool. It requires three things: checks tied to the contract, an owner on the producer side, and independent validation on the consumer side.
Make the contract an executable file: in DQOps, a contract is a YAML file with check definitions and severity rules. You run it via CLI or from a pipeline. If the
nulls_countcheck oncustomer_idexceeds a threshold, the pipeline gets a clear signal, not just a log entry.Set ownership and an escalation threshold: a contract without an owner is a contract without consequences. It should have an assigned owner and a rule that, on
fatalseverity, triggers an incident, not just a dashboard queue.Validate on the consumer side: do not assume data from the producer is good just because “it has a contract.” Run your own checks on the table you consume. DQOps lets you define checks on the source table and compare against expectations.
Share error samples: when a check detects a violation, export a few example rows and pass them to the data owner. Concrete examples work better than abstract thresholds. The same principle that works when monitoring conformity.
Do not validate everything in real time if you do not have to: full record-by-record validation requires streaming and an architecture change. In most cases, a batch check at the table or partition level, run after data is loaded but before it is exposed to consumers, is enough.
Summary
A data contract without enforcement is documentation that lulls you into a false sense of security. Fixing it comes down to three steps: tie the contract to executable checks, assign an owner, and validate data on the consumer side too. The rest is configuration detail.
Data quality best practices - a step-by-step guide to improve data quality
- Learn the best practices in starting and scaling data quality
- Learn how to find and manage data quality issues
What is the DQOps Data Quality Operations Center
DQOps is a data quality and observability platform designed to monitor data and assess the data quality trust score with data quality KPIs. DQOps provides extensive support for configuring data quality checks, applying configuration by data quality policies, detecting anomalies, and automatically finding patterns using machine learning. DQOps supports over 150+ data quality checks, which cover all popular formats used in data conformance testing.
DQOps is a platform that combines the functionality of a data quality platform to perform the data quality assessment of data assets. It is also a complete data observability platform that can monitor data and measure data quality metrics at a table level to measure its health scores with data quality KPIs.
You can set up DQOps locally or in your on-premises environment to learn how DQOps can monitor data sources and ensure data quality within a data platform. Follow the DQOps documentation, go through the DQOps getting started guide to learn how to set up DQOps locally, and try it.
You may also be interested in our free eBook, “A step-by-step guide to improve data quality.” The eBook documents our proven process for managing data quality issues and ensuring a high level of data quality over time. This is a great resource to learn about data quality.
