Data quality monitoring for data warehousing
Trust all the stages of your data warehouse
Incorrect data quality in data marts can result in misleading insights, wasted time fixing errors, and difficulty identifying trends. Furthermore, when stakeholders encounter inaccurate or unreliable information frequently, they lose credibility in the data mart.
Monitor the data quality in all tables of your data warehouse at every stage. Define the data dependencies between the tables and have a record of lineage and data quality alerts of the date in upstream tables. Detect potential issues in downstream tables before the data load happens. Trust all stages of your data warehouse.
Unified data quality process
DQOps platform has built-in data quality policies that automatically activate checks on all tables and columns. You can enable, disable, customize, and add new policies.
- Automatically activate data quality checks using the rule miner engine.
- Quickly modify multiple data quality checks using a search filter.
- Migrate the data quality rules across environments by just copying YAML files.
DQOps platform allows you to set a pattern for checks that will be activated by default. You can also activate, deactivate, and modify multiple data quality checks using a search filter.
- Automatically activate data quality checks without clicking.
- Quickly modify multiple data quality checks using a search filter.
- Migrate the data quality rules across environments by just copying YAML files.
Healthy data marts
Healthy data marts
DQOps platform provides operators for Apache Airflow to run data quality checks and detect the data quality status of any table before it is loaded to the data mart.
- Load the data mart tables incrementally only when there are no unresolved data quality issues in previous stages.
- Delay data mart refresh until data quality issues are resolved or accepted as minor issues.
- Protect your data marts from invalid data.
Incremental data loading
The DQOps platform can analyze the data using various time gradients. Analyze new data for incremental load and delay the incremental refresh when data quality issues are identified.
- Avoid a full refresh of a data mart if the fact table is likely to be loaded with incorrect data.
- Detect duplicated data that leads to invalid sums of aggregated columns.
- Delay or abandon an incremental refresh if it could potentially compromise the integrity of the data mart.
The DQOps platform can analyze the data using various time gradients. Analyze new data for incremental load and delay the incremental refresh when data quality issues are identified.
- Avoid a full refresh of a data mart if the fact table is likely to be loaded with incorrect data.
- Detect duplicated data that leads to invalid sums of aggregated columns.
- Delay or abandon an incremental refresh if it could potentially compromise the integrity of the data mart.
Data quality documentation
Data quality documentation
Data quality checks can be shared with BI developers and Data scientists through easy-to-read YAML files.
- Avoid questions about the data formats of columns from BI developers and Data Scientists.
- Build a knowledge base of your data warehouse.
- Verify the quality of your data warehouse by running the data quality checks.