Have you ever desperately stopped a data mart refresh to avoid loading invalid data?
Monitor the data quality of all tables at all stages in your data warehouse. Define the data dependencies between the tables. Having the data lineage and data quality alerts in the upstream tables, detect potential issues in downstream tables before the data load happens. Trust all the stages of your data warehouse
Monitor the data quality of all tables at all stages in your data warehouse. Define the data dependencies between the tables. Having the data lineage and data quality alerts in the upstream tables, detect potential issues in downstream tables before the data load happens. Trust all the stages of your data warehouse
Unified Data Quality Process
Unified Data Quality Process
Define all data quality rules for all stages of the data warehouse in the same way, in one place.
DQOps data quality rules are easy to attach as just YAML files with code complete. It is easy to connect hundreds or thousands of data quality rules to cover the whole data warehouse without a big effort. Define the dependencies of tables and you get the visibility of all data quality issues at the whole data lineage.
- Easily cover many tables with data quality checks, without clicking
- Track issues in related tables by following the table dependencies
- Migrate the data quality rules across environments by just copying files
Healthy Data Marts
Healthy Data Marts
Monitor all data quality issues in upstream tables that may cause the loading of invalid data to the data mart.
DQOps stores unique data quality alerts in its data quality database. Add code in your data mart loading pipeline that checks that all data quality issues in all tables in the data line path have been resolved.
- Load the data mart tables incrementally only when there are no unresolved data quality issues in previous stages
- Delay data mart refresh until data quality issues are resolved or accepted as minor issues
- Protect your data marts from invalid data
Incremental data loading
Incremental data loading
Avoid a full refresh by acting proactively on data quality issues in previous data stages. Delay the incremental load until the data is correct.
DQOps can analyze the data at multiple dates and time gradients. Analyze new data that you will use in the incremental load and just delay the incremental refresh when data quality issues are identified.
- Avoid full refresh of a data mart if the fact table would be loaded with wrong data
- Detect duplicate data that would cause invalid sums of aggregated columns
- Delay or abandon an incremental refresh if that might harm the data mart integrity
Data Quality documentation
Data Quality documentation
Document tables in your data warehouse as a set of easy-to-understand data quality checks.
Data quality checks that are defined in easy-to-read YAML files may be shared with Business Intelligence developers. BI developers and Data Scientists can just see which columns are unique, not null or what is the format of the tables.
- Avoid questions about the data formats of columns from BI developers and Data Scientists
- Build a knowledge base of your data warehouse
- Verify the quality of your data warehouse by running the data quality checks
