Data observability for DevOps
Keep the data quality checks along with the data pipeline code
It can be quite challenging for DevOps teams to maintain good data quality from testing to final use. The main issue is the difficulty in transferring data quality definitions between these stages, which can lead to inconsistencies. Data may work well in development but fail in production, causing delays and requiring rework.
DQOps addresses this challenge by storing data quality definitions as simple YAML files, which can be easily managed and moved between development, testing, and production environments. This approach ensures consistent data quality checks across the entire pipeline, allowing DevOps teams to focus on delivering reliable data more efficiently.
Source data quality checks
Stop source data issues from affecting your data warehouse or lake. DQOps enables you to easily define and manage data quality checks for all your source data in one place.
- Define data quality checks for source tables by editing simple YAML files or using the user interface.
- Similar data sources can leverage existing check definitions, saving time and ensuring consistency.
- Adding new tables to be observed is as simple as copying a YAML file.
Stop source data issues from affecting your data warehouse or lake. DQOps enables you to easily define and manage data quality checks for all your source data in one place.
- Define data quality checks for source tables by editing simple YAML files or using the user interface.
- Similar data sources can leverage existing check definitions, saving time and ensuring consistency.
- Adding new tables to be observed is as simple as copying a YAML file.
Data quality testing
Data quality testing
Define checks in code and develop pipelines using a test-driven development approach. Develop the pipeline, test it, refactor it, and then retest it after changes have been made.
- Data quality checks can be easily defined in the code.
- Data quality checks may be instantly executed.
- Enable Test Driven Development and Integration Testing for databases and data lakes.
Smooth transition from development to production
DQOps platform simplifies data quality management during environment migrations. No deployment is required to update the data quality checks.
- Data quality checks are stored in YAML files and can be stored within the code repository.
- Instantly upgrade data quality checks after migrating your pipelines to the production environment.
- Run data quality checks after migration to the production environment for instant validation.
DQOps platform simplifies data quality management during environment migrations. No deployment is required to update the data quality checks.
- Data quality checks are stored in YAML files and can be stored within the code repository.
- Instantly upgrade data quality checks after migrating your pipelines to the production environment.
- Run data quality checks after migration to the production environment for instant validation.
Data quality checks versioning
Data quality checks versioning
DQOps enables you to manage data quality checks with the same rigor as your code. With versioned data quality checks, you can track changes and maintain consistent data expectations.
- Storing data quality checks in YAML files makes them easy to version.
- Data quality checks can be released after a peer review (a pull request).
- Quickly verify who has changed the data quality checks.
Work with local environments
Ensure data quality by testing data preparation scripts on local databases before merging changes to a shared environment.
- Use DQOps command-line tools to run data quality checks directly on your local database, eliminating the need for a dedicated server.
- Develop and test custom data quality checks without affecting shared environments.
- Confirm changes to data quality checks locally to speed up your development and testing processes.
Ensure data quality by testing data preparation scripts on local databases before merging changes to a shared environment.
- Use DQOps command-line tools to run data quality checks directly on your local database, eliminating the need for a dedicated server.
- Develop and test custom data quality checks without affecting shared environments.
- Confirm changes to data quality checks locally to speed up your development and testing processes