Last updated: September 24, 2024
DQOps YAML files reference
DQOps stores all configuration of data sources, and the configuration of activated data quality checks in YAML files. The list of supported files is below.
DQOps file types
The list of supported files, their file extension and a link to the full reference of all fields is below. All file name patterns are relative to the DQOps user home folder.
Type | File location | Description |
---|---|---|
Data source connection | sources/<connection>/connection.dqoconnection.yaml | The data source configuration parameters, including the CRON schedule for running data quality checks, and all data source connection parameters. |
Table schema | sources/<connection>/*.dqotable.yaml | The table metadata with the configuration of activated data quality checks. |
Custom dashboard list | settings/dashboardslist.dqodashboards.yaml | The configuration of custom data quality dashboards organized as a dashboard folder tree shown in the Data quality dashboards section of the user interface. |
Default notification webhooks | settings/defaultnotifications.dqonotifications.yaml | The configuration of the default webhooks used for data quality incidents' notifications when custom webhook urls are not assigned in the configuration file of a data source. |
Table-level data quality policy | patterns/<pattern_name>.dqotablepattern.yaml | The selection and the parameters of the data observability checks that are activated on all imported tables. |
Column-level data quality policy | pattern/<pattern_name>.dqocolumnpattern.yaml | The selection and the parameters of the data observability checks that are activated on all imported tables. |
Default CRON schedules | settings/defaultschedules.dqoschedules.yaml | The default CRON schedules for running data quality checks. DQOps does not use these schedules directly and changing the file has no effect on data sources. DQOps only uses this file to copy the default CRON schedules to the data source connection file, when the data source is imported. |
Local settings | .localsettings.dqosettings.yaml | The local configuration settings that are specific only to the current instance of DQOps. Mainly used to store the DQOps Cloud Pairing API key. |
Sensor definition | sensors/**/*.dqosensor.yaml | The definition of a custom data quality sensor, with the configuration of the parameters that will be shown in on data quality check editor screen. |
Database specific sensor definition | sensors/**/<data_source_type>.dqoprovidersensor.yaml | The definition of a database specific custom data quality sensor. |
Data quality rule definition | rules/*/*.dqorule.yaml | The definition of a custom data quality rules. |
Custom check definition | checks/**/*.dqocheck.yaml | The definition of a custom data quality check. Specifies a pair of a data quality sensor and a data quality rule that make up a data quality check. |
Incident notification's webhook message
DQOps supports integration with external ticketing systems and messaging platforms, publishing notification messages of new or acknowledged data quality incidents using webhooks.
DQOps makes a HTTP POST operation, sending a json object that is a data quality incident notification message.
What's next
- You can review the documentation of the DQOps REST API client that uses the same objects in the request body or the response body.
- Look at the concept of the DQOps user home folder to understand where the DQOps configuration files are used.