Last updated: October 27, 2024

dqo check command-line command

The reference of the check command in DQOps. Commands related to checks and rules

dqo check run

Run data quality checks that match a given condition

Description

Run data quality checks on your dataset that match a given condition. The command output is a table with the results that provides insight into the data quality.

Command-line synopsis

$ dqo [dqo options...] check run [-deh] [-ces] [--daily-partitioning-include-today] [-fe] [-fw] [-hl]
           [--monthly-partitioning-include-current-month] [-c=<connection>]
           [-cat=<checkCategory>] [-ch=<check>] [-col=<column>]
           [-ct=<checkType>]
           [--daily-partitioning-recent-days=<dailyPartitioningRecentDays>]
           [-f=<failAt>] [--from-date=<fromDate>]
           [--from-date-time=<fromDateTime>]
           [--from-date-time-offset=<fromDateTimeOffset>] [-m=<mode>]
           [--monthly-partitioning-recent-months=<monthlyPartitioningRecentMonth
           s>] [-of=<outputFormat>] [-s=<sensor>] [-t=<table>]
           [--to-date=<toDate>] [--to-date-time=<toDateTime>]
           [--to-date-time-offset=<toDateTimeOffset>] [-tr=<executionTarget>]
           [-ts=<timeScale>] [--where-filter=<whereFilter>] [-l=<labels>]...
           [-tag=<tags>]...

DQOps shell synopsis

dqo> check run [-deh] [-ces] [--daily-partitioning-include-today] [-fe] [-fw] [-hl]
           [--monthly-partitioning-include-current-month] [-c=<connection>]
           [-cat=<checkCategory>] [-ch=<check>] [-col=<column>]
           [-ct=<checkType>]
           [--daily-partitioning-recent-days=<dailyPartitioningRecentDays>]
           [-f=<failAt>] [--from-date=<fromDate>]
           [--from-date-time=<fromDateTime>]
           [--from-date-time-offset=<fromDateTimeOffset>] [-m=<mode>]
           [--monthly-partitioning-recent-months=<monthlyPartitioningRecentMonth
           s>] [-of=<outputFormat>] [-s=<sensor>] [-t=<table>]
           [--to-date=<toDate>] [--to-date-time=<toDateTime>]
           [--to-date-time-offset=<toDateTimeOffset>] [-tr=<executionTarget>]
           [-ts=<timeScale>] [--where-filter=<whereFilter>] [-l=<labels>]...
           [-tag=<tags>]...

Command options

All parameters supported by the command are listed below.

Command argument	Description	Accepted values
`-cat` `--category`	Check category name (volume, nulls, numeric, etc.)
`-ch` `--check`	Data quality check name, supports patterns like '*_id'
`-ct` `--check-type`	Data quality check type (profiling, monitoring, partitioned)	profiling monitoring partitioned
`-ces` `--collect-error-samples`	Collects error samples for failed data quality checks
`-col` `--column`	Column name, supports patterns like '*_id'
`-c` `--connection`	Connection name, supports patterns like 'conn*'
`--daily-partitioning-include-today`	Analyze also today and later days when running daily partitioned checks. By default, daily partitioned checks will not analyze today and future dates. Setting true will disable filtering the end dates.
`--daily-partitioning-recent-days`	The number of recent days to analyze incrementally by daily partitioned data quality checks.
`-tag` `--data-grouping-level-tag`	Data grouping hierarchy level filter (tag)
`-d` `--dummy`	Runs data quality check in a dummy mode, sensors are not executed on the target database, but the rest of the process is performed
`-e` `--enabled`	Runs only enabled or only disabled sensors, by default only enabled sensors are executed
`-f` `--fail-at`	Lowest data quality issue severity level (warning, error, fatal) that will cause the command to return with an error code. Use 'none' to return always a success error code.	warning error fatal none
`-fe` `--fail-on-execution-errors`	Returns a command status code 4 (when called from the command line) if any execution errors were raised during the execution, the default value is true.
`-fw` `--file-write`	Write command response to a file
`--from-date`	Analyze the data since the given date (inclusive). The date should be an ISO 8601 date (yyyy-MM-dd). The analyzed table must have the timestamp column properly configured, it is the column that is used for filtering the date and time ranges. Setting the beginning date overrides recent days and recent months.
`--from-date-time`	Analyze the data since the given date and time (inclusive). The date and time should be an ISO 8601 local date and time without the time zone (yyyy-MM-dd HH:mm:ss). The analyzed table must have the timestamp column properly configured, it is the column that is used for filtering the date and time ranges. Setting the beginning date and time overrides recent days and recent months.
`--from-date-time-offset`	Analyze the data since the given date and time with a time zone offset (inclusive). The date and time should be an ISO 8601 date and time followed by a time zone offset (yyyy-MM-dd HH:mm:ss). For example: 2023-02-20 14:10:00+02. The analyzed table must have the timestamp column properly configured, it is the column that is used for filtering the date and time ranges. Setting the beginning date and time overrides recent days and recent months.
`-t` `--table` `--full-table-name`	Full table name (schema.table), supports wildcard patterns 'sch.tab'
`--headless` `-hl`	Starts DQOps in a headless mode. When DQOps runs in a headless mode and the application cannot start because the DQOps Cloud API key is missing or the DQOps user home folder is not configured, DQOps will stop silently instead of asking the user to approve the setup of the DQOps user home folder structure and/or log into DQOps Cloud.
`-h` `--help`	Show the help for the command and parameters
`-l` `--label`	Label filter
`-m` `--mode`	Reporting mode (silent, summary, info, debug)	silent summary info debug
`--monthly-partitioning-include-current-month`	Analyze also the current month and later months when running monthly partitioned checks. By default, monthly partitioned checks will not analyze the current month and future months. Setting true will disable filtering the end dates.
`--monthly-partitioning-recent-months`	The number of recent months to analyze incrementally by monthly partitioned data quality checks.
`-of` `--output-format`	Output format for tabular responses	TABLE CSV JSON
`-s` `--sensor`	Data quality sensor name (sensor definition or sensor name), supports patterns like 'table/validity/*'
`-tr` `--target`	Configures the data quality execution mode. The default option is to run both the sensor to collect metrics, and validate the results with data quality rules. Alternatively, it is possible to only run the sensors to collect metrics, or only run rules on existing data.	sensors_and_rules only_sensors only_rules
`-ts` `--time-scale`	Time scale for monitoring and partitioned checks (daily, monthly, etc.)	daily monthly
`--to-date`	Analyze the data until the given date (exclusive, the given date and the following dates are not analyzed). The date should be an ISO 8601 date (YYYY-MM-DD). The analyzed table must have the timestamp column properly configured, it is the column that is used for filtering the date and time ranges. Setting the end date overrides the parameters to disable analyzing today or the current month.
`--to-date-time`	Analyze the data until the given date and time (exclusive). The date should be an ISO 8601 date (yyyy-MM-dd). The analyzed table must have the timestamp column properly configured, it is the column that is used for filtering the date and time ranges. Setting the end date and time overrides the parameters to disable analyzing today or the current month.
`--to-date-time-offset`	Analyze the data until the given date and time with a time zone offset (exclusive). The date and time should be an ISO 8601 date and time followed by a time zone offset (yyyy-MM-dd HH:mm:ss). For example: 2023-02-20 14:10:00+02. The analyzed table must have the timestamp column properly configured, it is the column that is used for filtering the date and time ranges. Setting the end date and time overrides the parameters to disable analyzing today or the current month.
`--where-filter`	An additional filter which must be a valid SQL predicate (an SQL expression that returns 'true' or 'false') that is added to the WHERE clause of the SQL query that DQOps will run on the data source. The purpose of a custom filter is to analyze only a subset of data, for example, when a new batch of records is loaded, and the data quality checks are evaluated as a data contract. All the records in that batch must tagged with the same value, and the passed predicate to find records from that batch would use the filter in the form: "{alias}.batch_id = 1". The filter can use replacement tokens {alias} to reference the analyzed table.

dqo check activate

Description

Activates data quality checks matching specified filters

Command-line synopsis

$ dqo [dqo options...] check activate [-efhnow] [-de] [-df] [-dw] [-fw] [-hl] [-c=<connection>]
                [-cat=<checkCategory>] [-ch=<check>] [-col=<column>]
                [-ct=<checkType>] [-dt=<datatypeFilter>] [-of=<outputFormat>]
                [-sn=<sensor>] [-t=<table>] [-ts=<timeScale>]
                [-E=<String=String>]... [-F=<String=String>]...
                [-S=<String=String>]... [-W=<String=String>]...

DQOps shell synopsis

dqo> check activate [-efhnow] [-de] [-df] [-dw] [-fw] [-hl] [-c=<connection>]
                [-cat=<checkCategory>] [-ch=<check>] [-col=<column>]
                [-ct=<checkType>] [-dt=<datatypeFilter>] [-of=<outputFormat>]
                [-sn=<sensor>] [-t=<table>] [-ts=<timeScale>]
                [-E=<String=String>]... [-F=<String=String>]...
                [-S=<String=String>]... [-W=<String=String>]...

Command options

All parameters supported by the command are listed below.

Command argument	Description	Accepted values
`-cat` `--category`	Check category name (standard, nulls, numeric, etc.)
`-ch` `--check`	Data quality check name, supports patterns like '*_id'
`-ct` `--check-type`	Data quality check type (profiling, monitoring, partitioned)	profiling monitoring partitioned
`-col` `--column`	Column name, supports patterns like '*_id'
`-c` `--connection`	Connection name, supports patterns like 'conn*'
`-dt` `--data-type`	Datatype of columns on which to enable checks.
`-de` `--disable-error`	Disables an error severity rule. Use this option to update already activated data quality checks, together with the --override option.
`-df` `--disable-fatal`	Disables a fatal severity rule. Use this option to update already activated data quality checks, together with the --override option.
`-dw` `--disable-warning`	Disables a warning severity rule. Use this option to update already activated data quality checks, together with the --override option.
`-e` `--enable-error`	Enables an error severity rule. An error severity rule is also enabled when any warning rule parameters are passed using the -Eparam_nam=value parameters
`-f` `--enable-fatal`	Enables a fatal severity rule. A fatal severity rule is also enabled when any fatal rule parameters are passed using the -Fparam_nam=value parameters
`-w` `--enable-warning`	Enables a warning severity rule. A warning severity rule is also enabled when any warning rule parameters are passed using the -Wparam_nam=value parameters
`-fw` `--file-write`	Write command response to a file
`-t` `--table` `--full-table-name`	Full table name (schema.table), supports patterns like 'sch.tab'
`--headless` `-hl`	Starts DQOps in a headless mode. When DQOps runs in a headless mode and the application cannot start because the DQOps Cloud API key is missing or the DQOps user home folder is not configured, DQOps will stop silently instead of asking the user to approve the setup of the DQOps user home folder structure and/or log into DQOps Cloud.
`-h` `--help`	Show the help for the command and parameters
`-n` `--nullable`	Enable check only on nullable columns (false for explicitly non-nullable columns).
`-of` `--output-format`	Output format for tabular responses	TABLE CSV JSON
`-o` `--override`	Override existing configuration of selected checks.
`-sn` `--sensor`	Data quality sensor name (sensor definition or sensor name), supports patterns like 'table/validity/*'
`-ts` `--time-scale`	Time scale for monitoring and partitioned checks (daily, monthly, etc.)	daily monthly
`-E`	Error level rule options. Usage: -E=, --error-rule==
`-F`	Fatal level rule options. Usage: -F=, --fatal-rule==
`-S`	Configuration parameters for the sensor. Usage: -S=, --sensor-param==
`-W`	Warning level rule options. Usage: -W=, --warning-rule==

dqo check deactivate

Description

Deactivates data quality checks matching specified filters

Command-line synopsis

$ dqo [dqo options...] check deactivate [-hn] [-fw] [-hl] [-c=<connection>] [-cat=<checkCategory>]
                  [-ch=<check>] [-col=<column>] [-ct=<checkType>]
                  [-dt=<datatypeFilter>] [-of=<outputFormat>] [-s=<sensor>]
                  [-t=<table>] [-ts=<timeScale>]

DQOps shell synopsis

dqo> check deactivate [-hn] [-fw] [-hl] [-c=<connection>] [-cat=<checkCategory>]
                  [-ch=<check>] [-col=<column>] [-ct=<checkType>]
                  [-dt=<datatypeFilter>] [-of=<outputFormat>] [-s=<sensor>]
                  [-t=<table>] [-ts=<timeScale>]

Command options

All parameters supported by the command are listed below.

Command argument	Description	Accepted values
`-cat` `--category`	Check category name (standard, nulls, numeric, etc.)
`-ch` `--check`	Data quality check name, supports patterns like '*_id'
`-ct` `--check-type`	Data quality check type (profiling, monitoring, partitioned)	profiling monitoring partitioned
`-col` `--column`	Column name, supports patterns like '*_id'
`-c` `--connection`	Connection name, supports patterns like 'conn*'
`-dt` `--data-type`	Datatype of columns on which to disable checks.
`-fw` `--file-write`	Write command response to a file
`-t` `--table` `--full-table-name`	Full table name (schema.table), supports patterns like 'sch.tab'
`--headless` `-hl`	Starts DQOps in a headless mode. When DQOps runs in a headless mode and the application cannot start because the DQOps Cloud API key is missing or the DQOps user home folder is not configured, DQOps will stop silently instead of asking the user to approve the setup of the DQOps user home folder structure and/or log into DQOps Cloud.
`-h` `--help`	Show the help for the command and parameters
`-n` `--nullable`	Disable check only on nullable columns (false for explicitly non-nullable columns).
`-of` `--output-format`	Output format for tabular responses	TABLE CSV JSON
`-s` `--sensor`	Data quality sensor name (sensor definition or sensor name), supports patterns like 'table/validity/*'
`-ts` `--time-scale`	Time scale for monitoring and partitioned checks (daily, monthly, etc.)	daily monthly