Configure validation and quality

Configure a dataset’s validation rules, decide how a cleaner treats rejected values, and check the result before attaching processing to a live flow. Resource changes require ADMIN. Use a disposable entity or test data for changes that can remove, replace, or refuse values.

Define the data contract

1. Open the dataset resource and identify the field and semantic token.
2. Choose whether it permits an empty value and select its validation mode.
3. Choose IGNORE, CLEAR, REPLACE, FIX, or REFUSE deliberately. For REPLACE, provide errorReplacement.
4. Check all consumers before saving a shared dataset. Use the resource’s dependencies and the quality-impact preview to assess the affected tables.

A minimal email rule is:

{
  "key": "email",
  "type": "TOKEN",
  "token": "EMAIL",
  "validation": "DEFAULT",
  "empty": true,
  "error": "IGNORE"
}

REFUSE is not a mandatory-field check. Read Validation and error policies for its scope and for the behavior of an unsuccessful FIX.

Prepare and process records

1. Use a transformation when source and target fields differ.
2. Create a pipeline for the target dataset and include a CLEANER processor where error policies must apply.
3. Put transformations and scripts in the required order; each pipeline processor receives the previous processor’s result.
4. Test the pipeline before attaching it to the intended source or entity flow. A dataset rule alone does not run a cleaner.

Test before writing

Use the resource editor’s test action with custom records. Include a valid email, ana@, an absent value, and any supported repeated or nested shapes. Compare each input with the output after the cleaner.

For the example above, IGNORE retains ana@. CLEAR and an unsuccessful FIX remove it. REPLACE uses the configured replacement. REFUSE reports that the record would not be written. A successful test status means the test ran; inspect its per-record report for refusals.

Resource testing documents the API wrapper and test input options.

Verify the outcome

1. Run a small controlled load through the configured processing.
2. Follow its execution and read refusal or failure counts. A completed load does not imply every submitted record was written.
3. Read back the affected records and compare identity and business values.
4. Read their quality state and findings. Measuring a draft with POST /api/tables/{table}/records/check saves nothing and does not apply cleaner policies.
5. Expand the load only after the observed behavior matches the intended contract.

If the rule is wrong, restore the prior configuration and retest. Configuration rollback does not restore values already cleared or replaced; reload from an appropriate source or use a separately verified recovery procedure.

Monitor definition changes

Changing quality-relevant configuration can make existing measurements stale. Inspect qualityState, remeasurement progress, and measurement coverage rather than treating old scores as current. See Data quality and Investigate data quality.

After the affected records have been recalculated, an administrator can use Quality → Change → Measure now to save a new table measurement. Wait for completion and compare its definition and coverage with the older picture. Measure now does not recalculate records; taking a picture immediately after a rule change may therefore show incomplete coverage.