On this page
Validation and error policies
Understand how dataset validation and error policies relate to the values stored in a Golden record.
Validation asks whether a value satisfies the dataset’s rules. An error policy specifies what a cleaner does when its formatter rejects that value. Quality measurement reports conditions in a record; it does not itself repair or refuse the record.
For example, ana@ is an invalid email. Keeping it, removing it, replacing it,
and refusing the entire write are different business decisions. Choose the
behavior explicitly instead of treating every successful load as proof of
valid data.
Validation and error policy
A column selects one validation mode:
| Mode | Meaning |
|---|---|
NONE | No selected validation rule |
DEFAULT | Validation supplied by the semantic token |
PARSER | Parse with the configured format and locale where supported |
REGEX | Check the configured regular expression |
SCRIPT | Apply the configured validation script |
The token, emptiness setting, and validation mode describe different aspects
of the column. empty: false declares a mandatory value. It is not equivalent
to choosing an error policy.
What happens to an invalid value
The following cases use an EMAIL column with validation: "DEFAULT", a
CLEANER pipeline processor, and the input "email": "ana@".
| Policy | Cleaner result for this input |
|---|---|
IGNORE | Retains "email": "ana@" |
CLEAR | Removes the email value |
REPLACE | Uses errorReplacement; with "unknown@example.com", that becomes the value |
FIX | Uses a supported fix if one exists; this incomplete email has none, so its value is removed |
REFUSE | Refuses the record instead of allowing this write |
IGNORE means preserving the offending value, not declaring it correct.
A replacement is a configured default, not a recovered fact about the
customer. FIX cannot invent a missing domain or guarantee that the remaining
value is the one the customer intended.
These policies run where a cleaner processes the record. Declaring
REFUSE on a dataset without running a cleaner does not create a universal
write constraint. Nor does REFUSE turn a missing mandatory field into a
refused write: emptiness is measured as a quality condition, while refusal
responds to a value rejected by the formatter.
A refused record and a failed load are different
A refused record is not written. A load can skip refused records and continue with others, reporting the refusal count. Check the operation’s result and stored records, not just whether the background run succeeded.
A resource test can also finish successfully while its report says a record would be refused. The test completed; that is not a promise the record would be stored. Read the per-record report.
Verify the cleaned record
When testing a policy, keep the input, column settings, and output together. Check what remains after cleaning and then measure that result. Removing an invalid optional value can exchange an invalid-value finding for a missing-value warning; it has not supplied the missing business information.
For nested and repeated data, inspect the affected item and its path. A policy that removes one email must not be interpreted as deleting the customer.
Use Configure validation and quality to test these choices. The dataset reference lists the settings, and Data quality explains measurement.