Design identity, quality, and authority

Quality controls

Golden contributes to data quality by applying a declared record contract, processing configured transformations or pipelines, and exposing quality facts on managed records. It should be part of an organization’s quality control loop, not the only place where quality is owned.

Quality control loop

define the business contract
          ↓
validate and normalize incoming records
          ↓
store values and quality evidence
          ↓
inspect representative and exceptional records
          ↓
correct the authoritative source or approved Golden configuration
          └───────────────────────────────────────────────────────┘

Prefer correcting an invalid business fact in its authoritative source. A Golden-side correction may be appropriate for a mastered value or controlled exception, but should not silently hide a recurring source defect.

Controls available in Golden

Choose where to enforce quality

At source entry

Use source-system controls when the originating application can prevent an invalid value before it becomes operational data. Golden should still validate the exchanged contract because source and integration behavior can drift.

At Golden intake

Use datasets, transformations, and pipelines to reject, normalize, replace, or retain values according to an explicit exchange contract. Verify both the operation outcome and stored records; an asynchronous task and record-level quality evidence answer different questions.

During stewardship

Expose quality facts alongside source, recency, identifiers, and matching evidence. A high aggregate quality score does not prove that two records are the same subject, and a low score does not by itself prove that they are different.

At consumption

Downstream consumers should define whether they accept every mastered record, filter on documented metadata, or require additional domain checks. Use the returned quality explanation and the documented calculation to interpret scores.

Integration approaches

Measures and evidence

Define measures that can be reproduced from supported outputs, for example:

• records with one named quality fact;
• task completion and failure counts;
• candidates awaiting review;
• invalid destination events; and
• source defects corrected within an agreed period.

Do not present an undocumented aggregate as product health. Product metrics, record-quality facts, tasks, and destination events each describe a different scope.

Design questions

1. Who owns the business definition of each field?
2. Which invalid inputs must be rejected, preserved, cleared, or corrected?
3. Which normalization is deterministic and safe across all sources?
4. Where can a steward correct a value, and where must correction happen upstream?
5. How is a recurring defect assigned and verified after remediation?
6. Which supported evidence is retained for audit or service review?

Use Data quality in Golden for interpretation and Configure datasets and Configure pipelines for product configuration.

Master-data design

A Golden master-data flow connects multiple representations of a business subject to one governed, reusable outcome. Define identity, authority, evidence, survivorship, and recovery alongside the data flow.

Mastering lifecycle

1. Source systems provide records with stable source identity.
2. Golden maps them into a common dataset and table.
3. The entity makes records searchable and, when configured, groups possible duplicates.
4. The classifier or a steward decides whether candidates match.
5. The merger applies approved value precedence to create a golden record.
6. Consumers search or receive the mastered result.
7. New evidence or source changes begin the cycle again.

The resulting golden record is an outcome with contributing lineage. It does not mean that every source record was correct or that every source application must immediately discard its local identifier.

Choose an adoption shape

Consolidate and distribute

Load records from several sources on a schedule, master them in Golden, and deliver the result to analytics or operational consumers. This favors volume and controlled windows over immediate consistency.

Define the extraction watermark, task recovery, deletion behavior, destination contract, and safe redelivery strategy.

Shared lookup and duplicate prevention

Call Golden during an application workflow to look for an existing subject before creating another representation. This can reduce new duplicates, but it adds a synchronous dependency.

Define a timeout and decide explicitly whether the caller fails open, fails closed, or routes the decision for review. An empty search result is not proof that the subject is absent under every criterion.

Governed operational mastering

Use controlled upserts, stewardship, and mastered delivery as part of an operational process. Golden may become the authoritative mastered view for the defined subject while source applications remain authoritative for particular facts.

Define who may write, how conflicts are reconciled, when indexing is complete, and how downstream systems follow a surviving identifier after a merge.

Stewardship-first rollout

Begin with manual review to learn the data and validate evidence thresholds before enabling automatic actions. This reduces false-merge risk while policy is still being established.

Define queue ownership, decision turnaround, required comments, sampling, escalation, and recovery exercises before increasing automation.

Identity design

Keep these identities distinct:

Do not use a mutable display field as a durable integration identifier. Preserve source identity so a later load updates the intended representation instead of creating another one.

When an addressed record has already been merged, supported API conflicts can identify a surviving record. Consumers must reconcile to that identity rather than retrying the obsolete identifier indefinitely.

Authority and survivorship

For every mastered field, document:

• which sources may supply it;
• whether a source has precedence;
• whether recency or completeness affects the choice;
• whether a steward may override it;
• how a missing or invalid value is treated; and
• what evidence the consumer can use to understand the result.

The merger configuration implements approved choices. It is not a substitute for the policy that explains them.

Match governance

Architect around both error types:

• A false merge combines different subjects and can distribute incorrect mastered data.
• A missed match leaves duplicate representations unresolved and can preserve inconsistent outcomes.

Set thresholds and human-review boundaries according to business impact. A comparison score is evidence for a decision, not the decision itself.

Consistency and recovery

See Consistency and recovery for asynchronous completion, downstream verification, and recovery planning.