Automate duplicate decisions

On this page

Golden groups records into duplicate candidates. API callers with VIEWER can inspect granted groups. Merge, disconnect, delete, and escalation require STEWARD or ADMIN; ignore, unignore, and resolving an escalation require ADMIN.

Human stewards should use Steward duplicate candidates in the web application.

Every example on this page runs against the Aurelia Utilities sample, whose entity is sample-customer-entity. Install it in its initial state and configure the connection values. Mutating examples change the state required by later calls; reset the sample between independent cases.

Before you automate

Use cluster identifiers, not bucket identifiers, unless you have a reason not to. A cluster is the logical group a person sees; a bucket is one index’s contribution to it. Both families exist and they do not mix.
Test classification and action policy on representative non-production data.
Use VIEWER for inspection and a separate STEWARD token when an automated workflow is approved to change candidate state.
Preserve a decision comment, stable identifiers, and enough evidence to verify or escalate the result.

The entity must be enabled and its synchronization complete. If a queue appears stale, check the entity state and tasks before changing classifier or indexer configuration.

List the candidates

curl --fail-with-body --silent --show-error \
  --header "Authorization: Bearer ${GOLDEN_TOKEN}" \
  "${GOLDEN_URL}/api/golden/sample-customer-entity/duplicates/clusters?page=0&pageSize=100"

Each cluster reports the index that found it, its score and classification, and its escalation state. Read the structured classification, score, and mappingId fields. reasoning is human-readable and its wording can change. A group can also report mixedEvidence when its contributing comparisons do not all support the same conclusion.

For a cross-entity total, GET /api/golden/duplicates/record-counts returns the number of records carrying at least one possible duplicate, per entity.

Inspect comparison evidence

Use Duplicates → Expert to inspect the email group containing CRM-0001 and CRM-DUP-0001, and the name-and-postcode group containing CRM-NEAR-0001 and CRM-NEAR-0002. Set ANA_CLUSTER_ID and JAVIER_CLUSTER_ID from their current clusterId values and verify their members before taking an action.

read -r -p 'Ana cluster ID: ' ANA_CLUSTER_ID
read -r -p 'Javier cluster ID: ' JAVIER_CLUSTER_ID

evaluate compares every pair of records in a cluster and returns the per-column similarities behind the score. Start with Ana’s two CRM records:

curl --fail-with-body --silent --show-error \
  --header "Authorization: Bearer ${GOLDEN_TOKEN}" \
  "${GOLDEN_URL}/api/golden/sample-customer-entity/duplicates/clusters/${ANA_CLUSTER_ID}/evaluate"

The installed cleaner gives both CRM records the canonical phone +34611000000. They are classified MATCH; read the current pairs and similarities instead of reusing scores from an older sample installation. The score is evidence for review, not a probability of identity.

Compare a pair that must stay separate:

curl --fail-with-body --silent --show-error \
  --header "Authorization: Bearer ${GOLDEN_TOKEN}" \
  "${GOLDEN_URL}/api/golden/sample-customer-entity/duplicates/clusters/${JAVIER_CLUSTER_ID}/evaluate"

The two people share a name and postcode but have different tax identifiers. Their cluster remains REVIEW. Inspect the returned field comparisons before automating decisions. Candidate membership alone is not evidence of identity.

Choose an action

Action	Path suffix	Use it when
Merge	`/merge`	Every record represents the same subject
Merge a subset	`/merge/records`	Only some of them do
Preview a subset merge	`/merge/records/preview`	Before the above, to see the record it would produce
Disconnect	`/disconnect`	The records must no longer be treated as related
Disconnect selected	`/disconnect/records`	Record separations between selected and remaining records
Ignore, unignore	`/ignore`, `/unignore`	Defer the candidate without deleting its records
Escalate, resolve	`/escalate`, `/resolve-escalation`	A person with more context should rule
Delete	`/delete`, `/delete/records`	The records themselves should be removed

Delete is materially different from disconnect. Use it only when removal of the records is intended.

Merge a cluster

curl --fail-with-body --silent --show-error \
  --request POST \
  --header "Authorization: Bearer ${GOLDEN_TOKEN}" \
  --header "Content-Type: application/json" \
  --data '{"comment":"Matching NIF and a recognizable surname transposition"}' \
  "${GOLDEN_URL}/api/golden/sample-customer-entity/duplicates/clusters/${ANA_CLUSTER_ID}/merge"

The body takes comment and crossSeparations. Leave crossSeparations false unless the workflow explicitly permits overriding an existing separation.

Merge only part of a cluster

recordIds is required and must name at least two records, every one of them in the cluster.

curl --fail-with-body --silent --show-error \
  --request POST \
  --header "Authorization: Bearer ${GOLDEN_TOKEN}" \
  --header "Content-Type: application/json" \
  --data '{"recordIds":["CRM-0001","CRM-DUP-0001"],"comment":"Merge the two CRM records"}' \
  "${GOLDEN_URL}/api/golden/sample-customer-entity/duplicates/clusters/${ANA_CLUSTER_ID}/merge/records"

POST .../merge/records/preview takes the same body and computes the record the merge would produce without merging. Use it in an automated workflow that needs to check the outcome before committing to it.

Separations block a later merge

When a group holds two records somebody previously disconnected, the merge is refused:

{
  "errors": [
    "The records CRM-NEAR-0001 and CRM-NEAR-0002 were recorded as different, so they cannot be merged together. To merge them anyway, cross the separation explicitly and give a reason"
  ]
}

crossSeparations: true merges anyway and revokes that judgment.

Keep crossSeparations false unless the approved workflow permits an override. When overriding a separation, provide a comment identifying the rule and the reason for reversing the earlier decision.

Escalate a case

curl --fail-with-body --silent --show-error \
  --request PUT \
  --header "Authorization: Bearer ${GOLDEN_TOKEN}" \
  --header "Content-Type: application/json" \
  --data '{"comment":"Names agree but tax identifiers differ"}' \
  "${GOLDEN_URL}/api/golden/sample-customer-entity/duplicates/clusters/${JAVIER_CLUSTER_ID}/escalate"

The cluster’s escalated, escalatedBy, escalatedAt, and escalationNote fields are then set, and it appears in the web application’s Escalated cases queue. PUT .../resolve-escalation clears it and requires ADMIN.

Escalate when the available evidence does not meet the workflow’s decision criteria. The case remains available for an authorized reviewer.

Undo a merge

curl --fail-with-body --silent --show-error \
  --request POST \
  --header "Authorization: Bearer ${GOLDEN_TOKEN}" \
  --header "Content-Type: application/json" \
  --data '{"recordId":"<the surviving record>","comment":"Merged on a rule that has since been corrected"}' \
  "${GOLDEN_URL}/api/golden/sample-customer-entity/merge/undo"

recordId is required and names the record a merge produced. The absorbed records come back out of history, restored as they were before the merge plus any separation recorded against the survivor while it existed; the survivor is deleted and kept in history. Anything else edited on the survivor since the merge is not carried over.

Four refusals, each with its own message:

Condition	Message
The entity keeps no history	“the records a merge absorbed were not archived and the merge cannot be undone”
The record is not a merge survivor	“is not the result of a merge, so there is no merge to undo”
The survivor was merged again	“was itself merged into {other}. Undo that merge first, then this one”
A source is gone from history	“is no longer in history, so the merge that produced {record} cannot be undone”

Undo is not a general rollback. It reverses one merge, on a table that kept history, provided nothing has merged the survivor since. Do not design a recovery procedure that assumes it will always be available.

Verify the outcome

Re-read the cluster and confirm:

it has the expected state, or no longer appears under that filter;
the resulting mastered or disconnected records are correct; and
any task the action started reached a terminal success state.

If the result is unexpected, stop related decisions for the entity and follow the approved recovery or support process. Record the entity, cluster identifier, action, comment, and time.

Selected-record decisions

For a cluster, use POST .../merge/records/preview before applying POST .../merge/records. Both take recordIds; a merge requires at least two members of that cluster. POST .../disconnect/records separates the selected records from the remaining members in both directions, without separating the remaining members from each other. It preserves existing separations.

DELETE .../delete/records removes the selected records only. Ignore/unignore are group operations, with PUT .../ignore and PUT .../unignore, and require ADMIN. See the A/B/C exercise for a reproducible partial decision and checks on unselected records.

Steward duplicate candidates in the web application
How Golden resolves duplicates
Configure the classifier
Configure the merger
Configure a steward to run these rules on a schedule