Product event reference

When an entity’s records change, Golden publishes a product event to each of the entity’s destinations. The event’s payload is what the destination receives. This page describes that payload, the events a clear or a full re-delivery produces, the order a receiver can rely on, and the product events Golden keeps for each destination.

The delivered payload

A payload is the record as stored, with its business fields, _id, _source and _metadata, after the destination’s optional transformation. Two _metadata members say what happened:

• _operation says what happened to the record: insert, update, remove, or clear.
• _operation_bucket says which candidate-group operation caused the change: merge, split, disconnect or unmerge. It is absent on a plain write — an insert, an update or a delete.

_operation_bucket names the operation of this write only. A plain write to a record that a merge, split, disconnect or undo touched earlier carries no _operation_bucket; a survivor still lists what it absorbed in _metadata._merged.

A record a merge absorbed also carries _metadata._merged_into with the survivor’s identifier. _metadata._history is true exactly when the record was written to the table’s history.

Audit history in a payload

When the entity’s destination entry has audit on, the default, the payload also carries _audit: the record’s audit events, oldest first, each with user, instant, type, comment, records and changes. records lets a receiver follow a merge from an absorbed record to its survivor and back. With audit off, the payload carries no _audit. The API never returns _audit with a record; read audit there through the record audit endpoint.

A restored survivor reads merge

A restored record that was itself a merge survivor reads merge, not unmerge, so that its own undo remains available. For example, merge M1 and M2 into S1, merge S1 and M3 into S2, then undo S2. S1 returns as a survivor with _operation: insert, _operation_bucket: merge and _merged: [M1, M2] — what it absorbed, never the survivor it was absorbed into. M3, which absorbed nothing, is published as unmerge and insert.

A receiver tells this restoration from a new merge survivor by its identifier: a merge always creates a new identifier, and S1 is one the receiver was earlier told was removed.

Members a transformation cannot remove

A destination transformation can reshape the payload, but _operation, _operation_bucket, _merged, _history and _updated are always kept, so a delete never reads as an insert. When the transformation fails, the event is stored as invalid for that destination with a concise error, and its payload is the record before the transformation; see Revalidate an event.

Clear events

A clear event tells a destination to leave its mirror of the table empty. Golden publishes one clear per destination when:

• an entity’s table is cleared;
• an entity is deleted; and
• an entity synchronization runs with sinkMask: FULL, before it restates every live record.

Clearing a table that belongs to no entity removes its queued events and publishes nothing. A table cannot be deleted while an entity uses it. Deleting it afterwards removes its queued events except clears, so the clear an entity’s deletion queued is still delivered when the table is deleted right after the entity.

{
  "_metadata": {
    "_operation": "clear",
    "_updated": "<publication time>"
  },
  "_clear": {
    "table": "sample_customer",
    "entity": "sample-customer-entity",
    "records": 378
  }
}

A clear has no _id and no _source, and a destination transformation is not applied to it. records is the number of records the table held; a clear is sent even when it is 0, because it means “leave the mirror empty” whatever the destination holds.

Before a clear is published, Golden removes the table’s queued events for that destination, pending and invalid alike: an invalid event revalidated after a clear would re-apply an old write. An entity’s deletion also discards its pending asynchronous writes. After a clear there is nothing of that table left to revalidate.

Full re-delivery rebuilds the mirror

A synchronization with sinkMask: FULL is a mirror rebuild. Each destination receives one clear, then every live record as insert with no _operation_bucket, all under the run’s single operation, the clear first. Each record is read as it stands when it is published, so a write during the rebuild is never followed by an older copy. Deleted records leave the mirror. A receiver that does not implement the clear is not emptied: it receives every live record again and keeps the ones Golden deleted.

How each destination applies a clear

A clear empties the whole destination; see Use an exclusive target.

Delivery headers

HTTP requests and Kafka messages carry these headers when the value is known:

A Kafka message is keyed by the record’s _id, and a clear by the table identifier. JDBC and Golden-table destinations receive no headers. An event with no recorded command, such as one queued before Golden 3.1.0, carries only the event identifier.

Order and grouping

Each event describes itself. The events of one operation are not grouped: they can interleave with other writes, so a receiver must not rely on adjacency. The sequence can have gaps — an event a receiver refused or a transformation failed on stays undelivered, and a clear removes queued events — so it cannot tell a receiver that it holds a whole command.

On Kafka, ordering relative to later messages is guaranteed only on a single-partition topic: Kafka orders within a partition, and a merge, an undo or a clear spans several message keys.

A revalidated event is queued at the end of the queue, behind newer events of its record. JDBC and Golden-table destinations ignore an event older than the destination row, so a late or revalidated event does not overwrite a newer row; the stamp rules are in JDBC destinations and Golden-table destinations.

Asynchronous entities publish the net effect

On an entity that indexes asynchronously, several writes to one record before indexing runs become one event, attributed to the last command that wrote the record. That event is the net effect of the writes:

A receiver of an asynchronous entity sees what a synchronous entity would send, compressed. A full re-delivery’s clear does not discard these pending writes: they publish the current record after the rebuild.

Changes that publish nothing

• Ignoring, unignoring, escalating or resolving a bucket or cluster: it is a judgment about the group, not a change to a record.
• A disconnect repeated for records already separated. A disconnect publishes update with _operation_bucket: disconnect only for each record whose _metadata._unrelated grew, carrying the grown list.
• A quality re-measurement: a definition change, the daily run, an on-demand measurement or a recalculation.

A payload’s _metadata._quality and qualityState are the quality as of publication. Read current quality through the API; a full re-delivery refreshes a mirror.

Product events in Golden

Golden keeps each destination’s events until they are delivered. Every events operation and the Events screen require ADMIN. An event reports:

GET /api/events/sink/{id} lists one destination’s events in publication order, oldest first. It accepts filter (ALL, the default, VALID or INVALID) and, combined with it and with each other, entityId, tableId, recordId, operation and operationId. operation ignores case; a value other than insert, update, remove or clear is a 400. Page with pageNumber and pageSize (default 10, at most 1000). Events queued before Golden 3.1.0 can leave them empty, and a filter on them does not match such events.

A refused delivery’s error is concise: the status, the receiver’s own words (bounded), and each failure’s type and message. A stalled delivery and a destination that cannot start also write their reason and count the attempt.

Revalidate an event

PUT /api/events/valid/{id} queues an invalid event again, at the end of the queue.

On a JDBC or Golden-table destination, revalidation is protected by the destination’s stamp, within the limits described in JDBC destinations: in particular, after a physical delete an older update finds no row and brings the record back. Read what an invalid event would write before revalidating it.

For the screens and the recovery procedure, see Recover tasks and deliveries.

Related reading

• Destination configuration
• Configure and verify deliveries
• Record metadata reference
• Record write reference