On this page
Destination configuration
Configure supported Kafka, HTTP, JDBC, and Golden-table destinations for Trazadera Golden export flows.
Destination resources use the sink type prefix. Golden supports Kafka, HTTP,
JDBC, and Golden-table sinks. The current resource model does not define a file
sink.
Every destination receives the payloads and clear events described in the product event reference.
Kafka destinations
{
"type": "sink-kafka",
"_id": "customer_kafka_sink",
"description": "Mastered customer delivery",
"dataset": "customer_dataset",
"bootstrapServers": "kafka1.example.com:9092",
"topicName": "mastered-customers",
"credentials": "kafka_credentials",
"timeoutMs": 5000,
"properties": [{"key": "acks", "value": "all"}]
}
Use properties only for producer options approved for the deployment. Keep
authentication material in the referenced credentials resource. Properties
are an array of key and value pairs, and their accepted names come from the
Kafka client used by the target deployment.
Each Kafka message has the record’s _id as its key and the record as a plain
JSON object in its value; a clear event is keyed by the table identifier.
Consumers should parse the value once as JSON and use the key to keep versions
of one record in the same partition. See
Order and grouping for
ordering across keys.
HTTP destinations
{
"type": "sink-http",
"_id": "customer_http_sink",
"description": "Mastered customer API",
"dataset": "customer_dataset",
"url": "https://receiver.example.com/customers",
"credentials": "receiver_credentials",
"timeoutMs": 5000,
"method": "POST"
}
The supported methods are POST, PUT, and PATCH. The receiver should use
stable record identity and define how a repeated request is handled because a
timeout can make delivery outcome uncertain. Which answers count as delivered,
retried or refused is in
HTTP delivery outcomes.
JDBC destinations
{
"type": "sink-jdbc",
"_id": "customer_warehouse_sink",
"description": "Analytics customer table",
"dataset": "customer_dataset",
"url": "jdbc:postgresql://warehouse.example.com:5432/analytics",
"credentials": "warehouse_credentials",
"table": "public.customers",
"logicalDelete": false,
"writeTimestampFlag": false,
"writeIdFlag": false,
"properties": [{"key": "ssl", "value": "true"}]
}
JDBC sinks can additionally configure an operation column (operationColumn),
logical-delete behavior (logicalDelete), and columns that receive write
identifiers (writeIdFlag, writeIdColumn) or timestamps
(writeTimestampFlag, writeTimestampColumn). Match those choices to the
target table contract and test insert, update, delete and clear cases.
The sink protects the destination from an older event. Its stamp column is
writeTimestampColumn when writeTimestampFlag is on, otherwise a column
named _updated when the table has one; Golden writes it as fractional
epoch seconds from the event’s _metadata._updated. An event, including a
remove, whose stamp is strictly older than the row’s is ignored: it is
delivered and consumed, and the row is untouched. An equal stamp, as in a full
re-delivery, applies.
The protection has limits:
- A destination with no stamp column applies every event.
- It is only as fine as the column. An integer column keeps whole seconds,
so an older event within the same second as the row applies. A
TIMESTAMPcolumn is not compared at all. Use a numeric column that keeps the fraction. - A row that is gone cannot be compared. After a physical delete, a revalidated older update finds no row and applies, bringing the record back.
- Under logical deletes, a row already marked
removeis never compared, so a rebuild can bring it back — and a stale update revalidated after a logical remove also applies.
Read what an invalid event would write before revalidating it.
A clear deletes every row of the destination table, or, with logical deletes,
marks every row remove.
Golden-table destinations
{
"type": "sink-table",
"_id": "processed_customer_sink",
"description": "Processed customer table",
"table": "processed_customers"
}
A table sink writes to an existing Golden table and uses that table’s dataset.
It applies the delivered operation through the destination dataset’s identity,
lineage and audit rules. A refusal marks that delivery invalid and names the
reason. The whole source _metadata is removed before the destination write.
Without a transformation, source and destination must share a dataset. An
event’s _id addresses the destination only when both tables share one;
otherwise a COLUMN or SCRIPT destination computes its own identity.
A new row on a dataset requiring _src_id needs a keyed _source entry.
An event whose stamp is strictly older than the last one applied to the row
is ignored; that event stamp is kept in _sink_event, while the destination
row’s _updated is its own last write time. A clear empties the table and
its audit history; an invalid target relation makes the event invalid.
Test create, update, removal and clear with the target table.
The target must be a table that belongs to no entity, the same rule a table load follows. Saving a table sink onto an entity’s table is refused. A delivery to one, configured earlier or whose table later became an entity’s, marks the event invalid naming the entity; so does a target table that no longer exists or refuses the write. A table sink publishes no events of its own: neither the rows it writes nor its clears reach any other destination.
Use an exclusive target
A clear empties the whole target — it deletes every row of a JDBC or
Golden-table target or, on a JDBC target with logical deletes, marks every row
remove — including rows Golden did not write. When several entities share
one JDBC or table destination, clearing any of them
empties their common target, the other entities’ rows included. Give each
mirror a destination and a target of its own. The JDBC and table sinks’ test
report states this.
Verify a destination
- Test the resource without saving it.
- Deliver a controlled record set through an entity configured with this destination. A table file export does not deliver records to a sink.
- Follow the resulting task.
- Verify record identity, field structure, update, removal and clear behavior at the target.
- Enable recurring work only after the target result is correct.