Sinks

Purpose

Sinks write data from Golden to external systems.

Sink Types

SinkKafka - Apache Kafka Sinks

Stream records to Kafka topics.

JSON

{
  "type": "sink-kafka",
  "id": "customer_events_sink",
  "description": "Stream customer events",
  "dataset": "customer_event_dataset",
  "credentials": "kafka_credentials",
  "topicName": "customer-events",
  "bootstrapServers": "kafka1:9092,kafka2:9092",
  "timeoutMs": 5000,
  "properties": {
    "acks": "all",
    "compression.type": "snappy"
  }
}

Configuration:

topicName - Target Kafka topic
bootstrapServers - Kafka broker addresses
credentials - Reference to Kafka credentials
properties - Additional Kafka producer properties

SinkHttp - REST API Sinks

Push records to HTTP endpoints.

JSON

{
  "type": "sink-http",
  "id": "mdm_api_sink",
  "description": "Push to MDM API",
  "dataset": "customer_dataset",
  "url": "https://mdm.example.com/api/customers",
  "method": "POST",
  "credentials": "api_credentials",
  "timeoutMs": 10000
}

Configuration:

url - Target API endpoint
method - HTTP method (POST, PUT, PATCH)
credentials - Optional authentication
timeoutMs - Request timeout

SinkJdbc - Database Sinks

Write records to relational databases.

JSON

{
  "type": "sink-jdbc",
  "id": "warehouse_sink",
  "dataset": "customer_dataset",
  "url": "jdbc:postgresql://warehouse:5432/analytics",
  "credentials": "warehouse_credentials",
  "table": "public.customers"
}

SinkFile - File Export Sinks

Export records to files.

JSON

{
  "type": "sink-file",
  "id": "export_csv_sink",
  "dataset": "customer_dataset",
  "outputPattern": "/exports/customers_{date}.csv",
  "format": "CSV"
}

SinkTable - Internal Table Sinks

Write to Golden tables.

JSON

{
  "type": "sink-table",
  "id": "target_table_sink",
  "table": "processed_customers"
}

Common Sink Properties

All sinks support:

dataset - Output data structure
timeoutMs - Operation timeout (default: 2000ms)