On this page
Source configuration
Configure the supported file, JDBC, and Golden-table sources that supply records to Trazadera Golden data flows.
A source describes where a Golden table load reads records. Golden supports file, JDBC, and existing-table sources.
Provenance on a load
File, JDBC and table sources can set provenanceSource (the source name),
provenanceKey (the source field containing its record identifier) and
stampsProvenance (whether to add that pair). Golden no longer copies the
record’s identity into _src_id. A stamping source feeding a dataset that
requires _src_id must supply a valid provenanceKey; otherwise saving it or
starting its load is refused. For a copy that should keep the existing
origins, set stampsProvenance: false.
When provenanceKey: _id, the incoming rows must actually carry _id.
For a COLUMN or SCRIPT dataset, a JDBC source that reads _id must use
provenanceKey: _id; a JSON file with an unexpected _id fails when that
row is read. Test each source against representative records before a load.
File sources
source-file supports CSV and JSON formats.
{
"type": "source-file",
"_id": "customer_csv_source",
"description": "Customer CSV extracts",
"dataset": "customer_dataset",
"inputPattern": "customers*.csv",
"format": "CSV",
"separator": ",",
"header": true,
"ignoreQuotes": false
}
| Property | Purpose |
|---|---|
dataset | Structure produced by the source |
inputPattern | Stored filename or name pattern for files uploaded to Golden |
format | CSV or JSON |
separator | CSV separator; default ; |
header | Whether CSV includes a header row |
ignoreQuotes | Controls CSV quote handling; default false, so quoted separators and line breaks remain inside their fields |
encoding | File character encoding: UTF-8 (default), windows-1252, or ISO-8859-1 |
goldenExport | Treat the file as a Golden export, preserving its supported record members |
inputPattern is matched against complete stored filenames, not a filesystem
path. * matches any run of characters and ? matches exactly one character.
For example, customers*.csv matches customers-01.csv but not
a_customers.csv. Golden reads every matching file in turn as one record
sequence. Upload the files before running the source.
CSV files without a header map values to dataset columns by position. With a
header, names must match dataset column keys; ordinary undeclared columns are
dropped. An unmapped header beginning with _ is refused unless it is _id,
_source, _xml, _text or _binary.
On upgrade, stored file sources that ignored quotes are switched to
ignoreQuotes: false. If a source deliberately relied on ignoring loosely
used quotes, set it back explicitly and test a representative file.
CSV cannot represent nested dataset columns, so use JSON when the dataset has them. JSON input must be one array of objects; validate representative file sizes in the target environment before adopting it for a large exchange.
JDBC sources
{
"type": "source-jdbc",
"_id": "orders_jdbc_source",
"description": "Orders database",
"dataset": "order_dataset",
"url": "jdbc:postgresql://db.example.com:5432/orders",
"credentials": "orders_credentials",
"table": "public.orders",
"timestampFilteringFlag": true,
"timestampFilteringColumn": "updated_at",
"readTimestampFlag": false,
"readIdFlag": false,
"customScripts": false,
"properties": [{"key": "ssl", "value": "true"}]
}
| Property group | Purpose |
|---|---|
url, credentials, properties | Connection and driver settings |
table, count | Source table and optional expected count behavior |
timestampFilteringFlag, timestampFilteringColumn | Timestamp-based filtering support |
readTimestampFlag, readTimestampColumn | Read a timestamp value from the source |
readIdFlag, readIdColumn | Read a source identifier column |
customScripts, queryScript, countQueryScript | Use approved custom read queries |
Reference a credentials resource rather than embedding a user name or password.
Connection properties are name/value pairs passed to the JDBC driver; the
driver determines which names it accepts. See
Configure connection credentials for the supported
credential-type mapping.
When timestamp filtering is enabled, verify that the selected source column
changes for every source update you expect Golden to load.
The source’s stored lastExecution becomes its incremental watermark. After
the first run, timestamp filtering rereads an overlap below that watermark
(60 seconds by default) to avoid losing boundary rows. A transiently lost
row holds the watermark back; check the run’s messages and refusal counts.
A SQL NULL empties the stored value on update; see
Empty values in a load
before using a source that leaves columns NULL to mean “no change”.
Golden-table sources
{
"type": "source-table",
"_id": "staging_customer_source",
"description": "Customer staging table",
"table": "staging_customers"
}
A table source reads a table already managed by Golden. Its dataset comes from
that table, so it does not declare a separate dataset property.
When source and destination use the same dataset, a carried _id addresses
that identity. When they use different datasets, that _id is foreign: a
DEFAULT destination may use it as a candidate, while a COLUMN or
SCRIPT destination computes its own identity. Use
stampsProvenance: false for a copy that should retain its origins.
Verify a source
Test the resource before saving it, then run a controlled table load and confirm the returned record count and field shape. Source configuration alone does not start ingestion; an administrator invokes or schedules the associated data operation.