On this page

A source describes where a Golden table load reads records. Golden supports file, JDBC, and existing-table sources.

Provenance on a load

File, JDBC and table sources can set provenanceSource (the source name), provenanceKey (the source field containing its record identifier) and stampsProvenance (whether to add that pair). Golden no longer copies the record’s identity into _src_id. A stamping source feeding a dataset that requires _src_id must supply a valid provenanceKey; otherwise saving it or starting its load is refused. For a copy that should keep the existing origins, set stampsProvenance: false.

When provenanceKey: _id, the incoming rows must actually carry _id. For a COLUMN or SCRIPT dataset, a JDBC source that reads _id must use provenanceKey: _id; a JSON file with an unexpected _id fails when that row is read. Test each source against representative records before a load.

File sources

source-file supports CSV and JSON formats.

{
  "type": "source-file",
  "_id": "customer_csv_source",
  "description": "Customer CSV extracts",
  "dataset": "customer_dataset",
  "inputPattern": "customers*.csv",
  "format": "CSV",
  "separator": ",",
  "header": true,
  "ignoreQuotes": false
}
PropertyPurpose
datasetStructure produced by the source
inputPatternStored filename or name pattern for files uploaded to Golden
formatCSV or JSON
separatorCSV separator; default ;
headerWhether CSV includes a header row
ignoreQuotesControls CSV quote handling; default false, so quoted separators and line breaks remain inside their fields
encodingFile character encoding: UTF-8 (default), windows-1252, or ISO-8859-1
goldenExportTreat the file as a Golden export, preserving its supported record members

inputPattern is matched against complete stored filenames, not a filesystem path. * matches any run of characters and ? matches exactly one character. For example, customers*.csv matches customers-01.csv but not a_customers.csv. Golden reads every matching file in turn as one record sequence. Upload the files before running the source.

CSV files without a header map values to dataset columns by position. With a header, names must match dataset column keys; ordinary undeclared columns are dropped. An unmapped header beginning with _ is refused unless it is _id, _source, _xml, _text or _binary. On upgrade, stored file sources that ignored quotes are switched to ignoreQuotes: false. If a source deliberately relied on ignoring loosely used quotes, set it back explicitly and test a representative file.

CSV cannot represent nested dataset columns, so use JSON when the dataset has them. JSON input must be one array of objects; validate representative file sizes in the target environment before adopting it for a large exchange.

JDBC sources

{
  "type": "source-jdbc",
  "_id": "orders_jdbc_source",
  "description": "Orders database",
  "dataset": "order_dataset",
  "url": "jdbc:postgresql://db.example.com:5432/orders",
  "credentials": "orders_credentials",
  "table": "public.orders",
  "timestampFilteringFlag": true,
  "timestampFilteringColumn": "updated_at",
  "readTimestampFlag": false,
  "readIdFlag": false,
  "customScripts": false,
  "properties": [{"key": "ssl", "value": "true"}]
}
Property groupPurpose
url, credentials, propertiesConnection and driver settings
table, countSource table and optional expected count behavior
timestampFilteringFlag, timestampFilteringColumnTimestamp-based filtering support
readTimestampFlag, readTimestampColumnRead a timestamp value from the source
readIdFlag, readIdColumnRead a source identifier column
customScripts, queryScript, countQueryScriptUse approved custom read queries

Reference a credentials resource rather than embedding a user name or password. Connection properties are name/value pairs passed to the JDBC driver; the driver determines which names it accepts. See Configure connection credentials for the supported credential-type mapping. When timestamp filtering is enabled, verify that the selected source column changes for every source update you expect Golden to load. The source’s stored lastExecution becomes its incremental watermark. After the first run, timestamp filtering rereads an overlap below that watermark (60 seconds by default) to avoid losing boundary rows. A transiently lost row holds the watermark back; check the run’s messages and refusal counts.

A SQL NULL empties the stored value on update; see Empty values in a load before using a source that leaves columns NULL to mean “no change”.

Golden-table sources

{
  "type": "source-table",
  "_id": "staging_customer_source",
  "description": "Customer staging table",
  "table": "staging_customers"
}

A table source reads a table already managed by Golden. Its dataset comes from that table, so it does not declare a separate dataset property. When source and destination use the same dataset, a carried _id addresses that identity. When they use different datasets, that _id is foreign: a DEFAULT destination may use it as a candidate, while a COLUMN or SCRIPT destination computes its own identity. Use stampsProvenance: false for a copy that should retain its origins.

Verify a source

Test the resource before saving it, then run a controlled table load and confirm the returned record count and field shape. Source configuration alone does not start ingestion; an administrator invokes or schedules the associated data operation.

Golden 3.4.0 · Published 2026-10-29