Connect sources and load data

On this page

Golden file sources read CSV or JSON files that have first been uploaded to Golden. File operations require ADMIN.

Upload a file

Send the content as multipart/form-data with the required file and name parts. description is optional.

curl --fail-with-body --silent --show-error \
  --request POST \
  --header "Authorization: Bearer ${GOLDEN_TOKEN}" \
  --form "file=@tutorial-customers.csv" \
  --form "name=tutorial-customers.csv" \
  --form "description=Synthetic tutorial customers" \
  "${GOLDEN_URL}/api/files"

The supplied name, not the local filename, determines the stored filename and content type. Give it the .csv or .json extension expected by the file source.

When name ends in .gz, Golden decompresses the upload and removes the .gz suffix. For example, customers.csv.gz is stored and matched as customers.csv.

List and inspect files

curl --fail-with-body --silent --show-error \
  --header "Authorization: Bearer ${GOLDEN_TOKEN}" \
  "${GOLDEN_URL}/api/files?sorting=NAME&direction=ASC&pageNumber=0&pageSize=10"

sorting accepts NAME, SIZE, or CREATED; direction accepts ASC or DESC. Defaults are NAME, ASC, page 0, and page size 10.

Inspect one file’s metadata with GET /api/files/{file}. The response reports its identifier, stored name, description, creation time, content type, and length.

Download a file

curl --fail-with-body --silent --show-error \
  --header "Authorization: Bearer ${GOLDEN_TOKEN}" \
  "${GOLDEN_URL}/api/files/download/${FILE_ID}" \
  --output downloaded-file

Treat downloaded content according to its data classification. Do not place customer data in logs, documentation, or an unapproved local directory.

Remove a file

curl --fail-with-body --silent --show-error \
  --request DELETE \
  --header "Authorization: Bearer ${GOLDEN_TOKEN}" \
  "${GOLDEN_URL}/api/files/${FILE_ID}"

Deleting a file prevents future file-source runs from reading it. Confirm that no enabled flow still expects its stored name before removal.

Use the file in a source

Set a source-file resource’s inputPattern to the stored filename or a name pattern. It never accepts a filesystem path. See Configure sources for pattern and parsing rules.

Choose a source and run the load

Use ADMIN access for the load itself. Prepare a source and an existing target table with compatible datasets. Set SOURCE_ID and TABLE_ID to those reviewed objects; the target should be disposable while testing.

jq -n --arg source "$SOURCE_ID" --arg table "$TABLE_ID" \
  '{source:$source,sinkTable:$table,operation:"UPSERT",maxRecords:6,sampleRecords:-1}' |
curl --fail-with-body --silent --show-error \
  -H "Authorization: Bearer $GOLDEN_ADMIN_TOKEN" \
  -H "Content-Type: application/json" --data-binary @- \
  "$GOLDEN_URL/api/tables/load" > load-response.json
RUN_ID=$(jq -er '.run.id' load-response.json)

curl --fail-with-body --silent --show-error \
  -H "Authorization: Bearer $GOLDEN_ADMIN_TOKEN" \
  "$GOLDEN_URL/api/jobs/runs/$RUN_ID"

The example processes at most six input records. maxRecords:-1 removes that limit; choose it only after the controlled load is correct. sampleRecords:-1 disables sampling. Add transformation or pipeline only when the source and target flow requires them; transformation runs before the pipeline.

`operation`	Meaning
`INSERT`	Insert source records; the default if omitted
`UPSERT`	Insert or update according to the configured identity
`DELETE`	Remove the records identified by the input

A table load is different from entity synchronization. Use the entity phase masks when a flow also needs source-window control, indexing, classification or delivery.

Verify a source and its load

Test the resource and inspect parsing, identity and representative output.
Run the limited load above and follow its run to a terminal state.
Read the target records; compare counts, identifiers, values and quality.
Repeat the same UPSERT load in the disposable table and confirm stable identities do not create extra rows.
Inspect refusals and errors even when the run reports SUCCEEDED.
For an entity table, verify synchronization and search as required.

If the wrong records were written, stop recurring work and inspect the actual changes before retrying. A cancelled or failed run does not guarantee that no records were written. Restore or reload only through the recovery procedure approved for that target. Delete the temporary table and uploaded test file when the exercise is finished and no dependency still needs them.