Every data governance programme hits the same wall about four months in. The warehouse has three hundred Fivetran-landed tables, a dbt project on top, a BI layer on top of that, and when someone from finance asks "where does customer_arr actually come from?" the honest answer is a screen share of the Fivetran connector schema tab.
Fivetran already knows the answer. It knows every source schema, table and column it replicates, the destination object each one lands in, and when it last changed. What it does not do by default is publish that into whatever catalog your governance team bought. This tutorial covers how to get it out and where to put it.
What metadata Fivetran actually exposes
There are two supply routes, and mature setups use both.
1. The Metadata API (REST). Under /v1/metadata/connectors/{connector_id}/... you can list:
- schemas - source schemas the connection replicates, and the destination schema they map to
- tables - source table name, destination table name, whether it is enabled
- columns - source column, destination column, type, primary-key flag, and whether it is hashed or blocked
- lineage - the source-object to destination-object edges Fivetran maintains
This is the authoritative view of ingestion lineage. It answers "which source column became this warehouse column", including renames and the column-level governance rules you set when blocking or hashing PII.
2. The Fivetran Platform Connector. Fivetran replicates its own operational metadata into your warehouse as tables: connector, destination, schema_change, transformation_runs, log, incremental_mar, plus the source_table_metadata, source_column_metadata, destination_table_metadata and destination_column_metadata families that carry the same lineage edges the API returns.
The difference matters. The API is live and good for event-driven catalog pushes; the Platform Connector is SQL-queryable, historical, and joins natively against your dbt manifest and information schema. If you already stood it up for pipeline observability, the lineage tables are sitting there unused.
What Fivetran does not give you: anything downstream of the landing tables. Transformation and BI lineage has to come from dbt, your warehouse's access history, or the catalog's own crawlers. Fivetran is the first hop, not the whole graph.
Pulling the lineage from the API
Auth is the standard Basic auth with your API key and secret, base64 encoded. Start by listing connections in a group, then walk the metadata endpoints:
AUTH=$(printf '%s:%s' "$FIVETRAN_KEY" "$FIVETRAN_SECRET" | base64 -w0)
# connections in a destination group
curl -s -H "Authorization: Basic $AUTH" \
"https://api.fivetran.com/v1/groups/$GROUP_ID/connectors" | jq -r '.data.items[].id'
# column metadata for one connection
curl -s -H "Authorization: Basic $AUTH" \
"https://api.fivetran.com/v1/metadata/connectors/$CONNECTOR_ID/columns?limit=1000" \
| jq '.data.items[] | {parent_id, name_in_source, name_in_destination, type_in_destination, is_primary_key, is_foreign_key}'
# the lineage edges
curl -s -H "Authorization: Basic $AUTH" \
"https://api.fivetran.com/v1/metadata/connectors/$CONNECTOR_ID/lineage?limit=1000" \
| jq '.data.items[]'
Three operational details that decide whether your sync job survives contact with a real account:
- Everything is cursor paginated. Follow
data.next_cursoruntil it is absent. A 500-connection account with wide Salesforce objects will run to tens of thousands of column records; code that reads only the first page silently publishes a partial catalog, which is worse than publishing none. - Rate limits are per account. Back off on 429 and run the crawl on a schedule, not in a loop per catalog request. Nightly is almost always enough - schema metadata does not change by the minute.
- Metadata is populated after the first successful sync. A freshly created connection returns empty collections. If your catalog job runs before the initial sync finishes, expect gaps rather than errors.
Wrap this in a small job - a Cloud Run service, a Lambda on a schedule, or an Airflow DAG next to the ones already orchestrating your syncs - that writes normalised nodes and edges to a staging table. Do not push straight from the API into the catalog; the staging table is what makes the pipeline debuggable and re-runnable.
The same thing in SQL, from the Platform Connector
If the metadata schema is already landing in your warehouse, you can build the ingestion lineage edge list without writing any code:
select
c.connector_name,
sts.name as source_table,
scm.name as source_column,
dts.name as destination_table,
dcm.name as destination_column,
dcm.type_in_destination,
dcm.is_primary_key
from fivetran_platform.source_column_metadata scm
join fivetran_platform.source_table_metadata sts on sts.id = scm.table_id
join fivetran_platform.destination_column_metadata dcm on dcm.source_column_id = scm.id
join fivetran_platform.destination_table_metadata dts on dts.id = dcm.table_id
join fivetran_platform.connector c on c.id = sts.connector_id
where c.connector_name = 'salesforce_prod'
Column and table names differ a little by Platform Connector version, so check your own information schema before assuming the join keys - but the shape holds: source column to destination column, through a connection. Materialise it as a dbt model called something like lineage__fivetran_edges and you now have a versioned, tested artefact instead of a script's side effect.
That model is also the join point for the rest of the graph. Fivetran destination tables are exactly the objects your dbt sources point at, so lineage__fivetran_edges joined to the dbt manifest.json sources gives you end-to-end source-column to mart-column lineage with no manual mapping.
Pushing it into the catalog
Atlan has a native Fivetran integration that ingests connections, sync runs and lineage, and renders the Fivetran hop upstream of your warehouse assets. Prefer it to a custom build; keep the API crawl for anything the integration omits. If you do build custom, Atlan's /lineage bulk API takes typed process entities - create one process per connection, with the source and destination columns as inputs and outputs, and a stable qualifiedName so re-runs update rather than duplicate.
Collibra wants a lineage harvester file or a Data Catalog API push. The pragmatic route is to emit Collibra's technical-lineage CSV format from your staging table and drop it where the harvester picks it up, rather than driving the API asset-by-asset. Map Fivetran connections to a "System" asset type and connections' source schemas to "Database" so the existing stewardship model still applies.
Alation uses the Virtual Data Source and lineage APIs. One practical constraint: get your fully qualified names identical to what Alation's warehouse crawler already produced (db.schema.table.column, same case) or you will create a parallel set of orphan assets next to the real ones.
OpenMetadata / DataHub are the easiest custom targets because both accept an explicit lineage payload. OpenMetadata even ships a Fivetran connector in its ingestion framework; for anything custom, PUT /v1/lineage with entity FQNs and a columnsLineage block gets you column-level edges. DataHub's equivalent is a MetadataChangeProposal with upstreamLineage aspects, emitted through the Python SDK.
Unity Catalog and Snowflake Horizon are a different case. They already track lineage inside the warehouse automatically, so what they are missing is the pre-warehouse hop. Pushing Fivetran metadata in as tags and comments is usually the highest-value move: stamp source_system, source_table, pii_handling and fivetran_connection as table and column tags, and suddenly your existing Unity Catalog governance queries can answer provenance questions without a second tool.
Keeping it honest as schemas drift
A catalog that is three weeks stale is a liability, because people trust it. Two things keep it current:
- Subscribe to schema change. The Platform Connector's
schema_changetable and Fivetran's webhooks both fire on new tables and columns. Trigger the metadata crawl on that signal instead of hoping the nightly job catches it. This is the same plumbing as a schema drift playbook - reuse it. - Reconcile, do not just append. Each run, diff the crawled edge set against what the catalog holds and soft-delete edges that disappeared. Connections get paused, tables get de-selected, columns get blocked. Without a delete path your lineage graph slowly fills with objects that no longer exist, and one bad review kills confidence in the whole thing.
Also publish the governance flags, not only the shape. Which columns are hashed, which are blocked, which connection is in history mode - that metadata is what turns a lineage diagram into something a privacy officer can use for a DSAR or an audit response.
A sane scope for the first two weeks
Do not start with three hundred connections and a full governance model. Start with the two or three connections that feed your most-disputed dashboard:
- Stand up the Platform Connector if it is not already syncing.
- Build
lineage__fivetran_edgesas a dbt model and eyeball it against the Fivetran UI for one connection. - Join it to your dbt manifest sources so one column traces from source system to mart.
- Push that slice into the catalog and show it to the person who keeps asking the provenance question.
- Only then generalise: all connections, schema-change triggers, reconciliation, PII flags.
The value shows up at step 4, and step 4 is reachable in days. Governance programmes that try to boil the ocean first usually deliver nothing anyone uses.
Getting help with it
Lineage work is unglamorous and it is exactly the sort of thing that sits in a backlog for a year. SyncSpur builds these metadata pipelines as part of platform engagements - Metadata API crawlers, Platform Connector models, and catalog pushes into Atlan, Collibra, OpenMetadata and Unity Catalog, with the reconciliation logic that keeps them trustworthy.
If your catalog has a Fivetran-shaped hole in it, get in touch, or read how we approach Fivetran platform setup and management and Fivetran data warehousing consulting.