Every post on this blog so far assumes the source holds state: a database you can read a log from, a SaaS API you can page through, a bucket of files. Event streams do not work that way. A Kafka topic is a moving window of bytes with a retention clock on it, and Fivetran's batch sync model has to be mapped onto it deliberately or you end up with gaps, duplicates, or a schema that is one giant VARCHAR column.
This tutorial covers getting Kafka, Confluent Cloud, AWS Kinesis, Azure Event Hubs and Google Pub/Sub into a warehouse with Fivetran: how the connectors actually consume, what happens to JSON and Avro payloads, where offsets and retention bite you, what it does to your MAR bill, and when you should not use Fivetran for this at all.
First: should this be a Fivetran job?
Be honest about the requirement before you build anything.
| Requirement | Right tool |
|---|---|
| Analytics on event data, minutes of latency acceptable | Fivetran streaming connector — yes |
| Sub-second materialisation into an app or feature store | Stream processor (Flink, ksqlDB, Spark Structured Streaming) |
| Topic is a CDC feed produced by Debezium from a database you own | Point Fivetran at the database directly instead |
| Topic carries 50k+ events/second of telemetry | Native sink connector to object storage, then Fivetran's file or Iceberg path |
| Events must be joined to warehouse dimensions for reporting | Fivetran streaming connector — yes |
The third row is the one that saves the most money. Teams regularly ask us to sync a Debezium topic into Snowflake through Fivetran, when Fivetran's own Postgres or MongoDB log-based connectors would read the same database with less moving parts and a saner schema. If you already run the Kafka pipeline for other consumers, syncing it is fine. If you were going to build it for the warehouse, do not.
The fifth row is the escape hatch for volume. Fivetran prices on monthly active rows, and a high-volume telemetry topic is nothing but active rows. Sinking the topic to S3 or ADLS in Parquet and landing it with the managed data lake or a file connector is usually cheaper by an order of magnitude at that scale.
How the connectors consume
Fivetran's event connectors act as a consumer group against your stream. The shape differs slightly per platform:
- Apache Kafka / Confluent Cloud — Fivetran joins as a consumer group (default name is derived from the connection), subscribes to the topics you select, and commits offsets as it syncs. One topic becomes one destination table.
- AWS Kinesis — reads shards with a stored iterator position; one stream becomes one table.
- Azure Event Hubs — consumed via the Kafka-compatible endpoint in most deployments, so configure it like Kafka with SASL.
- Google Pub/Sub — Fivetran reads from a subscription, not the topic. Create a dedicated pull subscription for Fivetran; do not share one with another consumer, or you will each get half the messages.
The critical mental model: the connector only sees what is in the stream while it is running. There is no backfill from before the connection existed beyond what retention still holds, and there is no re-reading history after retention expires. Everything below follows from that.
Step 1 — Create a dedicated principal and subscription
Do not reuse an application's credentials.
For Confluent Cloud, create a service account with a dedicated API key and grant only:
READ on TOPIC events.orders (and each topic you sync)
READ on GROUP fivetran_prod_
DESCRIBE on CLUSTER
Grant the group ACL with a prefix so Fivetran can create its own consumer group. If you run Schema Registry, add READ on the relevant subjects too — without it, Avro payloads arrive as unreadable bytes.
For Pub/Sub, a dedicated subscription plus a service account with roles/pubsub.subscriber on that subscription only.
Step 2 — Decide retention before you decide anything else
Set topic retention to at least 3x your worst realistic outage. The arithmetic is unforgiving:
- Sync frequency 15 minutes, retention 24 hours: you can tolerate a broken connector over a weekend? No. You lose everything older than 24 hours, permanently, with no re-sync possible.
- Retention 7 days: a Friday-evening credential expiry that nobody notices until Monday is recoverable.
If the stream is the system of record for anything that matters, use compacted topics or tiered storage (Confluent) so that history survives. This is the single biggest difference between operating a streaming connector and a database connector: with CDC you can always re-sync from the source table; here, the source can genuinely disappear.
Step 3 — Network path
Public Confluent Cloud clusters take Fivetran's region IP allowlist plus SASL_SSL. Self-managed Kafka in a private VPC needs the same options as any private source — PrivateLink, VPN, SSH tunnel — covered in detail in the private networking comparison. One Kafka-specific trap: brokers advertise their own listener hostnames. If advertised.listeners returns internal DNS names that Fivetran cannot resolve through your tunnel, the bootstrap connection succeeds and every subsequent fetch fails. Fix the advertised listeners, not the tunnel.
Step 4 — Message format and schema
This is where most streaming syncs go wrong.
Avro with Schema Registry is the best case. Supply the registry URL and credentials, and Fivetran unpacks the record into typed columns and tracks schema evolution as the subject version moves. Use BACKWARD compatibility on the subject so added fields are additive; a breaking change here behaves like a source schema change and triggers the same handling described in the schema drift playbook.
JSON without a registry is the common case. Fivetran infers columns from the message body, which means:
- A field absent from the first messages it sees will be added as a column later, not backfilled into earlier rows.
- Mixed types across producers (
"order_id": 123and"order_id": "123") land as a string column or a type-change event. - Deeply nested objects are stored as JSON-typed columns, not exploded into tables. Plan to unpack them downstream with
PARSE_JSON/JSON_VALUEin dbt.
Protobuf or an internal binary format generally means writing a custom connector with the Connector SDK or decoding upstream. Do not plan on Fivetran parsing a bespoke envelope.
Practical advice regardless of format: insist producers put a message-level event timestamp and a stable event ID in the payload, not only in the broker metadata. You will need both for deduplication, and broker metadata columns vary by platform.
Step 5 — Primary keys and duplicates
Streaming connectors are at-least-once. A connector restart between a sync and an offset commit re-delivers messages. Fivetran typically keys event tables on a synthetic combination of partition and offset (Kafka) or sequence number (Kinesis), which deduplicates redelivery of the same broker record but does nothing about a producer that published the same logical event twice.
So add a deduplication layer in the warehouse. A minimal pattern:
-- models/staging/stg_orders_events.sql
with src as (
select
payload:event_id::string as event_id,
payload:occurred_at::timestamp_ntz as occurred_at,
payload as payload,
_fivetran_synced
from {{ source('kafka_events', 'events_orders') }}
),
ranked as (
select *,
row_number() over (
partition by event_id
order by occurred_at desc, _fivetran_synced desc
) as rn
from src
)
select * exclude (rn) from ranked where rn = 1
Run it incrementally on _fivetran_synced so you are not re-scanning the full event table every build. If you do not have a producer-supplied event_id, hash the business-meaningful fields and use that — but push back on the producers first; it is a five-minute change on their side and a permanent tax on yours.
Step 6 — Sync frequency and MAR
Every message that lands is an active row. There is no "it only counts once per month" relief the way there is for a slowly-changing dimension synced repeatedly — each event is a new primary key, so each is new MAR. A topic at 20 events/second is roughly 52 million rows a month. Price that against your plan before you turn it on; the mechanics are in the connection-level MAR guide.
Sync frequency does not change the MAR (the same events land either way), but it does change your warehouse compute bill, because each sync is another MERGE/COPY cycle against the destination. For event tables specifically:
- Append-only event data is cheaper to load than keyed upserts — avoid giving the table an update-heavy key design.
- 15 minutes is a reasonable default. Going to 5 minutes multiplies warehouse wake-ups for marginal freshness; see the sync frequency and Snowflake cost posts.
- Cluster or partition the destination table on the event timestamp, not on load time, or every downstream query scans everything.
Step 7 — Monitoring that is specific to streams
Standard Fivetran alerting tells you when a sync fails. For streams you need two extra signals, and only one of them comes from Fivetran:
- Consumer group lag, from the broker side (Confluent metrics API, Kafka
kafka-consumer-groups --describe, KinesisGetRecords.IteratorAgeMilliseconds, Pub/Subsubscription/oldest_unacked_message_age). Alert when lag approaches retention. This is your real early warning; a sync can be "succeeding" while falling steadily further behind. - Row-count-per-interval anomalies in the destination, from the Platform Connector dashboard. A topic that quietly drops to zero events because a producer was decommissioned looks identical to a healthy pipeline otherwise.
A useful alert threshold: oldest_unacked_message_age > 0.5 * retention. That gives you half your retention window to react before data loss becomes permanent.
Failure runbook
| Symptom | Likely cause | Action |
|---|---|---|
| Connector syncs, zero rows | Reading a shared Pub/Sub subscription, or another consumer in the same Kafka group | Give Fivetran its own subscription / group |
| Bootstrap OK, fetch times out | advertised.listeners returns unresolvable hosts | Fix broker advertised listeners |
| Payload lands as one base64-ish string column | Avro without Schema Registry access | Grant registry ACLs, re-create the connection |
| Duplicate business events downstream | Producer retries | Dedup model on event_id (Step 5) |
| Gap in the data after an outage | Retention expired before recovery | Unrecoverable from the stream; replay from the producer's own archive if one exists |
| MAR spike after a release | New high-volume topic selected, or a producer loop | Check connection-level MAR, deselect the topic, throttle the producer |
The fifth row is why Step 2 matters more than any other step here.
A sane target architecture
For most analytics teams the pattern that holds up looks like this:
- Business-event topics (orders, signups, payments, entitlement changes) — moderate volume, high value — go into Fivetran as streaming connections at 15-minute frequency.
- High-volume telemetry and clickstream — sink to object storage in Parquet, land through the managed data lake path, query as Iceberg.
- Anything that is really database CDC wearing a Kafka costume — replace with a direct Fivetran database connector.
- One dbt staging layer that deduplicates and flattens every event table identically, so analysts never touch raw JSON.
That split keeps the expensive, fragile part of the stack small, and it keeps your MAR attached to data that someone actually reports on.
Where SyncSpur fits
We implement and operate this pattern for clients running Confluent, MSK, Event Hubs and Pub/Sub alongside conventional database and SaaS sources — including the parts that are not strictly Fivetran work, like getting producers to emit a stable event ID and setting retention that matches your recovery expectations.
If you are weighing a streaming connection against a sink-to-lake approach, or you have a topic that is already costing more in MAR than it returns in insight, get in touch and we will size it with you before you build it.