Native connectors in pipelines
What certified routes handle for you: pagination, watermarks, and dependent endpoints.
When you build a pipeline on a Databasin Native connector, you pick endpoints from a list and Databasin already knows how to read each one. This article explains what that actually covers — because the difference between a certified route and "point a generic API connector at a URL" is mostly invisible until something goes wrong.
For the two-tier picture, start at How connectors work.
What a certified route already knows
Every route on a native connector ships with its extraction contract worked out and live-probed against the vendor:
| Handled for you | What it means |
|---|---|
| Pagination | How to walk every page, in the style this specific endpoint uses. |
| Incremental reads | Which field marks change, and how to ask the vendor for "only what's new." |
| Record extraction | Where the rows actually live in the response body. |
| Table naming | A stable target table name that doesn't change under you. |
| Dependent calls | Endpoints that need an id from another endpoint first. |
You don't configure any of it. You pick the endpoint and an ingestion mode; the rest is already in the profile.
Pagination, in the shapes vendors actually use
There's no single way APIs paginate, and getting it wrong means silently short reads. Certified routes declare which style applies:
- Page —
?page=1,2,3…, usually with a total-count header. - Offset —
?offset=0,100,200… - Stepped — page-based, but walked in a way that tolerates the vendor's quirks around page boundaries.
- Link — follow the
nextlink the response hands back. - Cursor / id cursor — keep asking for records after the last id you saw.
- Dependent — one call per parent record (below).
Incremental reads
A route that supports Delta declares its own change marker — the field the vendor updates when a record changes — and how to send it. Some vendors take a query parameter, some want a filter expression, some need it injected into a request body. That's part of the certified route, not something you write.
This is the practical difference from a generic connector: on a generic API you have to know that a vendor's modified_date only updates on insert, and design around it. On a certified route, that has already been probed.
Certification tells Databasin how to read incrementally. Whether a given table should be Delta, Snapshot, or Historical is still your call — see Ingestion modes.
Dependent routes (parent → child)
Some data can only be fetched one parent at a time. "Every work order for every property," "every time entry for every household" — the vendor has no endpoint that returns them all, so Databasin lists the parents, then makes one child call per parent. That's a dependent route, and it's where most of the recent work has gone.
Two things can go wrong with a fanout like this, and both are silent — you get a table with fewer rows than reality and no error to tell you.
Unstable parent ordering
If the parent listing isn't sorted on something stable and unique, records shift between pages while you're paging through it. A record that moves from page 3 to page 2 after you've read page 2 is never seen, so its children are never fetched.
Two things now prevent this:
- Dependent routes pin a stable, unique sort on the parent listing.
- A runtime integrity check watches for duplicate parent ids across pages — the tell-tale of a shifting window. It re-fetches once and raises a warning in the run's logs.
Result-window caps
Some vendors cap what a listing reports. Ask for a total and you get a round number like 9999 no matter the real count, and paging stops at a fixed window — 10,000 rows, say. Every parent past that window silently never fans out, and because nothing is duplicated, the integrity check has nothing to catch.
Affected routes now use cap recovery: instead of trusting the total, Databasin keeps asking for records past the last id it saw until the source genuinely runs dry.
Recovery makes a fanout complete, and complete costs time. One measured tenant went from 10,000 to roughly 28,800 child calls on a single route when recovery was switched on. If a dependent route's runtime jumps sharply after an upgrade, that's usually this — it was under-reading before.
Reading it in the run
Both guards surface in the pipeline's run logs as check-ins rather than failures. A warning about duplicate parent ids means the integrity guard fired — the run recovered, but it's worth a look at the source. See Monitoring and alerts.
When a connector is upgraded
Native connectors are versioned. When a route's extraction contract changes — a new pagination strategy, a cap-recovery fix — that's a new connector version.
Some route configuration is copied onto a pipeline's tables when the pipeline is created, so an already-built pipeline can keep running with the old settings after the connector is upgraded. Re-saving the connector doesn't necessarily push the change down either, because the setting lives at the table level.
If you're told a connector fix is out and your runs don't change, open the affected table in the pipeline and re-save it — or ask support to patch it. It's worth confirming rather than assuming.
What native connectors don't change
- There's still no transform step. Certified routes land faithful copies; reshaping happens downstream. See How pipelines work.
- Vendor rate limits still apply. Certification means Databasin paces itself correctly, not that the limits disappear.
- You still pick the tables. A certified connector exposes its useful routes; which ones you sync is your decision.