Setting up ingestion has always asked you to answer questions you can only really answer by digging: which of these tables can load incrementally, which column reliably tracks change, which columns make a row unique, is this file a CSV or something that merely looks like one.
Nobody enjoys that part. So Databasin now does the digging and shows you what it found.
Two screens instead of six
The database flow went from five or six screens down to two, and the file flow from three to two. You pick what you want to bring over, and the review screen arrives with a proposed plan for each object.
Sixteen database connectors are on the new flow, including Epic Clarity, along with five storage connectors for the file lane — SFTP, S3, ADLS, Box, and SharePoint.
It reads the evidence, it doesn't guess from names
The easy version of this feature is to look for a column called updated_at and hope. That approach breaks quietly and expensively.
Instead, detection reads real catalog metadata and runs bounded, read-only probes — a strict overall time budget, per-statement limits, and a hard rule that it never runs count(*) on your tables. Row counts come from statistics catalogs or they don't come at all.
The heuristics are the interesting bit, because they encode the mistakes:
- SQL Server's
rowversionis excluded as a watermark. It looks perfect and it isn't — it's a binary counter, not a time. - A candidate watermark that hasn't moved in 90 days while a sibling creation column has been updating gets flagged as stale rather than trusted.
- A nullable candidate isn't silently accepted — it's either paired with a creation-date sibling or probed for nulls first.
For files, Databasin samples several files per pattern rather than trusting the first one, and takes the consensus for format, delimiter, and header. When the files disagree with each other, it says so — mixed_dialect, header_disagreement — instead of picking one and letting you discover the problem in production.
"Files like this"
Turn any single file into a wildcard pattern with one click, and see how many files currently match before you commit to it.
And a small rule that removes an entire category of silent failure: a pattern that matches zero files won't save. The pipeline that runs green forever and never brings in a row is not a mystery we're willing to keep shipping.
You still decide
Detection is a recommendation, not an act. Every proposal is shown with what it was based on, and the file lane requires you to confirm before anything is created. Files default to a full snapshot; incremental loading is something you turn on, not something inferred on your behalf.
That's the line we're drawing: the machine does the tedious investigation, and you keep the judgment. You should be approving a plan you can see the reasoning for — not filling in six screens of fields you'd have to run experiments to answer honestly.
Connect a source and see what it proposes.