Live sources on Sail

A live source lets you query a system where its data lives: an API, files in cloud storage, a database or Databricks Unity Catalog. There's no pipeline, no copy and no schedule. Each query fetches fresh data from the source when it runs.

Live sources run on Databasin Sail clusters, and on Trino clusters where your environment is set up to run them. DuckDB and Databricks connections don't have them.

Live or a pipeline?

Live data isn't stored anywhere. To keep history, or to make a query fast and repeatable, build a pipeline for that source; there's no switch that turns a live source into a pipeline.

Which sources can be live

How to tell if an API connector can be live. Open the Live Sources panel (below). Each connector shows Make live if it can be, Live if it already is, or a Lakehouse only chip if it can't; hover the chip for the reason. Connectors that sign in with a user-consent OAuth flow (Salesforce and Google Analytics, for example) are Lakehouse only today. Some connectors can be live for only part of their tables: a table the live engine can't read is left out of the listings, and querying it by name tells you to sync it instead.

Files. Each file at the top of the container or folder becomes a table, named after the file. Subfolders aren't read, and files larger than 256 MB are skipped and don't appear as tables. Column types come from the first 200 rows.

Unity Catalog. Sail reads the Delta tables directly with credentials Unity Catalog issues, so no Databricks compute runs. Views, materialized views, streaming tables, non-Delta tables, tables with row filters or column masks, and storage the cluster can't reach are refused with a message saying why.

Make a source live

1. In the SQL editor or a notebook, open the Object Explorer, then Other Data Sources → Live Sources. (Or, from Integrations → Create integration, pick Set up a live connection.)
2. Click Make live on a connector.
3. On Sail, name the catalog. Names use lowercase letters, digits and _, up to 63 characters, and must be unique in your organization. Names such as live, sandbox and your lakehouse catalogs are reserved.

The source appears in the Object Explorer within about a minute. On Sail, open a new session if yours started before the source went live. Rename changes a Sail catalog's name later; queries, notebooks and BI connections that use the old name stop working.

Admins can query a source as soon as it's live. For everyone else, access comes from your project's access rules: for live databases, files and Unity Catalog, users an admin adds to the connection's permissions are given access. If you can't see a live source, ask your project admin.

Naming

On Sail there's no live catalog: each live source is a catalog of its own. BI tools on Sail can land in one with the HTTP path cliservice/<catalog>.

API tables are named after the connector's data model. Some have extra param_… columns: filtering on one sends that value to the API as a request parameter.

How a live query runs

• One streaming read per live table. Each live table in a query is read by one reader that pages through the source. More Sail workers don't speed up that read; they help with the rest of the query.
• A page budget. An API read stops after 1,000 pages by default and the query fails with a message that explains the options: filter on a column that's sent to the source, raise the budget (on Trino, SET SESSION live.page_budget = <n>), or sync the table.
• Rate limits. Requests to a source are limited per connection on each node, to 5 per second unless the connector sets its own limit.
• Timeouts and retries. Each request can take up to 180 seconds. Rate-limited requests wait as long as the source asks (Retry-After, up to 120 seconds); server errors and timeouts retry with a growing wait, up to 5 times.
• Starting up on Sail. The first live query on a node starts the live service. If it isn't ready within 120 seconds, you see "Live sources are starting; try again in a moment."
• Read-only. Live sources answer SELECT only.

Data types

• Timestamps from API sources arrive as a timestamp without a time zone, holding the UTC time to the millisecond (TIMESTAMP(3) on Trino).
• Arrays and objects arrive as one text column holding JSON. Use from_json or get_json_object to read inside them.
• A value that can't be read as its column's type becomes NULL.

A live table can therefore have a different shape from the synced table for the same source.

Pushdown: what's sent to the source

How fast a live query is depends mostly on how much of it the source does.

• Filters on columns the source can filter by are sent to it. A filter on an ID, a date or a param_… column can turn a full scan into a few requests.
• LIMIT is sent only when nothing else is left to filter in the engine.
• Not sent: column selection, aggregations (COUNT, SUM, …), ORDER BY … LIMIT, and joins. These run in the engine after the rows arrive.
• Files have no pushdown: each query reads the whole file.
• Databases on Sail get filters turned into SQL for the database, and LIMIT too when every filter could be sent.

The practical rule: filter the live side, and keep it on the small side of a join.

Joining live data with your lakehouse

Live catalogs sit in the same query as your lakehouse tables:

SELECT c.name, sum(o.total) AS revenue
FROM crm.public.customers c
JOIN gold.sales.orders o ON o.customer_id = c.id
WHERE c.region = 'west'
GROUP BY c.name;

The join and the aggregation run on the cluster. The filter on c.region reaches the database, so only the western customers are read.

Security

• Credentials stay with Databasin. Source credentials are kept in Azure Key Vault and handed to the cluster when a query needs them. They're never shown to users or written into catalog files.
• Access rules decide who can query. A user without access is refused before anything is sent to the source.
• Each Sail cluster has its own live token, scoped to its project, so a cluster sees only that project's live sources.

Turning live off

Click Turn off (✕) on the source in the Live Sources panel. Its catalog disappears within about a minute, and on Sail the name is free again. Nothing is deleted: pipelines that use the connector keep running, access rules stay in place, and you can make it live again at any time.

On Trino, a database you connected as its own catalog has no off switch here; remove it through catalog management.

Limits and what's planned

Today:

• API sources page through the source one request at a time, so large unfiltered tables are slow and use your API quota.
• Some pagination styles can't run live yet; those tables say to sync instead.
• There's no cache: every query fetches again.

Planned:

• Snowflake, BigQuery and Oracle as live databases on Sail.
• More pagination styles for API connectors.
• OAuth sources that use a user-consent sign-in, such as Salesforce and Google Analytics.
• More pushdown: aggregations, top-N and joins sent to the source.

Where to go next