How Sail works
How a Databasin Sail cluster runs a query: coordinator and workers on Azure Container Apps, open Iceberg and Delta tables, catalogs, access rules, and when workers sleep and wake.
This page explains what happens between pressing Run and seeing rows: which parts of a Sail cluster do what, where your tables live, which catalogs you see, how access is checked, and when the cluster sleeps and wakes.
The parts of a cluster
A Sail cluster runs in Azure Container Apps. Each node is its own container:
| Part | What it does |
|---|---|
| Front door | Runs on every node and is the only way in. It checks every connection before anything reaches the engine. |
| Coordinator | Plans every query, checks your access rules, answers metadata, and hosts the notebook service for Python cells. |
| Workers | Run the work the coordinator hands them: scans, joins, aggregations and writes. Every size comes with 1 to 5. |
The engine is written in Rust on Apache DataFusion and speaks Spark Connect, the protocol PySpark uses to talk to a remote Spark server. That's why Spark SQL and PySpark code runs on Sail without a Spark installation of your own.
How a query runs
- You send a statement from the SQL editor, a notebook, Databasin One, a BI tool or your own PySpark session.
- The coordinator plans it and checks every table the statement touches against your access rules. A statement that touches something you can't see stops here.
- Workers run the plan. Each worker runs several pieces of the query at once (its task slots). When one stage needs data from another, workers exchange it directly with each other (the shuffle).
- Results come back in partition order, the way Spark returns them. The SQL editor shows them as they arrive.
A few details worth knowing:
- Metadata doesn't need workers.
SHOW,DESCRIBE,USE,information_schemaand the catalog browsing BI tools do when they connect are answered by the coordinator alone, so they come back at once even while workers sleep. - Single-node mode. With the worker count set to 0, the coordinator runs every query itself. That suits light use and small data.
- Queries start on the first workers to arrive. After a quiet period, a query doesn't wait for every worker to start.
- Memory. When a query needs more memory than a node has, Sail spills to the node's local disk, joins included.
- Failures. A failed task is retried, up to three times. Losing a worker while a statement is writing its final output fails that statement, and you run it again.
- A local cache of file reads. Each node keeps recently read pieces of your table files on its local disk, so repeat queries read less from storage. It checks each file's version, so a rewritten file is never served stale.
df.cache(), df.persist() and CACHE TABLE answer the way Spark does, and your results are the same, but Sail doesn't keep a copy of the data in memory. The local file-read cache above is what speeds up repeat reads.
Where your data lives
Your lakehouse tables are open Apache Iceberg or Delta Lake tables: Parquet data files plus table metadata, in your project's lakehouse storage. Trino reads the tables Sail writes, and Sail reads the tables your pipelines write.
On Databasin Cloud we run that storage and the clusters for you. On Databasin Private, the same product runs in your own Azure subscription.
Shuffle data and spill files stay inside the cluster, on its nodes.
Each cluster keeps the Sail build it was created with. Spark compatibility explains what that means for recent fixes.
Catalogs you'll see
| Catalog | What it holds |
|---|---|
bronze, silver, gold, staging |
The tables your pipelines write. Writes follow your access rules: by default, admins can write and everyone else can read. |
sandbox |
Your own work: finished tables, scratch tables and experiments. |
| Other lake catalogs | Any other catalog in your lakehouse metastore. |
| Live catalogs | One per live connection, named by you. Read-only. See Live sources on Sail. |
Every catalog has an information_schema (schemata, tables, columns, views), and SHOW CATALOGS / SCHEMAS / TABLES / COLUMNS work everywhere. You only see what your access rules allow.
Security
- Access rules. Sail enforces the same file-based access rules as your Trino lakehouse, with the same meaning as Trino's file-based access control. Every table a statement touches is checked, first matching rule wins, and a rule change reaches open sessions within about 30 seconds. If the rules can't be read, Sail denies access rather than allowing it.
- Hidden columns. If your access rules hide a column, Sail leaves it out of results,
DESCRIBE,information_schemaand the column lists BI tools read, and refuses a query that names it. You also can't delete, update, merge into or overwrite that table. - Row filters and column masks aren't supported yet. A table protected by one is refused on Sail.
- No raw storage paths. Reading or writing storage by path, setting a table
LOCATION, and Spark'ssocketandconsolesources are blocked: use catalog tables. The one exception is reading your project's File Share as/Files/<path>, which only project members can do. - Python UDFs only on single-user clusters. On a shared cluster (the default), Python UDFs, pandas UDFs, UDTFs and Python data sources are refused, because they would run inside the engine itself. A single-user cluster allows them for one named project member.
- Your own session. A Spark session belongs to the user who opened it, and each notebook's Python runs in its own sandbox, apart from other users' sessions and without the storage credentials.
- Settings Databasin manages (catalogs, storage, connection and engine runtime settings) can't be changed with
SET. - Network access. Outside tools reach the cluster only when its network access is Specific IPs or Open. A Private cluster is reachable from inside Databasin only.
Idle and wake
The coordinator and the workers sleep separately.
- The coordinator is on all the time by default, so BI tools and metadata browsing connect at once. The exception is the first cluster set up for a new Databasin Cloud organization: its coordinator scales to zero too. You can change this on any cluster with Keep the coordinator always on when you edit it.
- Workers scale to zero about 30 minutes after the last request, unless you turn off Workers scale to zero.
- Any signed-in request wakes the workers: a query, a notebook cell, a BI tool's connection or a PySpark call. A tool that keeps polling the catalog keeps them awake.
- Waking takes time. The first query after a quiet period waits while workers start. Workers take about 2–3 minutes to start; a dedicated coordinator that has scaled to zero takes about 2–5 minutes. The editor, notebooks and Databasin One tell you while it's happening.
- Scheduled starts wake the cluster at times you choose and keep it warm for a window you set, for example weekday mornings before a dashboard refresh.