Databasin Sail

Databasin Sail is the query engine we recommend for most teams, and the one a new lakehouse gets by default. It runs Spark SQL and PySpark natively: no JVM to tune, no Spark configuration to babysit. It reads and writes your Iceberg lakehouse, queries live sources in place, and connects to notebooks and BI tools.

Under the hood, Sail is built on LakeSail Sail, an open-source Spark Connect server written in Rust on Apache DataFusion. Databasin adds the parts a shared lakehouse needs: table-level access rules, our lakehouse catalogs, live sources, and clusters that sleep when idle.

Create a Databasin Sail cluster→

Why Sail

• The Spark you already know. Spark SQL and PySpark (DataFrames, spark.sql, pyspark.sql.functions) work as they do on Apache Spark. Code you wrote for Databricks or open-source Spark mostly runs unchanged; see the Sail SQL reference for what's covered.
• Reads and writes your lakehouse. INSERT, CREATE TABLE AS, MERGE, UPDATE and DELETE on Iceberg tables, plus table maintenance (OPTIMIZE, VACUUM, snapshot expiry) from the SQL editor.
• Live sources. Query APIs, files, PostgreSQL / MySQL / SQL Server databases and Databricks Unity Catalog tables in place, and join them with your lakehouse tables in one statement.
• Everywhere you work. The SQL editor, notebooks (SQL and Python), Databasin One, Data Apps, automations, and outside tools over JDBC, ODBC and Spark Connect.
• Same access rules as Trino. Sail enforces the same catalog, schema, table and column rules your Trino lakehouse uses.
• Workers sleep when idle. They wake when a query needs them; a small coordinator stays on by default so tools connect instantly (you can let it sleep too).

Where you use it

Sizes

Pick a size when you create the cluster; you can resize it later. Medium is the default, and the right place to start for most teams.

A few things worth knowing:

• The coordinator answers metadata on its own: SHOW, DESCRIBE, USE, information_schema and the catalog browsing BI tools do when they connect. It stays on by default, so tools connect instantly. When you create or edit the cluster you can let it sleep too, or pick a bigger dedicated coordinator for many notebook users.
• Workers do the heavy lifting and sleep when idle (after about ten minutes with no work). The first query after a quiet period waits while they start, which can take a couple of minutes; the editor and Databasin One tell you when that's happening.
• Scheduled starts keep workers warm for a window you choose (for example, weekday mornings before a dashboard refresh). Set them when you edit the cluster.

Tables and catalogs

A Sail cluster sees the same lakehouse your pipelines write to:

• Pipeline catalogs (bronze, silver, gold, staging) are read-only on Sail. Pipelines own them.
• sandbox (and any catalog you create for your own work) is writable: build gold tables, scratch tables and experiments there.
• Table formats: Iceberg (the default for new lakehouses) and Delta.
• information_schema works on every catalog (schemata, tables, columns, views), along with SHOW CATALOGS / SCHEMAS / TABLES / COLUMNS and DESCRIBE. You only see what your access rules allow.
• Iceberg metadata tables such as my_table.snapshots and my_table.files can be queried directly.

USE sandbox.analytics;

CREATE TABLE daily_orders USING iceberg AS
SELECT order_date, count(*) AS orders, sum(total) AS revenue
FROM gold.sales.orders
GROUP BY order_date;

Writing and maintaining tables

Everything you'd expect for Iceberg tables in sandbox and your own catalogs:

• INSERT INTO / INSERT OVERWRITE, CREATE TABLE … AS SELECT, CREATE OR REPLACE TABLE … AS SELECT.
• MERGE INTO, UPDATE and DELETE, each reporting how many rows changed. On Iceberg v2 tables, UPDATE writes small delete files instead of rewriting whole data files, so changing a few rows in a big table is quick.
• ALTER TABLE: add, drop, rename and retype columns, rename the table, change partitioning, set a sort order (WRITE ORDERED BY).
• SHOW CREATE TABLE, SHOW TBLPROPERTIES, TRUNCATE.
• OPTIMIZE my_table [WHERE …] compacts small files on the cluster's workers (and sorts them, if the table has a sort order).
• VACUUM my_table [RETAIN n HOURS] [DRY RUN] removes files no snapshot needs any more (keeps 7 days by default, never less than 24 hours).
• CALL <catalog>.system.…: expire_snapshots, remove_orphan_files, rewrite_manifests, rewrite_data_files, rollback_to_snapshot, rollback_to_timestamp, set_current_snapshot.

If two writers commit to the same table at once, Sail retries the one that lost the race, and refuses a stale update rather than overwrite newer data.

Live sources

Make a connection live and it becomes its own catalog on Sail, named by you, queryable with nothing synced:

• Native API connectors and files in Azure Data Lake Storage.
• PostgreSQL, MySQL and SQL Server databases that sign in with a username and password.
• Databricks Unity Catalog tables (Azure Databricks; Delta managed and external tables).

Live sources are read-only, and they join with your lakehouse tables in one statement:

SELECT c.name, sum(o.total) AS revenue
FROM crm.public.customers c
JOIN gold.sales.orders o ON o.customer_id = c.id
GROUP BY c.name;

See Live connections for setting one up.

Connecting outside tools

Outside tools sign in with your lakehouse username and password, and only reach the cluster when its network access allows it (Specific IPs or Open; a Private cluster is reachable from inside Databasin only). Open the cluster card's details for the exact connection strings.

• Tableau: the Spark SQL connector with the Simba Spark ODBC 2.9 driver (newer Databricks-branded drivers refuse servers that aren't Databricks). To land in a catalog, add it to the HTTP path (cliservice/<catalog>) rather than using Initial SQL.
• Power BI: the Spark connector over HTTP.
• DBeaver and other JDBC tools: the Databasin Sail JDBC driver (download it from the cluster card), or the Hive JDBC URL shown there.
• PySpark, Jupyter, VS Code: the Spark Connect URL (sc://…) from the cluster card.

Security

• Access rules: Sail uses the same file-based rules as Trino. Every table a statement touches is checked, columns you can't see are hidden, and a rule change takes effect within about 30 seconds.
• Path-based reads and writes are blocked: use catalog tables, never raw storage paths.
• Python: a cluster is shared by default, and shared clusters don't run user-defined Python functions (Python UDFs, pandas UDFs, Python data sources). Choose single user under Advanced to allow them for one named person.
• Network access: Private, Specific IPs (your address by default) or Open, set at create and changeable later.

Monitoring

Query Activity shows every statement on the cluster, from the SQL editor, notebooks (SQL and Python), Databasin One, automations and BI tools: what's running now, with cancel, and the history. Members see their own statements; project admins see everyone's.

OLTP mode (coming soon)

The cluster card shows an OLTP mode switch marked Coming soon. It will add a managed PostgreSQL database beside your Sail cluster, for apps that read and write single rows fast, queryable from Sail like any other source. The switch can't be turned on yet.

Good to know

• No Scala cells. Notebooks run SQL and Python (PySpark) on Sail.
• No CREATE VIEW in lake catalogs. Use a temporary view, or CREATE TABLE … AS SELECT. Existing Iceberg views written in Spark SQL can be queried.
• No row filters or column masks. A table protected by one is refused on Sail.
• Not in scope: Structured Streaming, MLlib and the RDD API.
• Python packages: notebook cells come with pandas, NumPy, PyArrow, matplotlib, seaborn and scikit-learn.

Where to go next