Databasin Sail overview

Databasin Sail is the query engine we recommend for most teams, and the one a new lakehouse gets by default. It runs Spark SQL and PySpark, reads and writes your lakehouse tables, queries live sources in place, and connects to notebooks and BI tools. You don't tune executors or Spark memory settings: you pick a size and run queries.

Sail is built on LakeSail Sail, an open-source Spark Connect server written in Rust on Apache DataFusion. Databasin adds what a shared lakehouse needs: access rules, our lakehouse catalogs, live sources, Python notebooks, and clusters that sleep when idle.

Create a Databasin Sail cluster→

Who it's for

• Analysts and engineers who write SQL. Sail speaks Spark SQL, the dialect Apache Spark and Databricks use.
• Python users. PySpark DataFrames, pandas and notebooks, with packages from PyPI.
• Teams moving off Spark or Databricks. Spark SQL and PySpark run on Sail; Spark compatibility lists the differences.
• BI users. Tableau and DBeaver connect over ODBC and JDBC, and Power BI with its Spark connector, signed in with your lakehouse username and password.

The two-minute picture

A Sail cluster is a coordinator and a set of workers, running in Azure Container Apps. The coordinator plans each query and checks your access rules; the workers do the heavy lifting. Workers scale to zero when nobody is using them and start again on the next request. How Sail works walks through it.

Your tables are open Apache Iceberg or Delta Lake tables. Sail writes tables that Trino reads, and reads the tables your pipelines write.

Where you use it

Outside tools only reach the cluster when its network access allows it. Connect outside tools has the settings for each one.

What you can do with it

• Query and join. Spark SQL with joins, window functions, CTEs, PIVOT and about 500 built-in functions. The Sail SQL reference lists them.
• Write tables. INSERT, CREATE TABLE AS, MERGE, UPDATE and DELETE, plus persistent views, in the catalogs your access rules let you write to.
• Maintain tables. OPTIMIZE, VACUUM, snapshot expiry and rollback on Iceberg tables; OPTIMIZE, VACUUM, RESTORE and DESCRIBE HISTORY on Delta tables.
• Query live sources. APIs, files, PostgreSQL, MySQL, SQL Server and Databricks Unity Catalog tables, joined with your lakehouse tables in one statement. See Live sources on Sail.
• Work in Python. PySpark, pandas, toPandas(), your project's files, and packages from PyPI. See Python and PySpark on Sail.
• See what's running. Query Activity shows statements from every tool, running and in history. Members see their own; project admins see everyone's.

What Sail supports today has the full list, including what's coming next.

A first query

USE sandbox.analytics;

CREATE TABLE daily_orders AS
SELECT order_date, count(*) AS orders, sum(total) AS revenue
FROM gold.sales.orders
GROUP BY order_date;

A table created without USING takes the catalog's format, so this is an Iceberg table in an Iceberg catalog. Asking for a different format there (USING parquet, say) is refused.

Good to know

• Start with Medium. It's the default size and fits most teams. Sail clusters and sizes explains the others.
• The first query after a quiet period waits for workers. Workers take about 2–3 minutes to start; a dedicated coordinator that has scaled to zero takes about 2–5 minutes. The editor and Databasin One tell you while it's happening.
• Shared clusters don't run Python UDFs. Choose a single-user cluster for that.
• Not on Sail: Scala cells, Spark ML (MLlib), Structured Streaming and the RDD API.

Where to go next