Python and PySpark on Sail

Sail runs PySpark through Spark Connect, PySpark's client-server API. Your Python code builds DataFrames and sends them to the cluster; the cluster runs them and sends results back. You can do this from a Databasin notebook, or from your own Python in Jupyter, VS Code or a script.

The cluster runs Python 3.14 with PySpark 4.2.

In a notebook

Add a Spark - Python cell. It shares one Spark session with the notebook's Spark - SQL cells, so a temporary view made in one is there in the next. These names are ready to use without imports:

orders = spark.table("gold.sales.orders")
by_region = orders.groupBy("region").agg(F.sum("total").alias("revenue"))
display(by_region)

A DataFrame on the last line of a cell also shows as a grid. Matplotlib charts show below the cell. By default a result keeps up to 100,000 rows, and up to 20 images per cell.

Also installed: PyArrow, seaborn, scikit-learn, requests and openpyxl. A Spark - Python cell can be stopped while it runs, like a SQL cell. databasin.exit("message") ends a notebook run early, as a success.

Files from your project

Notebooks can read and write your project's File Share (Lakehouse → File share). Spark reads files by path starting with /Files/, and Python uses databasin.files:

sales = spark.read.option("header", True).csv("/Files/sales/2026-q3.csv")
events = spark.read.json("/Files/events/*.json")          # JSON Lines

budget = pd.read_excel(databasin.files.open("budget/fy27.xlsx"))
databasin.files.write_text("out/summary.txt", budget.describe().to_string())

Spark reads CSV, Parquet, JSON Lines, text, Avro, Arrow and binary files from /Files, but never writes there: save results as a table in sandbox, or write a file with databasin.files. The full helper list:

databasin.files.ls(path="", recursive=False)
databasin.files.exists(path)
databasin.files.read_bytes(path)        # or read_text(path, encoding="utf-8")
databasin.files.write_bytes(path, data, overwrite=True)   # or write_text(...)
databasin.files.upload(local_path, path, overwrite=True)
databasin.files.download(path, local_path=None)
databasin.files.open(path, mode="rb")   # "rb"/"r" to read, "wb"/"w" to write
databasin.files.rm(path)
databasin.files.spark_path(path)        # "/Files/<path>", for Spark

Everyone in the project can use these files, and only project members: table access rules don't apply to them. Each file can be up to 5 GB.

Python packages

• For one notebook session: run %pip install <package> in a cell of its own. The package is ready in the next cell and gone when the session ends. %pip list shows what's installed.
• For everyone on the cluster: a project admin lists packages under Advanced → Python packages, one per line. They install each time the cluster starts. If one fails, the cluster starts without them and the card shows the error.

Packages come from public PyPI as prebuilt wheels. Pin versions (tabulate==0.9.0) so every start gets the same ones. A package can't change the pandas, NumPy or PyArrow versions the cluster runs, and the cluster needs to reach PyPI over the internet.

Python UDFs

Python UDFs, pandas UDFs, Arrow UDFs, applyInPandas, mapInPandas, UDTFs and Python data sources run only on a single-user cluster (Access mode → Single user, under Advanced). A shared cluster refuses them, because they would run inside the engine itself. Notebook Python code isn't a UDF, so it runs on any cluster.

import pandas as pd
from pyspark.sql.functions import pandas_udf

@pandas_udf("double")
def with_tax(amount: pd.Series) -> pd.Series:
    return amount * 1.08

spark.table("sandbox.sales.orders").select(with_tax("total").alias("total_with_tax")).show()

On a multi-node cluster, UDFs run on the workers. A UDF that imports a package beyond pandas, NumPy and PyArrow needs that package in the cluster's Python packages list; %pip packages are for notebook code only.

From your own Python

Connect Jupyter, VS Code or a script with the Spark Connect URL on the cluster card. It's shown when the cluster's network access is Specific IPs or Open. Install the same PySpark version the cluster runs:

pip install "pyspark[connect]==4.2.0"

Then add your lakehouse username and password to the URL:

from pyspark.sql import SparkSession

spark = SparkSession.builder.remote(
    "sc://<host>:443/;use_ssl=true;user_id=<lakehouse username>;token=<lakehouse password>"
).getOrCreate()

spark.sql("SELECT current_user()").show()

To send Python UDFs from your own machine, run Python 3.14 there too: the cluster refuses a UDF built with a different Python version.

Getting data into Python

• df.toPandas() and df.toArrow() bring a result into local memory. Filter and aggregate on the cluster first.
• pandas API on Spark (pyspark.pandas): basic use such as ps.DataFrame(...) and df.pandas_api() works. Test anything beyond that before you rely on it.

Not available

• Spark ML (pyspark.ml).
• spark.addArtifacts() to ship .py, .zip or .jar files.
• df.observe().
• Structured Streaming. Use a scheduled notebook or pipeline instead.
• console and socket sources and sinks.
• The RDD API and SparkContext, which aren't part of Spark Connect.

Where to go next