Data Platform Notebooks

Browser-based Python notebooks with authenticated access to the OL data warehouse. Users sign in with MIT OL SSO, get a private environment, and query the warehouse as themselves.

• Production — https://nb.data.ol.mit.edu
• QA — https://nb-qa.data.ol.mit.edu

Audience

Anyone who needs to answer a question from warehouse data that a dashboard cannot express: analysts, researchers, graduate students, and engineers prototyping against the dimensional model. No local Python setup, no credential request, no VPN.

Getting started

1. Open the environment and sign in through MIT OL SSO.
2. Choose a resource profile. Standard (2 CPU / 8 GB) suits almost everything; Large (4 CPU / 32 GB) is for genuinely heavy local computation, not for making queries faster — query work happens on the warehouse cluster, not in the pod.
3. Open getting_started.py from ~/notebooks/ and run the cells top to bottom.
4. The first query prints a Starburst Galaxy login link. Open it, sign in with the same SSO account, and the query resumes on its own.

Three files are seeded into every user's ~/notebooks/:

marimo, not classic Jupyter

The environment runs JupyterLab, so .ipynb notebooks still work, but the templates are marimo notebooks.

A marimo notebook is a Python file, not JSON. It diffs and code-reviews like source, and marimo edit --sandbox demo.py runs it outside JupyterLab. (Plain python demo.py does not: it ignores the PEP 723 header described below, and the image carries none of the notebook packages, so the connection cell fails on import. Sandbox mode builds the environment from that header and adds marimo itself.) It also has no hidden state: marimo tracks which cells read which variables and re-runs the affected cells when a value changes, so out-of-order execution — the most common source of results that cannot be reproduced — is not possible. In exchange, two cells cannot both define the same name.

The practical upside is mo.ui: a dropdown or slider is a value other cells read, so a parameterised analysis needs no re-run button and no callback plumbing.

Sandbox mode: one environment per notebook

Each notebook gets its own isolated Python environment, built from a PEP 723 header at the top of the file:

# /// script
# requires-python = ">=3.11"
# dependencies = [
#   "polars>=1.0",
#   "trino[sqlalchemy]>=0.330",
# ]
# ///

pip install does not persist. Packages go in that list, then restart the kernel. This is deliberate: the header is a complete, versioned record of what the notebook needs, so it runs the same way for the next person.

The templates ship with polars, pandas, numpy, pyarrow, and altair. Heavier libraries (scikit-learn, statsmodels, scipy) are intentionally not pre-declared — adding them per notebook keeps notebooks that do not need them starting quickly.

How warehouse authentication works

Two separate sign-ins, which is worth understanding before something looks broken:

1. Into the notebook environment. JupyterHub authenticates the user against the ol-data-platform Keycloak realm via GenericOAuthenticator. This is the login at the start.
2. Into the warehouse. Starburst Galaxy authenticates query clients itself, through its own OAuth2 flow, federating the login to the same SSO. The notebook prints a link, the user opens it, Galaxy issues its own token.

The second step is not redundant, and this is the part that most often surprises people:

The upside of Galaxy owning the handshake is real: queries run as the individual user, so access is exactly what that user's Galaxy role grants, and queries are attributable in the audit log.

The token is cached in memory and reused for as long as Galaxy accepts it, so signing in is not per-query. It is not guaranteed to be once per session either: trino-python-client re-runs the redirect flow whenever a cached token is rejected — expiry or revocation — and this deployment sets no maximum session age, so a user can be prompted again mid-session. Restarting the kernel also means signing in again.

Only the endpoint is configured for users:

TRINO_USER is deliberately not set. marimo's environment scanner offers a one-click "Quick add" Trino connection when TRINO_HOST, TRINO_USER and TRINO_CATALOG are all present, and the snippet it generates uses BasicAuthentication or no auth at all — neither of which Galaxy accepts here. Setting it would surface a one-click connection that 401s.

Finding data

The Explore variables and data sources panel lists the warehouse engine; expanding it lists schemas and tables. Schemas load on expand rather than up front — marimo does not eagerly scan a remote warehouse (trino is not in its cheap-discovery list), so an unexpanded tree is not an empty one.

The warehouse is built by dbt in layers, and the schema suffix names the layer:

Within _dimensional: dim_* is something to group or filter by, tfact_* is a transaction fact (one row per event), afact_* is a pre-summarised aggregate fact, and bridge_* is a many-to-many link.

Most of what you see in the panel is not on that list. There are also dozens of ol_warehouse_production_<username>_* schemas — developer sandboxes from ol-dbt local, not authoritative and not maintained. If a schema name has a person in it, it is not for you.

Two ways to query

SQL cells — mo.sql("SELECT ...", engine=warehouse). The editor treats it as SQL with column completion, the result is a dataframe, and the cell joins the reactive graph so downstream cells update by themselves. One statement, one result, whole result in memory, no parameter binding.

The DB-API cursor — cur.execute("SELECT ..."). Ordinary Python: loop it, build the SQL from a list, stream with fetchmany. It also holds connection-scoped state — SET SESSION and USE survive on the cursor but not in a SQL cell, which opens a fresh connection per statement — and cur.stats is how to find out why a query is slow. Returns tuples, no SQL editing support, and marimo cannot see the dependency.

Reach for a SQL cell. It is not restricted to SELECT: SHOW, EXPLAIN and DDL run there too, and any statement returning rows comes back as a dataframe. Drop to the cursor for a loop, a stream, session state, or cur.stats.

Working with personal data

dim_user holds names and email addresses. Joining to it is sometimes right, but neither template does, on purpose — a getting-started file should not normalise SELECT * on a table full of identifiers.

• Aggregate where possible. The fact tables carry user_fk, so count(DISTINCT user_fk) answers most questions about people without returning a person.
• Select named columns, never *.
• Put a floor on group sizes. A pass rate over four learners identifies them to anyone who knows the cohort.
• Home directories are persistent volumes. A CSV of learner rows written there outlives the question it was written to answer.
• Queries through Galaxy run as the individual user and are logged. That does not extend to direct S3/Glue reads, which run under the shared pod role — see Known gaps.

Resource limits

After 4 hours idle the culler stops your whole single-user server — the pod and JupyterLab with it, not just the kernels — and there is no absolute session limit. Home directories survive; anything held only in memory does not, and you sign in to Galaxy again on the next server.

Sharing work

• marimo run --sandbox <file>.py serves the notebook as an app with the code hidden — mo.ui inputs become the controls. --sandbox is what builds the environment from the /// script header; without it marimo uses the base environment, which has none of the notebook packages.
• Export to self-contained HTML, or to .ipynb for a Jupyter collaborator.
• The notebook is a .py file, so commit it. It reviews like code.

Troubleshooting

Operating this environment

The image builds and publishes to ghcr.io/mitodl/marimo-jupyterlab on every push to main that touches images/marimo-jupyterlab/. Templates are baked into the image at /usr/local/share/marimo/templates/ and copied into ~/notebooks/ by a singleuser.lifecycleHooks.postStart hook using cp -n, so a user's edited copy on the persistent volume is never overwritten.

Changing the templates

Edit them in ol-data-platform, merge to main, wait for the image build, then restart notebook servers so users pull the new image. Because cp -n will not overwrite, users who already have a copy keep their old one — that is the right trade-off for not destroying someone's work, but it means template fixes do not reach existing users automatically. Tell them to delete or rename their copy and then restart their notebook server — the reseed runs only in the pod's postStart hook, so deleting a file in a running server does not bring the new one back and leaves them with neither.

Known gaps

• Published (run-mode) marimo apps cannot authenticate to Galaxy yet. The marimo_data stack provisions the ol-marimo-app-client service account for Trino access, but client credentials against Keycloak yield a Keycloak JWT, which Galaxy rejects for the same reason described above. Published apps have no user session, so the OAuth2 flow is not available to them either — they need a Galaxy service account. The credentials themselves are already synced — marimo_data creates an OLVaultK8SSecret that reads sso/marimo-app into the marimo-app-oidc-secret Kubernetes Secret — but no application consumes that Secret yet, so nothing is broken in production today.
• All three stacks point trino_host at the production Galaxy cluster while the IRSA role grants S3 and Glue on ol-data-lake-*-<environment>. Notebooks on nb-ci and nb-qa therefore query production through Trino while holding AWS credentials scoped to their own environment — the policy is parameterised by stack_info.env_suffix, so nb-ci gets CI-scoped and nb-qa QA-scoped access. Either way the two access paths disagree about which environment the pod is in.
• Direct lake access is not authorized or attributed per user. That IRSA policy is attached to one service-account role shared by every single-user pod, so a notebook can read a table from S3/Glue even where the user's Galaxy role denies it, and CloudTrail records the pod identity rather than their SSO identity. The notebook templates deliberately document no direct-read recipe, but that is documentation, not a control. Narrowing the role or issuing per-user credentials is the actual fix.