Unity Catalog, for real
+Catalogs, schemas, tables and volumes with full CRUD. Every catalog is its own DuckDB database file, so catalog.schema.table addressing is native — not a regex rewrite.
diff --git a/.github/workflows/pages.yml b/.github/workflows/pages.yml new file mode 100644 index 0000000..993ee60 --- /dev/null +++ b/.github/workflows/pages.yml @@ -0,0 +1,41 @@ +name: Pages + +# Publishes the marketing site in `site/` to GitHub Pages +# (https://dmux.github.io/minilake/). Requires the repository's Pages source to +# be set to "GitHub Actions" -- Settings -> Pages -> Build and deployment. +on: + push: + branches: [main] + paths: + - "site/**" + - ".github/workflows/pages.yml" + workflow_dispatch: + +permissions: + contents: read + pages: write + id-token: write + +concurrency: + group: pages + cancel-in-progress: false + +jobs: + deploy: + runs-on: ubuntu-latest + + environment: + name: github-pages + url: ${{ steps.deployment.outputs.page_url }} + + steps: + - uses: actions/checkout@v4 + + - uses: actions/configure-pages@v5 + + - uses: actions/upload-pages-artifact@v3 + with: + path: site + + - id: deployment + uses: actions/deploy-pages@v4 diff --git a/README.md b/README.md index a7d81f5..cb2341c 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,7 @@
- Documentation · GitHub · Container Image (GHCR) + Website · Documentation · GitHub · Container Image (GHCR)
--- diff --git a/site/.nojekyll b/site/.nojekyll new file mode 100644 index 0000000..e69de29 diff --git a/site/README.md b/site/README.md new file mode 100644 index 0000000..87a99c9 --- /dev/null +++ b/site/README.md @@ -0,0 +1,27 @@ +# `site/` — the project website + +The landing page published to GitHub Pages at
+ MiniLake is a local Databricks API emulator with real SQL,
+ real Delta Lake and real Spark job execution. Point
+ databricks-sdk, the Databricks CLI, Terraform, Asset Bundles or an LLM agent at
+ http://localhost:8000 — no account, no API key, no cloud bill.
+
pip install minilake && minilake --port 8000
+ PyPI
+
+ docker run -p 8000:8000 ghcr.io/dmux/minilake:latest
+ GHCR
+
+ Then open http://localhost:8000/ui/ for the built-in SQL workspace. The image downloads nothing at runtime — it works air-gapped.
+ Fake endpoints let broken code pass. MiniLake executes your SQL on DuckDB, writes real Delta files, and runs real spark-submit jobs in containers — so a green test locally means something.
Catalogs, schemas, tables and volumes with full CRUD. Every catalog is its own DuckDB database file, so catalog.schema.table addressing is native — not a regex rewrite.
Statement execution on real DuckDB: INLINE and EXTERNAL_LINKS dispositions, JSON_ARRAY / ARROW_STREAM / CSV formats, and a result manifest that carries true column types.
EXTERNAL Delta tables are genuine Parquet plus a _delta_log/. INSERT, UPDATE and DELETE are routed through a real Spark job; reads go through DuckDB's delta_scan().
spark_python_task runs spark-submit in a sibling container and returns actual stdout. Real DAG scheduling honours depends_on and run_if, with independent branches in parallel.
spark.table("cat.sch.tbl") resolves against MiniLake through the official UCSingleCatalog connector — the same protocol Spark speaks to a real workspace.
An Athena-style editor at /ui: Monaco with catalog-aware completion, a virtualized result grid, charts, CSV/JSON export, saved queries, query history — and an embedded JupyterLab.
Point the databricks provider at localhost and terraform apply creates real catalogs. databricks bundle deploy and bundle run work end to end — jobs execute for real.
67 tools, 8 resources and 4 prompts at /mcp — including composites that collapse a five-call sequence into one, and a SQL-dialect resource that stops agents writing Spark SQL.
One POST /_minilake/reset gives every test a clean workspace without restarting the container. /health, /ready and /services round out the control plane.
File-backed notebook and file storage with chunked uploads. Secrets are real CRUD whose values are only ever resolvable inside a job's environment — exactly like Databricks.
+ Real execution +DuckDB's delta extension and the Delta / Unity Catalog jars are baked in at build time. The image downloads nothing at runtime — provable with docker run --network none.
Failures come back as {"error_code", "message"}, so the SDK raises its normal typed exceptions. Anything out of scope returns a loud 501 NOT_IMPLEMENTED instead of a convincing lie.
+ MiniLake speaks the Databricks REST API on the wire, so every client you already use keeps working — + the same calls, the same response shapes, the same typed errors. +
+STRING, LONG, DECIMAL(10,2), ARRAY<T>, MAP and STRUCT map to DuckDB and come back as the SDK's ColumnInfo.MINILAKE_PERSIST=1 snapshots state on shutdown and restores it on start.from databricks.sdk import WorkspaceClient
+from databricks.sdk.service.catalog import ColumnInfo, TableType
+
+# The only line that differs from production
+w = WorkspaceClient(host="http://localhost:8000", token="dev")
+
+w.catalogs.create(name="sales")
+w.schemas.create(name="store", catalog_name="sales")
+w.tables.create(
+ name="orders", catalog_name="sales", schema_name="store",
+ table_type=TableType.MANAGED,
+ data_source_format="DELTA", storage_location="/data/sales/store/orders",
+ columns=[
+ ColumnInfo(name="id", type_text="BIGINT"),
+ ColumnInfo(name="customer", type_text="STRING"),
+ ColumnInfo(name="amount", type_text="DECIMAL(10,2)"),
+ ],
+)
+
+wh = w.warehouses.create(name="dev", cluster_size="Small")
+w.statement_execution.execute_statement(
+ warehouse_id=wh.id,
+ statement="INSERT INTO sales.store.orders VALUES (1, 'Ana', 459.90)",
+)
+
+result = w.statement_execution.execute_statement(
+ warehouse_id=wh.id,
+ statement="SELECT customer, amount FROM sales.store.orders",
+)
+print(result.result.data_array) # [['Ana', '459.90']] — really stored
+
+ from pyspark.sql import SparkSession
+
+UC = "http://localhost:8000"
+
+spark = (
+ SparkSession.builder
+ .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
+ # Unity Catalog connector — three-part names resolve against MiniLake
+ .config("spark.sql.catalog.spark_catalog", "io.unitycatalog.spark.UCSingleCatalog")
+ .config("spark.sql.catalog.spark_catalog.uri", UC)
+ .config("spark.sql.catalog.sales", "io.unitycatalog.spark.UCSingleCatalog")
+ .config("spark.sql.catalog.sales.uri", UC)
+ .getOrCreate()
+)
+
+# Real Delta files: parquet + _delta_log/, written by real Spark
+spark.createDataFrame([(1, "a"), (2, "b")], ["id", "name"]) \
+ .write.format("delta").mode("overwrite").save("/data/delta/sales/store/events")
+
+spark.sql("INSERT INTO sales.store.events VALUES (3, 'c')")
+print(spark.table("sales.store.events").count()) # 3
+
+# ...and the same bytes are readable through MiniLake's SQL API:
+# SELECT e.name, o.customer
+# FROM sales.store.events e JOIN sales.store.orders o ON o.id = e.id
+
+ provider "databricks" {
+ host = "http://localhost:8000"
+ token = "dev"
+}
+
+resource "databricks_catalog" "sandbox" {
+ name = "sandbox"
+ comment = "Local sandbox catalog"
+}
+
+resource "databricks_schema" "things" {
+ catalog_name = databricks_catalog.sandbox.name
+ name = "things"
+}
+
+resource "databricks_sql_endpoint" "compute" {
+ name = "local-compute"
+ cluster_size = "Small"
+}
+
+# terraform apply -> creates real catalogs and real DuckDB databases
+# terraform destroy -> removes them
+# Resources backed by APIs MiniLake does not emulate fail loudly with 501,
+# instead of silently pretending to succeed.
+
+ bundle:
+ name: my_pipeline
+
+targets:
+ dev:
+ mode: development
+ workspace:
+ host: http://localhost:8000
+
+resources:
+ jobs:
+ daily_load:
+ name: daily_load
+ tasks:
+ - task_key: main
+ spark_python_task:
+ python_file: ../src/load.py
+
+# $ databricks bundle deploy -t dev
+# $ databricks bundle run daily_load -t dev
+#
+# The CLI syncs files through the Workspace API, the provider creates the job,
+# and `bundle run` executes it on real Spark in a sibling container —
+# returning the job's actual output.
+
+ # Is it up, and which services answered?
+curl http://localhost:8000/_minilake/health
+
+# Create a catalog through the Databricks API surface
+curl -X POST http://localhost:8000/api/2.1/unity-catalog/catalogs \
+ -H 'Content-Type: application/json' \
+ -d '{"name": "sales"}'
+
+# Run SQL and get real rows back
+curl -X POST http://localhost:8000/api/2.0/sql/statements \
+ -H 'Content-Type: application/json' \
+ -d '{"warehouse_id": "dev", "statement": "SELECT 42 AS answer"}'
+
+# Give the next test a clean workspace — no restart needed
+curl -X POST http://localhost:8000/_minilake/reset
+ A FastAPI application serves the Databricks REST surface, the web UI and the MCP endpoint together. Behind it sit DuckDB, real Delta files on disk, and Docker for Spark.
++ A Databricks workspace is an awkward thing for an agent to drive: every useful action is four calls + deep, half the SQL it writes is the wrong dialect, and failures are silent. MiniLake's MCP server + closes those gaps — and it costs nothing when the agent gets it wrong. +
++ Plus 8 resources — including a SQL-dialect guide that stops agents writing Spark SQL at DuckDB — + and 4 prompts that front-load the rules. Tools run through MiniLake's own ASGI stack, so an agent's + error is exactly the error an SDK client would have seen. +
+ +# One line to register it with Claude Code
+claude mcp add --transport http minilake http://localhost:8000/mcp
+
+# Then just ask:
+# "Use minilake: create a sales catalog with a seeded
+# orders table, and tell me the total."
+
+# The agent does this — three calls, no glue code:
+# setup_fixture({"catalog": "sales", "schema": "store"})
+# seed_table({"full_name": "sales.store.orders", ...})
+# run_sql({"statement": "SELECT SUM(amount) FROM sales.store.orders"})
+
+# Enable it on the server side:
+MINILAKE_MCP=1 minilake --port 8000
+ Every row below is either really executed or clearly labelled. Anything out of scope answers 501 NOT_IMPLEMENTED rather than faking a success your production code will not get.
| Service | Status | What that means |
|---|---|---|
| Unity Catalog | Real | Catalogs, schemas, tables, volumes — each catalog is its own DuckDB database, with native three-part addressing |
| SQL statements | Real | Executed on DuckDB; all three formats and both dispositions, with a typed result manifest |
| EXTERNAL Delta tables | Real | Genuine Delta files; writes go through a generated Spark job, reads through delta_scan() |
| Jobs | Real | spark-submit in sibling containers (or a subprocess fallback), real DAG scheduling, real logs |
| SQL warehouses | Real | Full CRUD and lifecycle; statements run regardless of state, as they do locally |
| Query history & saved queries | Real | Server-backed, failures included — which is usually why you opened history |
| Workspace, DBFS & Files | Real | File-backed storage with chunked upload; raw-bytes sync is what makes bundle deploy work |
| Secrets | Real | Real CRUD; values are resolvable only inside job environment variables, never through the API |
| Web UI & notebooks | Real | Athena-style SQL workspace at /ui, with JupyterLab proxied at /jupyter |
| MCP server | Real | Opt-in with MINILAKE_MCP=1: 67 tools, 8 resources, 4 prompts on the same port |
| Persistence | Real | MINILAKE_PERSIST=1 snapshots state as JSON on shutdown and restores it on startup |
| Clusters | State machine | CRUD plus timed lifecycle transitions — deliberately no Spark compute; that lives in Jobs |
| Permissions | Allow-all CRUD | Real create/read/update, but nothing is ever enforced — this is a single-user tool |
| Identity (SCIM) | Static | A fixed current-user endpoint; there is exactly one user and any token is accepted |
| DLT, Model Registry, Vector Search, Dashboards, Repos | Not implemented | Returns 501 NOT_IMPLEMENTED — loudly, so you never mistake it for working |
These are deliberate design decisions, not a backlog. MiniLake targets one developer running it locally — not a shared, multi-tenant server.
+Any token is accepted and there is exactly one user. Never expose it beyond localhost.
The Permissions API is real CRUD that always allows everything — a passing test here says nothing about grants in a real workspace.
Statements run on DuckDB's dialect. The MCP server ships a dialect resource for exactly this reason; the docs carry the translation table.
The cluster lifecycle is a state machine. Real Spark happens through Jobs, in sibling containers.
DuckDB is single-writer, so concurrent load serializes on a lock. Fine for dev and CI; not a deployment target.
They are DuckDB tables with no files. Use EXTERNAL Delta when Spark has to see the data — that trade-off is what keeps SQL fast.
Being explicit about the edges is the point: an emulator that quietly fakes what it cannot do is worse than no emulator at all.
+
+ Free and MIT-licensed. No account, no API key, no cloud bill — and nothing to uninstall from your cloud account when you're done.
+ +