Skip to content

Add an ml extra for gradients in SQL, and two ML demos #269

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
alxmrs merged 29 commits into from
Oct 31, 2026
+2,540 −49
Merged

Add an ml extra for gradients in SQL, and two ML demos #269

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
53958d4
Add an ml extra for gradients in SQL, and three ML demos
alxmrs Oct 17, 2026
3eee986
Address review: complete SQL statements, nanoGPT's model, introductions
alxmrs Oct 18, 2026
a4ef3b2
Train the GPT with nanoGPT's AdamW behind --optimizer adamw
alxmrs Oct 18, 2026
8ddc585
Define attention once, as a view, and its loss separately
alxmrs Oct 18, 2026
e459a8c
Split the GPT into a model view and a separate loss
alxmrs Oct 18, 2026
403a1f5
Fix lint: format the GraphCast script, and type two values for mypy
alxmrs Oct 18, 2026
65706fd
Differentiate the softmax's window MAX directly with ddxdb 0.3.1
alxmrs Oct 18, 2026
a2eb466
Require ddxdb 0.3.2 and record its faster GPT steps
alxmrs Oct 18, 2026
a61842c
Require ddxdb 0.3.3 and record its faster gradients
alxmrs Oct 21, 2026
574e9c2
Require ddxdb 0.4.0, on DataFusion 55
alxmrs Oct 21, 2026
229b842
Record the demos on ddxdb 0.4.0 and in-memory tables
alxmrs Oct 21, 2026
03c1d51
Take gradients on DuckDB through ddx
alxmrs Oct 25, 2026
d5c03ab
Size the GPT from the command line, and record where its time goes
alxmrs Oct 25, 2026
ff7352b
Move the GraphCast demo to its own PR
alxmrs Oct 25, 2026
1c08ea4
Build the GPT's block from attention, mlp and residual, as nanoGPT does
alxmrs Oct 25, 2026
dc515a5
Run the ML demos in CI, and say how to contribute one
alxmrs Oct 25, 2026
508ae96
Say that a mask is absent rows, and an infinity only tests it
alxmrs Oct 25, 2026
28f3cb0
Require ddxdb 0.4.2, and record the demos on it
alxmrs Oct 25, 2026
38aaf91
Address review: fail on new refusals, run AdamW, lock JAX and ddx's D…
alxmrs Oct 27, 2026
883108b
Merge main, with Context.prepare, into ml-demos
alxmrs Oct 27, 2026
1361a59
Require ddxdb 0.4.3, which trains the GPT on DuckDB
alxmrs Oct 28, 2026
ef30a6e
Run the GPT's CTEs once each on DataFusion, so it trains past one layer
alxmrs Oct 28, 2026
f7c39f5
Require ddxdb 0.4.5, which frees each step's tables after their last …
alxmrs Oct 29, 2026
d543b55
Run the ML demos on einfold 0.1.0rc1, and require ddxdb 0.4.6
alxmrs Oct 30, 2026
8a7aa90
Address review: ignore grad( in literals, and skip JAX on Intel macOS
alxmrs Oct 30, 2026
cb935d0
Require einfold 0.1.0rc2, and install the test extra for the ML demos
alxmrs Oct 30, 2026
714ba63
Write the GPT as functions of relations, as nanoGPT's modules are
alxmrs Oct 30, 2026
6debe24
Write the GPT as plain SQL again, without the shared-CTE workaround
alxmrs Oct 30, 2026
f6cae08
Address review: route quoted identifiers past ddx, and comment demo 1…
alxmrs Oct 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 41 additions & 2 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ jobs:
# Use the --no-install-package to only install the dependencies
# but do not yet build the rust library
- name: Install xarray_sql
run: uv sync --dev --no-install-package xarray-sql
run: uv sync --locked --dev --no-install-package xarray-sql
- name: build rust
run: uv run --no-project maturin develop --uv
- name: Install in-process ADBC drivers
Expand All @@ -81,4 +81,43 @@ jobs:
- name: Run unit tests
env:
VIRTUAL_ENV: ${{ github.workspace }}/.venv
run: uv run --no-project pytest -v . -m "not integration"
# The ML demos run in their own job, below.
run: uv run --no-project pytest -v . -m "not integration" --ignore=tests/test_ml_demos.py

@d33bs d33bs Oct 31, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might need --frozen too.

@alxmrs alxmrs Oct 31, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🌀 Claude: Done in f6cae08: the unit-test job's uv sync is --locked too. The uv run --no-project steps don't sync.


ml-demos:
# Each ML demo in benchmarks/ml, at its smallest size, checked against
# JAX (tests/test_ml_demos.py).
name: "ml demos"
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

@d33bs d33bs Oct 31, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is pretty outdated. Consider bumping the versions of all actions using dependabot or similar (using a cooldown too). Here's an example: https://github.com/CU-DBMI/template-uv-python-research-software/blob/main/.github/dependabot.yml#L14

@alxmrs alxmrs Oct 31, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🌀 Claude: Agreed. It applies to every workflow, so it's tracked in #295, with your template as the example.


- uses: dtolnay/rust-toolchain@stable

- name: Setup sccache
uses: mozilla-actions/sccache-action@v0.0.9

- name: Configure sccache
run: |
echo "SCCACHE_GHA_ENABLED=true" >> $GITHUB_ENV
echo "RUSTC_WRAPPER=sccache" >> $GITHUB_ENV

- name: Cache Cargo
uses: actions/cache@v4
with:
path: ~/.cargo
key: cargo-cache-${{ steps.rust-toolchain.outputs.cachekey }}-${{ hashFiles('Cargo.lock') }}

- uses: astral-sh/setup-uv@v5
with:
python-version: "3.12"
enable-cache: true

- name: Install xarray_sql
run: uv sync --locked --dev --extra test --no-install-package xarray-sql
- name: build rust
run: uv run --no-project maturin develop --uv
- name: Run the demos
env:
VIRTUAL_ENV: ${{ github.workspace }}/.venv
run: uv run --no-project pytest -v tests/test_ml_demos.py

@d33bs d33bs Oct 31, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider using --frozen here to help ensure reproducibility alongside the lockfile.

@alxmrs alxmrs Oct 31, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🌀 Claude: Done in f6cae08, with --locked rather than --frozen. Both install from the lockfile, but --locked also fails when uv.lock is out of date with pyproject.toml, where --frozen would quietly install the old lock.

Loading
Loading