Skip to content

perf(model): relabel assign_coords containers without a label dict - #1030

Merged
FabianHofmann merged 2 commits into
PyPSA:masterfrom
olivier-lacroix:perf/assign-coords-relabel
Nov 3, 2026
Merged

FabianHofmann merged 2 commits into
PyPSA:masterfrom
olivier-lacroix:perf/assign-coords-relabel

Conversation

@olivier-lacroix

@olivier-lacroix olivier-lacroix commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Taking assign_coords for a spin, I realised it was way too slow. this PR fixes it. Sorry for not catching this sooner...

Note

The following content was generated by AI.

What

Model.assign_coords (from #955) relabelled containers by mapping each label through Index.map(dict(zip(master, new))), and rebuilt the dict once per container.

Two changes, both inside the #955 code:

  1. Positional relabel. master.get_indexer(index) locates each container's labels and new.take(positions) selects the new values. get_indexer also replaces the separate isin(master).all() check, so one lookup does both jobs.

    # before, per container
    index.map(dict(zip(master, new)))
    
    # after, per container
    new.take(master.get_indexer(index))

    On a tz-aware DatetimeIndex, the rolling-horizon case the API is built for, Index.map returns an object array of Timestamps and pandas infers datetime64 back. That inference accounts for 16 ms of the 20 ms per container on pandas 3.1.0 (objects_to_datetime64), and the dict rebuild adds 6 ms.

  2. One Dataset construction. reassign_coords writes the new index into the coordinate's original position and builds the dataset once, instead of Dataset.assign_coords(...) followed by a second Dataset(...) to restore the coordinate order. This is a small second gain (about 3 ms), not part of the speedup above.

Benchmark

30 containers (15 variables + 15 CSR constraints), 4380 tz-aware labels, Model(sparse=True), pandas 3.1.0, a fresh model per call, interleaved, best of 5:

change Model.assign_coords
none (master) 809 ms
one Dataset construction only 805 ms
positional relabel only 11.5 ms
both 8.8 ms
Reproduction
import time
import pandas as pd
import linopy
from linopy import Model

idx = pd.date_range("2024-01-01", periods=4380, freq="30min", tz="Australia/Melbourne", name="time")
new = pd.date_range("2024-04-01", periods=4380, freq="30min", tz="Australia/Melbourne", name="time")

m = Model(sparse=True)
vs = [m.add_variables(coords=[idx], name=f"x{i}") for i in range(15)]
for i, v in enumerate(vs):
    m.add_constraints(v >= 0, name=f"c{i}")

best = 1e9
for _ in range(3):
    t0 = time.perf_counter()
    m.assign_coords(time=new)
    best = min(best, time.perf_counter() - t0)
print(f"{best * 1000:.1f} ms")

Checklist

  • AI-generated content is marked (see AGENTS.md).
  • Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
  • Unit tests for new features were added (if applicable).
  • A note for the release notes doc/release_notes.rst of the upcoming release is included.
  • I consent to the release of this PR's code under the MIT license.

@codspeed

codspeed Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 181 untouched benchmarks
⏩ 181 skipped benchmarks1


Comparing olivier-lacroix:perf/assign-coords-relabel (314a6ed) with master (b9ec07b)2

Open in CodSpeed

Footnotes

  1. 181 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on master (2ace7b3) during the generation of this report, so b9ec07b was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

@olivier-lacroix
olivier-lacroix force-pushed the perf/assign-coords-relabel branch 3 times, most recently from 14e9d58 to 824d1f0 Compare October 11, 2026 12:21
@olivier-lacroix
olivier-lacroix marked this pull request as ready for review October 11, 2026 12:22
@olivier-lacroix
olivier-lacroix force-pushed the perf/assign-coords-relabel branch 2 times, most recently from dbf4c67 to 663a15c Compare October 27, 2026 03:11
@olivier-lacroix

olivier-lacroix commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor Author

Rebased onto master at 8736b35 (2026-10-29). Re-measured against a worktree of 8736b35, interleaved runs, best of 3 assign_coords calls:

checkout assign_coords
8736b35 914 ms (871 to 967)
this branch 9.4 ms (8.5 to 14.1)

The gain is about 100x, versus 280x in the original benchmark.

@olivier-lacroix
olivier-lacroix force-pushed the perf/assign-coords-relabel branch from 663a15c to de4ade1 Compare October 29, 2026 10:01
@FabianHofmann

Copy link
Copy Markdown
Collaborator

I fail to understand the changes when scanning over them. is the problem perhaps within maybe_infer_to_datetimelike being slow? could you remove the inline comments from the code please?

Olivier Lacroix and others added 2 commits November 3, 2026 11:39
Model.assign_coords rebuilt a master->new dict per container and mapped each
container's labels through Index.map. On a tz-aware DatetimeIndex that map
falls into pandas' object-to-datetime inference, costing milliseconds per
container. Map full-length containers straight to the new values and locate
subset containers' labels with get_indexer/take, and build a reassigned
Dataset once instead of twice.

~817 ms -> ~3 ms for 30 containers on a 4380-label tz-aware axis.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Drop the inline comments, the `equals` fast path and the redundant
`.rename(dim)`: `get_indexer` returns arange for a full-length index,
and `take` preserves the name. Update the test docstring and the
benchmark numbers in the release note.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@olivier-lacroix
olivier-lacroix force-pushed the perf/assign-coords-relabel branch from d1fba72 to 314a6ed Compare November 3, 2026 00:46
@olivier-lacroix

Copy link
Copy Markdown
Contributor Author

@FabianHofmann , yes, the slowness is pandas' datetimelike inference. Index.map(dict) on a tz-aware DatetimeIndex produces an object array of Timestamps and pandas converts it back to datetime64. The dict rebuild was adding to the costs.

Inline comments removed & code simplified. The 'equals' branch is gone as benefits are super tiny, not worth an extra branch.

Best of 5: 809 ms on master, 11.5 ms with the relabel alone, 8.8 ms with both changes. The PR description now has the before/after and this table.

@FabianHofmann
FabianHofmann merged commit 9c000ea into PyPSA:master Nov 3, 2026
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants