Remove the legacy heap region from snapshots by ludfjig · Pull Request #1968 · hyperlight-dev/hyperlight

ludfjig · 2026-10-21T18:10:33Z

Remove the fixed-size guest heap region from sandbox memory. The region was pure waste. It held only zeros that the guest in practice never reads, and the first write to each page copied it into scratch anyway. The guest now maps each heap page from scratch on first touch, the same way it maps stack pages, so snapshots no longer carry the zero-filled region and the heap is bounded only by scratch.

This benefits every guest workload. Every guest allocates heap, and each heap page it first writes after sandbox creation or a restore maps a fresh scratch page instead of copying a zero page out of the snapshot. The saving is per page, so it grows with the heap a call writes:

A call that writes 4 to 128 MiB of fresh heap gets 26 to 31% faster on MSHV and 40 to 67% faster on KVM.
guest_call_with_large_parameters, which passes 50 MiB arguments, gets 34% faster on MSHV and 36% faster on KVM.

Snapshots also shrink by the size of the heap. With a 256 MiB heap configured:

Saving, loading and cold starting from a snapshot get 98 to 99.7% faster.
Sandbox creation gets 93% faster on MSHV and 24% faster on KVM.

Closes #1697

How it works

At init the guest claims a fixed 512 GiB virtual range with talc. The page fault handler maps a fresh scratch page on each heap page's first touch, so untouched pages cost nothing.
This replaces a copy-on-write fault that also copied 4 KiB from a snapshot zero page, made the host populate that page, and flushed the TLB entry.
Heap pages written since the last snapshot use scratch, alongside the stack, page tables and transport. Allocations can exceed scratch. A write that needs a page when scratch is full aborts the guest with MallocFailed.
executable_heap stays a host feature. The host used to set heap permissions in the page tables it built. The guest now sets them when it maps each heap page, from a PEB flag the host writes, and heap pages stay non-executable by default. The host boundary does not change. The guest owns its page tables and scratch is mapped executable in the hypervisor, so a malicious guest could always make any page executable. A non-executable heap only stops a well-behaved guest's bugs from running injected data as code.

Alternatives considered

ZeroInit page-table entries for the heap. Rejected for performance, and because it needs a fixed heap size again. The ZeroInit marker exists only on leaf entries, one per 4 KiB page, so all of the heap's page tables exist up front, 1/512 of the heap size. A 256 MiB heap adds 512 KiB of page tables that every snapshot captures and every restore copies into scratch. Each first touch is also slower, because the fault handler must walk the page tables to find the marker, whereas this PR only needs a range check.
Keep a sized heap region but leave it out of snapshot memory. This keeps the heap size setting, and first touch stays a copy-on-write fault.
Grow the talc heap on demand. A talc source can extend the heap only while scratch has room, so oversized allocations return null. It adds growth state to the allocator. Scratch is shared with the stack and page tables, so a write can still find scratch full.

Breaking changes

SandboxConfiguration::set_heap_size, SandboxConfiguration::DEFAULT_HEAP_SIZE, SandboxBuilder::heap_size and SandboxBuilder::get_heap_size are removed. Size scratch for the guest's heap use.
Running out of memory aborts the guest with MallocFailed on the write that needs a page. Allocations no longer return null and Rust guests no longer panic with memory allocation of N bytes failed.
Snapshots use ABI 8 and config schema v5. Existing snapshots must be regenerated. Goldens move to v8.0.
HyperlightPEB::guest_heap is replaced by heap_executable.

Commits

Pass the expected arguments in test_initialise. The test relied on 0x8000 falling inside the heap region.
Check the stack range before walking guest page tables.
Map guest heap pages on first touch.
Remove the guest heap region and heap size configuration.
Bump the snapshot ABI to 8 and the config schema to v5.
Update tests for the scratch-backed heap.
Size benchmarks by scratch and add heap_first_touch.

Benchmarks

Compared to the PR below.

MSHV: Linux 6.6 mshv.
KVM: Linux 6.17.

Each size keeps its scratch and drops its heap: default has a 128 KiB heap, and small, medium and large have 8, 64 and 256 MiB. This PR redefines the sizes as 8, 64 and 256 MiB of scratch, so later results are not comparable with these by name. heap_first_touch is new. It writes half of 8, 64 or 256 MiB of scratch, and the PR below gets a heap as large as scratch.

Highlights

Benchmark	MSHV	KVM
`guest_calls/heap_first_touch/medium`	8.74 ms → 6.1 ms (−30%)	50.1 ms → 16.6 ms (−67%)
`guest_calls/heap_first_touch/large`	35.7 ms → 24.7 ms (−31%)	535 ms → 276 ms (−48%)
`guest_functions_with_large_parameters/guest_call_with_large_parameters`	81 ms → 53.4 ms (−34%)	863 ms → 549 ms (−36%)
`sandboxes/create_initialized/large`	27.7 ms → 1.83 ms (−93%)	3.06 ms → 2.32 ms (−24%)
`sandboxes/sandbox_from_snapshot/large`	97.5 ms → 1.11 ms (−99%)	1.07 ms → 762 µs (−29%)
`snapshots/create/large`	1.67 ms → 245 µs (−85%)	1.69 ms → 286 µs (−83%)
`snapshot_files/save_snapshot/large`	267 ms → 849 µs (−99.7%)	1.87 s → 36 ms (−98%)
`snapshot_files/cold_start_via_snapshot/large`	291 ms → 2.12 ms (−99%)	192 ms → 2.56 ms (−99%)

Regressions

Benchmark	MSHV	KVM	Cause
`snapshots/create/default`	230 µs → 248 µs (+7%)	265 µs → 282 µs (+7%)	Likely the heap's own page tables. Init writes both ends of the 512 GiB heap range, which needs 5 page-table pages. Capture walks and rebuilds them. The 128 KiB heap fit in an existing page table
`snapshot_files/cold_start_via_snapshot_unverified/default`	1.53 ms → 1.39 ms (~)	2.05 ms → 2.12 ms (+4%)	Likely the same page-table pages. Restore copies them into fresh scratch, and each fresh page costs a host fault on KVM

WHP and aarch64 were not benchmarked.

Full results

All benchmarks on both hosts (click to expand)

Before → after (change). Negative is faster. Bold is a regression. ~ is no consistent change. A change needs every run that measured it to agree: a 0.5 s run, a 3 s run, and for results only some runs flagged, a 3 s rerun with the two sides in reverse order. snapshots/create/default counts as a regression because both 3 s runs agree on both hosts. Values are from the 3 s run.

sandboxes

Benchmark	Size	MSHV	KVM
`create_initialized`	default	1.56 ms → 1.56 ms (~)	2.47 ms → 2.33 ms (−6%)
`create_initialized`	small	2.75 ms → 1.55 ms (−44%)	2.5 ms → 2.34 ms (~)
`create_initialized`	medium	8.51 ms → 1.62 ms (−81%)	2.61 ms → 2.5 ms (−4%)
`create_initialized`	large	27.7 ms → 1.83 ms (−93%)	3.06 ms → 2.32 ms (−24%)
`create_initialized_and_drop`	default	33.3 ms → 33.5 ms (~)	13.8 ms → 12.9 ms (~)
`create_initialized_and_drop`	small	33.4 ms → 29.6 ms (~)	13.2 ms → 14.6 ms (~)
`create_initialized_and_drop`	medium	42.4 ms → 33.6 ms (−21%)	12.6 ms → 14.4 ms (~)
`create_initialized_and_drop`	large	70.7 ms → 33.4 ms (−53%)	13.9 ms → 13.1 ms (~)
`sandbox_from_snapshot`	default	915 µs → 895 µs (~)	852 µs → 807 µs (~)
`sandbox_from_snapshot`	small	2.97 ms → 929 µs (−69%)	838 µs → 836 µs (~)
`sandbox_from_snapshot`	medium	14.7 ms → 960 µs (−93%)	907 µs → 791 µs (−13%)
`sandbox_from_snapshot`	large	97.5 ms → 1.11 ms (−99%)	1.07 ms → 762 µs (−29%)

guest_calls

Benchmark	Size	MSHV	KVM
`call`	default	42.7 µs → 45.2 µs (~)	18 µs → 18.6 µs (~)
`call`	small	46.7 µs → 42.9 µs (~)	17.7 µs → 17.7 µs (~)
`call`	medium	43.4 µs → 43.5 µs (~)	17.9 µs → 17.2 µs (~)
`call`	large	44.3 µs → 42.8 µs (~)	17.9 µs → 18.4 µs (~)
`call_with_host_function`	default	71.7 µs → 75 µs (~)	32.6 µs → 36.1 µs (~)
`call_with_host_function`	small	76.6 µs → 71.7 µs (−4%)	34.5 µs → 36.1 µs (~)
`call_with_host_function`	medium	75.4 µs → 73.7 µs (−4%)	34.8 µs → 36 µs (~)
`call_with_host_function`	large	72.9 µs → 74.1 µs (~)	34.8 µs → 36.5 µs (~)
`call_with_restore`	default	137 µs → 140 µs (~)	63 µs → 65.8 µs (~)
`call_with_restore`	small	138 µs → 138 µs (~)	66.4 µs → 65.3 µs (~)
`call_with_restore`	medium	146 µs → 145 µs (~)	69.2 µs → 68.1 µs (~)
`call_with_restore`	large	164 µs → 152 µs (−7%)	89.9 µs → 76.9 µs (−13%)
`different_thread`	-	68.2 µs → 66.2 µs (~)	40.2 µs → 40.8 µs (~)
`heap_first_touch`	small	825 µs → 619 µs (−26%)	957 µs → 640 µs (−40%)
`heap_first_touch`	medium	8.74 ms → 6.1 ms (−30%)	50.1 ms → 16.6 ms (−67%)
`heap_first_touch`	large	35.7 ms → 24.7 ms (−31%)	535 ms → 276 ms (−48%)
`interrupt_latency`	-	46.8 µs → 46.4 µs (~)	22.7 µs → 21.8 µs (~)

snapshots

Benchmark	Size	MSHV	KVM
`call_and_create`	default	333 µs → 343 µs (~)	404 µs → 415 µs (~)
`call_and_create`	small	372 µs → 337 µs (−12%)	453 µs → 411 µs (~)
`call_and_create`	medium	679 µs → 338 µs (−49%)	745 µs → 422 µs (−44%)
`call_and_create`	large	1.75 ms → 350 µs (−80%)	1.85 ms → 422 µs (−77%)
`create`	default	230 µs → 248 µs (+7%)	265 µs → 282 µs (+7%)
`create`	small	273 µs → 254 µs (~)	340 µs → 279 µs (−15%)
`create`	medium	580 µs → 244 µs (−58%)	622 µs → 270 µs (−55%)
`create`	large	1.67 ms → 245 µs (−85%)	1.69 ms → 286 µs (−83%)
`restore`	default	64.4 µs → 68.2 µs (~)	19.5 µs → 20.1 µs (~)
`restore`	small	64.1 µs → 66.5 µs (~)	20.7 µs → 20 µs (−3%)
`restore`	medium	71.5 µs → 66.7 µs (−9%)	25 µs → 22.4 µs (−11%)
`restore`	large	96.9 µs → 76.6 µs (−22%)	48.4 µs → 31.8 µs (−32%)
`restore_alternate`	default	79.1 µs → 78.5 µs (~)	93.3 µs → 93 µs (~)
`restore_alternate`	small	80.7 µs → 81.7 µs (~)	94.5 µs → 94.8 µs (~)
`restore_alternate`	medium	83.9 µs → 83.4 µs (~)	99.3 µs → 96.3 µs (~)
`restore_alternate`	large	110 µs → 92 µs (−17%)	122 µs → 107 µs (−13%)

snapshot_files

Benchmark	Size	MSHV	KVM
`cold_start_via_evolve`	default	1.67 ms → 1.6 ms (~)	2.58 ms → 2.73 ms (~)
`cold_start_via_evolve`	small	2.43 ms → 1.59 ms (−35%)	2.78 ms → 2.74 ms (~)
`cold_start_via_evolve`	medium	8.18 ms → 1.71 ms (−79%)	2.76 ms → 2.6 ms (−6%)
`cold_start_via_evolve`	large	27.3 ms → 1.8 ms (−93%)	3.24 ms → 2.68 ms (−17%)
`cold_start_via_snapshot`	default	2.09 ms → 2 ms (~)	2.52 ms → 2.49 ms (~)
`cold_start_via_snapshot`	small	11.9 ms → 1.95 ms (−84%)	8.41 ms → 2.5 ms (−70%)
`cold_start_via_snapshot`	medium	78.9 ms → 1.98 ms (−97%)	49.9 ms → 2.48 ms (−95%)
`cold_start_via_snapshot`	large	291 ms → 2.12 ms (−99%)	192 ms → 2.56 ms (−99%)
`cold_start_via_snapshot_unverified`	default	1.53 ms → 1.39 ms (~)	2.05 ms → 2.12 ms (+4%)
`cold_start_via_snapshot_unverified`	small	4.79 ms → 1.41 ms (−71%)	2.01 ms → 2 ms (~)
`cold_start_via_snapshot_unverified`	medium	26.8 ms → 1.5 ms (−94%)	2.08 ms → 2.15 ms (~)
`cold_start_via_snapshot_unverified`	large	96 ms → 1.64 ms (−98%)	2.16 ms → 2.17 ms (~)
`load_snapshot`	default	703 µs → 629 µs (−11%)	684 µs → 613 µs (−10%)
`load_snapshot`	small	6.75 ms → 631 µs (−91%)	6.35 ms → 612 µs (−90%)
`load_snapshot`	medium	51.9 ms → 630 µs (−99%)	47.7 ms → 611 µs (−99%)
`load_snapshot`	large	196 ms → 630 µs (−99.7%)	189 ms → 611 µs (−99.7%)
`load_snapshot_unverified`	default	108 µs → 111 µs (+3%)	106 µs → 107 µs (~)
`load_snapshot_unverified`	small	109 µs → 111 µs (~)	115 µs → 107 µs (~)
`load_snapshot_unverified`	medium	108 µs → 112 µs (~)	113 µs → 107 µs (~)
`load_snapshot_unverified`	large	110 µs → 112 µs (~)	117 µs → 106 µs (~)
`save_snapshot`	default	944 µs → 846 µs (−10%)	52.9 ms → 33.5 ms (−37%)
`save_snapshot`	small	8.97 ms → 843 µs (−91%)	58.2 ms → 45.2 ms (−22%)
`save_snapshot`	medium	67.7 ms → 849 µs (−99%)	469 ms → 44.8 ms (−90%)
`save_snapshot`	large	267 ms → 849 µs (−99.7%)	1.87 s → 36 ms (−98%)

guest_functions_with_large_parameters

Benchmark	Size	MSHV	KVM
`guest_call_with_large_parameters`	-	81 ms → 53.4 ms (−34%)	863 ms → 549 ms (−36%)

sample_workloads

Benchmark	Size	MSHV	KVM
`24K_in_8K_out_c`	-	60.1 µs → 60.7 µs (~)	29.9 µs → 30.2 µs (~)
`24K_in_8K_out_rust`	-	57.2 µs → 60.4 µs (~)	30.1 µs → 31.1 µs (+4%)

dummyguest halts only when it receives its expected PEB address and seed. Otherwise it reads 0x8000 to force an exit. The test passed a dummy PEB address and seed, so it relied on 0x8000 being mapped, which held only because that address fell inside the guest heap region. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

A not-present fault in the stack range always maps a fresh scratch page, so the page table walk in virt_to_phys is wasted work. Check the range first and walk only for faults outside it. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

The guest heap is a fixed 512 GiB virtual range. Guest init claims the whole range with talc, and the page fault handler maps each heap page from scratch on first touch, the same way it maps stack pages. Heap memory is bounded by scratch rather than a fixed heap size, and untouched heap pages cost nothing. The host passes the executable heap policy in the PEB, which replaces the heap address and size. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

The guest maps heap pages from scratch on first touch, so the host no longer reserves a zero-filled heap region in snapshot memory. This removes SandboxConfiguration::set_heap_size, DEFAULT_HEAP_SIZE, and SandboxBuilder::heap_size and get_heap_size. Scratch is the only guest memory setting. Snapshots no longer carry up to hundreds of MiB of zero pages, which speeds up sandbox creation and snapshot capture, save, and load. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

Snapshot memory no longer contains a heap region, and the PEB carries the executable heap policy in place of the heap address and size. The config schema drops the heap size. This is a hard break. Loaders reject config v4 and older, and the goldens move to v8.0. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

Tests that sized the heap now size scratch. Allocation succeeds beyond scratch, and the guest aborts with MallocFailed when a write needs a page and scratch is full, so heap exhaustion tests expect that abort. FillHeapAndCauseException is removed because the heap can no longer run out before scratch does. The shared sandbox sizes in hyperlight-testing become scratch sizes. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

Benchmark sandbox sizes set scratch, which is the only guest memory setting. The small, medium, and large variants use 8, 64, and 256 MiB of scratch. heap_first_touch measures writes to heap the guest has not touched since the last restore. Each variant writes half its scratch. The default size is skipped because fixed reservations take most of its scratch. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

hyperlight-gh-bot · 2026-10-21T21:48:26Z

Benchmark Results

Measured commit: 8b0f611b4101
Baseline commit: e92a00406601

kvm / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`757.65 ns` (➖ 1.00x slower)
`vec_bytes`	`588.05 ns` (➖ 1.00x faster)
		`374.02 µs` (➖ 1.00x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`552.05 ns` (➖ 1.04x slower)
`65536`	`141.49 ns` (➖ 1.00x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`13.56 ms` (➖ 1.01x slower)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`7.75 ns` (➖ 1.00x slower)	`7.76 ns` (➖ 1.00x faster)	`7.76 ns` (➖ 1.00x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`116.96 µs` (➖ 1.00x faster)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`7.36 µs` (➖ 1.01x slower)	`7.07 µs` (➖ 1.06x faster)
`65536`	`2.04 µs` (➖ 1.04x slower)	`1.93 µs` (➖ 1.09x faster)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`6.19 µs` (➖ 1.06x faster)	`6.22 µs` (➖ 1.03x faster)
`8192`	`1.05 µs` (➖ 1.02x faster)	`1.05 µs` (➖ 1.00x faster)
`262144`		`28.53 µs` (➖ 1.04x slower)

kvm / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`787.41 ns` (➖ 1.02x faster)
`vec_bytes`	`616.95 ns` (➖ 1.00x faster)
		`765.87 µs` (➖ 1.02x slower)

payload_allocation

	`slot_pool_segmented`
`262144`	`567.73 ns` (➖ 1.07x faster)
`65536`	`151.43 ns` (➖ 1.07x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`14.97 ms` (➖ 1.01x slower)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`7.75 ns` (➖ 1.06x faster)	`7.96 ns` (➖ 1.04x faster)	`7.83 ns` (➖ 1.05x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`65.24 µs` (➖ 1.05x slower)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`8.36 µs` (➖ 1.08x faster)	`8.46 µs` (➖ 1.05x faster)
`65536`	`2.40 µs` (➖ 1.05x faster)	`2.34 µs` (➖ 1.07x faster)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`7.72 µs` (➖ 1.08x faster)	`7.77 µs` (➖ 1.09x faster)
`8192`	`823.76 ns` (➖ 1.08x faster)	`796.13 ns` (➖ 1.08x faster)
`262144`		`34.11 µs` (➖ 1.00x faster)

mshv3 / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`974.81 ns` (➖ 1.05x slower)
`vec_bytes`	`712.41 ns` (➖ 1.00x faster)
		`285.96 µs` (➖ 1.07x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`725.14 ns` (➖ 1.00x faster)
`65536`	`192.49 ns` (➖ 1.01x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`56.48 ms` (➖ 1.12x faster)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`9.91 ns` (➖ 1.01x slower)	`9.81 ns` (➖ 1.00x slower)	`11.42 ns` (➖ 1.17x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`111.79 µs` (➖ 1.03x slower)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`10.87 µs` (➖ 1.10x faster)	`9.68 µs` (➖ 1.03x faster)
`65536`	`2.39 µs` (➖ 1.03x faster)	`2.25 µs` (➖ 1.00x faster)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`8.19 µs` (➖ 1.24x faster)	`8.38 µs` (➖ 1.18x faster)
`8192`	`1.31 µs` (➖ 1.01x slower)	`1.31 µs` (➖ 1.03x slower)
`262144`		`36.58 µs` (➖ 1.04x faster)

mshv3 / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`931.84 ns` (➖ 1.02x faster)
`vec_bytes`	`660.50 ns` (➖ 1.02x faster)
		`708.08 µs` (➖ 1.09x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`647.47 ns` (➖ 1.01x slower)
`65536`	`171.06 ns` (➖ 1.00x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`75.04 ms` (➖ 1.10x slower)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`8.81 ns` (➖ 1.00x slower)	`8.87 ns` (➖ 1.01x slower)	`8.83 ns` (➖ 1.01x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`56.53 µs` (➖ 1.06x faster)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`8.01 µs` (➖ 1.01x slower)	`8.10 µs` (➖ 1.01x slower)
`65536`	`2.34 µs` (➖ 1.03x slower)	`2.29 µs` (➖ 1.01x slower)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`7.56 µs` (➖ 1.01x slower)	`7.58 µs` (➖ 1.01x slower)
`8192`	`951.77 ns` (➖ 1.00x faster)	`932.64 ns` (➖ 1.02x slower)
`262144`		`39.68 µs` (➖ 1.02x faster)

hyperv-ws2025 / amd (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`1.18 µs` (➖ 1.00x faster)
`vec_bytes`	`821.32 ns` (➖ 1.04x slower)
		`1.96 ms` (➖ 1.05x slower)

payload_allocation

	`slot_pool_segmented`
`262144`	`797.17 ns` (➖ 1.00x slower)
`65536`	`233.50 ns` (➖ 1.00x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`11.88 ms` (➖ 1.10x slower)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`10.43 ns` (➖ 1.06x slower)	`10.15 ns` (➖ 1.01x slower)	`10.12 ns` (➖ 1.00x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`774.86 µs` (➖ 1.24x faster)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`9.24 µs` (➖ 1.04x faster)	`9.13 µs` (➖ 1.05x faster)
`65536`	`2.37 µs` (➖ 1.01x faster)	`2.35 µs` (➖ 1.01x faster)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`9.17 µs` (➖ 1.01x faster)	`8.76 µs` (➖ 1.02x faster)
`8192`	`1.32 µs` (➖ 1.07x faster)	`1.31 µs` (➖ 1.01x faster)
`262144`		`39.66 µs` (➖ 1.02x faster)

hyperv-ws2025 / intel (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`1.26 µs` (➖ 1.01x faster)
`vec_bytes`	`801.51 ns` (➖ 1.01x faster)
		`3.05 ms` (➖ 1.01x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`754.45 ns` (➖ 1.03x faster)
`65536`	`219.97 ns` (➖ 1.02x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`14.03 ms` (➖ 1.10x slower)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`10.72 ns` (➖ 1.00x slower)	`10.10 ns` (➖ 1.03x faster)	`10.26 ns` (➖ 1.00x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`672.55 µs` (➖ 1.15x faster)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`7.82 µs` (➖ 1.02x slower)	`7.95 µs` (➖ 1.02x slower)
`65536`	`2.35 µs` (➖ 1.02x slower)	`2.34 µs` (➖ 1.01x faster)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`8.33 µs` (➖ 1.03x faster)	`9.20 µs` (➖ 1.08x slower)
`8192`	`1.11 µs` (➖ 1.13x faster)	`1.09 µs` (➖ 1.00x slower)
`262144`		`44.18 µs` (➖ 1.10x slower)

_{Reported by cargo ci bench-report --candidate run:39675521234 --config-file bench_report.toml.}

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

3 open issues

Using Ordering::Relaxed to publish HEAP_EXECUTABLE is risky because the value is consumed from… · New
Using Ordering::Relaxed to publish HEAP_EXECUTABLE is risky because the value is consumed from… · New
The variable name init_data_offset here is misleading: this value is the offset after pushing the… · New

What changed in this PR

Removes the legacy fixed-size guest heap region from snapshot memory and switches the guest to mapping heap pages from scratch on first touch, reducing snapshot size and improving performance.

Changes:

Removes heap-size configuration and updates tests/fixtures/benchmarks to size by scratch instead.
Bumps snapshot ABI to 8 and config schema media type to v5, updating goldens and versioning docs.
Updates guest fault handling to map heap pages from scratch on demand and propagates heap executability via a PEB flag.

File	Description
src/tests/rust_guests/simpleguest/src/main.rs	Updates guest OOM/exhaustion behavior to reflect scratch-backed heap semantics.
src/hyperlight_testing/src/lib.rs	Renames benchmark/test size constants from heap sizes to scratch sizes and adjusts types.
src/hyperlight_host/tests/snapshot_goldens/goldens_version.rs	Bumps goldens version to v8.0 for the new snapshot ABI/schema.
src/hyperlight_host/tests/snapshot_goldens/fixtures.rs	Removes heap sizing from golden snapshot config fixture.
src/hyperlight_host/tests/sandbox_host_tests.rs	Updates leak-related tests to constrain scratch rather than heap.
src/hyperlight_host/tests/integration_test.rs	Updates integration tests to assert scratch exhaustion aborts with `MallocFailed`.
src/hyperlight_host/src/sandbox/uninitialized.rs	Updates sandbox/snapshot tests to remove heap sizing and validate scratch-only layout behavior.
src/hyperlight_host/src/sandbox/snapshot/tripwires.rs	Bumps expected ABI/schema and updates invariant commentary around layout offsets.
src/hyperlight_host/src/sandbox/snapshot/file_tests.rs	Updates snapshot file tests for schema v5 and ABI 8; removes heap layout assertions.
src/hyperlight_host/src/sandbox/snapshot/file/mod.rs	Updates loader to accept config v5 only and removes heap from persisted layout reconstruction.
src/hyperlight_host/src/sandbox/snapshot/file/media_types.rs	Adds config v5 media type and bumps snapshot ABI version to 8.
src/hyperlight_host/src/sandbox/snapshot/file/config.rs	Removes heap_size from config schema/layout and updates schema pins/validation.
src/hyperlight_host/src/sandbox/initialized.rs	Removes heap override propagation from snapshots and updates tests to size by scratch.
src/hyperlight_host/src/sandbox/config.rs	Removes heap size configuration API; clarifies scratch semantics and updates tests.
src/hyperlight_host/src/sandbox/builder.rs	Removes heap_size builder APIs; updates docs to show scratch sizing.
src/hyperlight_host/src/mem/mgr.rs	Updates memory manager tests to use scratch sizing constants (no heap sizing).
src/hyperlight_host/src/mem/layout.rs	Removes heap region from snapshot memory layout; updates offsets/PEB contents accordingly.
src/hyperlight_host/src/hypervisor/mod.rs	Adjusts initialise test parameters now that heap region assumptions are gone.
src/hyperlight_host/examples/map-file-cow-test/main.rs	Removes heap sizing from example; relies on scratch sizing only.
src/hyperlight_host/benches/benchmarks.rs	Re-keys benchmark configurations to scratch sizes and adds `heap_first_touch` benchmark.
src/hyperlight_guest_bin/src/lib.rs	Claims a fixed virtual heap range and reads heap executability from the PEB.
src/hyperlight_guest_bin/src/arch/amd64/layout.rs	Adds a static assert to ensure heap range doesn’t overlap processor control.
src/hyperlight_guest_bin/src/arch/amd64/exception/handle.rs	Maps heap pages from scratch on first touch; supports executable heap via PEB flag.
src/hyperlight_guest_bin/src/arch/aarch64/exception/handle.rs	Same as amd64: scratch-backed heap faults with executable flag support.
src/hyperlight_guest/src/layout.rs	Exposes heap virtual range constants and asserts no overlap with transport/stack.
src/hyperlight_guest/src/arch/amd64/layout.rs	Defines 512GiB-aligned heap virtual range for amd64.
src/hyperlight_guest/src/arch/aarch64/layout.rs	Defines 512GiB-aligned heap virtual range for aarch64.
src/hyperlight_common/src/mem.rs	Replaces PEB heap region field with a heap executability flag.
fuzz/fuzz_targets/host_call.rs	Removes heap sizing from fuzz target configuration.
docs/snapshot-versioning.md	Documents ABI 8/config v5 and notes heap region removal from snapshot memory.
docs/snapshot-oci-format.md	Updates config media type reference to v5.
docs/hyperlight-execution-details.md	Updates lifecycle docs to describe scratch-backed stack+heap mapping.
CHANGELOG.md	Records breaking changes: ABI/schema bump and removal of heap sizing APIs.

_{Review effort: Lite}

+/// Whether heap pages are mapped executable on first touch.
+pub(crate) static HEAP_EXECUTABLE: AtomicBool = AtomicBool::new(false);
+



-        let heap_start = (*peb_ptr).guest_heap.ptr as usize;
-        let heap_size = (*peb_ptr).guest_heap.size as usize;
+        HEAP_EXECUTABLE.store((*peb_ptr).heap_executable != 0, Ordering::Relaxed);


+        let init_data_offset =
            builder.push_page_aligned(size_of::<HyperlightPEB>(), MemoryRegionFlags::READ, Peb);


ludfjig added area/performance Addresses performance kind/refactor For PRs that restructure or remove code without adding new functionality. area/API Related to the API or public interface regen-goldens Regenerate snapshot golden fixtures labels Oct 21, 2026

ludfjig changed the title ~~Map guest heap pages on first touch~~ Remove the legacy heap region from snapshots Oct 21, 2026

ludfjig added this pull request to stack #1929 October 21, 2026 18:14

ludfjig added 3 commits October 21, 2026 11:19

This comment has been minimized.

Sign in to view

ludfjig force-pushed the ll/4-unbounded-heap branch from 1642d27 to f27c1ef Compare October 21, 2026 19:27

ludfjig added 4 commits October 21, 2026 13:19

This comment has been minimized.

Sign in to view

ludfjig force-pushed the ll/4-unbounded-heap branch from f27c1ef to 8b0f611 Compare October 21, 2026 21:02

ludfjig marked this pull request as ready for review October 23, 2026 21:25

ludfjig requested review from andreiltd, danbugs, dblnz, devigned, jprendes, jsturtevant, simongdavies, squillace and syntactically as code owners October 23, 2026 21:25

Copilot AI balanced review requested due to automatic review settings October 23, 2026 21:25

Copilot AI reviewed Oct 23, 2026

View reviewed changes

		/// Whether heap pages are mapped executable on first touch.
		pub(crate) static HEAP_EXECUTABLE: AtomicBool = AtomicBool::new(false);

		let init_data_offset =
		builder.push_page_aligned(size_of::<HyperlightPEB>(), MemoryRegionFlags::READ, Peb);

Conversation

ludfjig commented Oct 21, 2026

How it works

Alternatives considered

Breaking changes

Commits

Benchmarks

Highlights

Regressions

Full results

sandboxes

guest_calls

snapshots

snapshot_files

guest_functions_with_large_parameters

sample_workloads

Uh oh!

This comment has been minimized.

This comment has been minimized.

hyperlight-gh-bot Bot commented Oct 21, 2026

Benchmark Results

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects