codecbench

Whole-buffer deflate benchmarks across implementations.

One benchmark source builds once per backend, every executable registers the same benchmark names, and the JSON outputs compare directly with scripts/compare_runs.py. Every backend inflates identical zlib-ng level 9 streams, and all output is verified against the original data.

Backends

Executable	Backend	Option	Requires
codecbench_zlibng	zlib-ng (reference)	always
codecbench_libdeflate	libdeflate	WITH_LIBDEFLATE
codecbench_isal	ISA-L (igzip)	WITH_ISAL	nasm on x86
codecbench_slz	libslz (compress only)	WITH_SLZ
codecbench_chromium_zlib	Chromium zlib	WITH_CHROMIUM_ZLIB
codecbench_madler_zlib	madler zlib	WITH_MADLER_ZLIB
codecbench_zlib_rs	zlib-rs	WITH_ZLIB_RS	cargo
codecbench_miniz	miniz	WITH_MINIZ
codecbench_libcompression	libcompression	WITH_LIBCOMPRESSION	macOS
codecbench_zstd	zstd (format context)	WITH_ZSTD

Building

git clone https://github.com/zlib-ng/corpora test/data/corpora
cmake -B build
cmake --build build -j

Test data

The corpora clone provides the per-file corpora. For single-stream runs comparable with deflatebench, put its uncompressed Silesia tars in a corpora subdirectory (e.g. test/data/corpora/tars/):

• 203MiB full Silesia testcorpus
• 44MiB custom cropped Silesia testcorpus
• 16MiB custom cropped Silesia testcorpus

The original source of this testcorpus is Silesia.

Running

build/codecbench_zlibng --benchmark_list_tests=true
build/codecbench_zlibng --benchmark_filter="silesia" --benchmark_data_types=all

--benchmark_data_types=<type,...|all> selects the synthetic inputs. zlib API backends also report peak per-stream bytes as a mem counter. --benchmark_cooldown=<seconds> sleeps between benchmark families to mitigate thermal throttling.

Comparing

build/codecbench_zlibng --benchmark_out=zlibng.json --benchmark_out_format=json
build/codecbench_libdeflate --benchmark_out=libdeflate.json --benchmark_out_format=json
scripts/compare_runs.py zlibng.json libdeflate.json

Benchmarking a local zlib-ng

Point the reference backend at a checkout instead of the pinned release to measure work in progress:

cmake -B build -D ZLIBNG_SOURCE_DIR=~/Source/zlib-ng
cmake --build build -j

scripts/bench_pr.py <pr> benchmarks an upstream zlib-ng pull request against its merge-base with develop. It fetches both revisions, builds a zlib-ng-only codecbench for each under .pr-bench/, runs them sequentially, and prints the comparison:

scripts/bench_pr.py 2437 --graph pr2437.svg

Graphing

scripts/graph_runs.py turns two or more runs into a multi-panel speed versus ratio SVG and prints an aggregate table. It needs only the Python standard library.

scripts/graph_runs.py zlibng.json libdeflate.json -o zlibng_vs_libdeflate.svg

scripts/compare_levels.py charts the level ladder of an A/B pair, such as the base and head JSONs bench_pr.py leaves under .pr-bench/, above a table of the compressed size and cpu time at every level. Strategy variants found in the runs get their own lines and table, --strategies and --levels narrow the chart to some of them.

scripts/compare_levels.py .pr-bench/pr2437-base.json .pr-bench/pr2437-head.json \
    --names "PR base,PR head" -o pr2437-levels.svg

scripts/compare_inputs.py breaks the same A/B pair down input by input, a grid of the size and cpu time change per level for every corpus file and synthetic data type the runs share, tinted by direction. It shows where a result depends on the data.

scripts/insert_map.py <input> --pr <n> --level 6 -o out.svg maps which input positions two zlib-ng revisions insert into the hash table, as a raster of base, head and the difference with tar members marked. It builds traced copies of both revisions under .pr-bench, so it needs the toolchain the benchmarks need.

scripts/frontier.py base.json head.json -o out.svg draws one speed against size panel per input with both runs' level ladders, and rings any point the other run beats on both axes at some level, the trade a user of the level knob would take instead.

scripts/stream_anatomy.py base.json head.json --level 2 -o out.svg breaks the compressed streams down by what the bits were spent on, literals, lengths, distances, stored bytes and overhead, with the match mix by distance and by length beside them. It reads the bits_*, dist_* and len_* counters the deflate benchmarks emit from their stream walker, so a size-only run is enough.

scripts/symbol_cost.py base.json head.json --level 2 -o out.svg charts the price of each symbol instead, bits per literal, bits per match split into length and distance, and bits per byte a match produces, one scale per panel across the inputs, from the same counters.

scripts/compare_inputs.py base.json head.json --levels 2-6 -o inputs.svg

Results

All arm64 backends on silesia.tar, with zstd as format context, macOS on an Apple M5:

All eight x86_64 backends on silesia.tar, Windows 11 on an Intel Core i7-1185G7, clang 22:

Similar benchmarks

• deflatebench compares zlib-ng builds over single-stream runs.
• TurboBench benchmarks many compressors in one binary.
• lzbench is an in-memory benchmark of open-source compressors.
• squash-benchmark benchmarks the algorithms behind the Squash abstraction layer.