stone benchmarks¶
This crate times stone’s compiled output (stone build) against the same programs written in C, Rust, and Python, and with --interpreter, the interpreter (stone run) too.
cargo run --release --manifest-path bench/Cargo.toml -- [--runs 5] [--filter NAME] [--stone PATH] [--numpy PYTHON] [--min] [--interpreter] [--checks]
For each benchmark, the runner builds stone in release mode, compiles the .stn with stone build and the .c with gcc -O2 -fwrapv -ffp-contract=off and gcc -O0 -fwrapv -ffp-contract=off (linking -lm), the .rs (and a .fast.rs, if the benchmark has one) with rustc --edition 2024 -C opt-level=3 -C codegen-units=1 -C panic=abort, and runs the .py with python3. Every version runs with the arguments in the benchmark’s .args file, one per line, as the golden tests’ .args files are. The runner runs each implementation once untimed and requires its stdout to match the .out file exactly. It then times --runs more runs, checking the output of each, and prints a Markdown table of the median wall times, or the fastest with --min. Progress and warnings go to stderr, so > results.md captures only the table. It needs gcc and python3 on PATH, and leaves the Rust columns out with a warning when rustc is missing.
With --interpreter, the runner also times each .stn under stone run, checked against the .out file like the others, and adds stone run (ms) and stone run / Python columns: how many times slower than CPython the interpreter is, lower being better. The workloads are sized for compiled code, so the interpreter takes several seconds on most of them; --filter and --runs 1 keep a run short.
A benchmark of arithmetic on lists can also have a numpy version, name.np.py, which runs under the Python that --numpy names (python3 by default), such as a virtualenv’s bin/python, and adds numpy (ms) and numpy / stone columns. If that Python cannot import numpy, the runner warns and leaves numpy out.
Every benchmark has an idiomatic, safe Rust version, name.rs (Vec, Box, HashMap, iterators, and indexing with its bounds checks), and some have a tuned one, name.fast.rs, written the way an expert would for speed with the same algorithm: flat arrays, an arena of indexes, a fast hasher, numbers formatted into one buffer, all in safe Rust. They add Rust (ms), Rust fast (ms), stone / Rust, and stone / best Rust columns, the last against the faster of the two. Beating the tuned version, not only the idiomatic one, is the goal: that is what C or Rust written for speed looks like.
With --checks, the runner also builds each .stn with STONE_STATS=1 and adds a checks in loops column: how many index, division, call depth, and other checks the optimized code still has inside loops, counted in its IR rather than timed, so a change that proves checks away shows here even when timing is too noisy to. A loop that runs a copy without checks when one test before it allows still counts its checked copy.
cargo test --manifest-path bench/Cargo.toml checks the statistics, table, and .args code, and that every benchmark still passes stone’s checker, without running any workloads.
Reading the table¶
stone / C -O2is how many times slower compiled stone is than optimized C. Lower is better for stone.stone / Rustandstone / best Rustare how many times slower compiled stone is than idiomatic Rust and than the faster of idiomatic and tuned Rust. Lower is better for stone.Python / stoneis how many times faster compiled stone is than CPython. Higher is better for stone.C -O0is a useful anchor. gcc at-O0does no optimization at all and keeps every variable in memory, where stone allocates registers, inlines small functions, folds constants, moves invariant arithmetic out of loops, drops work nothing reads, and removes checks it can prove repeated, though it does far less than gcc at-O2.The geometric mean row summarizes the ratios, treating 2x faster and 2x slower symmetrically.
Times include process startup, which is about 1 ms for the native programs and tens of milliseconds for Python.
Benchmarks¶
name |
what it stresses |
workload |
|---|---|---|
|
calls and returns |
recursive |
|
integer arithmetic, division, and |
20 million steps of a linear congruential generator |
|
reading globals from a function |
|
|
branches and division |
the longest Collatz chain for starts below 300,000 |
|
list append, indexed load and store |
counting primes up to 5,000,000 |
|
nested list indexing |
multiplying two 300x300 |
|
float arithmetic and nested list indexing |
multiplying two 300x300 |
|
arithmetic on lists, element by element |
|
|
|
joining 200,000 numbers into a sentence and splitting it back into words, 10 times |
|
a |
counting 2 million words drawn from 50,000 |
|
a |
a million steps of the Sun and the four gas giants orbiting |
|
an enum that holds itself, |
building and walking binary trees of depth 4 to 15, as in the Benchmarks Game |
|
making and matching |
10 million steps that may find nothing, keeping every 16th option |
|
|
sorting 2 million 62-bit ints, then 2 million floats |
|
|
3 million steps of a point around a wobbling circle |
|
the |
writing 20,000 records as JSON text, reading them back, and summing what was read, 10 times |
Each workload size is in the benchmark’s .args file, so every version reads it at runtime and gcc cannot specialize on a constant size. Sizes are chosen so gcc -O2 takes at least about 40 ms and the whole suite finishes in a few minutes. The runner warns when any time is under 10 ms, which usually means gcc optimized the work away.
The versions are meant to be written naturally in each language, not to mimic each other instruction by instruction, so the C versions format numbers into a buffer instead of allocating a string for each, use a hand-written hash table in wordcount, and free trees with an explicit walk in binary_trees. Every program keeps integers nonnegative and within 64 bits, so stone’s / and %, which truncate like C’s, give the same output as Python’s // and %, which round down. Programs with floats print them scaled to an int, so C need not print floats the way stone does, and add them in the same order in every version, so they round identically. -ffp-contract=off keeps gcc from fusing a multiply and an add, which stone never does, on targets with FMA.
To add a benchmark, write name.stn, name.c, name.rs, and name.py (and name.fast.rs if a tuned Rust version would differ), put the hot loop inside a function, read the workload size from the arguments, and save the arguments as name.args and the shared output as name.out.
Baseline¶
Median of 9 runs on 2026-11-15, on an Intel Core i7-10750H under WSL2, pinned to one core with taskset -c 2, with gcc 13.3, rustc 1.99, and Python 3.12 from the distribution, and numpy 2.5.5 in a virtualenv for vector. Absolute times on a machine like this can differ by 2x between sessions, and by a fifth between runs of the suite, as CPU boost and background load change, so compare the ratio columns, which come from the same session, rather than times from different sessions. For a single change, time the old and new stone binaries against each other with --stone, as the History notes do.
program |
C -O2 (ms) |
C -O0 (ms) |
stone (ms) |
Python (ms) |
numpy (ms) |
Rust (ms) |
Rust fast (ms) |
stone / C -O2 |
stone / Rust |
stone / best Rust |
Python / stone |
numpy / stone |
checks in loops |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
binary_trees |
100.4 |
147.5 |
118.4 |
913.7 |
132.4 |
31.2 |
1.2x |
0.9x |
3.8x |
7.7x |
3 |
||
collatz |
55.3 |
104.6 |
66.4 |
2178.0 |
29.9 |
1.2x |
2.2x |
2.2x |
32.8x |
1 |
|||
fib |
17.8 |
57.9 |
42.1 |
1120.2 |
21.4 |
2.4x |
2.0x |
2.0x |
26.6x |
0 |
|||
globals |
211.4 |
223.5 |
48.5 |
4290.0 |
12.5 |
0.2x |
3.9x |
3.9x |
88.4x |
0 |
|||
json_records |
649.8 |
686.7 |
645.5 |
1234.9 |
426.4 |
1.0x |
1.5x |
1.5x |
1.9x |
130 |
|||
lcg |
44.6 |
58.2 |
53.5 |
4420.5 |
12.3 |
1.2x |
4.4x |
4.4x |
82.7x |
0 |
|||
matmul |
15.7 |
63.3 |
33.8 |
1939.2 |
33.5 |
19.1 |
2.2x |
1.0x |
1.8x |
57.3x |
9 |
||
matmul_float |
24.7 |
67.8 |
37.1 |
1266.6 |
32.2 |
24.9 |
1.5x |
1.2x |
1.5x |
34.1x |
9 |
||
nbody |
58.6 |
265.6 |
81.6 |
4436.1 |
37.2 |
1.4x |
2.2x |
2.2x |
54.4x |
4 |
|||
options |
44.2 |
88.1 |
65.1 |
1794.6 |
50.0 |
1.5x |
1.3x |
1.3x |
27.6x |
3 |
|||
sieve |
92.3 |
115.3 |
97.6 |
905.6 |
91.5 |
1.1x |
1.1x |
1.1x |
9.3x |
2 |
|||
sort |
473.9 |
726.5 |
112.5 |
3920.0 |
121.0 |
0.2x |
0.9x |
0.9x |
34.9x |
4 |
|||
strings |
90.7 |
112.3 |
167.4 |
1192.4 |
143.3 |
51.4 |
1.8x |
1.2x |
3.3x |
7.1x |
6 |
||
trig |
148.0 |
206.7 |
129.4 |
891.5 |
160.9 |
0.9x |
0.8x |
0.8x |
6.9x |
54 |
|||
vector |
56.8 |
112.1 |
53.3 |
2584.1 |
212.9 |
50.7 |
0.9x |
1.1x |
1.1x |
48.5x |
4.0x |
0 |
|
wordcount |
185.9 |
192.2 |
150.2 |
848.4 |
153.9 |
144.6 |
0.8x |
1.0x |
1.0x |
5.6x |
3 |
||
geometric mean |
1.0x |
1.4x |
1.8x |
20.8x |
4.0x |
Compiled stone is about as fast as gcc -O2, 1.4x slower than idiomatic Rust and 1.8x slower than the faster of the two Rust versions, and about 21x faster than CPython, as geometric means. It is faster than gcc -O2 on globals, sort, trig, wordcount, and vector, as fast on json_records, faster than gcc -O0 on every benchmark but strings, and about 4x faster than numpy on vector. The Python ratio is low on json_records, wordcount, trig, and strings because they spend their time in work CPython does in C: parsing and writing text, hashing in dict, the math module’s functions, and splitting and joining strings.
The tuned Rust versions show what is left: binary_trees keeps its nodes in one arena of indexes that every tree reuses, where stone allocates and frees each node, strings formats numbers into one buffer and takes words as slices of it instead of allocating a string for each, and matmul reads one flat array per matrix. Idiomatic Rust is slower than stone on binary_trees, sort, and trig, and about as fast on matmul and wordcount, but several times faster on globals, lcg, and collatz, which no change here has looked into yet.
sortbeats C because C’s version calls a comparison function throughqsortfor every comparison, while stone sorts ints and floats by radix, a pass per byte in which they differ.globalsbeats C because its constants are globals that the program assigns one literal each, once, so stone reads them as literals and% modulusshifts like% 2147483648. C’s globals could change, so gcc loads them and usesidiv. Assigning such a global anywhere else makes stone load it too.trigbeats C because stone expands each trigonometric function inline from the same code the interpreter runs, and computes a sine and a cosine of the same value with one argument reduction, where C calls glibc’ssinandcosseparately.wordcountbeats C because a key is hashed eight bytes at a time and the index keeps part of each key’s hash, so a lookup compares almost only the key it is after, andcounts.get(word).unwrap_or(0)makes no option, while C’s hand-written table hashes a byte at a time.sieveis close to C because allocating and touching 40 MB of memory dominates every native version.binary_treesis close to C because both spend their time allocating and freeing nodes, and stone’s allocator, which keeps a free list per block size, keeps up with glibc’smallocthere. EveryTree.Leafis one shared static value, so stone allocates exactly the nodes C does.
History¶
change |
collatz |
fib |
globals |
lcg |
matmul |
sieve |
geometric mean |
|---|---|---|---|---|---|---|---|
first baseline |
12.2x |
4.9x |
2.3x |
9.8x |
15.8x |
1.4x |
5.6x |
constant divisors skip checks, powers of two shift |
4.4x |
5.7x |
1.5x |
3.2x |
15.1x |
1.7x |
3.8x |
list index checks inline instead of calling |
4.1x |
5.8x |
1.6x |
3.2x |
13.3x |
2.0x |
3.8x |
IR, linear-scan register allocation, and compare-and-branch conditions |
1.8x |
3.7x |
1.1x |
1.4x |
4.4x |
1.0x |
1.9x |
programs use |
1.7x |
4.0x |
1.0x |
1.6x |
4.8x |
1.0x |
1.9x |
constant globals, multiply-high division, check hoisting and reuse, inlining |
1.6x |
4.0x |
0.2x |
1.2x |
4.7x |
1.2x |
1.5x |
floats in vector registers, invariant reads and element addresses before loops, loops tested at the bottom |
1.6x |
3.9x |
0.2x |
1.0x |
4.2x |
1.1x |
1.4x |
early returns before the frame, unchecked |
1.0x |
2.7x |
0.2x |
0.9x |
5.5x |
1.2x |
1.2x |
checks removed by inferred facts, one guard before a loop nest over rows (0.4.7; the mean now covers 16 programs) |
1.2x |
2.4x |
0.2x |
1.2x |
2.2x |
1.1x |
1.0x |
Each cell is stone’s time over gcc -O2’s from the same session, and the geometric mean covers the six columns shown. Benchmarks a change does not touch still move by a few tenths between rows, which shows the noise. The row in which programs read their sizes from .args changed the C versions too, since gcc can no longer specialize on a constant size. In the last row, matmul’s ratio rose because gcc -O2 ran faster in that session, while stone’s time stayed about the same. For a single change, timing the old and new stone binaries against each other in one session is more reliable: inlining index checks made matmul 1.25x faster that way, and left sieve unchanged, since memory traffic dominates it. Register allocation, timed the same way with --stone, made collatz 2.6x faster, matmul and matmul_float 2.5x, lcg 2.3x, fib 1.6x, globals 1.35x, and sieve 1.17x.
Rewriting x - (x / m) * m as x % m first made lcg and collatz about 3.5x slower, since % by a constant power of two still used idiv where / already shifted. Computing the remainder from the shifted quotient made lcg 3.7x faster than that and collatz 3.2x, timed the same way. Checking an index into a list of structs or tuples without dividing the list’s slot count by the element’s width made nbody 3.9x faster.
On 2026-10-23, timed against the previous binary in the same session: reading globals that hold one literal as literals made globals 4.7x faster (0.24 s to 0.05 s). Inlining chain into longest made collatz 1.3x faster. Checking each element of a loop once, moving the check of bodies[i] and the uniqueness check of bodies out of the inner loop, inlining drift, and reading fields at fixed offsets together made nbody 1.25x faster (360 ms to 296 ms, and 0.36 s to 0.28 s timed by hand). fib, matmul, sieve, binary_trees, lcg, strings, and wordcount did not change. The same change made popping a map’s entries constant time on average: popping half of 400,000 keys took 0.06 s, where the previous binary needed 10 s for half of 40,000.
Later the same day, timed against that binary: keeping floats in vector registers made nbody 2.2x faster (0.33 s to 0.15 s), and loading the bodies’ element address once per loop rather than for every field took it to 0.13 s. Reading a[i] once before matmul’s inner loop made it about 1.1x faster, and matmul_float gained about 1.15x from both. Testing loops at the bottom changed no benchmark measurably, since the jump back to the test was always predicted, but it leaves one branch per turn.
On 2026-10-27, for 0.3.6, timed against 0.3.5: fib took 60 ms rather than 100 ms, wordcount 183 ms rather than 330 ms, collatz 85 ms rather than 103 ms, strings about 13% less, and nbody, vector, and matmul 5 to 10% less. The changes: a function whose first step tests its parameters and returns one of them, as fib’s base case does, returns before setting up its frame; a for counter over range is known never to be negative, so indexing with it checks only the length; append stores without a call when the list has room; strings compare eight bytes at a time and ints print without dividing; and map keys hash eight bytes at a time, with part of each hash kept in the index, while m.get(k).unwrap_or(d) makes no option.
On 2026-11-07, timed against the same compiler with every enum kept as an object, pinned to one core: keeping enums that do not hold themselves as their tag and values side by side made the new options benchmark 2.1x faster (0.24 s to 0.11 s), since some(x) no longer allocates and step returns its option in two registers. binary_trees, whose tree holds itself, compiles to the same code, and wordcount, nbody, strings, and sort differ at most in how main reads its arguments.
Where the time goes¶
The generated assembly for each benchmark is left at bench/target/programs/<name>-stone.s, and STONE_DEBUG=1 with a debug build prints each function’s IR and register assignment. Values live in registers, and loop tests compare and jump directly, so the remaining gap to C comes mostly from:
Only five registers survive calls on x86-64. A value that lives across a call needs one of the callee-saved registers
rbxandr12tor15(ten,x19tox28, on arm64), and the rest spill to the stack, the least-used first.matmul’smultiplykeeps the matrices and inner counters in registers, but its outer loop counters live in memory, sinceappendis inside that loop, and it counts as a call even though it stores without one when the list has room, and readinga[i]before the inner loop costsbits register. Each vreg also gets one location for its whole life, with no splitting around calls.Floats only stay in vector registers between calls. A float lives in one of
xmm3toxmm15(d3tod7andd16tod31on arm64) when it crosses no call, and does its arithmetic there, as gcc does. Every vector register is one calls may clobber, so a float that lives across a call, such asmatmul_float’s running sum in a loop that also callsappend, falls back to a general register and moves intoxmm0for each operation.Calls do extra bookkeeping. Every call compares
stone.call_depthwith 1,000 first, so recursion stays within the language’s depth limit, and the function it calls increments the count and decrements it on return. A function whose first step tests its parameters and returns one of them, asfib’s base case does, returns before counting the call or setting up its frame, which leaves only the comparison for those calls; the rest of the bookkeeping is most of what separatesfibfrom C. Small functions that call no others are inlined, which also keeps only the comparison:collatz’schainandnbody’sdriftare no longer calls.Division by a variable uses
idiv. A literal divisor that is a power of two compiles to shifts, and any other literal to a multiplication by its magic number, for/and%alike, with no zero or overflow check. So does a global that holds one literal for the whole program. A divisor held in any other variable still gets both checks and anidiv, while C has no checks at all, since dividing by zero is undefined behavior there.List access chases two pointers and checks bounds. An index loads the list’s header, adjusts a negative index, compares it with the length, then loads the data pointer and the element. The counter of a
forloop overrangeis never negative, so indexing with it only compares. Forlist[list[int]]that happens twice pera[i][k], where C’s flat array needs one load. Checking the same element again is skipped when nothing in between could change the list’s length, a check or a read every turn of a loop repeats moves before the loop, and a loop that calls nothing loads each list’s data pointer once, butmatmul’sb[k][j]still checks both indexes every turn.Globals are checked on every read in a function. A function checks a global’s
g.<name>.setflag before loading it, since globals always live in memory, unless the global holds one literal for the whole program, which reads as that literal.
Structs, strings, maps, and enums¶
nbody is now within about 1.4x of C. A list[Body] stores each body’s seven floats side by side, as C’s array of structs does. Within an iteration of the pair loop, bodies[j] is checked once and bodies[i] not at all, drift is inlined, the bodies’ element address is loaded once per loop, and every float stays in a vector register. What is left beside gcc’s loop is checking bodies[j], the division by zero check before dt / (d2 * mag), the check that math.sqrt’s argument is not negative, and a load and a store for every field change, where gcc keeps bodies[i]’s fields in registers across the inner loop.
strings allocates a counted string for every str(), a list of them, the joined sentence, and then a string for every word split finds, where C formats each number straight into one buffer and walks it in place. Each of those strings is reference counted and freed when the round’s list goes away.
wordcount allocates a string for each str() key and then finds it twice, once for get and once to set the new count, as Python’s counts.get(word, 0) + 1 does, where C formats each key into a stack buffer and finds its slot once. Each lookup is cheap enough that stone still comes out ahead, as the baseline notes say.
Arithmetic on lists¶
vector measures arithmetic on whole lists, which numpy is built for. An expression such as 0.5 * x + y compiles to one loop with no list for any intermediate result, where numpy makes a temporary array for each operator. The loop runs over pairs of floats with SSE2 (NEON on arm64), then finishes any odd element one at a time, and writes over y in place, since y is assigned and nothing else holds its list. The baseline above shows the result. Built with --target x86_64v3, the loop runs over four floats at a time with AVX2: on 2026-11-01, timed against the baseline build of the same compiler (best of 7, pinned to one core), that took the default million elements from 191 ms to 178 ms, since lists that size wait on memory, and 10,000 elements 5,000 times from 24 ms to 16 ms. --target x86_64v4 runs eight at a time with AVX-512 and the last few under a mask, and --target aarch64v9-linux as many as the processor’s SVE vectors hold; neither has been timed yet, since the machine these numbers come from has neither (they were checked for correct output under Intel SDE and qemu).
The times include building the lists and summing them with ordinary loops, plus Python’s startup and importing numpy, which is about 100 ms of numpy’s time. Timing only the passes over the lists, stone took about 1.25x as long as gcc -O2, which also uses SSE2 here, and numpy about 2.5x as long as stone, or 1.9x with out= arguments that avoid its temporaries.