Testing¶
cargo test # everything in the stone and stone-lsp crates
cargo test simple_functions # tests whose name contains the filter
Golden programs¶
tests/programs.rs runs every .stn file in examples/, tests/programs/, and
docs/examples/ through both stone run and stone build, and compares stdout with the
sibling .out file. This is the main guarantee that the interpreter and the compiler agree.
stone build compiles for the processor the tests run on. The compiler for the other one is
tested too, by building each program with --target and running it under qemu-user, when the
emulator is on PATH: qemu-aarch64 on an x86-64 machine, or qemu-x86_64 on an arm64 one.
Otherwise that pass is skipped with a note. Compiled programs are static and need no libc, so
qemu needs nothing else. CI runs both backends: each Linux job installs qemu-user for the other
processor, and an arm64 job runs the suite natively.
Off macOS, every program is also built for aarch64-macos without running it, and off Windows,
a build-x86_64-windows pass builds every program as an .exe and runs it through WSL interop
or wine when either is there, and otherwise only links it. A program that uses net or
os.run is skipped for Windows. The macOS and Windows CI jobs run every program natively.
A build-verify pass builds every program with STONE_VERIFY_FACTS=1, which keeps each check
an optimization proved never fails and removed, as a check that stops the program with
compiler bug: <pass> removed a check that fails if the proof was wrong. Without it, a wrong
proof would read or write outside a list and could still print the right output, so this pass
is what tests each proof against the interpreter. The differential fuzzer builds the same way.
A pass that removes a check on a proof must leave an Inst::Never for it when
Options::verify asks (src/codegen/ir/opt.rs), and STONE_STATS=1 stone build counts the
checks that are left.
tests/assembler.rs checks the built-in assembler and linker against GNU’s. For every golden
program and both processors, it assembles the backend’s output with GNU as, links it with GNU
ld at the addresses the built-in linker chose, and requires every section to hold the same
bytes. It needs as and ld for the host plus aarch64-linux-gnu-as and -ld (or the
x86_64-linux-gnu- pair) for the other processor, and skips a processor whose tools are
missing.
To cover new behavior, add name.stn and the exact output it should print as name.out:
tests/programs/
strings.stn
strings.out
A program that should fail at runtime also gets a name.err file holding text its stderr must
contain, read with the program’s directory left out of every path: the position line without
the path, such as :3:7: error: division by zero, or the whole error with the calls that led to
it, as tests/programs/call_chain_recursion.err holds. It must
then exit with an error under both backends, after printing its .out. A few more sibling files
set a program up:
name.stderr: the exact stderr every backend must write, for programs that useeprint(without one, a successful run’s stderr is not checked)name.in: the program’s standard input, which is otherwise emptyname.args: its command-line arguments, one per linename.env:NAME=valuelines added to its environment, foros.envname.files/: a directory copied into the program’s working directory before it runs. Each backend runs a program in a new empty directory of its own, so programs that write files never meet
A program of several files is a directory instead, run from its main.stn, with the .out (and
any .err, .in, .args, .env, or main.files/) next to main.stn:
tests/programs/
module_cycle/
main.stn
main.out
parity/
even.stn
odd.stn
Both backends must agree on everything the checker accepts, so SKIPS in tests/programs.rs
should stay empty. If they disagree, either fix the backend or make the checker reject the
program.
Unit tests¶
Unit tests live in #[cfg(test)] modules next to the code:
file |
covers |
|---|---|
|
tokens, indentation, literals |
|
syntax trees and syntax errors |
|
inferred types and type errors |
|
IR produced from the AST |
|
live intervals |
|
linear scan and parallel moves |
|
target names and the leak check of a linked program |
|
the assembler’s parsing, directives, and jump relaxation |
|
instruction encodings, against bytes from GNU as |
|
executable layout, fixups, and the symbol table |
|
generated x86-64 assembly |
|
generated arm64 assembly |
|
the run, build, and check pipelines |
|
self-update (run with |
Parser tests build expected trees with .into() and compare them through without_spans.
Other test suites¶
tests/cli.rsruns thestonebinary to check error output and exit codes.tests/docs.rschecks that this documentation covers every builtin and keyword.cargo test -p stone-lspruns the language server’s tests. See Language server.cargo test --manifest-path fuzz/Cargo.tomlchecks the fuzzer’s program generator. See Fuzzing.cargo test --manifest-path bench/Cargo.tomlchecks the benchmark runner. See stone benchmarks.
Errors, never panics¶
Every stage must return an error rather than panic on bad input: LexError for bad literals, a
Diagnostic from the parser, parser::MAX_DEPTH for deep nesting, Limits in the interpreter,
and Err(String) from code generation. The fuzzers enforce this.