Testing

cargo test                  # everything in the stone and stone-lsp crates
cargo test simple_functions # tests whose name contains the filter

Golden programs

tests/programs.rs runs every .stn file in examples/, tests/programs/, and docs/examples/ through both stone run and stone build, and compares stdout with the sibling .out file. This is the main guarantee that the interpreter and the compiler agree.

stone build compiles for the processor the tests run on. The compiler for the other one is tested too, by building each program with --target and running it under qemu-user, when the emulator is on PATH: qemu-aarch64 on an x86-64 machine, or qemu-x86_64 on an arm64 one. Otherwise that pass is skipped with a note. Compiled programs are static and need no libc, so qemu needs nothing else. CI runs both backends: each Linux job installs qemu-user for the other processor, and an arm64 job runs the suite natively.

Off macOS, every program is also built for aarch64-macos without running it, and off Windows, a build-x86_64-windows pass builds every program as an .exe and runs it through WSL interop or wine when either is there, and otherwise only links it. A program that uses net or os.run is skipped for Windows. The macOS and Windows CI jobs run every program natively.

A build-verify pass builds every program with STONE_VERIFY_FACTS=1, which keeps each check an optimization proved never fails and removed, as a check that stops the program with compiler bug: <pass> removed a check that fails if the proof was wrong. Without it, a wrong proof would read or write outside a list and could still print the right output, so this pass is what tests each proof against the interpreter. The differential fuzzer builds the same way. A pass that removes a check on a proof must leave an Inst::Never for it when Options::verify asks (src/codegen/ir/opt.rs), and STONE_STATS=1 stone build counts the checks that are left.

tests/assembler.rs checks the built-in assembler and linker against GNU’s. For every golden program and both processors, it assembles the backend’s output with GNU as, links it with GNU ld at the addresses the built-in linker chose, and requires every section to hold the same bytes. It needs as and ld for the host plus aarch64-linux-gnu-as and -ld (or the x86_64-linux-gnu- pair) for the other processor, and skips a processor whose tools are missing.

To cover new behavior, add name.stn and the exact output it should print as name.out:

tests/programs/
  strings.stn
  strings.out

A program that should fail at runtime also gets a name.err file holding text its stderr must contain, read with the program’s directory left out of every path: the position line without the path, such as :3:7: error: division by zero, or the whole error with the calls that led to it, as tests/programs/call_chain_recursion.err holds. It must then exit with an error under both backends, after printing its .out. A few more sibling files set a program up:

  • name.stderr: the exact stderr every backend must write, for programs that use eprint (without one, a successful run’s stderr is not checked)

  • name.in: the program’s standard input, which is otherwise empty

  • name.args: its command-line arguments, one per line

  • name.env: NAME=value lines added to its environment, for os.env

  • name.files/: a directory copied into the program’s working directory before it runs. Each backend runs a program in a new empty directory of its own, so programs that write files never meet

A program of several files is a directory instead, run from its main.stn, with the .out (and any .err, .in, .args, .env, or main.files/) next to main.stn:

tests/programs/
  module_cycle/
    main.stn
    main.out
    parity/
      even.stn
      odd.stn

Both backends must agree on everything the checker accepts, so SKIPS in tests/programs.rs should stay empty. If they disagree, either fix the backend or make the checker reject the program.

Unit tests

Unit tests live in #[cfg(test)] modules next to the code:

file

covers

src/lexer.rs

tokens, indentation, literals

src/parser/tests.rs

syntax trees and syntax errors

src/checker/tests.rs

inferred types and type errors

src/codegen/ir/lower/tests.rs

IR produced from the AST

src/codegen/ir/liveness.rs

live intervals

src/codegen/regalloc.rs

linear scan and parallel moves

src/codegen.rs

target names and the leak check of a linked program

src/codegen/asm.rs

the assembler’s parsing, directives, and jump relaxation

src/codegen/asm/x64.rs, src/codegen/asm/arm64.rs

instruction encodings, against bytes from GNU as

src/codegen/elf.rs

executable layout, fixups, and the symbol table

src/codegen/x64.rs

generated x86-64 assembly

src/codegen/arm64.rs

generated arm64 assembly

src/driver.rs

the run, build, and check pipelines

src/update.rs

self-update (run with --features self-update)

Parser tests build expected trees with .into() and compare them through without_spans.

Other test suites

  • tests/cli.rs runs the stone binary to check error output and exit codes.

  • tests/docs.rs checks that this documentation covers every builtin and keyword.

  • cargo test -p stone-lsp runs the language server’s tests. See Language server.

  • cargo test --manifest-path fuzz/Cargo.toml checks the fuzzer’s program generator. See Fuzzing.

  • cargo test --manifest-path bench/Cargo.toml checks the benchmark runner. See stone benchmarks.

Errors, never panics

Every stage must return an error rather than panic on bad input: LexError for bad literals, a Diagnostic from the parser, parser::MAX_DEPTH for deep nesting, Limits in the interpreter, and Err(String) from code generation. The fuzzers enforce this.