(std bench)

Submodules:

Reference

Overview

Benchmarks: def/bench, run-benches as the file’s main, and time-it for one line at the REPL.

(import (std prelude))
(import (std bench))

(def/bench "list-sort"
  #:params ((n [1000 100000]))
  #:setup ((xs {listing (% (* i 7919) n) (:for i (range 0 n))}))
  (list-sort xs))

(def/bench "vecing" (vec-length {vecing i (:for i (range 0 1000))}))

(run-benches)

def/bench registers a benchmark where it is written, and (run-benches) is the file’s main, as deftest and (run-tests) are. bjo bench FILE builds the file in Release and runs it.

A run measures each benchmark in turn:

The table has a row per benchmark and combination of parameters, and each row is printed as soon as it is measured:

bench: 3 benchmarks, Release build, .NET 10.0.4, X64, 24 cores, gcServer=false
bench: AMD Ryzen 9 5900X 12-Core Processor, Fedora Linux 43

name        n          median        ±         min    alloc/op   gc/1k 0/1/2
list-sort   1000      41.2 µs     0.6%     40.8 µs     39.1 KB      3.33/0/0
list-sort   100000    8.31 ms     1.1%     8.12 ms     3.91 MB       333/0/0   noisy
vecing                1.24 µs     0.4%     1.22 µs     7.36 KB      0.88/0/0   2 outliers

The median per call, its median absolute deviation as a percentage (±), the fastest sample, the bytes allocated per call, and the collections per thousand calls of each generation. The marks are noisy (± above 3%), unsteady, outliers (beyond Tukey’s fences and 2% from the median), and for #:setup-every a body below the clock’s resolution.

Choosing. With names, the benchmarks whose names contain one of them run, and a name may be a parameter as the table shows it, n=1000. With none, at a terminal, a menu asks; --all runs everything without asking, and is what a run that is not at a terminal does.

Flags. --quick and --long halve and quadruple the time targets; --min-samples N, --max-time S, --params P=V,... (replacing a parameter’s values), --detail (more statistics, the samples in order and a histogram) and --scaling (how the time grows with a numeric parameter). --save FILE keeps every sample, and --compare FILE compares with what was kept: see (std bench store). --compare-within compares the benchmarks of each #:group with its first, in -N interleaved rounds. parse-bench-args lists them all.

Two versions. bjo bench FILE --against REF runs the file against itself at a git ref, or against another path, in interleaved rounds: the way to ask whether a change made something slower. Two processes of one program can differ by a few percent, more than the 2% a verdict asks for; --processes P measures that difference rather than leaving it out.

Not in the time. The body’s value is kept alive by a sink the JIT cannot remove, so a body that computes a value is not optimized away. A benchmark program runs everything in one process, one benchmark after another, and the collection before each sample is what keeps one benchmark’s garbage out of the next one’s time.

See also: def/bench, run-benches, time-it, parse-bench-args

Reference

Types

Functions

Macros

Types

BenchParam

trait ↑ Types

A type a parameter’s value can be read as from –params: int, long, double, string, Symbol and bool.

Implement it for a type of your own to let --params give values of it.

See also: bench-param-read, def/bench

Bench

record ↑ Types

A registered benchmark.

name
Its name.
module
The key of the module that registered it, from (current-module).
file
The source file.
line
Its line.
group
Its #:group, or empty.
skip
Why it is skipped, or empty.
once
The number of cold runs of a #:once benchmark; 0 for a calibrated one.
param-names
The names of its parameters, in order.
cases
Makes its cases: a function, so that the values are read after --params.

See also: BenchCase, registered-benches

BenchCase

record ↑ Types

A benchmark at one combination of its parameters: a row of the table.

labels
The parameters' names and values, as the table shows them.
prepare
Runs the setup, and answers what to run.

See also: Bench, CaseRun

CaseRun

record ↑ Types

What a prepared case runs.

batch
Runs the body k times, its value kept alive each time.
call
For #:setup-every: one call after its setup, answering the call's nanoseconds and bytes.

See also: measure-case

BenchOptions

record ↑ Types

What the arguments of a benchmark program asked for.

filters
The names: a benchmark runs when its name, or a parameter as n=10, contains one.
all
--all: everything, without a menu.
list
--list: the names, nothing run.
scale
What the time targets are multiplied by: 0.5 for --quick, 4 for --long.
min-samples
--min-samples: at least this many samples. 20.
max-time-ns
--max-time: the most time a benchmark may sample for, or 0 for three times the target.
detail
--detail: the mean, the deviation, the quartiles, the samples in order and a histogram.
params
--params: parameters whose values replace the file's.
save
--save: the file to save the run to.
compare
--compare: the saved run to compare with.
alpha
--alpha: the significance level of a verdict. 0.05.
threshold
--threshold: the smallest change a verdict counts, as a fraction. 0.02.
p-raw
--p-raw: the uncorrected p-values too.
fail-slower
--fail-slower: exit 1 when something is SLOWER by more than this fraction.
serve
--serve: driven over standard input and output.
cases
--cases: exactly these cases, numbered as --serve lists them.
scaling
--scaling: how the time grows with a numeric parameter.
compare-within
--compare-within: each #:group's cases against its first, in rounds.
rounds
-N: the rounds of --compare-within. 20.
seed
--seed: the order of the rounds; from the clock unless given.

See also: parse-bench-args, bench-settings

Settings

record ↑ Types

The targets of a run, in nanoseconds, from its options.

sample-ns
What a sample is calibrated to: 10 ms.
time-ns
How long a benchmark samples for at least: 2 s, scaled.
warmup-ns
The most a warm-up may take: 1 s, scaled. It takes at least half.
max-ns
The most a benchmark may sample for, taking more samples for a precise median.
min-samples
The fewest samples.
max-samples
The most samples.

See also: bench-settings

Measured

record ↑ Types

What measuring one case came to.

samples
Nanoseconds per call, in the order taken; for #:once, one per cold run.
k
The calls in a sample.
calls
Every call the samples timed, which the allocation and the collections are per.
bytes
The bytes the samples allocated.
gc0
Generation 0 collections during the samples.
gc1
Generation 1 collections.
gc2
Generation 2 collections.
steady
Whether the warm-up got steady.
once
#t for single cold runs.
per-call
#t when each call was timed on its own, for #:setup-every.

See also: measure-case

Selection

record ↑ Types

What a run is asked to measure: a benchmark, and the cases of it chosen.

bench
The benchmark.
cases
Its chosen cases.

See also: parse-choice, run-selection

Functions

bench-param-read

function ↑ Functions

(: bench-param-read (-> string (Option %a)))

Reads a parameter’s value from its text on the command line.

s
The text: 10, 2.5, sorted, 'sorted, #t, true.
returns
The value, or None when the text is not one.

registered-benches

function ↑ Functions

(: registered-benches (-> (List Bench)))
(registered-benches)

Every benchmark registered so far, in the order they were written.

returns
The benchmarks.

See also: module-benches

module-benches

function ↑ Functions

(: module-benches (-> string (List Bench)))
(module-benches module)

The benchmarks one module registered.

module
The key of the module, as (current-module) gives it.
returns
Its benchmarks, in the order they were written.

parse-bench-args

function ↑ Functions

(: parse-bench-args (-> (Vec string) (Result string BenchOptions)))
(parse-bench-args args)

Reads the arguments of a benchmark program.

args
The arguments: names, and the flags.
returns
The options, or what is wrong with the arguments.

The flags are --all, --list, --quick, --long, --detail, --scaling, --min-samples N, --max-time S, --params P=V,..., --save FILE, --compare FILE, --alpha P, --threshold PCT, --p-raw, --fail-slower PCT, --compare-within, -N ROUNDS, --seed N, --serve and --cases I,J,.... A percentage is written 2 or 2%. Anything else that starts with -- is an error; the rest are names.

See also: BenchOptions, run-registered-benches

bench-settings

function ↑ Functions

(: bench-settings (-> BenchOptions Settings))
(bench-settings o)

The targets that options ask for.

o
The options.
returns
The settings.

bench-params-set!

function ↑ Functions

(: bench-params-set! (-> (Map string (Vec string)) void))
(bench-params-set! overrides)

Replaces parameters’ values for the benchmarks’ cases made after it: what –params does.

overrides
Each parameter's name, and its values as text.

See also: bench-param-values

measure-case

function ↑ Functions

(: measure-case (-> BenchCase int Settings Measured))
(measure-case c once s)

Measures one case: calibrates, warms up and samples it, or times its cold runs.

c
The case.
once
The number of cold runs of a #:once benchmark, or 0.
s
The targets.
returns
What it measured.

See also: Measured, run-selection

time-text

function ↑ Functions

(: time-text (-> double string))
(time-text ns)

Nanoseconds in the unit that reads best, to three digits.

ns
The nanoseconds.
returns
"41.2 µs", "999 ns", "9.88 s".

bytes-text

function ↑ Functions

(: bytes-text (-> double string))
(bytes-text b)

Bytes in the unit that reads best, to three digits.

b
The bytes.
returns
"512 B", "1.50 KB", "10.0 MB".

header-lines

function ↑ Functions

(: header-lines (-> int (List string)))
(header-lines count)

The lines a run begins with: the build, the runtime and the machine, and warnings about them.

count
How many benchmarks the run has.
returns
The lines.

A Debug build, an attached debugger and a CPU frequency governor that is not performance are warnings.

header-facts

function ↑ Functions

(: header-facts (-> (List (Tuple string string))))
(header-facts)

The facts of this machine and build, by name, as a saved run keeps them.

returns
build, dotnet, arch, cores, gc-server, cpu, os and governor.

this-run

function ↑ Functions

(: this-run (-> (List SavedCase) SavedRun))
(this-run cases)

Measured cases as a saved run of this machine, saved now.

cases
The cases.
returns
The run.

See also: header-facts

sel-bench

function ↑ Functions

(: sel-bench (-> Selection Bench))
(sel-bench p)

A selection’s benchmark.

p
The selection.
returns
The benchmark.

sel-cases

function ↑ Functions

(: sel-cases (-> Selection (List BenchCase)))
(sel-cases p)

A selection’s cases.

p
The selection.
returns
The cases.

case-labels

function ↑ Functions

(: case-labels (-> BenchCase (List (Tuple string string))))
(case-labels c)

A case’s parameters, name and value.

c
The case.
returns
The labels.

run-selection

function ↑ Functions

(: run-selection (-> (List Selection) BenchOptions (List SavedCase)))
(run-selection chosen o)

Measures what is chosen and prints the table, a row as each case is done.

chosen
What to measure.
o
The options.
returns
What was measured, as a saved run's cases; a skipped case is not among them.

See also: run-and-report

run-and-report

function ↑ Functions

(: run-and-report (-> (List Selection) BenchOptions int))
(run-and-report chosen o)

Measures what is chosen, then saves, compares and compares within groups, as the options say.

chosen
What to measure.
o
The options.
returns
The exit code: 1 when --fail-slower fails the run, 2 when the file to compare with cannot be used, else 0.

See also: run-selection, run-with-options

run-with-options

function ↑ Functions

(: run-with-options (-> string BenchOptions int))
(run-with-options module o)

What a benchmark program does with its options: lists, serves, asks, or runs what they choose.

module
The key of the module whose benchmarks to run.
o
The options.
returns
The exit code.

See also: run-registered-benches

run-registered-benches

function ↑ Functions

(: run-registered-benches (-> string (Vec string) int))
(run-registered-benches module args)

The main of a benchmark file: reads the arguments and runs the module’s benchmarks. (run-benches) writes a call to it.

module
The key of the module, from (current-module).
args
The program's arguments.
returns
The exit code; 2 for wrong arguments.

See also: run-benches, parse-bench-args

parse-choice

function ↑ Functions

(: parse-choice (-> string (List Bench) (Result string (List Selection))))
(parse-choice text benches)

Reads a choice made in the menu.

text
What was typed: numbers (0), ranges (2-4), parts of names (sort), * for all, and after a benchmark p=v,w for some of its parameter's values.
benches
The benchmarks, numbered from 0.
returns
What it chooses, or why it chooses nothing.

menu-lines

function ↑ Functions

(: menu-lines (-> string (List Bench) (List string)))
(menu-lines file benches)

The menu: the benchmarks numbered, under their groups, each with its parameters’ values, and how to choose.

file
What the menu is of, for its first line.
benches
The benchmarks.
returns
The lines.

menu-loop

function ↑ Functions

(: menu-loop (-> (List string) (List Bench) (-> (Option string)) (-> (List Selection) void) void))
(menu-loop lines benches read run)

Shows the menu and asks until q, the end of the input, or nothing chosen with nothing to repeat.

lines
What to show first.
benches
The benchmarks a choice numbers.
read
Reads a line of input, None at its end.
run
What to do with a choice.

Enter repeats the last choice. bjo bench uses it over several files, running each file with the cases of a choice that are its own.

See also: bench-menu, parse-choice

bench-menu

function ↑ Functions

(: bench-menu (-> string (List Bench) BenchOptions (-> (Option string)) void))
(bench-menu file benches o read)

The menu of a benchmark program: asks what to run, runs it, and asks again.

file
The file's name, for the menu's first line.
benches
The benchmarks.
o
The options every choice runs with.
read
Reads a line of input, None at its end.

See also: menu-loop

bench-serve

function ↑ Functions

(: bench-serve (-> (List Bench) (-> (Option string)) void))
(bench-serve benches read)

Driven over standard input and output, a command and a reply a line, in bjodat: what –serve does.

benches
The benchmarks.
read
Reads a command line, None at the end of the input.

The cases are numbered from 0 in the order (list) gives them.

(list)                               → (cases (case 0 (name "sort") (labels ("n" "10")) (group "") (once 0) (skip "")) ...)
(prepare 3 :warmup 1000 :sample 10)  → (prepared 3 :k 4096 :steady #t)
(sample 3 :k 4096)                   → (sample 3 :ns 41213 :bytes 0 :gc (0 0 0))
(quit)

:warmup and :sample are milliseconds; a sample’s :ns is the whole sample’s, k calls. prepare runs the case’s setup and warms it up as a run would. Anything wrong is (error "why"), and the next command is read; (quit) or the end of the input ends it. This is what bjo bench --against drives two programs with.

print-comparison-table

function ↑ Functions

(: print-comparison-table (-> CompareResult BenchOptions string string string bool void))
(print-comparison-table result o was-head now-head other-side ratio?)

Prints a comparison: its warnings, the table, and the cases one side has alone.

result
The comparison.
o
The options: the threshold, --p-raw and --fail-slower.
was-head
The heading of the earlier side's medians.
now-head
The heading of the later side's.
other-side
What the earlier side is called, for the cases only it has.
ratio?
#t for a column of the ratio, ×2.10, as --compare-within shows.

See also: comparisons-text, fails-slower?

comparisons-text

function ↑ Functions

(: comparisons-text (-> int string))
(comparisons-text n)

How many comparisons a run made, and that they were corrected.

n
The number.
returns
"3 comparisons, Holm-corrected".

fails-slower?

function ↑ Functions

(: fails-slower? (-> CompareResult BenchOptions bool))
(fails-slower? result o)

Whether –fail-slower fails a run: something SLOWER by more than its percentage, against a run of this machine.

result
The comparison.
o
The options.
returns
#t to exit 1.

register-bench!

function ↑ Functions

(: register-bench! (-> string string string int string string int (List string) (-> (List BenchCase)) int))
(register-bench! name module file line group skip once param-names cases)

Adds a benchmark to the registry. def/bench writes calls to it.

name
The name.
module
The key of the module, from (current-module).
file
The source file.
line
The line.
group
The #:group, or empty.
skip
The #:skip reason, or empty.
once
The #:once count, or 0.
param-names
The parameters' names.
cases
Makes the cases.
returns
0, so that the call is a top-level def.

It is exported because the template of def/bench names it.

bench-keep!

function ↑ Functions

(: bench-keep! (-> %a void))

Keeps a value alive: a write the JIT cannot remove, which allocates nothing.

x
The value.

The runtime’s BjoBench.Consume: a write to a volatile static field of a generic class, one per type, so nothing is boxed. A benchmark’s batch loop writes each call’s value to it.

bench-param-values

function ↑ Functions

(: bench-param-values (-> string (Vec %a) (Vec %a)))
(bench-param-values name values)

A parameter’s values: the file’s, or those –params gave, read as the parameter’s type. def/bench writes calls to it.

name
The parameter's name.
values
The values the file gives.
returns
The values to run with.

bench-now

function ↑ Functions

(: bench-now (-> long))
(bench-now)

The clock, in nanoseconds: current-clock’s monotonic time, which a test can replace.

returns
The nanoseconds.

bench-allocated

function ↑ Functions

(: bench-allocated (-> long))
(bench-allocated)

The bytes this thread has allocated so far.

returns
The bytes.

time-it-run

function ↑ Functions

(: time-it-run (-> string (-> long void) void))
(time-it-run label batch)

What time-it expands to: measures a batch with –quick’s targets and prints its line.

label
The body's source text.
batch
Runs the body k times.

time-it-fixed

function ↑ Functions

(: time-it-fixed (-> string int int (-> long void) void))
(time-it-fixed label times warm batch)

What time-it with #:times or #:warm-up-rounds expands to: exactly that many runs.

label
The body's source text.
times
The timed runs.
warm
The untimed runs first.
batch
Runs the body k times.

time-it-expansion

function ↑ Functions

(: time-it-expansion (-> Syntax Syntax))
(time-it-expansion form)

The expansion of a time-it form, shared by (std bench) and (std stopwatch).

form
The form.
returns
Its expansion.

time-it-keep

function ↑ Functions

(: time-it-keep (-> %a void))
(time-it-keep x)

Keeps a time-it body’s value alive, as bench-keep! does.

x
The value.

Macros

def/bench

macro ↑ Macros

(def/bench name body ...)
(def/bench name #:setup ((setup-name setup-expr) ...) body ...)
(def/bench name #:setup-every ((every-name every-expr) ...) body ...)
(def/bench name #:params ((param values) ...) body ...)
(def/bench name #:once n body ...)
(def/bench name #:group group body ...)
(def/bench name #:skip why body ...)

Registers a benchmark. (run-benches) runs it.

name
A string: the benchmark's name, which the filters of a run match.
setup-name
A name the setup binds, which the body and the bindings after it can use.
setup-expr
Its value: made once, before the benchmark is warmed up, outside the time.
every-name
A name bound before every call.
every-expr
Its value, made again before every call, outside the time and the allocation.
param
The name of a parameter, which the setups and the body can use.
values
A vec literal of the parameter's values, evaluated once when the benchmark is registered.
n
How many cold runs of a #:once benchmark.
group
A string: a heading in the menu, and what --compare-within compares within.
why
A string: why the benchmark is skipped, which the run shows.
body ...
Forms run as one call, the value of the last kept alive. The parameters and the setups' names are in scope.
(def/bench "fib 20" (fib 20))

(def/bench "sum"
  #:setup ((xs {listing i (:for i (range 0 1000))}))
  (fold + 0 xs))

(def/bench "sort"
  #:params ((n [1000 100000])
            (order ['sorted 'random]))
  #:setup ((xs (make-list n order)))
  (list-sort xs))

;; The input is used up: a fresh copy before every call, out of the time.
(def/bench "sort!"
  #:setup ((unsorted (random-array 10000)))
  #:setup-every ((a (array-copy unsorted)))
  (array-sort! a))

(def/bench "sieve" #:once 3 (sieve 2000000000))

Write it at the top level of a module, as deftest. The options come after the name, in any order.

See also: run-benches, time-it

run-benches

macro ↑ Macros

(run-benches)

Ends a benchmark file: becomes its main, which runs the benchmarks the module registered.

The arguments of the program are names and flags, as parse-bench-args reads them. The exit code is 0, 1 when --fail-slower fails the run, and 2 for wrong arguments or a file that --compare cannot read.

See also: def/bench, run-registered-benches, parse-bench-args

time-it

macro ↑ Macros

(time-it body ...)
(time-it #:times n body ...)
(time-it #:warm-up-rounds w body ...)
(time-it #:times n #:warm-up-rounds w body ...)

Measures a body where it is written and prints one line: the median per call, its spread, the minimum and the allocation.

n
With #:times, exactly n timed runs: no calibration.
w
With #:warm-up-rounds, exactly w untimed runs first. 0 unless given.
body ...
The forms, run as one call; the value of the last is kept alive.
(time-it (list-sort big-list))
;; (list-sort big-list): 8.31 ms ± 1.1% (min 8.12 ms), 2.40 MB/call, 120 samples

(time-it (sieve 2000000000))
;; (sieve 2000000000): 10.0 s (one cold run: a body over a second is run once)

(time-it #:times 100 #:warm-up-rounds 3 (list-sort big-list))

An anonymous benchmark, run at once with --quick’s targets: warmed up, calibrated and sampled. The label is the body’s source text, cut short. A body whose first run takes more than a second is not run again: that run is the measurement. #:times and #:warm-up-rounds keep their old meaning, a known number of runs, for a program that depends on the body running that many times.

(std stopwatch) re-exports it, so an (import (std stopwatch)) that uses it goes on working.

See also: def/bench

(std bench stats)2026-10-16

(std bench store)2026-10-16