(std bench store)

Overview

Benchmark runs as data: written to a file and read back, and compared, with a saved run or in A/B rounds.

(import (std bench store))

(match (read-saved-run "base.bjodat")
  ((Ok saved) (compare-runs saved this-run 0.05 0.02))
  ((Err why) (println why)))

What --save FILE, --compare FILE and bjo bench --against are made of. (std bench) measures and prints; this module knows runs only as data, so that the two do not import each other.

The file is one bjodat form, a case to a line, so that a diff of two saved runs says which cases changed:

(bench-run
  (version 1)
  (saved "2026-10-16T12:00:00Z")
  (machine "95c59c6948245e2e")
  (header (build "Release") (dotnet ".NET 10.0.4") (cpu "...") ...)
  (case (name "sort") (labels ("n" "1000")) (k 512) (once #f) (bytes 0.0) (samples [41213.0 ...])))

Every sample is kept, in nanoseconds per call and in the order taken, rather than a summary: a comparison needs the distribution.

A comparison reports, per case, the change in the median, its 95% interval, a p-value corrected for the number of comparisons by Holm’s method, and a verdict. The verdict is Faster or Slower only when the corrected p is below the level and the change is larger than the threshold; a real change smaller than the threshold is SmallChange.

A comparison with a saved run is unpaired, and treats the samples of each run as independent. Two processes of one program can differ by more than a threshold of 2%, so such a comparison can call a difference between processes a change; compare-paired with several process pairs measures that difference instead.

Reference

Types

CompareResult
Comparison
PairedCase
SavedCase
SavedRun
Verdict

Functions

case-key
compare-paired
compare-runs
parse-saved-run
read-saved-run
saved-run->string
write-saved-run!

Types

`SavedCase`

record ↑ Types

One benchmark at one combination of its parameters, as a saved run keeps it.

name: The benchmark's name.
labels: Its parameters, name and value, as the report shows them.
k: The calls in each sample.
once: #t for a benchmark of single cold runs.
bytes: Bytes allocated per call.
samples: Nanoseconds per call, in the order the samples were taken.

`SavedRun`

record ↑ Types

A benchmark run: when and where it was taken, and its cases.

saved: When it was saved: UTC, ISO 8601, to the second.
machine: The machine key: a hash of the CPU, the core count, the memory and the operating system.
header: The facts of the machine and the build, by name: build, dotnet, arch, cores, gc-server, cpu, os and governor.
cases: What was measured.

`Verdict`

union ↑ Types

What a comparison concludes about one case.

Faster: Significant after the correction, and faster by more than the threshold.
Slower: Significant after the correction, and slower by more than the threshold.
SmallChange: Significant, and smaller than the threshold: there, and decided not to matter.
NoChange: Not significant.

`Comparison`

record ↑ Types

One case compared: the medians, the change and its interval, the p-values and the verdict.

name: The benchmark's name.
labels: Its parameters.
was: The median per call of the earlier side, A or the saved run, in nanoseconds.
now: The median per call of the later side, B or this run.
change: The ratio of now to was, less one: 0.05 is 5% slower.
lo: The low end of the change's 95% interval.
hi: The high end.
p-raw: The p-value as the test gave it.
p: The p-value corrected by Holm's method for the comparisons of the run.
verdict: What it comes to, with the run's level and threshold.
spread: With several process pairs, the standard deviation of their log ratios; -1 when it was not measured.

`CompareResult`

record ↑ Types

Two runs compared.

comparisons: The cases both have, compared, in the later run's order.
only-now: The keys of the cases only the later side has.
only-saved: The keys of the cases only the earlier side has.
warnings: The facts of the two runs that differ and move times: the build, the .NET version, server GC, the governor.
other-machine: #t when the machine keys differ: the numbers mean little and no verdict is to be believed.

`PairedCase`

record ↑ Types

One case of an A/B run: each process pair’s samples of A and of B, a sample of each a round.

name: The benchmark's name.
labels: Its parameters.
a: For each process pair, A's nanoseconds per call in the order of the rounds.
b: The same for B, round for round.

Functions

`case-key`

function ↑ Functions

(: case-key (-> SavedCase string))
(case-key c)

What a case is known by across runs: its name and parameters, as the report shows them.

c: The case.
returns: "sort n=1000 order=random", or the name alone without parameters.

`saved-run->string`

function ↑ Functions

(: saved-run->string (-> SavedRun string))
(saved-run->string r)

The text of a saved run: one (bench-run …) form, a case to a line.

r: The run.
returns: The text, ending in a newline.

`parse-saved-run`

function ↑ Functions

(: parse-saved-run (-> string (Result string SavedRun)))
(parse-saved-run text)

Reads a saved run from its text.

text: The text.
returns: The run, or what is wrong with the text: not bjodat, not a (bench-run ...), a newer version of the format, or a case that lacks something.

`write-saved-run!`

function ↑ Functions

(: write-saved-run! (-> string SavedRun void))
(write-saved-run! path r)

Writes a run to a file, replacing what was there.

path: The file.
r: The run.

`read-saved-run`

function ↑ Functions

(: read-saved-run (-> string (Result string SavedRun)))
(read-saved-run path)

Reads the run saved in a file.

path: The file.
returns: The run, or a sentence saying what is wrong: no such file, unreadable, or not a saved run.

`compare-runs`

function ↑ Functions

(: compare-runs (-> SavedRun SavedRun double double CompareResult))
(compare-runs saved now alpha threshold)

A run compared with a saved one, case by case: unpaired, by the Mann-Whitney U test and a bootstrap of the ratio of the medians.

saved: The earlier run.
now: The later run.
alpha: The significance level of a verdict: 0.05.
threshold: The smallest change a verdict counts, as a fraction: 0.02.
returns: The comparison.

`compare-paired`

function ↑ Functions

(: compare-paired (-> (List PairedCase) (List string) (List string) double double CompareResult))
(compare-paired cases only-a only-b alpha threshold)

An A/B run compared, A as was and B as now, paired by round or by process pair.

cases: The cases both sides have.
only-a: The keys of the cases only A has.
only-b: The keys of the cases only B has.
alpha: The significance level of a verdict.
threshold: The smallest change a verdict counts, as a fraction.
returns: The comparison.

With one process pair, the rounds are the pairs: the Wilcoxon signed-rank test on each round’s log ratio, and the paired bootstrap. A drift that moved both of a round’s samples moves neither. What it cannot see is a difference between the two processes as a whole.

With several, each pair gives one ratio of medians, and the test is a t-test on their logs, with a t-interval: the difference between processes is then in the test, and its size is spread.