diff --git a/_02_.Writing-programs.sz b/_02_.Writing-programs.sz index ea56320..e2c298f 100644 --- a/_02_.Writing-programs.sz +++ b/_02_.Writing-programs.sz @@ -1573,6 +1573,454 @@ random-int (std random) Type @code{:help} to view available REPL commands, or press @code{Ctrl-D} (or type @code{:quit}) to exit. +@subsection{Benchmarking} + +Sooner or later a program is fast enough everywhere except one place, and the question becomes how fast that place is, and whether a change made it better. Timing it with a stopwatch gives a number, but not one you can trust: the first calls run code the JIT has not optimized yet, a garbage collection can land in the middle of a measurement, and two runs of the same code on a busy laptop differ by more than most improvements do. + +@code{(std bench)} measures in a way that holds up against all of that, and says how sure it is. There are three ways in: + +@read-list{ + - @code{time-it}, for a quick answer at the REPL: one line, measured properly. + - Benchmark files, with @code{def/bench} and @code{(run-benches)}, for the code you care about: a table, a menu, saved runs to compare with later. + - @code{bjo bench FILE --against REF}, for the question "did my change make this slower": the file run against itself at another commit, the two versions measured in turns. +} + +All three are measured the same way, which the section "How a benchmark is measured" below describes. + +@subsubsection{time-it} + +@code{time-it} measures a body where you write it and prints one line. It is in @code{(std bench)}, and @code{(std stopwatch)} has it too: + +@codeblock[#:lang "text"]|{ +bjo> (import (cardlib deck)) +bjo> (import (std bench)) +bjo> (time-it (shuffle (full-deck))) +(shuffle (full-deck)): 1.06 µs ± 0.7% (min 1.05 µs), 3.64 KB/call, 67 samples +}| + +The line gives the median time per call, its spread (the median absolute deviation, as a percentage of the median), the fastest sample, how much the body allocated per call, and how many samples it took. The body is warmed up, calibrated and sampled just as a benchmark is, with shorter time targets, so it is done in a second or two. + +A body that takes more than a second the first time is not run again. That first run is the measurement, and the line says so. Here the body is a slow recursive Fibonacci: + +@codeblock[#:lang "text"]|{ +(fib 44): 1.46 s (one cold run: a body over a second is run once) +}| + +@code{#:times n} and @code{#:warm-up-rounds w} ask for an exact number of runs instead: @code{w} untimed runs, then @code{n} timed ones, with no calibration. Use them when the body has an effect that has to happen a known number of times: + +@codeblock[#:lang "text"]|{ +bjo> (time-it #:times 1000 (shuffle (full-deck))) +(shuffle (full-deck)): 3.82 µs (min 3.02 µs), 3.64 KB/call, 1000 runs +}| + +The median here is more than three times the first one: without a warm-up, the early runs are the JIT's unoptimized code. That is the reason to let @code{time-it} decide how often to run a body. + +@subsubsection{A benchmark file} + +For code you want to keep measuring, write a benchmark file. Benchmarks belong in @code{benches/} in a project, next to @code{src/} and @code{tests/}. Here is one for the deck from @code{cardlib}, in @code{benches/deck.bjo}: + +@codeblock[#:lang "bjolang"]|{ +(import (cardlib deck)) +(import (std bench)) + +(def/bench "full-deck" (full-deck)) + +(def/bench "shuffle" + #:setup (full-deck) + (fun (deck) (shuffle deck))) + +(def/bench "hand points" + #:params ((size [5 13 52])) + #:setup (fun (size) (vec-slice (shuffle (full-deck)) 0 size)) + (fun (hand) (vec-fold (fun (c total) (+ total (card-points c))) 0 hand))) + +;; Sorting changes the deck it sorts, so every call gets a freshly +;; shuffled one, made outside the time. +(def/bench "sort a shuffled deck" + #:setup-every (vec->list (shuffle (full-deck))) + (fun (cards) (list-sort cards))) + +(def/bench "deal with vec-slice" #:group "dealing" + #:setup (shuffle (full-deck)) + (fun (deck) (vec-slice deck 0 5))) + +(def/bench "deal with a loop" #:group "dealing" + #:setup (shuffle (full-deck)) + (fun (deck) {vecing (vec-ref deck i) (:for i (range 0 5))})) + +(def/bench "a million hands" #:once 3 + (loop (:for i (up-from 0 #:to 1000000)) + (:acc total (summing (card-points (vec-ref (full-deck) (% i 52))))) + => total)) + +(def/bench "deal from a stacked deck" #:skip "the cheat is not written yet" + (full-deck)) + +(run-benches) +}| + +It is laid out just like a test file. @code{def/bench} registers a benchmark where it is written, and @code{(run-benches)} at the end becomes the file's @code{main}. @code{bjo bench} builds every file under @code{benches/} and runs it: + +@codeblock[#:lang "text"]|{ +$ bjo bench --all +bench: 10 benchmarks, Release build, .NET 10.0.4, X64, 24 cores, gcServer=false +bench: AMD Ryzen 9 5900X 12-Core Processor, Fedora Linux 43 + +name size median ± min alloc/op gc/1k 0/1/2 +full-deck 309 ns 0.3% 307 ns 1.70 KB 0.20/0/0 1 outlier +shuffle 558 ns 0.3% 555 ns 1.94 KB 0.23/0/0 4 outliers +hand points 5 4.67 ns 1.9% 4.52 ns 0 B - 9 outliers +hand points 13 11.3 ns 1.4% 10.7 ns 0 B - 7 outliers +hand points 52 46.2 ns 0.5% 45.5 ns 0 B - 1 outlier +sort a shuffled deck 3.63 µs 0.5% 3.60 µs 3.30 KB 1.00/0/0 9 outliers +deal with vec-slice 19.4 ns 0.2% 19.3 ns 152 B 0.02/0/0 3 outliers +deal with a loop 51.7 ns 0.7% 51.2 ns 656 B 0.08/0/0 4 outliers +a million hands 320 ms - 320 ms 1.62 GB 208/0/0 + (324 ms, 320 ms, 320 ms) +deal from a stacked deck skipped: the cheat is not written yet +}| + +The first two lines say what the numbers were measured on, because a time is a fact about one machine. A Debug build, an attached debugger and a CPU frequency governor other than @code{performance} add a warning there. @code{bjo bench} always builds in Release, whatever @code{-d} says, since a Debug build measures code the JIT did not optimize. + +The columns are: + +@read-table{ +| Column | What it says | +|----------------------+--------------------------------------------------------------------------------------------------| +| @code{name}, @code{size} | the benchmark, and a column for each of its parameters | +| @code{median} | the median time per call. Not the mean: one collection can move a mean anywhere | +| @code{±} | the median absolute deviation, as a percentage of the median: how much the samples agree | +| @code{min} | the fastest sample, which is what a quiet machine converges to | +| @code{alloc/op} | the bytes allocated per call | +| @code{gc/1k 0/1/2} | the collections of each generation per thousand calls (per call for a @code{#:once} benchmark) | +} + +After the columns come the marks. @code{noisy} means a spread above 3%, @code{unsteady} means the warm-up ran out of time before the times settled, and outliers are samples far outside the rest. They are counted and shown, never dropped: a pause that happens once in fifty calls is part of what the call costs. + +Each row is printed as soon as its benchmark is measured, so a long run can be read as it goes. At a terminal, a line below the table says where the run is: which benchmark, which phase, the median so far and an estimate of the time left. It is rewritten in place, and is left out when the output goes to a pipe or a file, under @code{CI}, and in an Emacs buffer. + +@subsubsection{How a benchmark is measured} + +You do not have to know this to use @code{(std bench)}, but it explains its numbers. + +@read-list{ + - @strong{The body is kept.} The value of the body's last form is written to a sink the JIT cannot remove. A body that computes a value and throws it away is therefore not optimized away, and the sink allocates nothing. + - @strong{Calibration.} The body is run in batches of @code{k} calls, and @code{k} grows until a batch takes 10 ms. One sample is one batch, so the clock's resolution does not matter even for a body of a few nanoseconds. + - @strong{Warm-up.} Batches are run until the medians of the last two windows of five batches agree within 2%, for at least half a second and at most one. The minimum is there because of the JIT. .NET recompiles a hot method into faster code only after a pause with no new compilation, and a body can look steady at the first, slow tier before the faster code arrives. A warm-up that runs out of time leaves the row marked @code{unsteady}. + - @strong{Samples.} Before each sample a collection runs, so that one benchmark's garbage is not collected in another one's time. At least 20 samples are taken, for at least 2 s. While the median's 95% interval is wider than ±1%, it takes more, up to 100 samples or three times the time. A noisy benchmark costs more time, and gets a more precise median for it. +} + +Every benchmark of a file runs in the file's one process, one after another, so what one benchmark compiled is there for the next. That is usually what you want: it is how the code runs in a program. + +@subsubsection{Setup, parameters, and the rest} + +The body of a benchmark should be the thing you want to measure and nothing else. The options of @code{def/bench} are about keeping everything else out of the time: + +@read-table{ +| Option | What it does | +|------------------------------+--------------------------------------------------------------------------------------------------------| +| @code{#:setup e} | runs @code{e} once, outside the time; the body is then a function of what it made | +| @code{#:setup-every e} | runs @code{e} before every call, outside the time; for a body that uses up its input | +| @code{#:params ((p values) ...)} | a row for each combination of values; the body, or the setup, is a function of the parameters | +| @code{#:once n} | for a body of seconds: @code{n} cold runs, timed one at a time, no warm-up | +| @code{#:group "g"} | a heading in the menu, and what @code{--compare-within} compares within | +| @code{#:skip "why"} | not run; the reason is shown | +} + +@code{#:setup} is the one to reach for first. In the example, @code{shuffle} measures shuffling a deck, not making one: the deck is made once, before the warm-up, and every call shuffles that same deck. + +@code{#:setup-every} is for a body that changes or uses up what it is given, such as a sort that sorts in place, or a body that consumes a list. Each call gets a fresh input, and each call is then timed on its own, with its setup outside the time and outside the allocation. Timing calls one at a time has a cost, because the clock is read twice per call. A body that takes less than a microsecond is marked, since its time is close to the clock's resolution. + +@code{#:params} gives the parameters' names and values, like a @code{let}. The values are a vec of any expressions, evaluated once. There is a row for every combination, in the order written, with the last parameter varying fastest. Without a setup, the body is a function of the parameters. With one, the setup is, and the body gets what the setup made: + +@codeblock[#:lang "bjolang"]|{ +(def/bench "sort" + #:params ((n [1000 100000]) + (order ['sorted 'reversed 'random])) + #:setup (fun (n order) (make-cards n order)) + (fun (cards) (list-sort cards))) +}| + +Here @code{make-cards} stands for a function of your own that makes @code{n} cards in the given order. + +Each combination is a benchmark of its own for everything else: it is warmed up, calibrated, saved and compared on its own. To try a size the file does not list, give it on the command line: @code{--params n=500,5000} replaces the values of @code{n} for that run. A value is read as the parameter's values are typed: an @code{int}, a @code{double}, a string, a symbol or a @code{bool}. Implementing the trait @code{BenchParam} lets other types be read too. + +@code{#:once n} is for bodies too slow to run in batches. There is no calibration and no warm-up, just @code{n} runs timed one at a time, the first of them cold. Each time is shown under the row, as for @code{a million hands} above, and the median is the row's. + +@subsubsection{Choosing what runs} + +Names on the command line choose benchmarks. A benchmark runs when its name contains one of them, and a parameter written the way the table shows it chooses rows: + +@codeblock[#:lang "text"]|{ +bjo bench benches/deck.bjo shuffle # shuffle, and sort a shuffled deck +bjo bench benches/deck.bjo "hand points" size=52 +bjo bench --list # the names, and nothing run +}| + +Note that @code{shuffle} also matches @code{sort a shuffled deck}: a name is a part of a name. + +With no names, at a terminal, a benchmark file asks what to run: + +@codeblock[#:lang "text"]|{ +$ bjo bench +Benchmarks in deck.bjo: + + #0 full-deck + #1 shuffle + #2 hand points size: 5 13 52 + #3 sort a shuffled deck + dealing + #4 deal with vec-slice + #5 deal with a loop + #6 a million hands + #7 deal from a stacked deck + +Choose by number (0 2-4), by name (sort), with parameters (2 n=1000), +or * for all. Nothing quits, or repeats the last choice once there is one. +bench> 2 size=5,52 +}| + +A choice is numbers, ranges of numbers (@code{2-4}), parts of names, or @code{*} for everything, in any mix and separated by spaces. A parameter after a benchmark narrows that benchmark to some of its values: @code{2 size=5,52} runs @code{hand points} at sizes 5 and 52 only. The groups and the parameters' values are shown, so a choice can be made without opening the file. + +After a run the menu asks again. Pressing Enter repeats the last choice, so you can change the code in another terminal and run the same benchmarks again with one key. @code{q} or the end of the input leaves. + +With several files and no names, @code{bjo bench} makes one menu of all of them, numbered across the files under a heading for each, and runs each file with the part of the choice that is its own. Not at a terminal, when the output is redirected or @code{CI} is set, there is no menu: a run with no names runs everything, as @code{--all} does. + +@subsubsection{More numbers} + +@code{--detail} adds three lines under each row: the mean, the standard deviation and the quartiles; the samples in the order they were taken; and a histogram of them: + +@codeblock[#:lang "text"]|{ +$ bjo bench benches/deck.bjo "hand points" --detail +name size median ± min alloc/op gc/1k 0/1/2 +hand points 5 6.50 ns 6.6% 5.21 ns 0 B - noisy, 10 outliers + mean 6.69 ns, sd 1.02 ns, q1 6.04 ns, q3 6.90 ns, 100 samples of 1600000 calls + in order ██████▄▂▃▃▃▃▃▂▂▃▁▃▃▃▃▁▂▂▄▂▂▃▃▃▃▂▁▂▄▄▂▂▂▃▃▃▃▃▃▂▂▅▄▁▂▃▃▁▁▃▃▅▂▃ + histogram 5.21 ns ▃▂▅▄▄▆█▇▅▂▃▂▂▂ ▁ ▃▂▂▂ 9.64 ns +}| + +The "in order" line is the one that explains a strange row. Here the first six samples were slower than the rest, and the spread of 6.6% and the @code{noisy} mark come from those six; the other 94 agree closely. A second run tells whether it was the machine or the code: run again, the same benchmark had a spread of 1.9%. A line that rises from left to right is a body that slows down as it runs, for example because its heap grows. The histogram shows the shape: two humps are two different speeds, not one noisy speed. + +@code{--scaling} adds a column to a benchmark with one numeric parameter: how much the time grew from the row before, against how much the parameter grew. + +@codeblock[#:lang "text"]|{ +$ bjo bench benches/deck.bjo "hand points" --scaling +name size median ± min alloc/op gc/1k 0/1/2 scaling +hand points 5 4.94 ns 5.4% 4.47 ns 0 B - noisy +hand points 13 9.99 ns 2.7% 8.41 ns 0 B - ×2.02 for ×2.60 +hand points 52 41.5 ns 0.7% 40.5 ns 0 B - ×4.16 for ×4.00 3 outliers +}| + +From 13 to 52 cards the time grew by as much as the hand did, which is what a linear function does. From 5 to 13 it grew less than the hand, because a few nanoseconds of every call are not spent on the cards at all, and with five cards those nanoseconds are a large part of the time. A column that keeps growing faster than the parameter is the sign of something quadratic. + +@subsubsection{Which of two ways is faster} + +"Is dealing with @code{vec-slice} faster than with a loop, and by how much?" is a question about two benchmarks in the same file. Put them in a @code{#:group}, and run with @code{--compare-within}: + +@codeblock[#:lang "text"]|{ +$ bjo bench benches/deck.bjo deal --compare-within +name median ± min alloc/op gc/1k 0/1/2 +deal with vec-slice 20.0 ns 0.9% 19.0 ns 152 B 0.02/0/0 2 outliers +deal with a loop 47.3 ns 0.6% 46.8 ns 656 B 0.08/0/0 14 outliers +deal from a stacked deck skipped: the cheat is not written yet + +bench: within dealing, against deal with vec-slice: 20 rounds in this process, paired (Wilcoxon signed-rank); 1 comparison +bench: seed 11 (--seed repeats the order of the cases) + +name deal with vec-slice this change 95% interval p ratio verdict +deal with a loop 19.5 ns 47.0 ns +140.9% +140.0% .. +141.7% <0.001 ×2.41 SLOWER +}| + +After the usual table, the benchmarks of each group are measured again in rounds: every round takes one sample of each, in an order shuffled anew each round. Each one is then compared with the first of its group. Taking turns is what makes the comparison fair. If the machine gets slower halfway through, both sides get slower in the same rounds, and the comparison, which pairs the samples of a round, does not move. + +The ratio is the number to quote. "A loop is 2.4 times as slow as @code{vec-slice}" is roughly true on the next machine too, where "47 ns" is not. @code{-N 50} takes 50 rounds rather than 20, and @code{--seed 11} repeats the order of an earlier run. The columns of the comparison, and how a verdict is reached, are explained in the next section. + +@subsubsection{Saving a run and comparing with it} + +@code{--save FILE} keeps a run: every sample of every benchmark, with the facts of the machine it was taken on. @code{--compare FILE} measures again and compares with what was kept. Nothing is saved unless you ask, and nothing is saved anywhere you did not name. Where a baseline lives, and whether it is committed, is up to you. + +@codeblock[#:lang "text"]|{ +$ bjo bench benches/deck.bjo full-deck shuffle "hand points" --save before.bjodat +... +bench: saved to before.bjodat +}| + +The file is bjodat, a benchmark to a line, so a diff of two saved runs shows which benchmarks changed: + +@codeblock[#:lang "text"]|{ +(bench-run + (version 1) + (saved "2026-10-16T17:25:06Z") + (machine "95c59c6948245e2e") + (header (build "Release") (dotnet ".NET 10.0.4") (arch "X64") (cores "24") ...) + (case (name "full-deck") (labels) (k 60000) (once #f) (bytes 1744.0) (samples [371.62 312.41 ...])) + ...) +}| + +Now suppose @code{full-deck} is rewritten to collect the cards in a list and turn the list into a vec at the end: + +@codeblock[#:lang "bjolang"]|{ +(defun (full-deck) + (list->vec + (loop (:for s all-suits) + (:subloop) + (:for r all-ranks) + (:acc deck (listing (Card (rank r) (suit s)))) + => deck))) +}| + +The comparison prints the run's own table, and then: + +@codeblock[#:lang "text"]|{ +$ bjo bench benches/deck.bjo full-deck shuffle "hand points" --compare before.bjodat +... +bench: compared with before.bjodat, saved 2026-10-16T17:25:06Z; unpaired (Mann-Whitney U); 6 comparisons, Holm-corrected +bench: one process each: a difference between processes is not in the p-values + +name size was now change 95% interval p verdict +full-deck 311 ns 597 ns +92.1% +91.7% .. +92.5% <0.001 SLOWER +shuffle 623 ns 548 ns -12.0% -12.0% .. -11.8% <0.001 faster +hand points 5 4.58 ns 5.07 ns +10.7% +10.1% .. +11.3% <0.001 SLOWER +hand points 13 10.6 ns 10.2 ns -4.2% -5.0% .. -3.7% <0.001 faster +hand points 52 44.2 ns 44.7 ns +1.0% +0.4% .. +1.4% <0.001 no change (< 2%) +sort a shuffled deck 3.68 µs 3.92 µs +6.8% +6.6% .. +7.0% <0.001 SLOWER +}| + +Each benchmark gets four answers: + +@read-list{ + - @strong{The change} is the ratio of the two medians, as a percentage. + - @strong{The interval} is a bootstrap of that ratio: the samples are resampled 10 000 times, and the middle 95% of the resampled ratios is shown. It makes no assumption about the shape of the timings, which are lopsided and have long tails. + - @strong{The p-value} is the probability of a difference at least this large if nothing had changed, from the Mann–Whitney U test. A run that compares many benchmarks at once will show some differences by chance alone: with 40 benchmarks at a level of 5%, two are expected to look different when nothing changed. So the p-values are corrected for the number of comparisons, by Holm's method, and the corrected value is the one shown and used. @code{--p-raw} shows the uncorrected values beside them. + - @strong{The verdict} is @code{faster} or @code{SLOWER} only when the corrected p is below 0.05 (@code{--alpha}) @em{and} the change is larger than 2% (@code{--threshold}). A change that is real but smaller is @code{no change (< 2%)}: it is there, and it was decided not to matter. Anything else is @code{no change}. +} + +The reason for asking for both is that a p-value says how unlikely a difference is, not how large it is: with enough samples, a change of 0.1% has a tiny p. + +Now look at the table again. @code{full-deck} is 92% slower, as it should be: that is the change. But @code{shuffle} did not change at all, since its deck is made in its setup, outside the time. Yet it says @code{faster}, by 12%. The rows for @code{hand points} and the sorting moved too, and they never call @code{full-deck} in their time at all. + +This is the weakness of comparing with a saved run. The two runs are two processes, and @strong{two processes of the same program can differ by a few percent}, in every benchmark at once. Where the code ends up in memory, which core the process runs on, and what the JIT decided all differ from one process to the next. Within one run the samples agree to a fraction of a percent, so the test is very sure about a difference that is really between the processes. The report says this in its second line. Treat a saved-run comparison as a tool for large changes, like @code{full-deck}'s. For small ones, use @code{--against} with several process pairs, as described next. + +A saved run from another machine is still compared, but under a warning above the table, and every verdict is marked @code{(other machine)}: the numbers are shown, but no verdict is to be believed. Differences in the build, the .NET version, the garbage collector's mode or the CPU governor are warned about too. + +@code{--fail-slower PCT} makes the run exit with 1 when anything is @code{SLOWER} by more than @code{PCT}, for a CI job that guards a hot path. It does not apply against a run from another machine. A file that cannot be read stops the run with exit code 2 before anything is measured. + +@subsubsection{Two versions against each other: @code{--against}} + +The real question behind most benchmarking is "did my change make this slower?". @code{--against} answers it in one run. It builds the benchmark file as it is and as it was at a git ref, and measures the two versions in turns: + +@codeblock[#:lang "text"]|{ +$ bjo bench benches/deck.bjo full-deck shuffle "hand points" --against HEAD~1 +bench: 6 benchmarks, Release build, .NET 10.0.4, X64, 24 cores, gcServer=false +bench: AMD Ryzen 9 5900X 12-Core Processor, Fedora Linux 43 +bench: A/B against HEAD~1, 20 rounds, paired (Wilcoxon signed-rank); 6 comparisons, Holm-corrected +bench: one process per side: a difference between processes is not in the p-values (--processes P measures it) +bench: seed 42 (--seed repeats the order of the cases) + +name size HEAD~1 now change 95% interval p verdict +full-deck 360 ns 649 ns +80.2% +73.9% .. +93.9% <0.001 SLOWER +shuffle 659 ns 616 ns -6.5% -63.1% .. +2.4% 0.097 no change +hand points 5 5.60 ns 5.46 ns -2.4% -5.5% .. +1.5% 0.046 faster +hand points 13 11.5 ns 12.8 ns +11.0% +5.7% .. +17.6% <0.001 SLOWER +hand points 52 47.2 ns 50.3 ns +6.6% +4.7% .. +10.9% <0.001 SLOWER +sort a shuffled deck 4.09 µs 4.14 µs +1.3% -4.5% .. +8.4% 0.50 no change +}| + +This is what happens: + +@read-list{ + - @code{REF} is checked out in a temporary git worktree, which is removed afterwards, also when something fails. It can also be a path: another checkout, or another @code{.bjo} file. + - Both sides are built in Release, by the same compiler. Both programs are started, and @code{bjo bench} drives them over a small protocol on their standard input and output. + - Every benchmark both sides have is warmed up and calibrated on both, with the smaller of the two batch sizes, so that a sample of each is the same number of calls. + - Then come @code{-N} rounds, 20 by default. Each round takes one sample of every benchmark from each side, in an order shuffled from @code{--seed}, which is printed so that a run can be repeated. Which side goes first alternates every round, so that neither side always runs on a cache the other one warmed. + - Each round's two samples are a pair. The p-value is the Wilcoxon signed-rank test on the pairs, and the interval a bootstrap that resamples whole rounds. If the machine slows down during the run, both samples of a round slow down, and the comparison does not move. +} + +A benchmark that only one side has is listed, not compared. + +Look at the table, though. Rounds cancel a slowdown @em{during} the run, but not the difference between the two processes as a whole, and that difference is still there: @code{hand points} is called @code{SLOWER} by 11% at size 13, and it has not changed. The report says so in its fourth line. This is why there is @code{--processes}: + +@codeblock[#:lang "text"]|{ +$ bjo bench benches/deck.bjo full-deck shuffle "hand points" --against HEAD~1 -N 60 --processes 6 +bench: A/B against HEAD~1, 60 rounds, 6 process pairs (t-test on their log ratios); 6 comparisons, Holm-corrected +bench: seed 42 (--seed repeats the order of the cases) + +name size HEAD~1 now change 95% interval p between processes verdict +full-deck 344 ns 614 ns +81.2% +72.4% .. +90.4% <0.001 4.7% SLOWER +shuffle 596 ns 585 ns -2.1% -10.3% .. +6.9% 0.73 8.3% no change +hand points 5 5.55 ns 5.13 ns -9.6% -18.7% .. +0.6% 0.24 10.1% no change +hand points 13 12.1 ns 11.4 ns -7.6% -20.8% .. +7.7% 0.73 14.7% no change +hand points 52 45.2 ns 45.0 ns -3.1% -10.5% .. +4.9% 0.73 7.6% no change +sort a shuffled deck 3.91 µs 4.04 µs +3.6% +0.5% .. +6.7% 0.15 2.8% no change +}| + +With @code{--processes 6}, the rounds are spread over six pairs of processes. Each pair gives one ratio of medians, and the test is a t-test on the six ratios, so a difference between processes is now part of what is tested rather than hidden from it. The "between processes" column says how large it was. For these tiny benchmarks it was 3 to 15%, more than any of the "changes" the single pair reported. Now only @code{full-deck} gets a verdict, which is the right answer. + +So which to use? + +@read-table{ +| Question | Use | +|----------------------------------------------------------+-------------------------------------------------------------| +| is it about this fast | @code{time-it}, or a benchmark file | +| which of two ways is faster, and by how much | @code{#:group} and @code{--compare-within}: one process | +| did my change make something a lot slower | @code{--against REF}, or @code{--compare} with a saved run | +| did my change make something a few percent slower | @code{--against REF --processes 6} or more, and more rounds | +} + +@code{--against} has a few more options. @code{--against-warmup S} and @code{--against-sample MS} set the warm-up limit and the sample length of both sides. A warm-up much shorter than the default second can mistake the JIT's first tier for steady, and a benchmark that did not get steady is named in a warning. @code{--against-build "CMD"} builds the other side with a command of your own, for a side that is not built the way this one is. @code{--save FILE} keeps both sides' samples: this code's in @code{FILE}, and the other side's beside it, with @code{.against} before the extension. @code{--fail-slower}, @code{--alpha}, @code{--threshold} and @code{--p-raw} work as they do for @code{--compare}. + +A CI runner is one machine for the length of one job, which is exactly what @code{--against} needs. A job that guards performance runs @code{bjo bench --against origin/main --processes 6 --fail-slower 5} and fails when something got more than 5% slower. + +@subsubsection{The flags} + +@read-table{ +| Flag | What it does | +|----------------------------+-----------------------------------------------------------------------------| +| @code{NAME} | runs the benchmarks whose names contain it; @code{n=10} chooses rows | +| @code{--all} | runs every benchmark, without the menu | +| @code{--list} | names the benchmarks and their parameters, and runs nothing | +| @code{--quick}, @code{--long} | halves or quadruples the time targets | +| @code{--min-samples N} | takes at least @code{N} samples (20) | +| @code{--max-time S} | lets a noisy benchmark sample for up to @code{S} seconds | +| @code{--params P=V,...} | replaces a parameter's values for the run | +| @code{--detail} | the mean, the deviation, the quartiles, the samples in order, a histogram | +| @code{--scaling} | how the time grows with a numeric parameter | +| @code{--compare-within} | compares each @code{#:group}'s benchmarks with its first, in rounds | +| @code{-N ROUNDS} | the rounds of @code{--compare-within} and @code{--against} (20) | +| @code{--seed N} | repeats the order of an earlier run's rounds | +| @code{--save FILE} | writes every sample of the run to @code{FILE} | +| @code{--compare FILE} | compares with the run saved in @code{FILE} | +| @code{--alpha P} | the significance level of a verdict (0.05) | +| @code{--threshold PCT} | the smallest change a verdict counts (2%) | +| @code{--p-raw} | shows the uncorrected p-values too | +| @code{--fail-slower PCT} | exits with 1 when something is @code{SLOWER} by more than @code{PCT} | +| @code{--against REF} | runs the file against itself at a git ref, or against a path | +| @code{--processes P} | with @code{--against}: @code{P} pairs of processes (1) | +| @code{--against-warmup S} | with @code{--against}: the warm-up limit of both sides (1 s) | +| @code{--against-sample MS} | with @code{--against}: the sample length of both sides (10 ms) | +| @code{--against-build CMD} | with @code{--against}: builds the other side with @code{CMD} | +} + +A percentage can be written @code{2} or @code{2%}. A benchmark file is an ordinary program, so @code{bjo run benches/deck.bjo --quick shuffle} runs it just as well. @code{bjo bench} adds the Release build, the files under @code{benches/}, the menu over several files and @code{--against}. + +@subsubsection{Driven by another program} + +@code{--against} works because a benchmark program can be driven. Started with @code{--serve}, it reads commands on its standard input and answers each with one line of bjodat on its standard output. Below, @code{>} marks what is sent and @code{<} what comes back: + +@codeblock[#:lang "text"]|{ +> (list) +< (cases (case 0 (name "full-deck") (labels) (group "") (once 0) (skip "")) (case 1 (name "shuffle") (labels) ...) ...) +> (prepare 1 :warmup 1000 :sample 10) +< (prepared 1 :k 20000 :steady #t) +> (sample 1 :k 20000) +< (sample 1 :ns 13663495 :bytes 39680000 :gc (4 0 0)) +> (sample 9 :k 1) +< (error "case 9 is not prepared") +> (quit) +}| + +@code{prepare} runs a benchmark's setup and warms it up and calibrates it, as a run would. @code{sample} takes one sample of @code{k} calls, and @code{:ns} is the whole sample's time. Anything wrong is answered with @code{(error "why")}, and the session goes on. A served program prints nothing else, so its output is the protocol and nothing more. A script that compares two builds made by two different compilers, which is not something @code{bjo bench} does by itself, can drive both of them this way, or use @code{--against PATH --against-build "CMD"}. + @subsection{Performance} Bjolang is compiled into pretty straight-forward c#, but without spans or ref structs. It performs pretty similar to c#, except for some small corners of the language that are different. diff --git a/_03.Standard-library.sz b/_03.Standard-library.sz index be81ded..2703162 100644 --- a/_03.Standard-library.sz +++ b/_03.Standard-library.sz @@ -97,9 +97,17 @@ For more details, see @link["modules/std/ports.html"]{@code{(std ports)}} and @l For more details, see @link["modules/std/test/"]{@code{(std test)}}, @link["modules/std/test/prop.html"]{@code{(std test prop)}}, @link["modules/std/test/gen.html"]{@code{(std test gen)}}, @link["modules/std/test/expect.html"]{@code{(std test expect)}} and @link["modules/std/simpletest.html"]{@code{(std simpletest)}}. +@subsection{std/bench} + +Benchmarks, written like tests. @code{def/bench} registers a benchmark, with a setup outside the time, parameters that make a row for each combination of values, cold runs for a slow body, and groups. @code{(run-benches)} is the @code{main} of the file. A run warms each benchmark up until its times are steady, calibrates how many calls make a sample, and takes samples until the median is known to ±1%. It prints a table of the median, the spread, the minimum, the allocation and the collections, a row as each benchmark finishes. With no names at a terminal, a menu asks what to run. @code{--save} and @code{--compare} keep a run in a file and compare with it, with p-values corrected for the number of comparisons and a verdict that needs both a significant and a large enough change. @code{bjo bench FILE --against REF} runs a file against itself at a git ref, the two versions measured in turns. @code{time-it} is here too: a body measured in place, on one line. + +@code{(std bench stats)} has the statistics: medians, quantiles, bootstrap intervals of a ratio, and the Wilcoxon, Mann–Whitney and t tests, each checked against SciPy. @code{(std bench store)} has saved runs as data, and the two kinds of comparison. + +The section on benchmarking in "Writing programs" shows all of it with the card game. For more details, see @link["modules/std/bench/"]{@code{(std bench)}}, @link["modules/std/bench/stats.html"]{@code{(std bench stats)}} and @link["modules/std/bench/store.html"]{@code{(std bench store)}}. + @subsection{std/stopwatch} -Stopwatches that read the ambient clock, so a test can bind a clock and get an exact time. @code{time-it} runs a body a number of times and reports the time and memory that it used. +Stopwatches that read the ambient clock, so a test can bind a clock and get an exact time. The module also has @code{time-it}, which is @code{(std bench)}'s: a body measured as a benchmark is, reported on one line. For more details, see @link["modules/std/stopwatch.html"]{@code{(std stopwatch)}}. @subsection{std/syntax-match} diff --git a/modules/std/bench/index.sz b/modules/std/bench/index.sz new file mode 100644 index 0000000..eadb6a9 --- /dev/null +++ b/modules/std/bench/index.sz @@ -0,0 +1,10 @@ +(meta (title "(std bench)") + (sort title)) + +@strong{Submodules:} +@itemlist{ + @item{@link["stats.html"]{@code{(std bench stats)}}: medians, bootstrap intervals, and the tests a comparison is decided by.} + @item{@link["store.html"]{@code{(std bench store)}}: saved runs as data, and comparing them.} +} + +@generate-docs[#:module "std bench"] diff --git a/modules/std/bench/stats.sz b/modules/std/bench/stats.sz new file mode 100644 index 0000000..714be74 --- /dev/null +++ b/modules/std/bench/stats.sz @@ -0,0 +1,3 @@ +(meta (title "(std bench stats)")) + +@generate-docs[#:module "std bench stats"] diff --git a/modules/std/bench/store.sz b/modules/std/bench/store.sz new file mode 100644 index 0000000..060ce36 --- /dev/null +++ b/modules/std/bench/store.sz @@ -0,0 +1,3 @@ +(meta (title "(std bench store)")) + +@generate-docs[#:module "std bench store"]