Benchmarking: (std bench) and bjo bench, with the card game
A new section 2.9 between Testing and Performance: time-it, a benchmark file for the deck, how a benchmark is measured, setup and parameters, the menu, --detail and --scaling, --compare-within, saving and comparing, --against with --processes, the flags, and --serve. Every output is from a real run. It shows the weakness of comparing two processes too: unchanged benchmarks called faster or SLOWER by a few percent, and how --processes 6 settles it. The standard library chapter gets std/bench, std/stopwatch says time-it is (std bench)'s, and modules/std/bench/ has the reference pages for (std bench), (std bench stats) and (std bench store). 🤖 Generated with [ECA](https://eca.dev) (anthropic/claude-opus-4-7) Co-Authored-By: eca-agent <git@eca.dev>
This commit is contained in:
parent
3dc1518638
commit
f8cb9d3459
5 changed files with 473 additions and 1 deletions
|
|
@ -97,9 +97,17 @@ For more details, see @link["modules/std/ports.html"]{@code{(std ports)}} and @l
|
|||
|
||||
For more details, see @link["modules/std/test/"]{@code{(std test)}}, @link["modules/std/test/prop.html"]{@code{(std test prop)}}, @link["modules/std/test/gen.html"]{@code{(std test gen)}}, @link["modules/std/test/expect.html"]{@code{(std test expect)}} and @link["modules/std/simpletest.html"]{@code{(std simpletest)}}.
|
||||
|
||||
@subsection{std/bench}
|
||||
|
||||
Benchmarks, written like tests. @code{def/bench} registers a benchmark, with a setup outside the time, parameters that make a row for each combination of values, cold runs for a slow body, and groups. @code{(run-benches)} is the @code{main} of the file. A run warms each benchmark up until its times are steady, calibrates how many calls make a sample, and takes samples until the median is known to ±1%. It prints a table of the median, the spread, the minimum, the allocation and the collections, a row as each benchmark finishes. With no names at a terminal, a menu asks what to run. @code{--save} and @code{--compare} keep a run in a file and compare with it, with p-values corrected for the number of comparisons and a verdict that needs both a significant and a large enough change. @code{bjo bench FILE --against REF} runs a file against itself at a git ref, the two versions measured in turns. @code{time-it} is here too: a body measured in place, on one line.
|
||||
|
||||
@code{(std bench stats)} has the statistics: medians, quantiles, bootstrap intervals of a ratio, and the Wilcoxon, Mann–Whitney and t tests, each checked against SciPy. @code{(std bench store)} has saved runs as data, and the two kinds of comparison.
|
||||
|
||||
The section on benchmarking in "Writing programs" shows all of it with the card game. For more details, see @link["modules/std/bench/"]{@code{(std bench)}}, @link["modules/std/bench/stats.html"]{@code{(std bench stats)}} and @link["modules/std/bench/store.html"]{@code{(std bench store)}}.
|
||||
|
||||
@subsection{std/stopwatch}
|
||||
|
||||
Stopwatches that read the ambient clock, so a test can bind a clock and get an exact time. @code{time-it} runs a body a number of times and reports the time and memory that it used.
|
||||
Stopwatches that read the ambient clock, so a test can bind a clock and get an exact time. The module also has @code{time-it}, which is @code{(std bench)}'s: a body measured as a benchmark is, reported on one line.
|
||||
For more details, see @link["modules/std/stopwatch.html"]{@code{(std stopwatch)}}.
|
||||
|
||||
@subsection{std/syntax-match}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue