(std bench stats)

Reference

Overview

The statistics of benchmark samples: medians and spreads, bootstrap intervals, and the tests a comparison is decided by.

(import (std bench stats))

(def before [41.2 41.0 41.9 40.8 41.3])
(def after  [44.1 44.6 43.9 44.0 44.8])
(median after)                       ; 44.1
(bootstrap-ratio before after)       ; an interval around 1.07
(mann-whitney-p before after)        ; below 0.05
(holm [0.01 0.04 0.03])              ; [0.03 0.06 0.06]

Pure functions over (Vec double), for (std bench) and (std bench store), and for anyone with samples of their own. Nothing here assumes the shape of a distribution except t-test-p and t-critical, which are for a few values that are each a median of many.

Each test is checked against SciPy with its defaults: wilcoxon, mannwhitneyu(method="asymptotic"), ttest_1samp and t.ppf.

See also: median, bootstrap-ratio, wilcoxon-p, mann-whitney-p, t-test-p, holm

Reference

Functions

Functions

median

function ↑ Functions

(: median (-> (Vec double) double))
(median xs)

The median: the middle value, or the mean of the two middle values.

xs
The samples, in any order. Not changed.
returns
The median; 0 for no samples.

See also: quantile, mad

quantile

function ↑ Functions

(: quantile (-> (Vec double) double double))
(quantile xs q)

The value below which a fraction q of the samples lie, interpolated between neighbours.

xs
The samples, in any order. Not changed.
q
The fraction, from 0 to 1.
returns
The quantile, as NumPy's default (linear) method gives it.

See also: median

mean

function ↑ Functions

(: mean (-> (Vec double) double))
(mean xs)

The arithmetic mean.

xs
The samples.
returns
The mean; 0 for no samples.

std-dev

function ↑ Functions

(: std-dev (-> (Vec double) double))
(std-dev xs)

The sample standard deviation, over n - 1.

xs
The samples.
returns
The standard deviation; 0 for fewer than two samples.

mad

function ↑ Functions

(: mad (-> (Vec double) double))
(mad xs)

The median absolute deviation from the median, unscaled.

xs
The samples.
returns
The median of the distances from the median.

What (std bench) shows as ±, as a percentage of the median. One slow sample moves it by at most one place, where it can move a standard deviation arbitrarily far.

tukey-outliers

function ↑ Functions

(: tukey-outliers (-> (Vec double) int))
(tukey-outliers xs)

How many samples lie beyond Tukey’s fences, 1.5 inter-quartile ranges outside the quartiles.

xs
The samples.
returns
The count.

bootstrap-ratio

function ↑ Functions

(: bootstrap-ratio (-> (Vec double) (Vec double) (#:resamples int) (#:seed ulong) (Tuple double double)))
(bootstrap-ratio a b #:resamples 10000 #:seed 1UL)

A 95% interval of median(b) / median(a), from resampling each set on its own.

a
The samples of the first side.
b
The samples of the second side.
#:resamples
How many resamplings. 10 000 unless given.
#:seed
The seed of the resampling, so that a run is repeatable. 1 unless given.
returns
The 2.5th and the 97.5th percentiles of the resampled ratios.

See also: bootstrap-ratio/paired

bootstrap-ratio/paired

function ↑ Functions

(: bootstrap-ratio/paired (-> (Vec double) (Vec double) (#:resamples int) (#:seed ulong) (Tuple double double)))
(bootstrap-ratio/paired a b #:resamples 10000 #:seed 1UL)

A 95% interval of median(b) / median(a) for paired samples, resampling whole pairs.

a
The first side's samples; a[i] goes with b[i].
b
The second side's samples.
#:resamples
How many resamplings. 10 000 unless given.
#:seed
The seed of the resampling. 1 unless given.
returns
The 2.5th and the 97.5th percentiles of the resampled ratios.

For an A/B run’s rounds: a drift that moved both samples of a round stays in the round, rather than being spread over both sides.

See also: bootstrap-ratio

wilcoxon-p

function ↑ Functions

(: wilcoxon-p (-> (Vec double) double))
(wilcoxon-p ds)

The two-sided p-value of the Wilcoxon signed-rank test that the differences are centred on 0.

ds
The differences of paired samples; zeros are dropped.
returns
The p-value; 1 when nothing is left to test.

Exact, from the distribution of the signed-rank sum, when at most 50 differences are left and none are tied; otherwise the normal approximation, its variance corrected for ties, without a continuity correction. SciPy’s wilcoxon with its defaults gives the same.

See also: mann-whitney-p, t-test-p

mann-whitney-p

function ↑ Functions

(: mann-whitney-p (-> (Vec double) (Vec double) double))
(mann-whitney-p xs ys)

The two-sided p-value of the Mann-Whitney U test that two sets of samples come from one distribution.

xs
The first set.
ys
The second set.
returns
The p-value; 1 when either set is empty.

The normal approximation with a continuity correction and the variance corrected for ties, as SciPy’s mannwhitneyu with method="asymptotic" computes it.

See also: wilcoxon-p

t-test-p

function ↑ Functions

(: t-test-p (-> (Vec double) double))
(t-test-p xs)

The two-sided p-value of a one-sample t-test that the values’ mean is 0.

xs
The values: for an A/B run, the logs of the process pairs' ratios.
returns
The p-value; 1 for fewer than two values.

For a handful of values that are each a median of many samples: too few for a rank test to reach a small p, which with n values is 2/2^n at best, and close enough to normal for the t-test to be fair. SciPy’s ttest_1samp(xs, 0) gives the same.

See also: t-critical, wilcoxon-p

t-critical

function ↑ Functions

(: t-critical (-> double int double))
(t-critical alpha df)

The t with P(|T| > t) = alpha for Student’s T: what a mean’s standard error is multiplied by for its interval.

alpha
The two-sided level: 0.05 for a 95% interval.
df
The degrees of freedom.
returns
The critical value, as SciPy's t.ppf(1 - alpha/2, df).

See also: t-test-p

holm

function ↑ Functions

(: holm (-> (Vec double) (Vec double)))
(holm ps)

Holm’s step-down correction of p-values for the number of comparisons made at once.

ps
The p-values.
returns
The corrected p-values, in the order given.

The smallest is multiplied by m, the next by m - 1, and so on, each at least the one before it and none above 1. A run that compares 40 benchmarks at 0.05 expects two to look different when nothing changed; these are what keeps it from saying so.

normal-sf

function ↑ Functions

(: normal-sf (-> double double))
(normal-sf z)

P(Z > z) for a standard normal Z.

z
The point.
returns
The upper tail probability, to a relative error below 1.2e-7.