(std test prop)

Property tests: for-all runs a body with many generated values, shrinks a failing case, and stores it for the next run.

(import (std prelude))
(import (std test))
(import (std test gen))
(import (std test prop))

(deftest "reverse twice is the identity"
  (for-all ((: xs (List int)))
    (check (= (list-reverse (list-reverse xs)) xs))))

(deftest "a die roll is in range"
  (for-all ((n (gen-int-range 1 6)))
    (def r (roll-with n))
    (check (>= r 1))
    (check (<= r 6))))

(deftest "orderedmap-set then ref"
  (for-all #:runs 500 ((: m (OrderedMap string int)) (: k string) (: v int))
    (check (= (orderedmap-ref (orderedmap-set m k v) k) v))))

(run-tests)

for-all is a form in the body of a deftest. It runs its body many times, each time with new values from the generators of (std test gen). The checks of (std test) in the body tell whether a run passes. When a run fails, for-all shrinks it to a small case, reports that case under the test, and stores it beside the test file. The next run of the test tries the stored case first.

Runs. A for-all runs its body 100 times, or --runs n times, or #:runs n times. Each run has a size, from 1 at the first run to 100 at the last, so the first runs try small values. A run fails when a check in it fails, when it raises an exception, or when it leaves a resource open. The body of each run runs in a scope of its own, inside the scope of the test. The scope waits for the fibers that the run started, so the next run does not share them. The test has #:open-ok to allow open resources.

The first run that fails stops the random runs. A failed for-all does not stop its test, as a failed check does not. A test can have more than one for-all.

The report. A failure shows where the for-all is, which run failed and the seed of the run, the values of the bindings after shrinking, and the failures of the shrunk case:

FAIL reverse is not the identity (list_test.bjo:6)
  for-all at list_test.bjo:7 failed on run 4 of 100 (seed 1234567890)
  shrunk 9 time(s) to:
    xs = (0 1)
  list_test.bjo:8: (check (= (list-reverse xs) xs))
    (list-reverse xs) => (1 0)
    xs                => (0 1)

The second line says with: when shrinking found no smaller case. It says shrunk n time(s), then stopped, out of time or replays, at: when a limit stopped shrinking. When the generators raise an exception that is not a discard, the report shows the message and no values.

Discarded runs. assume discards a run, and so does a generator that gives up, such as gen-such-that. A discarded run is not counted. When more than 10 times the runs are discarded, the for-all fails, because the property was hardly tested.

Shrinking. Shrinking does not change the values. It changes the draws that the generators recorded, and replays them. Because a smaller draw gives a simpler value (see (std test gen)), a smaller vector of draws gives a simpler case. One round of shrinking does these steps, in this order:

• Delete blocks of 8, then 4, 2 and 1 draws, at each place.
• Set blocks of 8, then 4, 2 and 1 draws to 0, at each place.
• Make each draw smaller: try 0, and else search by halving for the smallest value that fails between 0 and the draw.
• Swap adjacent draws that are out of order.
• Only when the steps above found nothing, two steps that cost more replays: delete a block of 2 or 1 draws and add their sum to a draw near it, and move all or half of a draw to a later draw near it. “Near” is at most 8 places away. These steps shrink a list whose elements must add up to a value.

A change is kept when the replay fails the same way as the first failed run, and the draws that the replay used are smaller than the best so far. Two failures are the same when the first check that failed is at the same place, or when the run raised an exception of the same type. A smaller case that fails another way shows another bug, and reporting it would hide the bug that was found. Fewer draws are smaller. Of two vectors of the same length, the one that is smaller at the first place where they differ is smaller. Every kept change makes the draws smaller, so shrinking ends. The rounds repeat until a round keeps nothing.

Two limits stop shrinking early: --shrink-limit n replays, 2000 when not given, and the deadline of the test. Shrinking stops when the time left is not more than a fifth of the time that was left when shrinking started. The report then shows the smallest case found.

Flaky properties. Before it shrinks, for-all replays the failed run. When the replay passes, the property depends on something other than its draws, such as the time, a shared state or the scheduler. The report then says that the case passed when it ran again, and shows the values and the failures without shrinking. Nothing is stored.

Seeds. Each for-all draws from a source made from the seed of its test, the source text of its bindings, and its index among the for-all forms of the test with the same bindings. The seed of the test comes from the seed of the run and the name of the test (see test-seed). So choosing tests with filters, or adding a test, does not change what the other tests draw. A report shows the seed of the run. Run the test file again with --seed and that seed, and the name of the test as a filter, to repeat the run.

The failure file. A shrunk failure is stored in a file beside the test file: list_test.bjo stores to list_test.failed.bjodat. The file holds the draws, not the values, so no value has to be written as text:

(failed
  (case (test "reverse is not the identity") (bindings "((: xs (List int)))") (index 0) (seed "1234567890") (size 4) (draws [1 0 1 1 0]))
  (case (test "parse round trip") (bindings "((e gen-expr))") (index 0) (seed "1234567890") (size 37) (draws [2 7 0])))

• test is the name of the test, bindings the source text of the bindings of the for-all, and index its position, from 0, among the for-all forms with the same bindings that the test ran. A case is found by these three, not by a line number, because lines change when the file is edited. A change to the body keeps the case, so that it is tried while the bug is fixed. A change to the bindings drops it, because the draws no longer mean what they meant. A for-all added to the test does not take over the cases of another, unless its bindings are written the same.
• seed is the seed of the run that found the case, for a person to read.
• size is the size of the run. A replay uses it.
• draws are the shrunk draws. A draw above long-max is written as a ulong, 18446744073709551615UL, and the others as plain numbers.

A for-all first replays each case that the file has for it. The cases that pass are removed, and the cases that fail stay. The first that fails is shrunk again and reported, its draws are updated, and the random runs do not run. When no case fails, the random runs run, and a new shrunk failure is added at the end.

The file is read when a for-all first needs it. It is written at the end of (run-tests), through at-run-end!, and only when its text changed. When it has no case left, it is deleted. A run without filters also removes the cases of tests that no longer exist. run-test alone does not write the file: the cases stay in memory, and a later run in the same program sees them. With --no-store, the file is not read or written.

A file that does not read as cases is treated as empty. Whether to commit the file is the choice of the project. When it is committed, CI tries every case that anyone found.

Statistics. classify and collect count the runs that passed. With --stats, each for-all prints its counts when it ends:

for-all at list_test.bjo:7: 100 run(s)
  23% empty
  4% long

See also: for-all, assume, classify, label

(: assume (-> bool void))
(assume ok)

(for-all ((: xs (List int)))
  (assume (not (list-empty? xs)))
  (check (= (list-length (list-tail xs)) (- (list-length xs) 1))))

(: classify (-> string bool void))
(classify name ok)

(for-all ((: xs (List int)))
  (classify "empty" (list-empty? xs))
  (classify "long" (> (list-length xs) 10))
  (check (= (list-reverse (list-reverse xs)) xs)))

(: collect (-> %a void))
(collect value)

(for-all ((: xs (List int)))
  (collect (list-length xs))
  (check (= (list-reverse (list-reverse xs)) xs)))

(: run-for-all! (-> string int int string (List string) Property void))
(run-for-all! file line runs bindings names prop)

(: with-label (-> string (-> void) void))
(with-label name body)

(: prop-draw (-> (Gen %a) Choices %a))
(prop-draw g c)

(: prop-arbitrary (-> Choices %a))
(prop-arbitrary c)

(for-all ((name gen) ...) body ...)
(for-all ((: name Type) ...) body ...)
(for-all #:runs n ((name gen) ...) body ...)

(deftest "sort keeps the length"
  (for-all ((: xs (List int)))
    (check (= (list-length (list-sort xs)) (list-length xs)))))

(deftest "a sum of small numbers"
  (for-all #:runs 1000 ((a (gen-int-range 0 100)) (b (gen-int-range 0 100)))
    (check (<= (+ a b) 200))))

(label name body ...)

(for-all ((: m (OrderedMap string int)) (: k string) (: v int))
  (def m2 (orderedmap-set m k v))
  (label "after insert"
    (check (= (orderedmap-ref m2 k) v))))