While the first part focused on the core language constructs in single files, this part covers the wider ecosystem: breaking programs into modules, managing project dependencies, handling I/O, concurrency with bjoroutines, .NET interop, macros, and testing.
Throughout this part we will continue building on the card game from Part 1, gradually turning it into a full project complete with a deck module, a persistent high-score file, concurrent players, and a custom @code{#card(Q hearts)} literal.
In Bjolang, every @code{.bjo} file is implicitly a module named after its file. There are no namespace declarations or enclosing module blocks; instead, a module simply exports whatever public API it wants other files to see, leaving everything else private by default.
- Relative imports: @code{(import "deck.bjo")} specifies a file path relative to the importing file. Bjolang compiles @code{deck.bjo} into @code{deck.dll} ahead of time, automatically rebuilding it whenever it (or any of its dependencies) changes.
- Type exports are complete: Exporting a type exposes all its constructors, union cases, and fields. Because @code{Card} is exported, @code{game.bjo} can freely construct @code{(Card (rank Ace) (suit Spades))} and pattern-match on its fields.
- Trait implementations are automatic: Although @code{->str} was not explicitly listed in the export list, its implementations still work in @code{game.bjo}. Because @code{impl} blocks define anonymous implementations rather than named bindings, they cannot be exported individually; they are automatically carried along whenever the module defining them is loaded.
- Private definitions: Unexported bindings like @code{all-suits} remain private to @code{deck.bjo} and are inaccessible from outside.
Every exported definition must have an explicit top-level type signature. The compiler needs this signature to write the metadata into the compiled @code{.dll}:
A list-shaped module path such as @code{(std random)} refers to a standard library module. The compiler resolves these relative to the Bjolang installation rather than the current directory, so @code{(std random)} always points to the same library regardless of where you invoke @code{bjo}. A few notable modules include:
By default, @code{(std prelude)} is imported implicitly into every module, providing core functions like @code{println}, @code{list-map}, and the basic operations introduced in Part 1. If you need to hide or override names from the prelude, you can import it explicitly using import modifiers (described below), which replaces the default implicit import:
Types are private unless listed in @code{export}. Furthermore, the compiler forbids exporting a function whose signature references an unexported private type, since importers would have no way to understand or satisfy that signature. The compiler catches this as an export error in the defining module.
When you want external modules to use a type without inspecting or constructing its internal representation directly, mark the type definition with @code{#:opaque}. For example, here is a card draw pile in @code{pile.bjo} that only permits drawing from the top:
With this setup, another module can receive @code{Pile} values, use @code{Pile} in type annotations, and call functions like @code{new-pile} or @code{draw}. However, it cannot construct a pile directly with @code{(Pile ...)}, access its internal @code{cards} field, or pattern-match on its structure:
Even though the type's representation is private, any trait implementations defined in @code{pile.bjo} remain available to consumers. For instance, an @code{(impl (->str Pile) ...)} inside @code{pile.bjo} allows any importing module to convert a @code{Pile} to a string.
Modifiers can be nested and evaluate from the inside out. For example, @code{(prefix (except (std hashset) hashset-map) "s/")} excludes @code{hashset-map} and prefixes all remaining names with @code{s/}.
Note that @code{only} and @code{except} apply exclusively to functions and macros, not types. A module's exported types are always imported, ensuring that signatures referencing those types remain valid and resolvable:
Types are strictly scoped to the module that declared them. If two imported libraries each declare their own @code{Card} type, they remain distinct types; the unqualified name @code{Card} simply resolves to whichever import came last. To disambiguate between them, use @code{prefix-types}:
- Local definitions take precedence over imported names.
- For collisions between two imports, the later import shadows the earlier one. This shadowing behavior is how modules can freely override names from the prelude, which is loaded as the very first import.
- Explicit renames or aliases created by import modifiers or @code{:alias} must be unique. If an alias collides with an existing name, the compiler raises an error to prevent ambiguity.
While @code{export} publishes definitions authored in the current module, @code{re-export} publishes definitions that were imported from elsewhere. This makes it easy to create aggregator modules that bundle several sub-modules behind a single import. The prelude uses this technique to pass through @code{=}, @code{compare}, and @code{list-sort} from @code{(std eq)}. In our card game:
Re-exporting a type exposes the exact original type, not a distinct copy; a @code{Card} referenced via @code{cards.bjo} is identical to one from @code{deck.bjo}. However, trait implementations (@code{impl}) do not transfer through re-exports. They remain bound to the module where they were defined, meaning any file that wants to format cards with @code{->str} must still import @code{deck.bjo}.
You can provide alternative names for bindings or macros using @code{(:alias new old)}. Combining @code{:alias} with @code{export} or @code{re-export} allows libraries to offer cleaner or more idiomatic public APIs:
Unlike @code{import}, which compiles a separate module, @code{(include "file.bjo")} textually splices the included file's forms directly into the current file at compile time. No separate module boundary is created, no exports are required, and all declarations in the included file share the enclosing module's scope:
As a general guideline: use @code{include} when you want to divide a single large module across multiple files, and use @code{import} when the target file represents an independent component with its own encapsulation boundaries.
Up to this point, our programs have consisted of standalone files importing neighboring modules directly by relative path. While that approach is convenient for quick scripts and small experiments, any software that pulls in external dependencies or is meant to be consumed by other developers should be structured as a project.
A project is simply a directory containing a @code{manifest.bjodat} file. When operating inside a project directory, @code{bjo} understands the package's identity, resolves dependencies, and manages builds automatically. Outside of a project, the CLI falls back to single-file execution just as before.
All source files in a package live under @code{src/}, where each module's name mirrors its relative path within that directory. For example, moving @code{deck.bjo} into @code{src/} exposes it as the module @code{(cards deck)}, which @code{src/main.bjo} can import just like any library module:
Because @code{bjo} automatically detects the project root by walking upward until it finds a @code{manifest.bjodat}, you can invoke CLI commands from any sub-directory within the project:
In @code{bjo run . one two}, the dot specifies the project itself. Any arguments following the target are forwarded directly to the program's @code{main} function, binding @code{args} to @code{[one two]}. You can also point @code{bjo run} at a specific test file, such as @code{bjo run tests/deck-test.bjo}; the file will be compiled and linked against the project's declared packages.
A package that lacks an executable entry point (@code{src/main.bjo}) is treated as a library. Running @code{bjo init --lib} initializes a library project, and running @code{bjo build} inside it compiles all modules found under @code{src/}.
Path dependencies are resolved relative to the manifest file that specifies them. Because a path dependency points straight to source files on disk, it does not require a version number—the build simply consumes whatever code is currently in the directory, making local development and iteration seamless.
Package versions correspond to Git tags formatted strictly as @code{vX.Y.Z}. Git dependencies specify version constraints using @code{(version-at-least "1.2")}, @code{(version-between "1.2" "2.0")}, or @code{(version-at-most "2.0")}.
Bjolang resolves versions using Minimal Version Selection (MVS). Rather than greedily selecting the latest release available in the wild, MVS inspects the minimum version requested by each package across the entire dependency graph, and then selects the highest of those minimum requirements. In other words, you get the oldest possible release that satisfies everyone's stated requirements. This guarantees reproducible builds: new upstream releases won't alter your build until someone explicitly raises their requirement in a manifest, ensuring your code compiles identically a year from now. Any upper bound constraints are treated strictly as validation checks, never as criteria to pick an older package.
Commands like @code{bjo fetch}, @code{build}, and @code{run} clone external dependencies into a local @code{.bjo/} directory and record the exact resolved commits in @code{bjo.lock}. You should always check @code{bjo.lock} into version control, while leaving @code{.bjo/} ignored (as configured in the default @code{.gitignore}). If an upstream repository rewires a tag to point to a different commit after it was locked, the build aborts immediately. On CI servers, pass @code{--locked} to guarantee that the build fails if any dependency deviates from the locked snapshot.
You can also override dependency sources: if your root manifest defines an explicit source for a package, that source overrides any locations requested deeper in the dependency tree. This makes it straightforward to substitute a local checkout or an emergency fork while debugging.
Once restored, types and methods from NuGet packages can be consumed through Bjolang's .NET interop mechanisms (described in the .NET section below). Version strings follow standard NuGet versioning semantics: @code{"9.0.3"} matches version 9.0.3 or higher, while bracketed syntax like @code{"[9.0.3]"} locks to that exact version. Bjolang delegates package restoration to the .NET SDK via the @code{dotnet} tool, storing the resolution graph in @code{packages.lock.json} (which should also be committed). Currently, compiled binaries load dependencies directly from the local NuGet cache, so executables expect to run in an environment with the restored packages available.
A program usually runs on .NET: @code{bjo run} starts the runtime, and the JIT compiles each method the first time it runs. With @code{(aot #t)} in the manifest, the program is also compiled ahead of time, with .NET's Native AOT, to one native executable:
@codeblock[#:lang "bjolang"]|{
(package
(name (cards))
(version "0.1.0")
(aot #t))
}|
The executable is beside the source, with no extension: @code{src/main}, or @code{src/main.native.exe} on Windows. It holds the program, the standard library and the .NET runtime, so it runs on a machine without .NET installed.
@read-list{
- @strong{It starts in a few milliseconds.} A program on .NET spends about 70 ms starting the runtime, and then compiles its methods as it runs them. On 17 days of Advent of Code, the solutions together took 3.0 s on .NET and 1.2 s as native executables.
- @strong{@code{bjo run} starts the native executable, and @code{bjo publish} copies it} into the output directory. The ordinary @code{.exe} is still built, for the debugger and for @code{-d}.
- @strong{The build takes longer}, about 10 s more, and needs clang or gcc. The first build downloads .NET's ILCompiler package from NuGet.
- @strong{The runtime settings are built into the executable.} The GC settings, such as @code{(runtime (gc-server #t))}, take effect. The settings of tiered compilation do nothing, because there is no JIT.
- @strong{A long-running program can be slower.} The JIT optimizes the code with what it sees the program do (dynamic PGO), and a native executable can not. This matters for programs where the call graph stops the monomorphization or when trait calls cannot be determined at compile time.
- @strong{Reflection that is resolved at run time may fail.} Native AOT keeps only the code that it can see the program use. Bjolang does not use reflection by default, so this should be a small problem.
By default the executable runs on any x86-64 CPU, so it uses only the instructions that all of them have: no AVX, and 128-bit vectors. @code{aot-cpu} names a CPU to build for instead:
@codeblock[#:lang "bjolang"]|{
(aot #t)
(aot-cpu "native")
}|
@code{"native"} is the CPU of the machine that builds. A level, such as @code{"x86-64-v3"}, is every CPU that has its instructions: @code{x86-64-v3} has AVX2, and @code{x86-64-v4} has AVX-512. An executable built for a CPU stops at start on a CPU that does not have the instructions, so use @code{"native"} for a program that runs where it is built. This matters most to @code{(std array)} and @code{(std simd)}; see the section on vectors under performance.
Only the project being built reads @code{aot} and @code{aot-cpu}, as with @code{runtime}: a dependency is not a program. A change to either rebuilds the program. The compiler itself takes them as @code{--aot} and @code{--aot-cpu}, for a file built without a project.
- Organize your source files under @code{src/}, and create a @code{manifest.bjodat} defining the package @code{name} and @code{version}.
- Run @code{bjo build} to verify that all modules compile cleanly without errors.
- Tag your commit with the semantic version—e.g. @code{git tag -a v0.1.0 -m "First release"}—and push the tag to your remote repository. Dependencies require valid @code{vX.Y.Z} tags to resolve.
Because these wrap the underlying .NET I/O methods directly, any I/O failure (such as missing permissions or a missing file on read) throws a .NET exception. If you expect a failure under normal conditions, wrap the call in a @code{try} expression.
To illustrate, we can add a persistent high-score ledger to our card game. Each line records a player name and their score separated by a space. Malformed lines are safely discarded rather than crashing the program:
Paths in Bjolang are represented as standard strings. The path manipulation utilities perform purely lexical operations on the string representations without touching the filesystem:
When dealing with large files, streaming data, or incremental I/O, Bjolang uses *ports*. An input port provides a stream from which characters and lines can be read incrementally, while an output port collects written data. File ports and in-memory string ports share the same underlying interface, so functions written against ports work seamlessly with both files and string buffers.
Attempting to open a file returns a @code{Result}, treating potential I/O errors (such as missing files) as expected outcomes. The @code{Err} variant wraps the underlying .NET exception, allowing you to match on specific exception types with @code{:is}:
To avoid leaking resources, you will usually want to use @code{call-with-input-file} or @code{call-with-output-file}. These higher-order functions open the port, pass it to your callback, and guarantee the port is cleanly closed when the function finishes—even if an exception is thrown. For example, counting lines in an arbitrary file looks like:
Convenience helpers in @code{(std ports)} can ingest an entire port into collections: @code{port->lines}, @code{port->list}, @code{port->vec}, as well as lazy sequences via @code{port->seq} and @code{file->seq}:
In Bjolang, open file ports are also tracked by their enclosing structured concurrency scope (explained in the Concurrency section below). When a scope exits, any unclosed ports opened within it are automatically finalized. Even @code{main} acts as a root scope, ensuring leaked handles are collected when your process terminates. For long-lived processes, however, you should always explicitly close ports or rely on @code{call-with-input-file}.
The @code{(std run)} module lets you invoke external processes. Commands are written as S-expression forms using a concise DSL where pipelines and shell-style redirections are integrated directly into the syntax:
Prefixing an identifier with a comma (e.g. @code{,who}) splices the variable's value into the command arguments. Each list element is treated as an individual command-line argument, preserving spaces without fragile string quoting or shell injection hazards. The DSL primitives @code{pipe}, @code{into-file}, @code{from-file}, @code{append-to-file}, and @code{errors-into-file} are validated at compile time; any unrecognized head identifier is executed as an external executable.
All of these functions return a @code{Result} and execute synchronously, blocking the calling thread until completion. When running inside a bjoroutine, non-blocking asynchronous alternatives are available; refer to the @code{(std run)} module documentation for details.
For one line of output, an interpolated string or @code{println*} is enough. @code{println*} is in @code{(std fmt)}, and takes strings, ints, doubles, chars and bools in one vec:
@codeblock[#:lang "bjolang"]|{
(import (std fmt))
(println #"${who} has ${points} points") ;; ada has 31 points
(println* [who " has " points " points"]) ;; ada has 31 points
}|
When the shape of the output matters, such as columns that line up, padding, a paragraph wrapped to a width or a table, use the layout part of @code{(std fmt)}. This section is a guide to it. The module's page has the reference.
@subsubsection{A block and a vec of blocks}
Layout is made of two types, and nearly every type error in layout code is one of them where the other is wanted:
@read-list{
- A @code{Block} is one piece of output: a string, an int, a char, or a combinator such as @code{(padded/left 5 ["x"])}.
- A @code{(Vec Block)} is pieces of output in order, written as a vec literal: @code{["x = " 7 nl]}.
}
There is one rule. Every combinator takes its content as a @code{(Vec Block)} and gives back one @code{Block}. @code{show}, @code{render} and @code{render->string}, which write the output, take a @code{(Vec Block)}. So the brackets go around content, and a combinator call is one element in them:
@codeblock[#:lang "bjolang"]|{
(show [(padded/left 5 ["x"])]) ;; " x"
(show [(padded/left 5 ["x"]) (padded/left 5 ["y"]) nl]) ;; " x y", then a newline
Read @code{(padded/left 5 ["x"])} as "pad this content to 5 columns". The content is a vec even when it is one thing, as the body of a @code{begin} is a body even when it is one form. These are refused:
@codeblock[#:lang "bjolang"]|{
(show (padded/left 5 "x")) ;; "x" is not content, and show wants a vec
(show (padded/left 5 ["x"])) ;; show wants a vec, and this is one block
(show [["x"]]) ;; a vec in a vec: a vec is not a block
}|
More brackets do not help. A vec of vecs is right in one place only: a table, which is a vec of rows or of columns, each of them a @code{(Vec Block)}.
Nothing is written until @code{show} runs the blocks. A block is a value: you can bind it, pass it to a function and return it.
@subsubsection{What goes in the brackets}
A vec literal is read element by element when it stands where a @code{(Vec Block)} is wanted: as an argument of a function that takes one, as the return value of a function declared to return one, or as the value of a binding whose type is written. Each element is one of these:
| A @code{(List %a)} @code{xs} | @code{(joined f xs [", "])}, where @code{f} makes the block of one element |
}
@strong{A number bound by @code{def}.} The @code{(: n int)} above is needed. @code{(def n 3)} alone does not say which kind of number @code{n} is, as it could still become a long or a double, so @code{[n]} cannot read it as a block, and the compiler says so at @code{n}. The same holds for @code{(def xs [1 2 3])} and @code{(def total (+ 1 2))}. A number that comes from a typed parameter, a record field or a function such as @code{vec-length} has its type already, and a number written in the vec itself, as in @code{["n = " 3]}, is an int.
@strong{A vec bound by @code{def}.} @code{(def parts ["a" "b"])} is a @code{(Vec string)}, which is not a @code{(Vec Block)}: write its type, and it is read as blocks.
@codeblock[#:lang "bjolang"]|{
(: parts (Vec Block))
(def parts ["a" 1 #\b])
(show parts) ;; a1b
(show ["<" (each parts) ">" nl]) ;; <a1b>
}|
@subsubsection{From one type to the other}
@read-table{
| You have | You want a @code{Block} | You want a @code{(Vec Block)} |
Outside a vec literal nothing is converted: a function declared to return a @code{Block} returns @code{(BStr s)} or @code{(each [s])}, not @code{s}. @code{BStr}, @code{BInt} and @code{BChar} are the cases of @code{Block}, and like any constructor they are functions, so @code{(vec-map BStr names)} is a @code{(Vec Block)}.
@subsubsection{Blocks from your data}
Most layout code is a function from a value of your own to a block. Give it a signature: the signature is what tells the compiler that the literal inside is blocks.
(padded/left 9 [(numeric (record-ref i price) #:precision 2 #:comma 3)])
(if (= (record-ref i stock) 0) " sold out" nothing)
nl]))
(show [(each (vec-map item-line stock))])
}|
@codeblock[#:lang "text"]|{
Green tea 3.50
Gaiwan 1,240.00
Kettle 89.90 sold out
}|
@code{item-line} returns one block, so its pieces are under @code{each}. @code{vec-map} gives a vec of blocks, and @code{each} makes that one block too. A function can return content instead, a @code{(Vec Block)}, and the caller decides where it goes:
A lambda has no signature, so a vec literal returned from one is not read as blocks: in @code{(vec-map (fun (i) [(record-ref i name) (record-ref i stock)]) stock)} the vec has a string and an int in it, and is refused. Put the literal under @code{each}, which says that it is content, or write a function with a signature:
@codeblock[#:lang "bjolang"]|{
(show [(each (vec-map (fun (i) (each [(record-ref i name) ": " (record-ref i stock) nl]))
stock))])
;; Green tea: 12
;; Gaiwan: 2
;; Kettle: 0
}|
@code{nothing} is the block that writes nothing, for the branch that has nothing to say. In a vec literal, what an @code{if}, a @code{match} or a @code{cond} gives is read as any element is, so an arm can be a string. In a function declared to return a @code{Block}, nothing is converted, so there the arms are blocks:
@code{padded/left}, @code{padded/right} and @code{padded/both} pad content to a width: aligned right, left, or centred. Padding never cuts. @code{trimmed/left}, @code{trimmed/right} and @code{trimmed/both} cut content to a width, and keep its end, its start or its middle. @code{fitted/left}, @code{fitted/right} and @code{fitted/both} do both, so the content is exactly the width.
@code{space-to} pads to a column, whatever was written before it, and @code{with-pad} changes what the padding is made of. Together they make a leader:
@code{nl} is a newline, and @code{fl} a newline unless the cursor is already at the start of a line.
@subsubsection{Numbers}
An int is a block. A double is not, as there is no one way to write one: @code{numeric} says how, with a precision, digits in groups, and the separators. @code{bytes} and @code{percent} are blocks for two common kinds of number.
The digits are always the invariant culture's, so the machine's locale never shows. The separators are settings, described below.
@subsubsection{Paragraphs}
@code{wrapped} fills content to @code{current-width}, and @code{justified} makes it flush against both margins. Every run of whitespace in the content is a break between words.
@codeblock[#:lang "bjolang"]|{
(show [(with-fmt current-width 20
[(justified ["Bjolang lays text out with blocks, and blocks nest."])])
nl])
;; Bjolang lays text
;; out with blocks, and
;; blocks nest.
}|
The first line starts where the cursor is, so after a label it has less room. The lines after it start at column 0.
@subsubsection{Tables}
A table is the one place where a vec of vecs is right. @code{tabular} takes a vec of columns, and @code{tabular/rows} a vec of rows. Each column or row is a @code{(Vec Block)}, so the inner vecs are read as blocks as any other content is:
In @code{tabular} a column's cells are separated by @code{nl}. In @code{tabular/rows} each element of a row is one cell, so a cell of several pieces is under @code{each}, and a newline in a cell gives the row another line:
The tables here are drawn with @code{StyleAscii}, whose characters every font has. @code{#:style} takes a @code{TableStyle}: @code{StyleNone}, the default, @code{StyleSimple}, @code{StyleAscii}, @code{StyleLight}, @code{StyleHeavy}, @code{StyleDouble}, @code{StyleRounded}, @code{StyleMarkdown}, or @code{StyleCustom} with pieces of your own. @code{#:align} takes an @code{Align} for each column, from the first: @code{AlignLeft}, @code{AlignRight}, @code{AlignCentre}, or @code{(AlignOn #\.)}, which puts every decimal point of the column in one place. @code{#:widths} and @code{#:overflow} make a table fit @code{current-width}, by cutting or wrapping its cells.
A table whose rows are not all known yet, such as results that come one at a time, is laid out a row at a time with @code{table-layout} and @code{table-row}. The widths are fixed beforehand, so each row is written when it is known:
The rows can be spliced, @code{(rows ,scores)}, from a @code{(Vec (Vec TableCell))}. See @code{Table} on the module's page.
@subsubsection{Settings}
The pad character, the right margin, the ellipsis and the number formats are parameters. @code{with-fmt} binds one around some blocks while they run, and @code{with-pad} is @code{with-fmt} for @code{current-pad}. Settings nest:
@codeblock[#:lang "bjolang"]|{
(show [(with-fmt current-comma 3
[(with-fmt current-comma-sep #\space
[(with-fmt current-decimal-sep #\,
[(numeric 1234567.5)])])])
nl])
;; 1 234 567,5
}|
The settings are @code{current-pad}, @code{current-width}, @code{current-ellipsis}, @code{current-string-width}, @code{current-precision}, @code{current-comma}, @code{current-comma-sep}, @code{current-decimal-sep} and @code{current-signed}. For text with wide characters or colours in it, which a terminal shows in more or fewer columns than it has characters, set @code{current-string-width} to @code{terminal-width}.
@subsubsection{When it does not compile}
The messages for the common mistakes, and what to write instead:
| expected @code{(Vec fmt/Block)}, found @code{string} | @code{["x"]} for @code{"x"} |
| expected @code{(Vec fmt/Block)}, found @code{fmt/Block} | @code{[b]} for @code{b} |
| expected @code{fmt/Block}, found @code{(Vec ...)} | @code{(each bs)} for @code{bs} |
| They differ inside the Vec: @code{fmt/Block} against @code{string} | @code{(: bs (Vec Block))} above @code{(def bs ...)} |
| Block: expected a string, an int, a char ...; found a vec | @code{(each [...])}, or no inner brackets |
| Block: ... found the number 1.5 | @code{(numeric 1.5)} |
| Block: ... found a boolean | @code{(if b "yes" "no")} |
| Block: 'n' is a number, and nothing says yet whether it is an int ... | @code{(: n int)} above @code{(def n 3)} |
| what @code{f} returns has the wrong type: expected @code{fmt/Block}, found @code{string} | @code{(BStr s)} or @code{(each [s])} |
}
When none of these fits, find the place where a @code{Block} and a @code{(Vec Block)} meet, and check it against the rule: content in brackets, one block out.
@subsubsection{A block of your own}
@code{captured} renders content to a string and gives it to a function, which returns the block to write in its place. It is how a block that needs the whole of its content is made, and how padding and trimming are written:
@codeblock[#:lang "bjolang"]|{
(: boxed (-> (Vec Block) Block))
(defun (boxed bs)
(captured (fun (s) (each ["[" s "]"])) bs))
(show [(boxed ["x = " 7]) nl]) ;; [x = 7]
}|
A block that writes as it goes is a @code{BRun}: a function given the sink to write to and the column the cursor is at, which returns the column it ends at. @code{run-blocks} runs content inside one. The module's page describes both.
Bjolang's concurrency model rests on three foundations: lightweight fibers called *bjoroutines*, communication channels, and structured concurrency scopes ensuring background work is cleanly contained. In addition, Bjolang adopts the Concurrent ML (CML) paradigm, where communication operations—such as sending or receiving on a channel—are first-class values that can be composed and combined before being synchronized.
Functions that may suspend execution—whether waiting on a channel, sleeping, or performing asynchronous I/O—are declared with @code{defbjo} rather than @code{defun}, and their type signatures use the @code{-bjo->} arrow. Suspending and waiting for an event is performed using @code{sync}:
Bjoroutines execute as lightweight fibers scheduled across a managed thread pool. When a fiber suspends at a @code{sync} point, it yields its underlying thread back to the pool. As a result, maintaining thousands of waiting fibers incurs the memory overhead of small heap objects rather than expensive OS threads.
Because calling a suspending function means the caller may also have to wait, function coloring applies: bjoroutines can only be directly invoked from within other bjoroutines:
In practice, this coloring is less infectious than in many other languages. Most standard I/O functions (such as @code{file-read-text}) are automatically emitted with dual implementations: a synchronous blocking path and an asynchronous suspending path. An ordinary @code{defun} that calls standard I/O similarly gets both versions generated under the hood; when called from a bjoroutine, it transparently executes the non-blocking suspending version:
The @code{(spawn (f args ...))} form creates a new concurrent fiber running the invocation @code{(f args ...)}. Arguments are evaluated eagerly in the spawning fiber before transferring execution to the new fiber.
Fibers communicate over channels created with @code{make-chan}. Rather than performing immediate I/O, expressions like @code{(chan-send ch v)} and @code{(chan-recv ch)} construct first-class *events*, which are then synchronized using @code{sync}. Unbuffered channels operate as rendezvous points: sending blocks until a receiver is ready, and receiving blocks until a sender arrives.
We can model our card players as concurrent bjoroutines. Each player receives a channel representing their seat at the table and plays one card per trick (descending by value). The table (@code{main}) draws cards round-robin from each seat:
Crucially, calling @code{(chan-recv seat)} does not perform the receive immediately; it simply constructs an event value describing the operation. The actual communication occurs only when the event is passed to @code{sync}. Because events are first-class values, they can be parameterized, composed, and transformed before being awaited:
With these combinators, deadlines and timeouts become simple functions that compose over any event. For instance, we can enforce a response timeout on our card players:
Notice that @code{within} is defined with standard @code{defun} rather than @code{defbjo}. Simply constructing or composing event values does not suspend execution; suspending only occurs when an event is submitted to @code{sync}.
Also notice how the first timeout behaves: when the 50 ms deadline expires before the card arrives, the @code{chan-recv} event is cleanly cancelled without consuming the message. The card remains on the channel, available to be received by the subsequent 500 ms wait.
An operation that is synced right where it is written does not need to be an event first. @code{(chan-put ch v)} does what @code{(sync (chan-send ch v))} does, and @code{(chan-get ch)} what @code{(sync (chan-recv ch))} does. They wait, and are cancelled, exactly as @code{sync} is, but no event value is built, so they are cheaper. Use the event forms when you need an event: to @code{choose} between operations, to @code{wrap} one, or to hand one to a function such as @code{within}. Our players could just as well be written with @code{chan-put}:
While @code{spawn} runs a fiber for its side effects and discards its return value, @code{bjo} returns a joinable promise. You can await this promise using @code{(promise-join p)}, which yields a @code{Result} wrapping either the computed value or an exception if the fiber failed:
Because both fibers execute concurrently across thread pool workers, the combined operations complete in roughly 50 ms rather than running sequentially for 100 ms.
Every fiber runs within an enclosing structured concurrency scope. A scope cannot complete until all fibers spawned within it have terminated, ensuring background tasks never outlive their caller or leak into other components. Furthermore, unhandled exceptions inside spawned fibers propagate outward when the scope terminates rather than being silently dropped.
The top-level @code{main} function acts as an implicit root scope, which is why our earlier card game example cleanly awaited all player fibers. You can create nested scopes with fine-grained cancellation and deadlines using these forms:
Cancellation in Bjolang is cooperative rather than preemptive. Cancelling a scope signals its fibers, which check for cancellation whenever they next perform a suspending operation (such as waiting on a channel or timer). A fiber that exits due to cancellation is treated as a clean completion rather than a failure.
The scope forms work in both colours. In a bjoroutine, the wait at the end of the scope suspends. In ordinary code, a @code{defun} or a @code{(fun ...)}, it parks the thread until the children have ended, as @code{sync/blocking} does. So an ordinary function can start fibers and know that they are done when it returns:
@codeblock[#:lang "bjolang"]|{
(: fetch-all (-> (Vec string) void))
(defun (fetch-all names)
(with-scope
(ignore (own! (fun () (println "released"))))
(loop (:for n names)
(:do (spawn (fetch n))))
(println "all started")))
(defun (main)
(fetch-all ["a" "b"])
(println "back in main")
0)
}|
@codeblock[#:lang "text"]|{
all started
fetched a
fetched b
released
back in main
}|
A scope also owns resources. @code{(own! thunk)} registers a release on the innermost scope, and @code{(own-disposable x)} registers @code{x}'s @code{.Dispose}. When the scope ends, the releases run after the fibers have ended, newest first. @code{(release! h)} runs one early, with the handle @code{own!} returned. These are ordinary functions, so they also work in both colours. A release is a plain thunk: it cannot suspend, and a release that blocks also blocks the end of the scope.
Do not park a bjoroutine's thread by accident. A scope in an ordinary @code{(fun ...)} that a bjoroutine calls parks the thread of that bjoroutine, and the compiler warns about it. Write the scope in the bjoroutine itself. Or, if the function the lambda is given to takes a @code{-?->} parameter, write the lambda as a @code{(bjoroutine ...)}.
Performing long CPU-bound computations or calling legacy blocking .NET APIs directly inside a fiber can monopolize thread pool threads, preventing other fibers from running. To avoid blocking the pool, you can offload such operations onto dedicated background threads; both helpers return synchronization events:
Conversely, you may occasionally need to wait on a channel or event from inside synchronous code. While an ordinary @code{defun} cannot suspend with @code{sync}, it can block its caller thread using @code{sync/blocking}. Avoid calling @code{sync/blocking} inside a bjoroutine, as doing so freezes a thread pool worker that other fibers rely on:
@subsubsection{Shared state: atomics and transactions}
Channels move values from one fiber to another. When many fibers must read and change the same data, a channel means one fiber that owns the data and answers requests for it. That is often the clearest design, but it is not the only one. Two modules let fibers share data directly:
| @code{(std atomic)} | One value that threads change without a lock: a counter, a flag, a cached result |
| @code{(std stm)} | Changes to several values as one step, and waiting until the values are right |
}
An atomic is the simple case and is described in Part III. The rest of this section is about transactions: software transactional memory, built on the design of the OCaml library kcas.
@strong{Locations and transactions.} A @code{(Loc T)} holds one value. A transaction is a function of one argument, a @code{Tx}, which is the log of the transaction. The function reads and writes locations through the log with @code{tx-ref} and @code{tx-set!}, and @code{atomically} runs it and commits it:
@codeblock[#:lang "bjolang"]|{
(import (std stm))
(def pot (make-loc 0))
(def table (make-loc (list)))
;; Put a card on the table and add its points to the pot, as one step.
(: play! (-> Card int void))
(defun (play! card points)
(atomically
(fun (tx)
(tx-set! tx table (Cons card (tx-ref tx table)))
(tx-set! tx pot (+ (tx-ref tx pot) points)))))
}|
The writes stay in the log until the transaction commits, and the commit makes all of them visible at once. No other fiber sees the card on the table without its points in the pot. If another transaction changed a location that this one read before this one commits, this one runs again with the new values. So a transaction never acts on values that changed under it, and nothing is ever locked.
@strong{The rules for a transaction function.} Because it can run more than once, a transaction function must have no side effects except through its @code{Tx}: no printing, no I/O, no changes to other state. Do those after @code{atomically} returns. A transaction function also can not @code{sync}. It is an ordinary function, and the compiler says so:
@codeblock[#:lang "text"]|{
Type Error at play.bjo:7: calling 'sync' is a yield point, and a yield point is not allowed here.
}|
If a transaction raises an exception, it writes nothing, and @code{atomically} raises the exception. Store immutable values in locations: a change inside a mutable vec is not part of any transaction.
@strong{Waiting with tx-retry.} A transaction that can not go on yet calls @code{tx-retry}. That stops it, and it runs again when a location that it read has changed. There is no condition variable to signal and no wakeup to forget:
@codeblock[#:lang "bjolang"]|{
;; The top card of a pile. Waits while the pile is empty.
(: take-top (-> Tx (Loc (List Card)) Card))
(defun (take-top tx pile)
(match (tx-ref tx pile)
((Cons card rest) (tx-set! tx pile rest) card)
(Nil (tx-retry tx))))
}|
@strong{A transaction is an event.} @code{(atomically-evt f)} is the event that runs and commits @code{f}. While @code{f} retries, the event is not available. So waiting for a condition on shared data combines with everything else that @code{choose} combines: a timeout, a channel, another transaction, and the cancellation of the fiber's scope. A transaction whose branch loses has written nothing. @code{atomically} is @code{(sync (atomically-evt f))}:
Inside a transaction, @code{(tx-or-else tx a b)} runs @code{a}, and if @code{a} retries, undoes its writes and runs @code{b}. If both retry, the whole transaction waits for a change to anything either of them read.
@code{atomically} works in both colours: in a bjoroutine it syncs, and in an ordinary function it blocks the thread. @code{try-atomically} never waits: it gives @code{None} when the transaction retries, which makes it safe in a callback that .NET calls.
@strong{Queues, maps and counters.} Three modules build data structures out of locations, so that their operations are part of the transaction that calls them:
Two dealers can run at the same time. Neither card can end up in two hands, or be counted without being dealt. The queue also has the channel names for use outside a transaction: @code{queue-send} and @code{queue-recv} are events, and @code{queue-put} and @code{queue-get} sync them. What a queue adds to a channel is the transaction: take from one queue and put in another, or take from whichever of two queues has a card, as one step.
@strong{Costs.} A transaction that reads and writes the same locations as many others runs again often. Many fibers that write one location cost the most. A @code{TMap} or @code{TCounter} spreads the writes over many locations for that reason. A transaction that only reads never makes another one run again. Transactions also allocate, and a program in which many fibers allocate at once is often limited by the garbage collector: set @code{(runtime (gc-server #t))} in the manifest to give it one heap per CPU. With it, sixteen fibers that add keys to one @code{TMap} have eight times the throughput of one fiber.
Go and Hopac are the two points of comparison. Go is the obvious one: goroutines and unbuffered channels are the same model of fibers meeting at a rendezvous. Hopac is the tougher one. It is a Concurrent ML library for F#, running on the same .NET runtime with the same JIT and garbage collector, so the differences between it and Bjolang are in the design rather than in the platform. The main repository has the same benchmarks in all three (@code{bench/bjolang/cmlbench.bjo}, @code{go run . -suite cmlbench} in @code{bench/go}, and @code{dotnet run -c Release -- cmlbench} in @code{bench/hopac}).
Every Bjolang channel operation races the cancellation token of its scope. Neither Go nor Hopac has that built in, so each is measured twice: plain, and with every operation raced against what a program in it has to use to stop its work. In Go that is a @code{select} against @code{ctx.Done()}; in Hopac it is an @code{IVar} (Hopac's write-once variable) that is filled to stop the jobs. Bjolang is measured twice as well: with events and @code{sync}, and with @code{chan-put} and @code{chan-get}, so the difference between the two Bjolang columns is what building an event and syncing it costs. The best of five runs, in nanoseconds per operation, on a Ryzen 9 5900X under Linux with .NET 10 and Go 1.25:
In the choose row only the sender can use @code{chan-put}, since the receiving side has to use channel-receive (and thus allocates). On the ring, syncing an event costs nothing measurable any more: a send or a receive synced alone does not build an event. For some reason, this ring inside a nested scope is faster in bjolang, and I haven't figured out why yet.
Against Go, Bjolang is ahead on the ring and the choose, and further ahead once Go has to watch @code{ctx.Done()}, which doubles its ring. Go spawns a little faster. Against Hopac it is the other way round without cancellation: Hopac's ring and choose take about 30% less time than Bjolang's, and its spawns about a third. Bjolang's numbers include watching the scope's token at every operation, which Hopac's plain rows do not do. Once cancellation is added, the ring turns. A thousand jobs racing one shared @code{IVar} all take that IVar's lock at every operation, and the ring slows to three and a half microseconds a message. Giving each job an IVar of its own brings it back to 195 ns, at the price of keeping track of a thousand IVars and filling all of them to stop the ring. The choose does not turn: with a cancel IVar Hopac's takes 136 ns, against Bjolang's 181.
The spawn row is about twice what it was with the scheduler Bjolang used before, which handed every fiber to the .NET thread pool, and the choose row about one and a half times. With it (@code{BJO_SCHEDULER=pool} in the environment, kept as a way back) a spawn costs 105 ns and the choose 113. The spawn row is a single fiber spawning a million fibers that do nothing, and it is slow only on many cores: on two or four cores the current scheduler spawns faster than the pool, 61 and 72 ns against 87 and 112. The scheduler that replaced it runs fibers from run queues of its own, in the shape of Go's, because the pool let work that had waited longest wait longer still under load. That is worth more in a loaded service, as the service benchmark below shows, than these two rows on an idle machine.
These are small benchmarks on one machine, and they only measure the cost of the concurrency primitives, not of the work between rendezvous. They do not show what each operation allocates either. A Go channel operation never allocates and a goroutine costs 16 bytes. A Bjolang send or receive synced alone allocates nothing, a @code{choose} 56 bytes and a spawn 136. Hopac allocates 208 bytes a message on the ring, 424 a choose and 184 a spawn. Go is ahead in other ways too: any Go function can block without being written as a bjoroutine and the Go scheduler preempts a goroutine that computes for too long, where a fiber has to be moved off the pool with @code{spawn/thread}.
What Bjolang offers that Go does not is in the CML vocabulary rather than in the numbers. A Go @code{select} is a statement, its cases are fixed in the source, and a channel operation cannot be passed to a function, wrapped, or combined with others. In Bjolang an event is a value, which is what made @code{within} above a two-line function that works on any event, and @code{choose}, @code{wrap}, @code{guard} and @code{with-nack} build new events out of old ones. Go has no structured concurrency either. Waiting for goroutines, stopping them, and reporting their failures are a @code{sync.WaitGroup}, a @code{context.Context} passed to every function and checked at every channel operation, and an @code{errgroup}, written by hand each time. A goroutine that nothing waits for leaks without a sound. In Bjolang a scope does all of that by construction: it waits for its fibers, raises their failures when it ends, releases the files they opened, and cancels them through a token that every @code{sync} watches without being asked. Hopac can run jobs in parallel and wait for all of them, but a job cannot be stopped from the outside, so stopping jobs when one of them fails, or when their owner is done with them, is the same hand-written race against an IVar as in the table.
The rows above time one primitive at a time on an idle machine. @code{bench/service} is closer to a program: a sharded key-value store in all three languages, with 16 shard fibers that each serve a part of the keys, and clients that send gets, puts and four-key multi-gets, each with a fresh reply channel and a one-second deadline raced against the reply. The three programs do the same work and agree on a checksum. @code{bench/service/drive.py} runs them and takes the median of 15 reps.
The two .NET programs run with the server GC, which keeps a few hundred megabytes and collects a few times a rep. Go's collector keeps the heap small and collects about twenty times as often, so Go runs twice: with its default, and with as much memory as the .NET programs took (@code{GOGC=off} and @code{GOMEMLIMIT} at their peak).
In a closed loop, 1000 clients send 1000 requests each, each one as soon as the last is answered, so the machine is always saturated. Wall time, CPU time, p99.9 latency and peak memory:
@read-table{
| Program | all cores: wall | CPU | p99.9 | memory | 4 cores: wall | CPU | p99.9 |
On all cores Bjolang and Go with the same memory finish together, and Go uses a fifth less CPU and has the shorter tail. On four cores Hopac is clearly the fastest, and Bjolang and Go are close.
In an open loop, 10000 clients send on a fixed schedule at a given rate, so the three are compared at the same load. Latency is taken from when a request was due, so a client that woke late, waited for a core, or slept through a GC pause counts that time too. p99 and p99.9 in milliseconds:
A client that is early sleeps until its request is due, in whole milliseconds rounded up, the same in all three programs, so every latency here has up to a millisecond of that in it, and below about 2 ms the numbers say little. At 250k/s on four cores the three are close, and Go with the same memory has the shortest tail. At every higher load Bjolang has the shortest, and at a million requests a second its tail is a quarter of Hopac's or less, and a fifth of Go's or less. Some rows move from run to run: over three runs at 500k/s on all cores, Go with the same memory had a p99 of 2.1 to 8.0 ms and Bjolang of 3.6 to 5.0, so that row is not settled. Go's tail at 500k/s on four cores is long with either memory, and its time once a request has started is as long, so the delay is in running the requests, not before them. Hopac keeps all 24 cores busy on all cores at either rate, and why has not been looked into.
The tables in @code{BjolangRuntime/Cml/design.md} in the main repository have every column, including the time a request took once its client had started it.
Our deck is shuffled using @code{shuffle-vec}, which is non-deterministic. While randomness is ideal during gameplay, it makes automated tests brittle when verifying deterministic outcomes like trick winners. The traditional object-oriented or functional workaround is dependency injection—passing an explicit shuffle function down through every layer of the call stack.
Bjolang provides an alternative mechanism: *tail-resumptive algebraic effects*. With effects, a function simply declares and performs an abstract operation, allowing an enclosing caller higher up the call stack to dynamically decide how that operation is handled.
@code{shuffle-deck} is exposed as a regular function matching the declared signature. When called without an active handler in the dynamic scope, it falls back to the default implementation—in this case, the real shuffle. In a test, you can override this behavior using @code{with-handler}:
The handler intercepts operations executed anywhere within the dynamic extent of the @code{with-handler} block, including inside deeply nested helper calls and child fibers, without requiring @code{deal-hand} to pass any parameters around.
- Default implementations: If an effect omits @code{#:default} and is performed without an active handler, the runtime raises an exception. Omit defaults for operations that always require explicit environmental context.
- One-shot execution: Handlers in Bjolang behave like standard functions and return a single value. Full multi-shot delimited continuations are not supported: a handler cannot resume its caller multiple times or abort resumption except by throwing an exception.
- Effect coloring: Operations declared with @code{->} can be invoked from any standard function. Operations declared with @code{-bjo->} may only be invoked from within bjoroutines, and their handlers must likewise be bjoroutines.
- Handler composition: Invoking an operation from inside its own handler invokes that handler recursively. To delegate to an enclosing outer handler, retrieve it using @code{(handler-of op)}:
Several core runtime facilities in the prelude are implemented as effects out of the box, allowing you to intercept or virtualize them in tests without modifying application code:
Because all prelude file I/O routes through the built-in @code{FS} effect operations, tests can replace the entire filesystem with an in-memory mock using @code{with-fake-fs} (covered in the Testing section below).
Keep in mind that @code{with-handler} only establishes a dynamic binding; unlike a concurrency scope, it does not await background fibers. If code inside the handler spawns asynchronous fibers that emit logs, ensure the concurrency scope enclosing those fibers is placed *inside* the @code{with-handler} block so the handler remains active until all fibers finish writing.
Because Bjolang compiles directly to C#, you have full, seamless access to the entire .NET ecosystem. By declaring the CLR classes, methods, and properties you need along with their signatures, the compiler exposes them as ordinary Bjolang functions and types.
Each entry in @code{import/extern} specifies a local Bjolang identifier, the fully qualified CLR member, and its signature. For overloaded methods (such as @code{Console.Write}), the compiler determines which overload to bind based on the provided signature. Instance methods accept the target object as their first parameter. Properties and fields can be accessed via @code{#:get} and modified with @code{#:set}:
.NET's own string, @code{System.String}, is UTF-16 and a different type from @code{string}. A .NET method that takes or returns one, or an array of them, is converted where it is called, so @code{trim} above takes and gives a Bjolang string; a @code{null} from .NET arrives as @code{""}. A .NET string inside another type, such as a @code{(Seq System.String)}, is not converted: @code{string->clr-string} and @code{clr-string->string} do it by hand.
Avoid binding an external method to the name of an existing prelude function. Locally bound names and prelude symbols take precedence over @code{import/extern} declarations, which means an alias named @code{println} would simply be shadowed and ignored.
The @code{import/class} form maps a .NET class to a Bjolang type, exposing its constructor as a function suffixed with a dot (e.g. @code{StringBuilder.}). CLR enums map similarly, with individual members accessed using dot notation like @code{Type.Member} (seen with @code{ConsoleColor.Red} above).
A method that @code{string} does not have, like @code{ToUpper}, is called on the string converted to .NET's, so its indices and lengths are .NET's UTF-16 ones and not @code{string-length}'s bytes.
As shown in the example above, you do not need to declare every method up front with @code{import/extern}. Once an object's type is known to the type checker, you can invoke any public method using @code{(.Method obj args ...)} and inspect properties using @code{(.-Property obj)}. Because @code{Append} returns the builder itself and Bjolang enforces explicit handling of return values, discard unused returns intentionally using @code{ignore}.
- Generic @code{try} blocks: Wrap arbitrary code in @code{try}, as introduced in Part 1.
- Exception-mapped constructors: Annotating a constructor with @code{#:exceptions} in @code{import/class} causes the constructor to return a @code{Result} instead of throwing on specified exceptions.
- Output parameter conversion: Common .NET methods adhering to the @code{bool TryParse(..., out T result)} pattern automatically transform into Bjolang functions that return an @code{(Option T)}.
A .NET method returning a @code{Task} or @code{Task<T>} can be imported using the @code{#:async} modifier. Calling an async method from within a bjoroutine suspends the fiber until the task completes without blocking an OS thread, with no explicit @code{await} required:
Conversely, CLR methods known to perform synchronous, thread-blocking I/O should be tagged with @code{#:blocking}. Calling a blocking method from within a bjoroutine triggers a compiler warning; you can resolve this warning by offloading the call to a dedicated thread pool thread via @code{(sync (blocking (fun () ...)))}.
At the boundary between systems, Bjolang types map cleanly to standard .NET equivalents: a @code{Func<A, B>} corresponds to @code{(-> %a %b)}, @code{IEnumerable<T>} maps to @code{(Seq %a)}, C# tuples map to @code{Tuple}, and native arrays @code{T[]} correspond to @code{(Array %a)}. Note that constructing generic .NET classes dynamically and pattern matching on .NET objects via constructor patterns are not currently supported.
Bjolang programs execute using the invariant culture by default. For example, floating-point numbers format with a decimal point (@code{2.5}), avoiding regional discrepancies on machines where local culture settings would otherwise produce @code{2,5}. If your application requires localized formatting, you can configure @code{CultureInfo.CurrentCulture} explicitly.
Every Bjolang module compiles to a public static class named @code{<module>_Module}, exposing top-level functions defined with @code{defun} as static methods and @code{def} values as static fields. Identifiers with characters invalid in C# are automatically mangled (for example, hyphenated names like @code{full-deck} become @code{fullsubdeck}), while collections remain Bjolang's persistent data structures. You can inspect the emitted C# source code directly by passing the debug flag: @code{bjo build -d}, which generates @code{out.cs}.
Macros are functions executed by the compiler during the expansion phase. A macro receives an unevaluated syntax tree as input data and returns a new syntax tree that replaces the invocation form. In fact, many constructs in Bjolang that appear to be built-in language keywords—such as @code{cond}, @code{when}, @code{type/derive}, and @code{with-test}—are actually implemented as standard library macros.
Macros are defined using @code{def/macro}. A macro transformer accepts three parameters: @code{form}, @code{inject}, and @code{compare}. Most macros are written using pattern matching via @code{syntax-match} from the @code{(std syntax-match)} library:
The @code{syntax-match} form decomposes syntax by structural shape. A wild-card pattern @code{_} matches the macro name itself, variable names bind matched syntax fragments, and ellipsis @code{...} matches repeated sequences. Templates are constructed using syntax quotation @code{#'(...)}, where comma @code{,x} unquotes an individual syntax value and @code|{,@xs}| splices a list of syntax objects into the form. Malformed invocations can be rejected at compile time using @code{syntax-error}, which highlights the offending source location.
Pattern fragments like @code{((name value) ...)} match lists of pairs and bind @code{name} and @code{value} to lists of syntax elements. Quoted symbols such as @code{'else} match that literal identifier.
Because macro transformers are compiled code executed directly by the compiler while reading subsequent source files, a macro cannot be used in the same module where it is defined:
For our card game, we place macro definitions in a dedicated module, @code{cardmacros.bjo}. Importing that module with @code{(import "cardmacros.bjo")} automatically makes its macros available. Unlike functions, macros do not require an explicit export list—every macro defined in a module is exported automatically.
The input passed to a macro is a value of type @code{Syntax}, a union type with variants like @code{(SSym s)} for symbols, @code{(SInt text)} for numbers, and @code{(SList items)} for parenthesized forms. Because macros are ordinary Bjolang functions, you can invoke helper functions, inspect @code{Syntax} records directly, and use recursion.
Bjolang macros are hygienic by default. Identifiers introduced within a syntax template are automatically renamed to ensure they never collide with local bindings in the calling scope:
Here, the template's temporary variable @code{tmp} and the caller's outer variable @code{tmp} are treated as two distinct symbols, allowing the expression to evaluate correctly to @code{9}. Furthermore, free identifiers referenced in the template (such as @code{let}, @code{if}, and @code{=}) always resolve to their prelude definitions, even if the calling scope has shadowed those names locally.
If a macro intentionally needs to introduce an unhygienic binding into the caller's environment, it can explicitly opt out of renaming using @code{(inject 'name)}:
A macro returns a single syntax form. If you need a macro to expand into multiple definitions or expressions at the top level, return a @code{(begin ...)} form, whose child elements are spliced directly into the enclosing scope:
In this expansion, the internal counter variable @code{count} is generated hygienically and cannot collide with any @code{count} variable in the importing module, while @code{next-id} takes on the identifier provided by the caller.
The @code{def/pattern} form defines macros intended for use within pattern-matching positions rather than ordinary expressions. Nullary pattern macros can be referenced simply by their bare capitalized name:
Pattern macros expand before the compiler performs exhaustiveness checking, meaning the compiler inspects the expanded @code{or} pattern just as if it had been written by hand.
Literal prefixes like @code{#map(...)} are reader-level hash macros. You can register custom hash literals using @code{def/hash-extend}. For example, we can implement a custom literal syntax for playing cards, converting ranks and suits into their typed constructors:
Notice the @code{re-export} in @code{cardmacros.bjo}: because the macro expansion emits constructor symbols like @code{Card} and @code{Queen}, those types must be resolvable in the module invoking the macro. Re-exporting them ensures that importing @code{cardmacros.bjo} alone brings the required types into scope. Importing @code{deck.bjo} alongside it remains recommended so that the corresponding @code{->str} implementations are also present.
Many embedded domain-specific languages do not need macros. A grammar says how a quoted literal of a type is written, and declares the type. Where the type checker expects the type, a quoted literal becomes its constructors:
@code{Action} is an ordinary union, and @code{match} takes its cases. The type checker reads the literal, so a misspelled keyword or a wrong number of items is a compile error that says what can come at that place:
A grammar reads a literal only where the type checker already expects one of its types, which is why @code{script} and @code{more} have signatures. A grammar can also have several productions, items in any order with @code{:each}, optional items, repeats with @code{...} and @code{...+}, parameters, and atoms that a function checks when the literal is compiled. @code{(std run)}, @code{(std fmt)} and @code{(std url)} use grammars for pipelines, layouts and URLs. The language reference describes every form: see @code{Docs/Syntax.org} and @code{Docs/Grammar-design.org} in the Bjolang repository.
A macro often builds a value out of constants, and that value is then built again every time the expansion is evaluated. When that is expensive, wrap it in @code{please-hoist-i-promise-i-am-not-naughty}: the form is evaluated the first time it is reached, and every later evaluation gets that same value. This is what @code{#rx(...)} does, so a regex is compiled once rather than once per match.
@codeblock[#:lang "bjolang"]|{
;; (tag-set "a" "b" ...) is a HashSet of the strings, built once.
(def/macro (tag-set form inject compare)
(syntax-match form
((_ tag ...)
#'(please-hoist-i-promise-i-am-not-naughty
(list->hashset (list ,@tag))))))
}|
If computing the value throws, nothing is kept, and the next evaluation tries again. The same form written in two places is one value.
The long name is a promise about what the compiler cannot check: that the form has no side effects worth repeating, and that it does not depend on things that change from call to call, such as a parameter or an effect handler. Everything it can check, it does, and a form that breaks a rule is a compile error that lists every reason. The form cannot:
@itemlist{
@item{refer to a local variable or a function parameter;}
@item{read a @code{def} of the module it is in, or any @code{def/mutable};}
@item{use a type variable of the function around it;}
@item{call a bjoroutine, @code{yield} to a sequence, or leave a @code{with-return};}
@item{be void, or produce something that can be changed in place, such as an array, a box or a mutable collection.}
}
It may use literals, functions, and the defs of other modules. A macro that puts its caller's code inside the form passes these rules on to its caller, so keep the form to parts the macro built itself.
@subsubsection{One expansion for both colours: by-colour}
A macro does not know if its expansion will land in a bjoroutine or in ordinary code. When the two need different code, the expansion writes @code{(by-colour (#:sync body ...) (#:bjo body ...))}, and the compiler keeps the body for the colour of the code the expansion is in. This is how @code{with-scope} works in both colours: it ends the scope with a suspending wait in a bjoroutine and with a parking wait in ordinary code.
@codeblock[#:lang "bjolang"]|{
;; (await ev) syncs on ev, in code of either colour.
(def/macro (await form inject compare)
(syntax-match form
((_ ev) #'(by-colour (#:sync (sync/blocking ,ev))
(#:bjo (sync ,ev))))))
}|
The colour is the colour of the innermost of these:
| a @code{defun} inside a body | the body around it |
}
A @code{defun} with a @code{by-colour} in it gets a suspending copy, as a @code{defun} that calls @code{file-read-text} does, and a bjoroutine calls that copy. If the @code{#:bjo} body never suspends, the copy is dropped, and the @code{#:sync} body runs everywhere. So write a @code{#:bjo} body that suspends and a @code{#:sync} body that does the same work without suspending.
A lambda applied where it is written, @code{((fun () body))}, is replaced by its body before colours are found. A @code{by-colour} in it has the colour of the code around it.
- Macros cannot be invoked within the module that defines them; they must be imported from a separate module.
- Macros can only appear in expression or definition positions (or in pattern positions when defined with @code{def/pattern}). They cannot be used as loop clauses, let bindings, or type annotations.
- Macros cannot override built-in special forms; a macro named @code{if} will never shadow the compiler primitive.
- Macros cannot inspect types, as macro expansion occurs prior to type checking.
- Macros are not first-class values and cannot be passed as function arguments.
- Macros cannot inject imports into the calling module; any imported dependencies must be declared explicitly in the consuming file.
Bjolang includes a lightweight test harness in @code{(std simpletest)}. Its core assertions are straightforward: @code{(expect label actual wanted)} checks equality between values, @code{(expect-true label ok)} asserts a boolean condition, and @code{(fail label)} explicitly flags an unreachable branch. Passing checks output @code{ok: label}, while test failures emit descriptive @code{FAILURE:} diagnostics:
Whenever an assertion fails, the output reports both the computed value and the expected value: @code{FAILURE: deliberately wrong gave 2 but wanted 3}.
For asynchronous or resource-intensive testing, @code{(with-test "name" body ...)} executes the test body within an isolated structured concurrency scope configured with a default deadline. Any background fibers spawned during the test are automatically awaited, and the test fails if unclosed file ports remain open when the scope exits. Because @code{with-test} forms a concurrency scope, it must be invoked from within a bjoroutine.
You can also mock filesystem interactions using @code{(with-fake-fs ((path contents) ...) body ...)}, which redirects file operations to an in-memory map for the duration of the enclosed block. Here is how we can verify our earlier high-score tracking functions without touching real files on disk:
Because @code{with-fake-fs} is implemented via algebraic effects, test modules using it must also import @code{(std effect)}. In standard project layouts, test files live in @code{tests/} and can be executed individually using @code{bjo run tests/<test-name>.bjo}.
You can launch an interactive Read-Eval-Print Loop by running @code{bjo repl}. When invoked inside a project directory, the REPL automatically discovers and allows importing any of the project's local modules. At a terminal the REPL edits its own lines and keeps a history across sessions. Pressing Enter while a bracket is still open starts a new line, so a long form can be written and edited over several lines. Submitting an expression evaluates it and prints the result, while entering definitions compiles them and prints the bound names:
Unlike many Lisp REPLs, Bjolang's REPL does not interpret forms. Instead, each entered interaction is compiled on the fly into an in-memory assembly using the exact same compiler pipeline applied to files on disk, ensuring identical semantics between interactive sessions and compiled production builds.
Top-level functions declared in the REPL require explicit type signatures, mirroring the rules for modules. You can supply parameter and return types inline directly on the @code{defun}, or submit a standalone signature on its own line immediately preceding the definition:
Redefining a symbol shadows the previous definition for subsequent interactions; however, any existing code already compiled against the earlier version continues to invoke the original binding:
The same incremental compilation model applies to trait implementations (@code{impl}): an implementation entered at the prompt is visible to subsequent entries, but cannot retroactively modify calls compiled earlier.
@code{:show name} prints the documentation of a name you can use at the prompt: what it is, where it comes from, how it is called, and the text its author wrote in a @code{(:doc ...)} form. It then lists the other usable names containing the text, in any case, with their modules and one-line summaries. Only names that an entry typed now could use are considered, that is builtins, names your imports bring in, and names defined at the prompt. So @code{:show random-bool} prints nothing until @code{(std random)} is imported:
returns One random bit. They are drawn 64 at a time.
See also: random-chance?
bjo> :show random-u
random-u32 (std random) 32 random bits.
random-u64 (std random) 64 random bits.
}|
Import modifiers apply as they would in an entry: after @code{(import (prefix (std random) "r/"))} the name to ask about is @code{r/random-bool}, and @code{:show} says that it is @code{random-bool} from @code{(std random)}. A name with no documentation is shown with where it comes from and "with no doc".
Tab completes the name at the cursor from the same names, together with the special forms and the names the entry you are typing has already bound. When only one name fits it is inserted, and when several do, the part they share is inserted and pressing Tab again opens a menu showing each name's signature and summary. Inside an import, Tab completes module paths instead: @code{(import (std ra} becomes @code{(import (std random}.
The same completions are available as a command, @code{:complete text}, which prints the names that could be written at the end of @code{text}, one per line followed by where each comes from. It is meant for editors rather than people: an editor can send the entry up to the cursor and offer what comes back.
Sooner or later a program is fast enough everywhere except one place, and the question becomes how fast that place is, and whether a change made it better. Timing it with a stopwatch gives a number, but not one you can trust: the first calls run code the JIT has not optimized yet, a garbage collection can land in the middle of a measurement, and two runs of the same code on a busy laptop differ by more than most improvements do.
@code{(std bench)} measures in a way that holds up against all of that, and says how sure it is. There are three ways in:
@read-list{
- @code{time-it}, for a quick answer at the REPL: one line, measured properly.
- Benchmark files, with @code{def/bench} and @code{(run-benches)}, for the code you care about: a table, a menu, saved runs to compare with later.
- @code{bjo bench FILE --against REF}, for the question "did my change make this slower": the file run against itself at another commit, the two versions measured in turns.
}
All three are measured the same way, which the section "How a benchmark is measured" below describes.
@subsubsection{time-it}
@code{time-it} measures a body where you write it and prints one line. It is in @code{(std bench)}, and @code{(std stopwatch)} has it too:
The line gives the median time per call, its spread (the median absolute deviation, as a percentage of the median), the fastest sample, how much the body allocated per call, and how many samples it took. The body is warmed up, calibrated and sampled just as a benchmark is, with shorter time targets, so it is done in a second or two.
A body that takes more than a second the first time is not run again. That first run is the measurement, and the line says so. Here the body is a slow recursive Fibonacci:
@codeblock[#:lang "text"]|{
(fib 44): 1.46 s (one cold run: a body over a second is run once)
}|
@code{#:times n} and @code{#:warm-up-rounds w} ask for an exact number of runs instead: @code{w} untimed runs, then @code{n} timed ones, with no calibration. Use them when the body has an effect that has to happen a known number of times:
The median here is more than three times the first one: without a warm-up, the early runs are the JIT's unoptimized code. That is the reason to let @code{time-it} decide how often to run a body.
@subsubsection{A benchmark file}
For code you want to keep measuring, write a benchmark file. Benchmarks belong in @code{benches/} in a project, next to @code{src/} and @code{tests/}. Here is one for the deck from @code{cardlib}, in @code{benches/deck.bjo}:
It is laid out just like a test file. @code{def/bench} registers a benchmark where it is written, and @code{(run-benches)} at the end becomes the file's @code{main}. The body of a benchmark is ordinary forms, like a function body, and the names its options bind are in scope in it. @code{bjo bench} builds every file under @code{benches/} and runs it:
deal with vec-slice 19.4 ns 0.2% 19.3 ns 152 B 0.02/0/0 3 outliers
deal with a loop 51.7 ns 0.7% 51.2 ns 656 B 0.08/0/0 4 outliers
a million hands 320 ms - 320 ms 1.62 GB 208/0/0
(324 ms, 320 ms, 320 ms)
deal from a stacked deck skipped: the cheat is not written yet
}|
The first two lines say what the numbers were measured on, because a time is a fact about one machine. A Debug build, an attached debugger and a CPU frequency governor other than @code{performance} add a warning there. @code{bjo bench} always builds in Release, whatever @code{-d} says, since a Debug build measures code the JIT did not optimize.
| @code{name}, @code{size} | the benchmark, and a column for each of its parameters |
| @code{median} | the median time per call. Not the mean: one collection can move a mean anywhere |
| @code{±} | the median absolute deviation, as a percentage of the median: how much the samples agree |
| @code{min} | the fastest sample, which is what a quiet machine converges to |
| @code{alloc/op} | the bytes allocated per call |
| @code{gc/1k 0/1/2} | the collections of each generation per thousand calls (per call for a @code{#:once} benchmark) |
}
After the columns come the marks. @code{noisy} means a spread above 3%, @code{unsteady} means the warm-up ran out of time before the times settled, and outliers are samples far outside the rest. They are counted and shown, never dropped: a pause that happens once in fifty calls is part of what the call costs.
Each row is printed as soon as its benchmark is measured, so a long run can be read as it goes. At a terminal, a line below the table says where the run is: which benchmark, which phase, the median so far and an estimate of the time left. It is rewritten in place, and is left out when the output goes to a pipe or a file, under @code{CI}, and in an Emacs buffer.
@subsubsection{How a benchmark is measured}
You do not have to know this to use @code{(std bench)}, but it explains its numbers.
@read-list{
- @strong{The body is kept.} The value of the body's last form is written to a sink the JIT cannot remove. A body that computes a value and throws it away is therefore not optimized away, and the sink allocates nothing.
- @strong{Calibration.} The body is run in batches of @code{k} calls, and @code{k} grows until a batch takes 10 ms. One sample is one batch, so the clock's resolution does not matter even for a body of a few nanoseconds.
- @strong{Warm-up.} Batches are run until the medians of the last two windows of five batches agree within 2%, for at least half a second and at most one. The minimum is there because of the JIT. .NET recompiles a hot method into faster code only after a pause with no new compilation, and a body can look steady at the first, slow tier before the faster code arrives. A warm-up that runs out of time leaves the row marked @code{unsteady}.
- @strong{Samples.} Before each sample a collection runs, so that one benchmark's garbage is not collected in another one's time. At least 20 samples are taken, for at least 2 s. While the median's 95% interval is wider than ±1%, it takes more, up to 100 samples or three times the time. A noisy benchmark costs more time, and gets a more precise median for it.
}
Every benchmark of a file runs in the file's one process, one after another, so what one benchmark compiled is there for the next. That is usually what you want: it is how the code runs in a program.
@subsubsection{Setup, parameters, and the rest}
The body of a benchmark should be the thing you want to measure and nothing else. The options of @code{def/bench} are about keeping everything else out of the time:
@code{#:setup} is the one to reach for first. Its bindings are written like a @code{let*}'s, so a binding can use the ones before it, and the body uses them by name. In the example, @code{shuffle} measures shuffling a deck, not making one: @code{deck} is made once, before the warm-up, and every call shuffles that same deck.
@code{#:setup-every} is for a body that changes or uses up what it is given, such as a sort that sorts in place, or a body that consumes a list. Its bindings are made again before every call, so each call gets a fresh input, and each call is then timed on its own, with its setup outside the time and outside the allocation. Timing calls one at a time has a cost, because the clock is read twice per call. A body that takes less than a microsecond is marked, since its time is close to the clock's resolution.
The two can be used together. What @code{#:setup} binds is made once, and @code{#:setup-every} can use it to make each call's input. That is the way to keep an expensive input out of the time and still give every call its own copy:
@codeblock[#:lang "bjolang"]|{
(def/bench "sort a big shuffled deck"
#:setup ((cards (shuffle (repeat-deck 1000)))) ; 52 000 cards, made once
#:setup-every ((copy (vec->list cards))) ; a fresh list for each call
(list-sort copy))
}|
@code{#:params} gives the parameters' names and values, like a @code{let}. The values are a vec of any expressions, evaluated once. There is a row for every combination, in the order written, with the last parameter varying fastest. The setups and the body use the parameters by name:
Each combination is a benchmark of its own for everything else: it is warmed up, calibrated, saved and compared on its own. To try a size the file does not list, give it on the command line: @code{--params n=500,5000} replaces the values of @code{n} for that run. A value is read as the parameter's values are typed: an @code{int}, a @code{double}, a string, a symbol or a @code{bool}. Implementing the trait @code{BenchParam} lets other types be read too.
@code{#:once n} is for bodies too slow to run in batches. There is no calibration and no warm-up, just @code{n} runs timed one at a time, the first of them cold. Each time is shown under the row, as for @code{a million hands} above, and the median is the row's.
@subsubsection{Choosing what runs}
Names on the command line choose benchmarks. A benchmark runs when its name contains one of them, and a parameter written the way the table shows it chooses rows:
@codeblock[#:lang "text"]|{
bjo bench benches/deck.bjo shuffle # shuffle, and sort a shuffled deck
bjo bench benches/deck.bjo "hand points" size=52
bjo bench --list # the names, and nothing run
}|
Note that @code{shuffle} also matches @code{sort a shuffled deck}: a name is a part of a name.
With no names, at a terminal, a benchmark file asks what to run:
@codeblock[#:lang "text"]|{
$ bjo bench
Benchmarks in deck.bjo:
#0 full-deck
#1 shuffle
#2 hand points size: 5 13 52
#3 sort a shuffled deck
dealing
#4 deal with vec-slice
#5 deal with a loop
#6 a million hands
#7 deal from a stacked deck
Choose by number (0 2-4), by name (sort), with parameters (2 n=1000),
or * for all. Nothing quits, or repeats the last choice once there is one.
bench> 2 size=5,52
}|
A choice is numbers, ranges of numbers (@code{2-4}), parts of names, or @code{*} for everything, in any mix and separated by spaces. A parameter after a benchmark narrows that benchmark to some of its values: @code{2 size=5,52} runs @code{hand points} at sizes 5 and 52 only. The groups and the parameters' values are shown, so a choice can be made without opening the file.
After a run the menu asks again. Pressing Enter repeats the last choice, so you can change the code in another terminal and run the same benchmarks again with one key. @code{q} or the end of the input leaves.
With several files and no names, @code{bjo bench} makes one menu of all of them, numbered across the files under a heading for each, and runs each file with the part of the choice that is its own. Not at a terminal, when the output is redirected or @code{CI} is set, there is no menu: a run with no names runs everything, as @code{--all} does.
@subsubsection{More numbers}
@code{--detail} adds three lines under each row: the mean, the standard deviation and the quartiles; the samples in the order they were taken; and a histogram of them:
hand points 5 6.50 ns 6.6% 5.21 ns 0 B - noisy, 10 outliers
mean 6.69 ns, sd 1.02 ns, q1 6.04 ns, q3 6.90 ns, 100 samples of 1600000 calls
in order ██████▄▂▃▃▃▃▃▂▂▃▁▃▃▃▃▁▂▂▄▂▂▃▃▃▃▂▁▂▄▄▂▂▂▃▃▃▃▃▃▂▂▅▄▁▂▃▃▁▁▃▃▅▂▃
histogram 5.21 ns ▃▂▅▄▄▆█▇▅▂▃▂▂▂ ▁ ▃▂▂▂ 9.64 ns
}|
The "in order" line is the one that explains a strange row. Here the first six samples were slower than the rest, and the spread of 6.6% and the @code{noisy} mark come from those six; the other 94 agree closely. A second run tells whether it was the machine or the code: run again, the same benchmark had a spread of 1.9%. A line that rises from left to right is a body that slows down as it runs, for example because its heap grows. The histogram shows the shape: two humps are two different speeds, not one noisy speed.
@code{--scaling} adds a column to a benchmark with one numeric parameter: how much the time grew from the row before, against how much the parameter grew.
name size median ± min alloc/op gc/1k 0/1/2 scaling
hand points 5 4.94 ns 5.4% 4.47 ns 0 B - noisy
hand points 13 9.99 ns 2.7% 8.41 ns 0 B - ×2.02 for ×2.60
hand points 52 41.5 ns 0.7% 40.5 ns 0 B - ×4.16 for ×4.00 3 outliers
}|
From 13 to 52 cards the time grew by as much as the hand did, which is what a linear function does. From 5 to 13 it grew less than the hand, because a few nanoseconds of every call are not spent on the cards at all, and with five cards those nanoseconds are a large part of the time. A column that keeps growing faster than the parameter is the sign of something quadratic.
@subsubsection{Which of two ways is faster}
"Is dealing with @code{vec-slice} faster than with a loop, and by how much?" is a question about two benchmarks in the same file. Put them in a @code{#:group}, and run with @code{--compare-within}:
deal with vec-slice 20.0 ns 0.9% 19.0 ns 152 B 0.02/0/0 2 outliers
deal with a loop 47.3 ns 0.6% 46.8 ns 656 B 0.08/0/0 14 outliers
deal from a stacked deck skipped: the cheat is not written yet
bench: within dealing, against deal with vec-slice: 20 rounds in this process, paired (Wilcoxon signed-rank); 1 comparison
bench: seed 11 (--seed repeats the order of the cases)
name deal with vec-slice this change 95% interval p ratio verdict
deal with a loop 19.5 ns 47.0 ns +140.9% +140.0% .. +141.7% <0.001 ×2.41 SLOWER
}|
After the usual table, the benchmarks of each group are measured again in rounds: every round takes one sample of each, in an order shuffled anew each round. Each one is then compared with the first of its group. Taking turns is what makes the comparison fair. If the machine gets slower halfway through, both sides get slower in the same rounds, and the comparison, which pairs the samples of a round, does not move.
The ratio is the number to quote. "A loop is 2.4 times as slow as @code{vec-slice}" is roughly true on the next machine too, where "47 ns" is not. @code{-N 50} takes 50 rounds rather than 20, and @code{--seed 11} repeats the order of an earlier run. The columns of the comparison, and how a verdict is reached, are explained in the next section.
@subsubsection{Saving a run and comparing with it}
@code{--save FILE} keeps a run: every sample of every benchmark, with the facts of the machine it was taken on. @code{--compare FILE} measures again and compares with what was kept. Nothing is saved unless you ask, and nothing is saved anywhere you did not name. Where a baseline lives, and whether it is committed, is up to you.
- @strong{The change} is the ratio of the two medians, as a percentage.
- @strong{The interval} is a bootstrap of that ratio: the samples are resampled 10 000 times, and the middle 95% of the resampled ratios is shown. It makes no assumption about the shape of the timings, which are lopsided and have long tails.
- @strong{The p-value} is the probability of a difference at least this large if nothing had changed, from the Mann–Whitney U test. A run that compares many benchmarks at once will show some differences by chance alone: with 40 benchmarks at a level of 5%, two are expected to look different when nothing changed. So the p-values are corrected for the number of comparisons, by Holm's method, and the corrected value is the one shown and used. @code{--p-raw} shows the uncorrected values beside them.
- @strong{The verdict} is @code{faster} or @code{SLOWER} only when the corrected p is below 0.05 (@code{--alpha}) @em{and} the change is larger than 2% (@code{--threshold}). A change that is real but smaller is @code{no change (< 2%)}: it is there, and it was decided not to matter. Anything else is @code{no change}.
}
The reason for asking for both is that a p-value says how unlikely a difference is, not how large it is: with enough samples, a change of 0.1% has a tiny p.
Now look at the table again. @code{full-deck} is 92% slower, as it should be: that is the change. But @code{shuffle} did not change at all, since its deck is made in its setup, outside the time. Yet it says @code{faster}, by 12%. The rows for @code{hand points} and the sorting moved too, and they never call @code{full-deck} in their time at all.
This is the weakness of comparing with a saved run. The two runs are two processes, and @strong{two processes of the same program can differ by a few percent}, in every benchmark at once. Where the code ends up in memory, which core the process runs on, and what the JIT decided all differ from one process to the next. Within one run the samples agree to a fraction of a percent, so the test is very sure about a difference that is really between the processes. The report says this in its second line. Treat a saved-run comparison as a tool for large changes, like @code{full-deck}'s. For small ones, use @code{--against} with several process pairs, as described next.
A saved run from another machine is still compared, but under a warning above the table, and every verdict is marked @code{(other machine)}: the numbers are shown, but no verdict is to be believed. Differences in the build, the .NET version, the garbage collector's mode or the CPU governor are warned about too.
@code{--fail-slower PCT} makes the run exit with 1 when anything is @code{SLOWER} by more than @code{PCT}, for a CI job that guards a hot path. It does not apply against a run from another machine. A file that cannot be read stops the run with exit code 2 before anything is measured.
@subsubsection{Two versions against each other: @code{--against}}
The real question behind most benchmarking is "did my change make this slower?". @code{--against} answers it in one run. It builds the benchmark file as it is and as it was at a git ref, and measures the two versions in turns:
sort a shuffled deck 4.09 µs 4.14 µs +1.3% -4.5% .. +8.4% 0.50 no change
}|
This is what happens:
@read-list{
- @code{REF} is checked out in a temporary git worktree, which is removed afterwards, also when something fails. It can also be a path: another checkout, or another @code{.bjo} file.
- Both sides are built in Release, by the same compiler. Both programs are started, and @code{bjo bench} drives them over a small protocol on their standard input and output.
- Every benchmark both sides have is warmed up and calibrated on both, with the smaller of the two batch sizes, so that a sample of each is the same number of calls.
- Then come @code{-N} rounds, 20 by default. Each round takes one sample of every benchmark from each side, in an order shuffled from @code{--seed}, which is printed so that a run can be repeated. Which side goes first alternates every round, so that neither side always runs on a cache the other one warmed.
- Each round's two samples are a pair. The p-value is the Wilcoxon signed-rank test on the pairs, and the interval a bootstrap that resamples whole rounds. If the machine slows down during the run, both samples of a round slow down, and the comparison does not move.
}
A benchmark that only one side has is listed, not compared.
Look at the table, though. Rounds cancel a slowdown @em{during} the run, but not the difference between the two processes as a whole, and that difference is still there: @code{hand points} is called @code{SLOWER} by 11% at size 13, and it has not changed. The report says so in its fourth line. This is why there is @code{--processes}:
hand points 5 5.55 ns 5.13 ns -9.6% -18.7% .. +0.6% 0.24 10.1% no change
hand points 13 12.1 ns 11.4 ns -7.6% -20.8% .. +7.7% 0.73 14.7% no change
hand points 52 45.2 ns 45.0 ns -3.1% -10.5% .. +4.9% 0.73 7.6% no change
sort a shuffled deck 3.91 µs 4.04 µs +3.6% +0.5% .. +6.7% 0.15 2.8% no change
}|
With @code{--processes 6}, the rounds are spread over six pairs of processes. Each pair gives one ratio of medians, and the test is a t-test on the six ratios, so a difference between processes is now part of what is tested rather than hidden from it. The "between processes" column says how large it was. For these tiny benchmarks it was 3 to 15%, more than any of the "changes" the single pair reported. Now only @code{full-deck} gets a verdict, which is the right answer.
| is it about this fast | @code{time-it}, or a benchmark file |
| which of two ways is faster, and by how much | @code{#:group} and @code{--compare-within}: one process |
| did my change make something a lot slower | @code{--against REF}, or @code{--compare} with a saved run |
| did my change make something a few percent slower | @code{--against REF --processes 6} or more, and more rounds |
}
@code{--against} has a few more options. @code{--against-warmup S} and @code{--against-sample MS} set the warm-up limit and the sample length of both sides. A warm-up much shorter than the default second can mistake the JIT's first tier for steady, and a benchmark that did not get steady is named in a warning. @code{--against-build "CMD"} builds the other side with a command of your own, for a side that is not built the way this one is. @code{--save FILE} keeps both sides' samples: this code's in @code{FILE}, and the other side's beside it, with @code{.against} before the extension. @code{--fail-slower}, @code{--alpha}, @code{--threshold} and @code{--p-raw} work as they do for @code{--compare}.
A CI runner is one machine for the length of one job, which is exactly what @code{--against} needs. A job that guards performance runs @code{bjo bench --against origin/main --processes 6 --fail-slower 5} and fails when something got more than 5% slower.
| @code{--all} | runs every benchmark, without the menu |
| @code{--list} | names the benchmarks and their parameters, and runs nothing |
| @code{--quick}, @code{--long} | halves or quadruples the time targets |
| @code{--min-samples N} | takes at least @code{N} samples (20) |
| @code{--max-time S} | lets a noisy benchmark sample for up to @code{S} seconds |
| @code{--params P=V,...} | replaces a parameter's values for the run |
| @code{--detail} | the mean, the deviation, the quartiles, the samples in order, a histogram |
| @code{--scaling} | how the time grows with a numeric parameter |
| @code{--compare-within} | compares each @code{#:group}'s benchmarks with its first, in rounds |
| @code{-N ROUNDS} | the rounds of @code{--compare-within} and @code{--against} (20) |
| @code{--seed N} | repeats the order of an earlier run's rounds |
| @code{--save FILE} | writes every sample of the run to @code{FILE} |
| @code{--compare FILE} | compares with the run saved in @code{FILE} |
| @code{--alpha P} | the significance level of a verdict (0.05) |
| @code{--threshold PCT} | the smallest change a verdict counts (2%) |
| @code{--p-raw} | shows the uncorrected p-values too |
| @code{--fail-slower PCT} | exits with 1 when something is @code{SLOWER} by more than @code{PCT} |
| @code{--against REF} | runs the file against itself at a git ref, or against a path |
| @code{--processes P} | with @code{--against}: @code{P} pairs of processes (1) |
| @code{--against-warmup S} | with @code{--against}: the warm-up limit of both sides (1 s) |
| @code{--against-sample MS} | with @code{--against}: the sample length of both sides (10 ms) |
| @code{--against-build CMD} | with @code{--against}: builds the other side with @code{CMD} |
}
A percentage can be written @code{2} or @code{2%}. A benchmark file is an ordinary program, so @code{bjo run benches/deck.bjo --quick shuffle} runs it just as well. @code{bjo bench} adds the Release build, the files under @code{benches/}, the menu over several files and @code{--against}.
@subsubsection{Driven by another program}
@code{--against} works because a benchmark program can be driven. Started with @code{--serve}, it reads commands on its standard input and answers each with one line of bjodat on its standard output. Below, @code{>} marks what is sent and @code{<} what comes back:
@code{prepare} runs a benchmark's setup and warms it up and calibrates it, as a run would. @code{sample} takes one sample of @code{k} calls, and @code{:ns} is the whole sample's time. Anything wrong is answered with @code{(error "why")}, and the session goes on. A served program prints nothing else, so its output is the protocol and nothing more. A script that compares two builds made by two different compilers, which is not something @code{bjo bench} does by itself, can drive both of them this way, or use @code{--against PATH --against-build "CMD"}.
Bjolang is compiled into pretty straight-forward c#, but without spans or ref structs. It performs pretty similar to c#, except for some small corners of the language that are different.
@subsubsection{Strings}
Strings in bjolang are utf8, as such they do @em{not} support random access. This means we navigate them differently, using string-cursors, which is explained in the part about strings. Navigating a string using cursors is slower than blasting through integer indices, and if your code needs to be really fast, there are some things you can think about:
@read-list{
- Use the provided search functions and the loop form. They work on raw memory, and can thus work around some of the limits that the safety of string-cursor give.
- use @code{string-cursor-next+ref} instead of next and then ref.
- some of the higher order functions are also faster, depending on use case. Benchmark and see.
- Use slices when copying is the bottleneck: an operation ending with @code{/slice} will return a view into the string instead of copying. This will retain the whole string in memory, so use it wisely.
When parsing strings with a known format, (std scan) will be a lot faster than (std rx) and the good old string-split. Often by an order of magnitude. The only cost is IL code size, since it builds the parser in the code, which will have other costs, such as no inlining.
Most of the time you should use the @code{loop} form. It will be the fastest, and is usually flexible enough to do what you need. The only case is for hot loops (ones that are run often) that are performance sensitive. For those, using the various @code{-fold}, @code{-for-each}, and @code{-iter} functions will be faster for the collections that are tree-based. Those collections are: @code{Vec}, @code{Map}, @code{HashSet}, @code{OrderedMap}, and @code{OrderedMap}. The overhead of loop vs. fold is often something like 3x, with fold being as fast or slightly faster thana @code{foreach} on a C# @code{List<T>}. It does not, however, beat using indexed iteration over the @code{List<T>} due to the @code{foreach} doing some safety checks to detect concurrent mutation.
The absolutely fastest collection to loop over is, unsurprisingly, the Array. In bjolang it should be just as fast as in c# (again, without spans).
The loop form gets turned into a kind of goto soup. For the single level case, it is just as efficient as a c# @code{while} loop, but for really complex cases with many levels and subloops with clauses the JIT in dotnet may struggle. In that case you either live with the clarity and simplicity it gives, or you reach for the glorified gotos of names lets. This is probably not going to be a problem though. I tried to rewrite the worst loop form in bjolang (a 16 line monstrosity with one large multi-level claused subloop and a regular subloop) and there was no difference.
Bjolang has two pairs of collections that do mostly the same job: @code{List} and @code{Vec}, and @code{Map} and @code{OrderedMap}. Which one you pick matters less for small collections than you think, and more for large ones than you would hope. The tables below come from the benchmarks in @code{bench/collections/} in the bjolang repository (run them with @code{./run.sh}). They were measured on one machine with .NET 10 on Linux, so trust the ratios more than the exact times.
| recursive match that binds the rest | 1 ns per item | 30 ns and 230 B per item |
| match the first items, ignore the rest | 2 ns | 2 ns |
}
Use a @code{List} when the work happens at the front and goes item by item: a stack, a result built with @code{cons}, or a recursive function that matches @code{(Cons h t)} and continues with @code{t}. That is what a list does in constant time, and there it is 3 to 30 times faster than a vec.
Use a @code{Vec} for everything else. It knows its length and last item, it reads any index in a couple of nanoseconds, it is faster to iterate, and it uses about a fifth of the memory. Some things to keep in mind:
@read-list{
- Build vecs with an accumulator (@code{vecing} in @code{loop}, or a vec builder), not with @code{vec-add} in a loop. Each @code{vec-add} copies the tail of the vec, which makes a @code{vec-add} loop 3–4 times slower than a @code{cons} loop.
- Do not bind the rest of a vec in a recursive match. @code{[a rest ...]} slices the vec at every step. @code{[a b _ ...]} ignores the rest and does not slice. To walk a vec, use @code{loop}, a fold or an index.
- @code{list-ref}, @code{length} and the last item of a list read the list from the start. 64 random reads in a list of 100,000 items take 3 ms. In a vec they take 240 ns.
| update, insert, remove | 1 | 1.2–1.9 times slower, 2× the allocation |
| build | 1 | up to 2.9 times slower |
| read every entry, 10,000+ entries | 1 | 2–6 times faster |
| smallest key | reads every entry | 2 ns |
| 64 entries in a key range | reads every entry | about 400 ns at any size |
| every key, in order | collect and sort | about 4 times faster |
}
Use a @code{Map} to look up, update, insert and remove by key. It wins at all of those at every size. The difference is largest for string keys, because a map hashes the key once and an ordered map compares it at every step down the tree.
Use an @code{OrderedMap} when the order of the keys matters: the smallest or largest key, the entries between two keys, or the entries in key order. A map can only answer those by reading every entry. An ordered map is also faster to iterate in full, so a program that walks a large map more often than it looks things up in it may prefer one.
A CPU can do one operation on several numbers at once: 4 doubles, 8 ints or 32 bytes in one 256-bit register with AVX2. Bjolang does not do this for you, because the .NET JIT does not turn a loop into vector code. Two modules do it explicitly.
Try @code{(std array)} first. @code{array-sum}, @code{array-dot}, @code{array-min}, @code{array-max}, @code{array-count}, @code{array-index-of}, @code{array-add!} and @code{array-scale!} are each one vector loop in the runtime, with the tail handled for you. On 4 million elements a sum of doubles took 0.8 ms instead of 2.6 ms for a loop, and a sum of ints 0.21 ms instead of 0.88 ms.
For a loop that the kernels do not cover, write it with @code{(std simd)}. A loop over an array has two parts: whole vectors, and the elements after the last whole vector, one at a time:
@codeblock[#:lang "bjolang"]|{
(import (std simd))
(: dot (-> (Array double) (Array double) double))
(defun (dot a b)
(def w (simd256-lanes a)) ; 4 doubles in a Float64x4
(def whole (simd256-whole a)) ; where the tail starts
(def acc
(loop (:for i (up-from 0 #:to whole #:by w))
(:acc v (folding (simd256-zero)
(simd-add v (simd-mul (simd256-load a i) (simd256-load b i)))))
=> v))
(+ (simd-sum acc)
(loop (:for i (up-from whole #:to (array-length a)))
(:acc s (folding 0.0 (+ s (* (array-ref a i) (array-ref b i)))))
=> s)))
}|
A comparison gives a mask instead of a bool: @code{(simd-lt v limit)} has every bit of a lane set where the lane is less. Use @code{simd-select} to choose lanes by a mask, @code{simd-count-true} to count them, and @code{simd-and} and @code{simd-or} to combine masks. A branch in a scalar loop is often a select in a vector loop.
Some things to keep in mind:
@read-list{
- Use more than one accumulator in a sum. An addition waits for the one before it. On 4 million doubles the loop above took 2.9 ms, the same with four @code{Float64x4} accumulators 1.4 ms, and a scalar loop 9.9 ms.
- Put a margin of zeros round a grid. A load at a neighbour's offset then needs no bounds test, and the zeros match nothing. Make each row longer than the grid by a whole vector, so that the last vector of a row does not read the next row.
- A vector loop pays when the work is dense and does not stop early. On Advent of Code 2025 day 4, a neighbour count over a 3000x3000 grid, the vectors were 9 times faster than the same loop one byte at a time. On day 9, a search that usually stops after a few comparisons, they were 1.6 times faster. On a small input, reading the input takes almost all of the time, and vectors change nothing. The @code{-avx} solutions in @code{Examples/aoc} show all three.
- Use 128 bits for code that must run everywhere. A 256-bit vector is emulated on a CPU without AVX2, and on ARM64, and is then much slower than scalar code. @code{simd256-accelerated?} says which you have.
- A Native AOT build targets a baseline x86-64 CPU, without AVX2, unless the manifest says @code{(aot-cpu "native")} or a level such as @code{(aot-cpu "x86-64-v3")}. The executable then does not start on a CPU without those instructions. A program that runs on .NET uses what the CPU has.
- Shifts, conversions between element types and shuffles are not there yet.
