bjoli/BjoManual
SHA256
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
BjoManual/_02_.Writing-programs.sz
Linus Björnstam f8cb9d3459 Benchmarking: (std bench) and bjo bench, with the card game 
A new section 2.9 between Testing and Performance: time-it, a benchmark file
for the deck, how a benchmark is measured, setup and parameters, the menu,
--detail and --scaling, --compare-within, saving and comparing, --against
with --processes, the flags, and --serve. Every output is from a real run.
It shows the weakness of comparing two processes too: unchanged benchmarks
called faster or SLOWER by a few percent, and how --processes 6 settles it.

The standard library chapter gets std/bench, std/stopwatch says time-it is
(std bench)'s, and modules/std/bench/ has the reference pages for (std
bench), (std bench stats) and (std bench store).

🤖 Generated with [ECA](https://eca.dev) (anthropic/claude-opus-4-7)

Co-Authored-By: eca-agent <git@eca.dev>
2026-10-16 19:42:23 +02:00

2093 lines
136 KiB
Text
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

@section{Writing programs}
While the first part focused on the core language constructs in single files, this part covers the wider ecosystem: breaking programs into modules, managing project dependencies, handling I/O, concurrency with bjoroutines, .NET interop, macros, and testing.
Throughout this part we will continue building on the card game from Part 1, gradually turning it into a full project complete with a deck module, a persistent high-score file, concurrent players, and a custom @code{#card(Q hearts)} literal.
@subsection{Modules}
In Bjolang, every @code{.bjo} file is implicitly a module named after its file. There are no namespace declarations or enclosing module blocks; instead, a module simply exports whatever public API it wants other files to see, leaving everything else private by default.
@subsubsection{Splitting the game into files}
Here is the deck from the first part as a module of its own. Put it in @code{deck.bjo}:
@codeblock[#:lang "bjolang"]|{
(import (std random))
(export Suit Rank Card full-deck shuffle card-points)
(type/derive (Eq Ord)
(: Suit (Union Clubs Diamonds Hearts Spades))
(: Rank (Union (: Pip int) Jack Queen King Ace))
(: Card (Struct (: rank Rank) (: suit Suit))))
(impl (->str Suit)
(defun (->str s)
(match s (Clubs "♣") (Diamonds "♦") (Hearts "♥") (Spades "♠"))))
(impl (->str Rank)
(defun (->str r)
(match r
((Pip n) (int->string n))
(Jack "J") (Queen "Q") (King "K") (Ace "A"))))
(impl (->str Card)
(defun (->str c) #"${(record-ref c rank)}${(record-ref c suit)}"))
;; Not exported: only this module needs them.
(def all-suits [Clubs Diamonds Hearts Spades])
(def all-ranks [(Pip 2) (Pip 3) (Pip 4) (Pip 5) (Pip 6) (Pip 7) (Pip 8)
(Pip 9) (Pip 10) Jack Queen King Ace])
(: full-deck (-> (Vec Card)))
(defun (full-deck)
(loop (:for s all-suits)
(:subloop)
(:for r all-ranks)
(:acc deck (vecing (Card (rank r) (suit s))))
=> deck))
(: shuffle (-> (Vec Card) (Vec Card)))
(defun (shuffle deck) (shuffle-vec deck))
(: card-points (-> Card int))
(defun (card-points c)
(match c
((Card (rank (Pip n))) n)
((Card (rank Ace)) 11)
(_ 10)))
}|
And the game, in @code{game.bjo} next to it:
@codeblock[#:lang "bjolang"]|{
(import "deck.bjo")
(defun (main)
(def hand (vec-slice (shuffle (full-deck)) 0 5))
(println #"Your hand: ${hand}")
(println #"Points: ${(vec-fold (fun (c acc) (+ acc (card-points c))) 0 hand)}")
0)
}|
Running @code{bjo run game.bjo} outputs something like @code{Your hand: [3♥ 10♠ 8♥ 10♦ 7♣]}.
Notice how the imports and exports interact here:
@read-list{
- Relative imports: @code{(import "deck.bjo")} specifies a file path relative to the importing file. Bjolang compiles @code{deck.bjo} into @code{deck.dll} ahead of time, automatically rebuilding it whenever it (or any of its dependencies) changes.
- Type exports are complete: Exporting a type exposes all its constructors, union cases, and fields. Because @code{Card} is exported, @code{game.bjo} can freely construct @code{(Card (rank Ace) (suit Spades))} and pattern-match on its fields.
- Trait implementations are automatic: Although @code{->str} was not explicitly listed in the export list, its implementations still work in @code{game.bjo}. Because @code{impl} blocks define anonymous implementations rather than named bindings, they cannot be exported individually; they are automatically carried along whenever the module defining them is loaded.
- Private definitions: Unexported bindings like @code{all-suits} remain private to @code{deck.bjo} and are inaccessible from outside.
}
Every exported definition must have an explicit top-level type signature. The compiler needs this signature to write the metadata into the compiled @code{.dll}:
@codeblock[#:lang "text"]|{
Export Error: Exported item 'helper' is missing a mandatory type signature at bad1.bjo:1
}|
@subsubsection{The standard library, and the prelude}
A list-shaped module path such as @code{(std random)} refers to a standard library module. The compiler resolves these relative to the Bjolang installation rather than the current directory, so @code{(std random)} always points to the same library regardless of where you invoke @code{bjo}. A few notable modules include:
@read-table{
| Module | What it has |
|---------------------------------+------------------------------------------------------|
| @code{(std random)} | random numbers, @code{shuffle-vec} |
| @code{(std hashset)} | sets; @code{(std orderedset)}, @code{(std orderedmap)} |
| @code{(std ports)} | reading a whole port: @code{port->lines} and friends |
| @code{(std fmt)} | text layout |
| @code{(std rx)} | regular expressions |
| @code{(std run)} | running other programs |
| @code{(std effect)} | effect handlers |
| @code{(std simpletest)} | tests |
| @code{(std syntax-match)} | writing macros |
| @code{(text json)} | JSON |
}
By default, @code{(std prelude)} is imported implicitly into every module, providing core functions like @code{println}, @code{list-map}, and the basic operations introduced in Part 1. If you need to hide or override names from the prelude, you can import it explicitly using import modifiers (described below), which replaces the default implicit import:
@codeblock[#:lang "bjolang"]|{
(import (except (std prelude) list-map))
}|
@subsubsection{Exporting types}
Types are private unless listed in @code{export}. Furthermore, the compiler forbids exporting a function whose signature references an unexported private type, since importers would have no way to understand or satisfy that signature. The compiler catches this as an export error in the defining module.
When you want external modules to use a type without inspecting or constructing its internal representation directly, mark the type definition with @code{#:opaque}. For example, here is a card draw pile in @code{pile.bjo} that only permits drawing from the top:
@codeblock[#:lang "bjolang"]|{
(import "deck.bjo")
(export Pile new-pile draw pile-size)
;; Importers can hold a Pile, but only this module can look inside.
(type (: Pile #:opaque (Record (: cards (Vec Card)))))
(: new-pile (-> Pile))
(defun (new-pile) (Pile (cards (shuffle (full-deck)))))
(: pile-size (-> Pile int))
(defun (pile-size p) (vec-length (record-ref p cards)))
;; The top card and the rest of the pile, or None when it is empty.
(: draw (-> Pile (Option (Tuple Card Pile))))
(defun (draw p)
(match (record-ref p cards)
([] None)
([top rest ...] (Some (Tuple top (Pile (cards rest)))))))
}|
With this setup, another module can receive @code{Pile} values, use @code{Pile} in type annotations, and call functions like @code{new-pile} or @code{draw}. However, it cannot construct a pile directly with @code{(Pile ...)}, access its internal @code{cards} field, or pattern-match on its structure:
@codeblock[#:lang "text"]|{
Type Error at game3.bjo:4: 'cards' cannot be read here. pile/Pile is exported #:opaque,
so its representation is visible only to the code of pile. A value of it can be held and
passed on here, and built and taken apart through the functions that module exports.
}|
Even though the type's representation is private, any trait implementations defined in @code{pile.bjo} remain available to consumers. For instance, an @code{(impl (->str Pile) ...)} inside @code{pile.bjo} allows any importing module to convert a @code{Pile} to a string.
@subsubsection{Import modifiers}
Imports can be wrapped in modifiers to filter which definitions are brought into scope or rename them to prevent naming collisions:
@read-table{
| Modifier | Does |
|---------------------------------+-------------------------------------------------------|
| @code{(only m a b ...)} | only these functions and macros |
| @code{(except m a b ...)} | everything but these |
| @code{(prefix m "p/")} | puts @code{p/} in front of every name |
| @code{(postfix m "/p")} | the same, at the end |
| @code{(prefix-defs m "p/")} | a prefix on functions and macros only |
| @code{(prefix-types m "P/")} | a prefix on types, constructors and traits only |
| @code{(rename m (old new) ...)} | renames functions and macros |
}
@codeblock[#:lang "bjolang"]|{
(import "deck.bjo"
(prefix "pile.bjo" "pile/"))
(defun (main)
(match (pile/draw (pile/new-pile))
((Some (Tuple c rest)) (println #"Drew ${c}, ${(pile/pile-size rest)} left"))
(None (println "empty")))
0)
}|
Modifiers can be nested and evaluate from the inside out. For example, @code{(prefix (except (std hashset) hashset-map) "s/")} excludes @code{hashset-map} and prefixes all remaining names with @code{s/}.
Note that @code{only} and @code{except} apply exclusively to functions and macros, not types. A module's exported types are always imported, ensuring that signatures referencing those types remain valid and resolvable:
@codeblock[#:lang "bjolang"]|{
(import (only "deck.bjo" full-deck)
(rename (std random) (shuffle-vec mix)))
(defun (main)
(println (vec-length (mix (full-deck)))) ;; 52
(println (Card (rank Ace) (suit Spades))) ;; A♠, Card came along anyway
0)
}|
Types are strictly scoped to the module that declared them. If two imported libraries each declare their own @code{Card} type, they remain distinct types; the unqualified name @code{Card} simply resolves to whichever import came last. To disambiguate between them, use @code{prefix-types}:
@codeblock[#:lang "bjolang"]|{
(import (prefix-types "poker.bjo" "P/")
(prefix-types "bridge.bjo" "B/"))
(: convert (-> P/Card B/Card))
}|
@subsubsection{Which name wins}
When multiple definitions share the same identifier, name resolution follows three precedence rules:
@read-list{
- Local definitions take precedence over imported names.
- For collisions between two imports, the later import shadows the earlier one. This shadowing behavior is how modules can freely override names from the prelude, which is loaded as the very first import.
- Explicit renames or aliases created by import modifiers or @code{:alias} must be unique. If an alias collides with an existing name, the compiler raises an error to prevent ambiguity.
}
@subsubsection{re-export and :alias}
While @code{export} publishes definitions authored in the current module, @code{re-export} publishes definitions that were imported from elsewhere. This makes it easy to create aggregator modules that bundle several sub-modules behind a single import. The prelude uses this technique to pass through @code{=}, @code{compare}, and @code{list-sort} from @code{(std eq)}. In our card game:
@codeblock[#:lang "bjolang"]|{
;; cards.bjo: one import for everything about cards
(import "deck.bjo" "pile.bjo")
(re-export Card Rank Suit full-deck shuffle Pile new-pile draw)
}|
Re-exporting a type exposes the exact original type, not a distinct copy; a @code{Card} referenced via @code{cards.bjo} is identical to one from @code{deck.bjo}. However, trait implementations (@code{impl}) do not transfer through re-exports. They remain bound to the module where they were defined, meaning any file that wants to format cards with @code{->str} must still import @code{deck.bjo}.
You can provide alternative names for bindings or macros using @code{(:alias new old)}. Combining @code{:alias} with @code{export} or @code{re-export} allows libraries to offer cleaner or more idiomatic public APIs:
@codeblock[#:lang "bjolang"]|{
(:alias deal full-deck)
(export deal)
}|
@subsubsection{include}
Unlike @code{import}, which compiles a separate module, @code{(include "file.bjo")} textually splices the included file's forms directly into the current file at compile time. No separate module boundary is created, no exports are required, and all declarations in the included file share the enclosing module's scope:
@codeblock[#:lang "bjolang"]|{
;; helpers.bjo
(: double (-> int int))
(defun (double x) (* x 2))
}|
@codeblock[#:lang "bjolang"]|{
(include "helpers.bjo")
(defun (main)
(println (double 21)) ;; 42
0)
}|
As a general guideline: use @code{include} when you want to divide a single large module across multiple files, and use @code{import} when the target file represents an independent component with its own encapsulation boundaries.
@subsection{Projects and packages}
Up to this point, our programs have consisted of standalone files importing neighboring modules directly by relative path. While that approach is convenient for quick scripts and small experiments, any software that pulls in external dependencies or is meant to be consumed by other developers should be structured as a project.
A project is simply a directory containing a @code{manifest.bjodat} file. When operating inside a project directory, @code{bjo} understands the package's identity, resolves dependencies, and manages builds automatically. Outside of a project, the CLI falls back to single-file execution just as before.
@subsubsection{Making a project}
@codeblock[#:lang "text"]|{
mkdir cards && cd cards
bjo init
bjo run
}|
Running @code{bjo init} scaffolds a new project layout:
@codeblock[#:lang "text"]|{
cards/
manifest.bjodat package metadata and dependencies
src/
main.bjo executable entry point
tests/
.gitignore
}|
The generated manifest defines the package identity:
@codeblock[#:lang "bjolang"]|{
(package
(name (cards))
(version "0.1.0"))
}|
All source files in a package live under @code{src/}, where each module's name mirrors its relative path within that directory. For example, moving @code{deck.bjo} into @code{src/} exposes it as the module @code{(cards deck)}, which @code{src/main.bjo} can import just like any library module:
@codeblock[#:lang "bjolang"]|{
(import (cards deck))
(defun (main args)
(println #"Your hand: ${(vec-slice (shuffle (full-deck)) 0 5)}")
(println #"Arguments: ${args}")
0)
}|
Because @code{bjo} automatically detects the project root by walking upward until it finds a @code{manifest.bjodat}, you can invoke CLI commands from any sub-directory within the project:
@read-table{
| Command | What it does |
|---------------------------+----------------------------------------------------------------|
| @code{bjo init [--lib]} | Initializes a new project in the current directory |
| @code{bjo fetch} | Fetches external dependencies without compiling |
| @code{bjo build} | Fetches dependencies and compiles the program or library |
| @code{bjo run . args ...} | Builds and runs the project with the specified arguments |
| @code{bjo check} | Type-checks the entire project without emitting output binaries |
| @code{bjo repl} | Starts the REPL preloaded with access to the project's modules |
}
In @code{bjo run . one two}, the dot specifies the project itself. Any arguments following the target are forwarded directly to the program's @code{main} function, binding @code{args} to @code{[one two]}. You can also point @code{bjo run} at a specific test file, such as @code{bjo run tests/deck-test.bjo}; the file will be compiled and linked against the project's declared packages.
@subsubsection{Libraries}
A package that lacks an executable entry point (@code{src/main.bjo}) is treated as a library. Running @code{bjo init --lib} initializes a library project, and running @code{bjo build} inside it compiles all modules found under @code{src/}.
To see this in action, we can split out our card deck into a separate library residing alongside the game:
@codeblock[#:lang "text"]|{
mkdir cardlib && cd cardlib
bjo init --lib
mv ../cards/src/deck.bjo src/
}|
We can then declare a path dependency in @code{cards/manifest.bjodat} and import the library module as @code{(cardlib deck)}:
@codeblock[#:lang "bjolang"]|{
(package
(name (cards))
(version "0.1.0")
(depends
(package (name (cardlib))
(source (path (dir "../cardlib"))))))
}|
@codeblock[#:lang "bjolang"]|{
(import (cardlib deck))
(defun (main)
(println (vec-slice (shuffle (full-deck)) 0 5))
0)
}|
Path dependencies are resolved relative to the manifest file that specifies them. Because a path dependency points straight to source files on disk, it does not require a version number—the build simply consumes whatever code is currently in the directory, making local development and iteration seamless.
@subsubsection{Dependencies from git}
Once a library is hosted in a Git repository, you can switch from a local path to a Git dependency:
@codeblock[#:lang "bjolang"]|{
(depends
(package (name (cardlib))
(version (version-at-least "0.1.0"))
(source (git (url "https://github.com/someone/cardlib")))))
}|
Package versions correspond to Git tags formatted strictly as @code{vX.Y.Z}. Git dependencies specify version constraints using @code{(version-at-least "1.2")}, @code{(version-between "1.2" "2.0")}, or @code{(version-at-most "2.0")}.
Bjolang resolves versions using Minimal Version Selection (MVS). Rather than greedily selecting the latest release available in the wild, MVS inspects the minimum version requested by each package across the entire dependency graph, and then selects the highest of those minimum requirements. In other words, you get the oldest possible release that satisfies everyone's stated requirements. This guarantees reproducible builds: new upstream releases won't alter your build until someone explicitly raises their requirement in a manifest, ensuring your code compiles identically a year from now. Any upper bound constraints are treated strictly as validation checks, never as criteria to pick an older package.
Commands like @code{bjo fetch}, @code{build}, and @code{run} clone external dependencies into a local @code{.bjo/} directory and record the exact resolved commits in @code{bjo.lock}. You should always check @code{bjo.lock} into version control, while leaving @code{.bjo/} ignored (as configured in the default @code{.gitignore}). If an upstream repository rewires a tag to point to a different commit after it was locked, the build aborts immediately. On CI servers, pass @code{--locked} to guarantee that the build fails if any dependency deviates from the locked snapshot.
You can also override dependency sources: if your root manifest defines an explicit source for a package, that source overrides any locations requested deeper in the dependency tree. This makes it straightforward to substitute a local checkout or an emergency fork while debugging.
@subsubsection{NuGet packages}
You can pull in third-party .NET packages directly from NuGet by listing them in the @code{packages} section:
@codeblock[#:lang "bjolang"]|{
(package
(name (shop))
(version "0.1.0")
(packages (nuget (id "Npgsql") (version "9.0.3"))))
}|
Once restored, types and methods from NuGet packages can be consumed through Bjolang's .NET interop mechanisms (described in the .NET section below). Version strings follow standard NuGet versioning semantics: @code{"9.0.3"} matches version 9.0.3 or higher, while bracketed syntax like @code{"[9.0.3]"} locks to that exact version. Bjolang delegates package restoration to the .NET SDK via the @code{dotnet} tool, storing the resolution graph in @code{packages.lock.json} (which should also be committed). Currently, compiled binaries load dependencies directly from the local NuGet cache, so executables expect to run in an environment with the restored packages available.
@subsubsection{Publishing a package}
Publishing a Bjolang package requires three basic steps:
@read-list{
- Organize your source files under @code{src/}, and create a @code{manifest.bjodat} defining the package @code{name} and @code{version}.
- Run @code{bjo build} to verify that all modules compile cleanly without errors.
- Tag your commit with the semantic version—e.g. @code{git tag -a v0.1.0 -m "First release"}—and push the tag to your remote repository. Dependencies require valid @code{vX.Y.Z} tags to resolve.
}
@subsection{Files and I/O}
@subsubsection{Whole files}
For files that comfortably fit into memory, the prelude provides straightforward functions to read or write entire files in a single call:
@read-table{
| Function | Description |
|-------------------------------------------+----------------------------------------------------|
| @code{(file-read-text path)} | Reads the entire file into a string |
| @code{(file-write-text path s)} | Writes @code{s}, overwriting any existing contents |
| @code{(file-append-text path s)} | Appends @code{s} to the file, creating it if needed |
| @code{(file-read-lines path)} | Reads lines as a @code{(Vec string)} |
| @code{(file-write-lines path lines)} | Writes a @code{(Vec string)}, one line per element |
| @code{(file-read-bytes path)} | Reads the raw bytes as an @code{(Array byte)} |
| @code{(file-exists? path)} | Checks whether a file exists at the given path |
| @code{(file-delete path)} | Deletes a file; succeeds silently if file is absent |
| @code{(file-copy from to)} | Copies a file, raising an error if target exists |
| @code{(file-move from to)} | Moves or renames a file or directory |
| @code{(file-info path)} | Returns size, modification time, and kind as an @code{Option} |
}
Because these wrap the underlying .NET I/O methods directly, any I/O failure (such as missing permissions or a missing file on read) throws a .NET exception. If you expect a failure under normal conditions, wrap the call in a @code{try} expression.
To illustrate, we can add a persistent high-score ledger to our card game. Each line records a player name and their score separated by a space. Malformed lines are safely discarded rather than crashing the program:
@codeblock[#:lang "bjolang"]|{
(: score-file string)
(def score-file "scores.txt")
(: parse-score (-> string (Option (Tuple string int))))
(defun (parse-score line)
(match (string-split line " ")
([name points]
(match (try (string->int points) #:catch (System.FormatException))
((Ok n) (Some (Tuple name n)))
((Err _) None)))
(_ None)))
(: read-scores (-> (Vec (Tuple string int))))
(defun (read-scores)
(if (file-exists? score-file)
(loop (:for line (file-read-lines score-file))
(:when-let (Some score) (parse-score line))
(:acc (vecing score)))
[]))
(: save-score (-> string int void))
(defun (save-score name points)
(file-append-text score-file #"${name} ${points}\n"))
(defun (main)
(save-score "ada" 31)
(save-score "bo" 27)
(println (read-scores)) ;; [(ada, 31) (bo, 27)]
0)
}|
@subsubsection{Paths, directories and the environment}
Paths in Bjolang are represented as standard strings. The path manipulation utilities perform purely lexical operations on the string representations without touching the filesystem:
@codeblock[#:lang "bjolang"]|{
(path-combine "a" "b" "c.txt") ;; "a/b/c.txt"
(path-file-extension "deck.bjo") ;; (Some ".bjo")
(path-filename "/tmp/x/") ;; None: it names a directory
(path-directory "/tmp/x/y.txt") ;; (Some "/tmp/x")
(path-absolute "deck.bjo") ;; the full path
}|
When you do need to inspect or modify the filesystem layout, use the directory functions:
@codeblock[#:lang "bjolang"]|{
(directory-create "saves/2026") ;; creates every missing directory on the way
(directory-exists? "saves") ;; #t
(directory-files ".") ;; the files directly in ".", as a (Vec string)
(directory-subdirectories ".") ;; and the directories
(directory-delete-tree "saves") ;; everything under it, and it
;; There is no pattern argument. Filter instead:
(vec-filter #(string-ends-with? & ".bjo") (directory-files "."))
;; Every file under a directory, lazily, without going into .git:
(filter #(string-ends-with? & ".bjo")
(directory-walk "." #:into? (fun (d) (not (= (path-filename d) (Some ".git"))))))
}|
Process environment variables and working directory state can be inspected and updated similarly:
@codeblock[#:lang "bjolang"]|{
(get-environment-variable "HOME") ;; (Some "/home/linus"), or None
(set-environment-variable! "MODE" "x") ;; for this process and the ones it starts
(current-directory)
}|
@subsubsection{Ports}
When dealing with large files, streaming data, or incremental I/O, Bjolang uses *ports*. An input port provides a stream from which characters and lines can be read incrementally, while an output port collects written data. File ports and in-memory string ports share the same underlying interface, so functions written against ports work seamlessly with both files and string buffers.
Attempting to open a file returns a @code{Result}, treating potential I/O errors (such as missing files) as expected outcomes. The @code{Err} variant wraps the underlying .NET exception, allowing you to match on specific exception types with @code{:is}:
@codeblock[#:lang "bjolang"]|{
(match (open-input-file "nope.txt")
((Ok port) (read-all port))
((Err (:is System.IO.FileNotFoundException e)) #"missing: ${(.-FileName e)}")
((Err e) "some other error"))
}|
To avoid leaking resources, you will usually want to use @code{call-with-input-file} or @code{call-with-output-file}. These higher-order functions open the port, pass it to your callback, and guarantee the port is cleanly closed when the function finishes—even if an exception is thrown. For example, counting lines in an arbitrary file looks like:
@codeblock[#:lang "bjolang"]|{
(: count-lines (-> string (Result Exception int)))
(defun (count-lines path)
(call-with-input-file path
(fun (port)
(loop (:until (port-eof? port))
(:let line (read-line port))
(:acc lines (counting line))
=> lines))))
(count-lines "scores.txt") ;; (Ok 2)
}|
@read-table{
| Function | Description |
|-----------------------------------+---------------------------------------------------------|
| @code{(open-input-file path)} | Returns a @code{(Result Exception TextInputPort)} |
| @code{(open-output-file path)} | Opens for writing; @code{#:mode} can be @code{Truncate}, @code{Append}, or @code{CreateNew} |
| @code{(read-line/opt p)} | Reads the next line, or returns @code{None} at EOF |
| @code{(read-char/opt p)} | Reads the next character, or returns @code{None} at EOF |
| @code{(read-all p)} | Reads the remaining contents into a single string |
| @code{(port-eof? p)} | Checks if the end of the input stream has been reached |
| @code{(write-string p s)} | Writes string @code{s} to the port |
| @code{(writeln p s)} | Writes string @code{s} followed by a newline |
| @code{(close-input-port p)} | Closes an input port |
| @code{(close-output-port p)} | Closes an output port |
| @code{(open-input-string s)} | Creates a port reading from an in-memory string |
| @code{(open-output-string)} | Creates a string-builder port; read with @code{get-output-string} |
}
The variants @code{read-line} and @code{read-char} (without the @code{/opt} suffix) also exist, but throw an end-of-file exception if called at EOF.
Convenience helpers in @code{(std ports)} can ingest an entire port into collections: @code{port->lines}, @code{port->list}, @code{port->vec}, as well as lazy sequences via @code{port->seq} and @code{file->seq}:
@codeblock[#:lang "bjolang"]|{
(import (std ports))
(call-with-input-file "scores.txt" port->lines) ;; (Ok ["ada 31" "bo 27"])
}|
In Bjolang, open file ports are also tracked by their enclosing structured concurrency scope (explained in the Concurrency section below). When a scope exits, any unclosed ports opened within it are automatically finalized. Even @code{main} acts as a root scope, ensuring leaked handles are collected when your process terminates. For long-lived processes, however, you should always explicitly close ports or rely on @code{call-with-input-file}.
@subsubsection{Running other programs}
The @code{(std run)} module lets you invoke external processes. Commands are written as S-expression forms using a concise DSL where pipelines and shell-style redirections are integrated directly into the syntax:
@codeblock[#:lang "bjolang"]|{
(import (std run))
(run/string '(echo "hello")) ;; (Ok "hello\n")
(run/strings '(pipe (cat "scores.txt") (sort -r))) ;; (Ok ["bo 27" "ada 31"])
(run/status '(into-file "sorted.txt"
(pipe (cat "scores.txt") sort))) ;; (Ok 0), the exit code
(def who "ada")
(run/strings '(grep ,who "scores.txt")) ;; (Ok ["ada 31"])
}|
Prefixing an identifier with a comma (e.g. @code{,who}) splices the variable's value into the command arguments. Each list element is treated as an individual command-line argument, preserving spaces without fragile string quoting or shell injection hazards. The DSL primitives @code{pipe}, @code{into-file}, @code{from-file}, @code{append-to-file}, and @code{errors-into-file} are validated at compile time; any unrecognized head identifier is executed as an external executable.
@read-table{
| Function | Returns |
|---------------------------+------------------------------------------------------|
| @code{(run/string form)} | Standard output as a single trimmed string |
| @code{(run/strings form)} | Standard output split into a vector of lines |
| @code{(run/status form)} | Process exit code as an integer |
| @code{(run/output form)} | Tuple of exit code and standard output string |
| @code{(run form)} | A running @code{Proc} handle with streaming I/O ports |
}
All of these functions return a @code{Result} and execute synchronously, blocking the calling thread until completion. When running inside a bjoroutine, non-blocking asynchronous alternatives are available; refer to the @code{(std run)} module documentation for details.
@subsection{Concurrency}
Bjolang's concurrency model rests on three foundations: lightweight fibers called *bjoroutines*, communication channels, and structured concurrency scopes ensuring background work is cleanly contained. In addition, Bjolang adopts the Concurrent ML (CML) paradigm, where communication operations—such as sending or receiving on a channel—are first-class values that can be composed and combined before being synchronized.
@subsubsection{Bjoroutines}
Functions that may suspend execution—whether waiting on a channel, sleeping, or performing asynchronous I/O—are declared with @code{defbjo} rather than @code{defun}, and their type signatures use the @code{-bjo->} arrow. Suspending and waiting for an event is performed using @code{sync}:
@codeblock[#:lang "bjolang"]|{
(: think (-bjo-> int int))
(defbjo (think n)
(sync (timeout 50)) ;; wait 50 ms, without holding up a thread
(* n n))
}|
Bjoroutines execute as lightweight fibers scheduled across a managed thread pool. When a fiber suspends at a @code{sync} point, it yields its underlying thread back to the pool. As a result, maintaining thousands of waiting fibers incurs the memory overhead of small heap objects rather than expensive OS threads.
Because calling a suspending function means the caller may also have to wait, function coloring applies: bjoroutines can only be directly invoked from within other bjoroutines:
@codeblock[#:lang "text"]|{
Type Error at colour.bjo:7: calling 'think' is a yield point, and a yield point is not allowed here.
'twice' is defined with (defun ...), which is emitted as an ordinary C# method, and an ordinary
method cannot await.
Define it with (defbjo ...), or move the suspending call out of it. Note that (defbjo ...) spreads:
whoever calls 'twice' needs to be one too.
}|
To enter concurrent execution from the start, @code{main} can itself be defined with @code{defbjo}.
In practice, this coloring is less infectious than in many other languages. Most standard I/O functions (such as @code{file-read-text}) are automatically emitted with dual implementations: a synchronous blocking path and an asynchronous suspending path. An ordinary @code{defun} that calls standard I/O similarly gets both versions generated under the hood; when called from a bjoroutine, it transparently executes the non-blocking suspending version:
@codeblock[#:lang "bjolang"]|{
(: size-of (-> string int))
(defun (size-of path) (string-length (file-read-text path))) ;; an ordinary defun
(defbjo (main)
(println (size-of "deck.bjo")) ;; the file is read without blocking a thread
0)
}|
@subsubsection{spawn and channels}
The @code{(spawn (f args ...))} form creates a new concurrent fiber running the invocation @code{(f args ...)}. Arguments are evaluated eagerly in the spawning fiber before transferring execution to the new fiber.
Fibers communicate over channels created with @code{make-chan}. Rather than performing immediate I/O, expressions like @code{(chan-send ch v)} and @code{(chan-recv ch)} construct first-class *events*, which are then synchronized using @code{sync}. Unbuffered channels operate as rendezvous points: sending blocks until a receiver is ready, and receiving blocks until a sender arrives.
We can model our card players as concurrent bjoroutines. Each player receives a channel representing their seat at the table and plays one card per trick (descending by value). The table (@code{main}) draws cards round-robin from each seat:
@codeblock[#:lang "bjolang"]|{
(import "deck.bjo")
(: player (-bjo-> (Vec Card) (Chan Card) void))
(defbjo (player hand seat)
(loop (:for c (list-reverse (list-sort (vec->list hand))))
(:do (sync (chan-send seat c)))))
;; One trick: a card from every seat, in turn.
(: play-trick (-bjo-> (Vec (Chan Card)) (Vec Card)))
(defbjo (play-trick seats)
(loop (:for seat seats)
(:acc (vecing (sync (chan-recv seat))))))
(defbjo (main)
(def deck (shuffle (full-deck)))
(def seats [(make-chan) (make-chan) (make-chan)])
(loop (:for seat seats)
(:for p (range 0 3))
(:do (spawn (player (vec-slice deck (* p 4) 4) seat))))
(loop (:for trick (range 1 5))
(:do (println #"Trick ${trick}: ${(play-trick seats)}")))
0)
}|
@codeblock[#:lang "text"]|{
Trick 1: [K♥ K♦ J♥]
Trick 2: [8♣ 7♣ 4♣]
Trick 3: [6♠ 4♥ 3♣]
Trick 4: [6♦ 3♦ 2♦]
}|
Because channel sends and receives synchronize as a rendezvous, a player cannot get ahead of the table and play multiple cards into the same trick.
@subsubsection{Events are values}
Crucially, calling @code{(chan-recv seat)} does not perform the receive immediately; it simply constructs an event value describing the operation. The actual communication occurs only when the event is passed to @code{sync}. Because events are first-class values, they can be parameterized, composed, and transformed before being awaited:
@read-table{
| Function | Description |
|---------------------------+-------------------------------------------------------------|
| @code{(chan-recv ch)} | Receives a value from channel @code{ch} |
| @code{(chan-send ch v)} | Sends @code{v} on channel @code{ch} |
| @code{(timeout ms)} | Fires after @code{ms} milliseconds |
| @code{(choose e1 e2 ...)} | Fires when the first of the supplied events occurs |
| @code{(wrap e f)} | Fires when @code{e} occurs, applying @code{f} to its value |
}
With these combinators, deadlines and timeouts become simple functions that compose over any event. For instance, we can enforce a response timeout on our card players:
@codeblock[#:lang "bjolang"]|{
(: within (-> int (Event %a) (Event (Option %a))))
(defun (within ms ev)
(choose (wrap ev #(Some &))
(wrap (timeout ms) (fun (u) None))))
(: slow-player (-bjo-> (Chan string) void))
(defbjo (slow-player seat)
(sync (timeout 200))
(sync (chan-send seat "7♣")))
(defbjo (main)
(def seat (make-chan))
(spawn (slow-player seat))
(println (sync (within 50 (chan-recv seat)))) ;; None
(println (sync (within 500 (chan-recv seat)))) ;; (Some 7♣)
0)
}|
Notice that @code{within} is defined with standard @code{defun} rather than @code{defbjo}. Simply constructing or composing event values does not suspend execution; suspending only occurs when an event is submitted to @code{sync}.
Also notice how the first timeout behaves: when the 50 ms deadline expires before the card arrives, the @code{chan-recv} event is cleanly cancelled without consuming the message. The card remains on the channel, available to be received by the subsequent 500 ms wait.
An operation that is synced right where it is written does not need to be an event first. @code{(chan-put ch v)} does what @code{(sync (chan-send ch v))} does, and @code{(chan-get ch)} what @code{(sync (chan-recv ch))} does. They wait, and are cancelled, exactly as @code{sync} is, but no event value is built, so they are cheaper. Use the event forms when you need an event: to @code{choose} between operations, to @code{wrap} one, or to hand one to a function such as @code{within}. Our players could just as well be written with @code{chan-put}:
@codeblock[#:lang "bjolang"]|{
(defbjo (player hand seat)
(loop (:for c (list-reverse (list-sort (vec->list hand))))
(:do (chan-put seat c))))
}|
@subsubsection{Collecting fiber results: bjo}
While @code{spawn} runs a fiber for its side effects and discards its return value, @code{bjo} returns a joinable promise. You can await this promise using @code{(promise-join p)}, which yields a @code{Result} wrapping either the computed value or an exception if the fiber failed:
@codeblock[#:lang "bjolang"]|{
(def a (bjo (think 3)))
(def b (bjo (think 4)))
(sync (promise-join a)) ;; (Ok 9)
(sync (promise-join b)) ;; (Ok 16)
}|
Because both fibers execute concurrently across thread pool workers, the combined operations complete in roughly 50 ms rather than running sequentially for 100 ms.
@subsubsection{Scopes}
Every fiber runs within an enclosing structured concurrency scope. A scope cannot complete until all fibers spawned within it have terminated, ensuring background tasks never outlive their caller or leak into other components. Furthermore, unhandled exceptions inside spawned fibers propagate outward when the scope terminates rather than being silently dropped.
The top-level @code{main} function acts as an implicit root scope, which is why our earlier card game example cleanly awaited all player fibers. You can create nested scopes with fine-grained cancellation and deadlines using these forms:
@read-table{
| Form | Description |
|---------------------------------------+------------------------------------------------------------|
| @code{(with-scope body ...)} | Waits for child fibers and finalizes open file ports |
| @code{(with-cancel (cancel) body ...)} | Binds a @code{cancel} procedure to signal child fibers |
| @code{(with-deadline ms body ...)} | Cancels the scope automatically after @code{ms} ms |
| @code{(with-shield body ...)} | Shields cleanup operations from parent cancellation |
}
@codeblock[#:lang "bjolang"]|{
(: fetch (-bjo-> string void))
(defbjo (fetch name)
(sync (timeout 30))
(println #"fetched ${name}"))
(: count-up (-bjo-> (Chan int) void))
(defbjo (count-up ch)
(let go ((i 0))
(sync (chan-send ch i))
(go (+ i 1))))
(defbjo (main)
;; Both are fetched when this returns, or the deadline raises.
(with-deadline 1000
(spawn (fetch "a"))
(spawn (fetch "b")))
(println "both done")
;; count-up never ends on its own. Cancelling the scope stops it
;; the next time it waits.
(def numbers (make-chan))
(with-cancel (cancel)
(spawn (count-up numbers))
(println (sync (chan-recv numbers))) ;; 0
(println (sync (chan-recv numbers))) ;; 1
(cancel (Requested "enough")))
0)
}|
Cancellation in Bjolang is cooperative rather than preemptive. Cancelling a scope signals its fibers, which check for cancellation whenever they next perform a suspending operation (such as waiting on a channel or timer). A fiber that exits due to cancellation is treated as a clean completion rather than a failure.
Depending on the desired lifecycle and supervision strategy, fibers can be started using four distinct forms:
@read-table{
| Form | Scope behavior |
|------------------------------------+----------------------------------------------------------|
| @code{(spawn (f x))} | Awaited by scope; propagates failure on error |
| @code{(bjo (f x))} | Awaited by scope; yields a promise holding value/failure |
| @code{(spawn/daemon (f x))} | Cancelled on scope exit without being awaited |
| @code{(spawn/detached (f x))} | Runs detached; unmonitored by the enclosing scope |
}
The scope forms work in both colours. In a bjoroutine, the wait at the end of the scope suspends. In ordinary code, a @code{defun} or a @code{(fun ...)}, it parks the thread until the children have ended, as @code{sync/blocking} does. So an ordinary function can start fibers and know that they are done when it returns:
@codeblock[#:lang "bjolang"]|{
(: fetch-all (-> (Vec string) void))
(defun (fetch-all names)
(with-scope
(ignore (own! (fun () (println "released"))))
(loop (:for n names)
(:do (spawn (fetch n))))
(println "all started")))
(defun (main)
(fetch-all ["a" "b"])
(println "back in main")
0)
}|
@codeblock[#:lang "text"]|{
all started
fetched a
fetched b
released
back in main
}|
A scope also owns resources. @code{(own! thunk)} registers a release on the innermost scope, and @code{(own-disposable x)} registers @code{x}'s @code{.Dispose}. When the scope ends, the releases run after the fibers have ended, newest first. @code{(release! h)} runs one early, with the handle @code{own!} returned. These are ordinary functions, so they also work in both colours. A release is a plain thunk: it cannot suspend, and a release that blocks also blocks the end of the scope.
Do not park a bjoroutine's thread by accident. A scope in an ordinary @code{(fun ...)} that a bjoroutine calls parks the thread of that bjoroutine, and the compiler warns about it. Write the scope in the bjoroutine itself. Or, if the function the lambda is given to takes a @code{-?->} parameter, write the lambda as a @code{(bjoroutine ...)}.
@subsubsection{Work that is not a fiber}
Performing long CPU-bound computations or calling legacy blocking .NET APIs directly inside a fiber can monopolize thread pool threads, preventing other fibers from running. To avoid blocking the pool, you can offload such operations onto dedicated background threads; both helpers return synchronization events:
@codeblock[#:lang "bjolang"]|{
;; Work that waits: a call that parks its thread.
(sync (blocking #(file-read-text "deck.bjo"))) ;; (Ok "...")
;; Work that computes: gets a thread of its own.
(sync (spawn/thread #(loop (:for i (range 0 1000)) (:acc (summing i))))) ;; (Ok 499500)
}|
Conversely, you may occasionally need to wait on a channel or event from inside synchronous code. While an ordinary @code{defun} cannot suspend with @code{sync}, it can block its caller thread using @code{sync/blocking}. Avoid calling @code{sync/blocking} inside a bjoroutine, as doing so freezes a thread pool worker that other fibers rely on:
@codeblock[#:lang "bjolang"]|{
(: wait-for-card (-> (Chan Card) Card))
(defun (wait-for-card ch)
(sync/blocking (chan-recv ch)))
}|
@subsubsection{Shared state: atomics and transactions}
Channels move values from one fiber to another. When many fibers must read and change the same data, a channel means one fiber that owns the data and answers requests for it. That is often the clearest design, but it is not the only one. Two modules let fibers share data directly:
@read-table{
| Module | What it is for |
|--------------------+-----------------------------------------------------------------------------------|
| @code{(std atomic)} | One value that threads change without a lock: a counter, a flag, a cached result |
| @code{(std stm)} | Changes to several values as one step, and waiting until the values are right |
}
An atomic is the simple case and is described in Part III. The rest of this section is about transactions: software transactional memory, built on the design of the OCaml library kcas.
@strong{Locations and transactions.} A @code{(Loc T)} holds one value. A transaction is a function of one argument, a @code{Tx}, which is the log of the transaction. The function reads and writes locations through the log with @code{tx-ref} and @code{tx-set!}, and @code{atomically} runs it and commits it:
@codeblock[#:lang "bjolang"]|{
(import (std stm))
(def pot (make-loc 0))
(def table (make-loc (list)))
;; Put a card on the table and add its points to the pot, as one step.
(: play! (-> Card int void))
(defun (play! card points)
(atomically
(fun (tx)
(tx-set! tx table (Cons card (tx-ref tx table)))
(tx-set! tx pot (+ (tx-ref tx pot) points)))))
}|
The writes stay in the log until the transaction commits, and the commit makes all of them visible at once. No other fiber sees the card on the table without its points in the pot. If another transaction changed a location that this one read before this one commits, this one runs again with the new values. So a transaction never acts on values that changed under it, and nothing is ever locked.
@strong{The rules for a transaction function.} Because it can run more than once, a transaction function must have no side effects except through its @code{Tx}: no printing, no I/O, no changes to other state. Do those after @code{atomically} returns. A transaction function also can not @code{sync}. It is an ordinary function, and the compiler says so:
@codeblock[#:lang "text"]|{
Type Error at play.bjo:7: calling 'sync' is a yield point, and a yield point is not allowed here.
}|
If a transaction raises an exception, it writes nothing, and @code{atomically} raises the exception. Store immutable values in locations: a change inside a mutable vec is not part of any transaction.
@strong{Waiting with tx-retry.} A transaction that can not go on yet calls @code{tx-retry}. That stops it, and it runs again when a location that it read has changed. There is no condition variable to signal and no wakeup to forget:
@codeblock[#:lang "bjolang"]|{
;; The top card of a pile. Waits while the pile is empty.
(: take-top (-> Tx (Loc (List Card)) Card))
(defun (take-top tx pile)
(match (tx-ref tx pile)
((Cons card rest) (tx-set! tx pile rest) card)
(Nil (tx-retry tx))))
}|
@strong{A transaction is an event.} @code{(atomically-evt f)} is the event that runs and commits @code{f}. While @code{f} retries, the event is not available. So waiting for a condition on shared data combines with everything else that @code{choose} combines: a timeout, a channel, another transaction, and the cancellation of the fiber's scope. A transaction whose branch loses has written nothing. @code{atomically} is @code{(sync (atomically-evt f))}:
@codeblock[#:lang "bjolang"]|{
;; A card from the pile, or None after 100 ms.
(sync (choose (wrap (atomically-evt (fun (tx) (take-top tx pile))) Some)
(wrap (timeout 100) (fun (_) None))))
}|
Inside a transaction, @code{(tx-or-else tx a b)} runs @code{a}, and if @code{a} retries, undoes its writes and runs @code{b}. If both retry, the whole transaction waits for a change to anything either of them read.
@code{atomically} works in both colours: in a bjoroutine it syncs, and in an ordinary function it blocks the thread. @code{try-atomically} never waits: it gives @code{None} when the transaction retries, which makes it safe in a callback that .NET calls.
@strong{Queues, maps and counters.} Three modules build data structures out of locations, so that their operations are part of the transaction that calls them:
@read-table{
| Module | Type | What it is |
|----------------------------+--------------------+--------------------------------------------------------------------|
| @code{(std stm queue)} | @code{(TQueue T)} | A FIFO queue, bounded or not. Producers and consumers seldom conflict |
| @code{(std stm map)} | @code{(TMap K V)} | A hash map. Changes to different keys seldom conflict |
| @code{(std stm counter)} | @code{(TCounter T)} | A number that fibers on different CPUs add to without conflicts |
}
Their operations take the @code{Tx} first. Here a dealer takes the top card of a shared deck into a player's hand, and counts it, in one step:
@codeblock[#:lang "bjolang"]|{
(import (std stm queue))
(import (std stm map))
(import (std stm counter))
(def deck (make-queue)) ;; a (TQueue Card)
(def hands (make-tmap)) ;; a (TMap string (List Card))
(def dealt (make-tcounter 0)) ;; a (TCounter int)
;; Waits while the deck is empty: queue-pop! retries.
(: deal! (-> Tx string Card))
(defun (deal! tx player)
(def card (queue-pop! tx deck))
(tmap-update! tx hands player
(fun (hand) (Some (Cons card (match hand ((Some cards) cards) (None Nil))))))
(tcounter-add! tx dealt 1)
card)
(: dealer (-bjo-> string int void))
(defbjo (dealer player n)
(loop (:for i (range 0 n))
(:do (ignore (atomically (fun (tx) (deal! tx player)))))))
}|
Two dealers can run at the same time. Neither card can end up in two hands, or be counted without being dealt. The queue also has the channel names for use outside a transaction: @code{queue-send} and @code{queue-recv} are events, and @code{queue-put} and @code{queue-get} sync them. What a queue adds to a channel is the transaction: take from one queue and put in another, or take from whichever of two queues has a card, as one step.
@strong{Costs.} A transaction that reads and writes the same locations as many others runs again often. Many fibers that write one location cost the most. A @code{TMap} or @code{TCounter} spreads the writes over many locations for that reason. A transaction that only reads never makes another one run again. Transactions also allocate, and a program in which many fibers allocate at once is often limited by the garbage collector: set @code{(runtime (gc-server #t))} in the manifest to give it one heap per CPU. With it, sixteen fibers that add keys to one @code{TMap} have eight times the throughput of one fiber.
@subsubsection{Compared with Go and Hopac}
Go and Hopac are the two points of comparison. Go is the obvious one: goroutines and unbuffered channels are the same model of fibers meeting at a rendezvous. Hopac is the tougher one. It is a Concurrent ML library for F#, running on the same .NET runtime with the same JIT and garbage collector, so the differences between it and Bjolang are in the design rather than in the platform. The main repository has the same benchmarks in all three (@code{bench/bjolang/cmlbench.bjo}, @code{go run . -suite cmlbench} in @code{bench/go}, and @code{dotnet run -c Release -- cmlbench} in @code{bench/hopac}).
Every Bjolang channel operation races the cancellation token of its scope. Neither Go nor Hopac has that built in, so each is measured twice: plain, and with every operation raced against what a program in it has to use to stop its work. In Go that is a @code{select} against @code{ctx.Done()}; in Hopac it is an @code{IVar} (Hopac's write-once variable) that is filled to stop the jobs. Bjolang is measured twice as well: with events and @code{sync}, and with @code{chan-put} and @code{chan-get}, so the difference between the two Bjolang columns is what building an event and syncing it costs. The best of five runs, in nanoseconds per operation, on a Ryzen 9 5900X under Linux with .NET 10 and Go 1.25:
@read-table{
| Benchmark | Bjolang, @code{sync} | Bjolang, @code{chan-put} | Go | Go, @code{ctx.Done()} | Hopac | Hopac, cancel @code{IVar} |
|----------------------------------------+----------------------+--------------------------+-----+-----------------------+-------+---------------------------|
| Ring: 1000 fibers, a million messages | 70 | 73 | 117 | 241 | 51 | 3487 |
| The same ring, one cancel IVar per job | — | — | — | — | — | 195 |
| The same ring inside a nested scope | 63 | — | — | 242 | — | 3427 |
| Spawning a million fibers and waiting | 210 | — | 190 | — | 80 | — |
| An 8-way choose where one branch fires | 181 | 205 | 403 | 451 | 127 | 136 |
}
In the choose row only the sender can use @code{chan-put}, since the receiving side has to use channel-receive (and thus allocates). On the ring, syncing an event costs nothing measurable any more: a send or a receive synced alone does not build an event. For some reason, this ring inside a nested scope is faster in bjolang, and I haven't figured out why yet.
Against Go, Bjolang is ahead on the ring and the choose, and further ahead once Go has to watch @code{ctx.Done()}, which doubles its ring. Go spawns a little faster. Against Hopac it is the other way round without cancellation: Hopac's ring and choose take about 30% less time than Bjolang's, and its spawns about a third. Bjolang's numbers include watching the scope's token at every operation, which Hopac's plain rows do not do. Once cancellation is added, the ring turns. A thousand jobs racing one shared @code{IVar} all take that IVar's lock at every operation, and the ring slows to three and a half microseconds a message. Giving each job an IVar of its own brings it back to 195 ns, at the price of keeping track of a thousand IVars and filling all of them to stop the ring. The choose does not turn: with a cancel IVar Hopac's takes 136 ns, against Bjolang's 181.
The spawn row is about twice what it was with the scheduler Bjolang used before, which handed every fiber to the .NET thread pool, and the choose row about one and a half times. With it (@code{BJO_SCHEDULER=pool} in the environment, kept as a way back) a spawn costs 105 ns and the choose 113. The spawn row is a single fiber spawning a million fibers that do nothing, and it is slow only on many cores: on two or four cores the current scheduler spawns faster than the pool, 61 and 72 ns against 87 and 112. The scheduler that replaced it runs fibers from run queues of its own, in the shape of Go's, because the pool let work that had waited longest wait longer still under load. That is worth more in a loaded service, as the service benchmark below shows, than these two rows on an idle machine.
These are small benchmarks on one machine, and they only measure the cost of the concurrency primitives, not of the work between rendezvous. They do not show what each operation allocates either. A Go channel operation never allocates and a goroutine costs 16 bytes. A Bjolang send or receive synced alone allocates nothing, a @code{choose} 56 bytes and a spawn 136. Hopac allocates 208 bytes a message on the ring, 424 a choose and 184 a spawn. Go is ahead in other ways too: any Go function can block without being written as a bjoroutine and the Go scheduler preempts a goroutine that computes for too long, where a fiber has to be moved off the pool with @code{spawn/thread}.
What Bjolang offers that Go does not is in the CML vocabulary rather than in the numbers. A Go @code{select} is a statement, its cases are fixed in the source, and a channel operation cannot be passed to a function, wrapped, or combined with others. In Bjolang an event is a value, which is what made @code{within} above a two-line function that works on any event, and @code{choose}, @code{wrap}, @code{guard} and @code{with-nack} build new events out of old ones. Go has no structured concurrency either. Waiting for goroutines, stopping them, and reporting their failures are a @code{sync.WaitGroup}, a @code{context.Context} passed to every function and checked at every channel operation, and an @code{errgroup}, written by hand each time. A goroutine that nothing waits for leaks without a sound. In Bjolang a scope does all of that by construction: it waits for its fibers, raises their failures when it ends, releases the files they opened, and cancels them through a token that every @code{sync} watches without being asked. Hopac can run jobs in parallel and wait for all of them, but a job cannot be stopped from the outside, so stopping jobs when one of them fails, or when their owner is done with them, is the same hand-written race against an IVar as in the table.
@subsubsection{A service under load}
The rows above time one primitive at a time on an idle machine. @code{bench/service} is closer to a program: a sharded key-value store in all three languages, with 16 shard fibers that each serve a part of the keys, and clients that send gets, puts and four-key multi-gets, each with a fresh reply channel and a one-second deadline raced against the reply. The three programs do the same work and agree on a checksum. @code{bench/service/drive.py} runs them and takes the median of 15 reps.
The two .NET programs run with the server GC, which keeps a few hundred megabytes and collects a few times a rep. Go's collector keeps the heap small and collects about twenty times as often, so Go runs twice: with its default, and with as much memory as the .NET programs took (@code{GOGC=off} and @code{GOMEMLIMIT} at their peak).
In a closed loop, 1000 clients send 1000 requests each, each one as soon as the last is answered, so the machine is always saturated. Wall time, CPU time, p99.9 latency and peak memory:
@read-table{
| Program | all cores: wall | CPU | p99.9 | memory | 4 cores: wall | CPU | p99.9 |
|------------------+-----------------+---------+--------+--------+---------------+---------+--------|
| Go | 397 ms | 5552 ms | 5.7 ms | 27 MB | 475 ms | 1832 ms | 3.4 ms |
| Go, same memory | 258 | 4076 | 3.2 | 364 | 414 | 1641 | 3.5 |
| Hopac | 411 | 8903 | 9.8 | 387 | 296 | 1161 | 4.2 |
| Bjolang | 254 | 5211 | 5.5 | 369 | 407 | 1571 | 4.7 |
}
On all cores Bjolang and Go with the same memory finish together, and Go uses a fifth less CPU and has the shorter tail. On four cores Hopac is clearly the fastest, and Bjolang and Go are close.
In an open loop, 10000 clients send on a fixed schedule at a given rate, so the three are compared at the same load. Latency is taken from when a request was due, so a client that woke late, waited for a core, or slept through a GC pause counts that time too. p99 and p99.9 in milliseconds:
@read-table{
| Load | Go | Go, same memory | Hopac | Bjolang |
|----------------------+-------------+-----------------+-------------+-------------|
| 4 cores, 250k/s | 5.6 / 9.0 | 2.0 / 2.2 | 2.8 / 4.3 | 2.2 / 3.1 |
| 4 cores, 500k/s | 28.0 / 32.6 | 41.9 / 48.6 | 6.9 / 8.3 | 3.1 / 3.7 |
| all cores, 500k/s | 7.0 / 9.1 | 8.0 / 12.7 | 24.5 / 31.1 | 3.6 / 6.3 |
| all cores, 1M/s | 33.9 / 37.1 | 69.4 / 73.5 | 27.5 / 32.3 | 5.9 / 8.6 |
}
A client that is early sleeps until its request is due, in whole milliseconds rounded up, the same in all three programs, so every latency here has up to a millisecond of that in it, and below about 2 ms the numbers say little. At 250k/s on four cores the three are close, and Go with the same memory has the shortest tail. At every higher load Bjolang has the shortest, and at a million requests a second its tail is a quarter of Hopac's or less, and a fifth of Go's or less. Some rows move from run to run: over three runs at 500k/s on all cores, Go with the same memory had a p99 of 2.1 to 8.0 ms and Bjolang of 3.6 to 5.0, so that row is not settled. Go's tail at 500k/s on four cores is long with either memory, and its time once a request has started is as long, so the delay is in running the requests, not before them. Hopac keeps all 24 cores busy on all cores at either rate, and why has not been looked into.
The tables in @code{BjolangRuntime/Cml/design.md} in the main repository have every column, including the time a request took once its client had started it.
@subsection{Effects}
Our deck is shuffled using @code{shuffle-vec}, which is non-deterministic. While randomness is ideal during gameplay, it makes automated tests brittle when verifying deterministic outcomes like trick winners. The traditional object-oriented or functional workaround is dependency injection—passing an explicit shuffle function down through every layer of the call stack.
Bjolang provides an alternative mechanism: *tail-resumptive algebraic effects*. With effects, a function simply declares and performs an abstract operation, allowing an enclosing caller higher up the call stack to dynamically decide how that operation is handled.
@subsubsection{Declaring an effect}
@codeblock[#:lang "bjolang"]|{
(import (std effect) "deck.bjo")
(defeffect Dealing
(shuffle-deck (-> (Vec Card) (Vec Card)))
#:default ((shuffle-deck shuffle)))
(: deal-hand (-> int (Vec Card)))
(defun (deal-hand n)
(vec-slice (shuffle-deck (full-deck)) 0 n))
}|
@code{shuffle-deck} is exposed as a regular function matching the declared signature. When called without an active handler in the dynamic scope, it falls back to the default implementation—in this case, the real shuffle. In a test, you can override this behavior using @code{with-handler}:
@codeblock[#:lang "bjolang"]|{
(deal-hand 5) ;; five random cards
(with-handler ((shuffle-deck (fun (deck) deck)))
(deal-hand 5)) ;; [2♣ 3♣ 4♣ 5♣ 6♣], every time
}|
The handler intercepts operations executed anywhere within the dynamic extent of the @code{with-handler} block, including inside deeply nested helper calls and child fibers, without requiring @code{deal-hand} to pass any parameters around.
Several key semantics govern effect handlers:
@read-list{
- Default implementations: If an effect omits @code{#:default} and is performed without an active handler, the runtime raises an exception. Omit defaults for operations that always require explicit environmental context.
- One-shot execution: Handlers in Bjolang behave like standard functions and return a single value. Full multi-shot delimited continuations are not supported: a handler cannot resume its caller multiple times or abort resumption except by throwing an exception.
- Effect coloring: Operations declared with @code{->} can be invoked from any standard function. Operations declared with @code{-bjo->} may only be invoked from within bjoroutines, and their handlers must likewise be bjoroutines.
- Handler composition: Invoking an operation from inside its own handler invokes that handler recursively. To delegate to an enclosing outer handler, retrieve it using @code{(handler-of op)}:
}
@codeblock[#:lang "bjolang"]|{
(with-handler ((log (let ((outer (handler-of log)))
(fun (s) (outer (str "[game] " s))))))
(log "dealing")) ;; "[game] dealing" on stderr
}|
@subsubsection{The prelude's own effects}
Several core runtime facilities in the prelude are implemented as effects out of the box, allowing you to intercept or virtualize them in tests without modifying application code:
@read-table{
| Operation | Default |
|----------------------------+----------------------------------------------------------|
| @code{(log s)} | Writes a line to standard error |
| @code{(warn s)} | Writes a warning line to standard error |
| @code{(getenv name)} | Reads an environment variable as an @code{Option} |
| @code{(monotonic-ms)} | High-resolution monotonic clock for elapsed timing |
| The @code{FS} operations | Physical filesystem operations |
}
Because all prelude file I/O routes through the built-in @code{FS} effect operations, tests can replace the entire filesystem with an in-memory mock using @code{with-fake-fs} (covered in the Testing section below).
Similarly, you can capture diagnostic output in memory instead of dumping it to the terminal by installing a custom handler for @code{log}:
@codeblock[#:lang "bjolang"]|{
(def lines (make-box (list)))
(with-handler ((log (fun (s) (box-set! lines (Cons s (box-ref lines))))))
(log "one")
(log "two"))
(box-ref lines) ;; '("two" "one")
}|
Keep in mind that @code{with-handler} only establishes a dynamic binding; unlike a concurrency scope, it does not await background fibers. If code inside the handler spawns asynchronous fibers that emit logs, ensure the concurrency scope enclosing those fibers is placed *inside* the @code{with-handler} block so the handler remains active until all fibers finish writing.
@subsection{.NET interop}
Because Bjolang compiles directly to C#, you have full, seamless access to the entire .NET ecosystem. By declaring the CLR classes, methods, and properties you need along with their signatures, the compiler exposes them as ordinary Bjolang functions and types.
@subsubsection{Methods: import/extern}
For example, we can render suit symbols in color in the terminal using .NET's @code{System.Console}:
@codeblock[#:lang "bjolang"]|{
(import "deck.bjo")
(import/class
(ConsoleColor (: System.ConsoleColor)))
(import/extern
(set-foreground! (: System.Console.ForegroundColor (-> ConsoleColor void) #:set))
(reset-colour! (: System.Console.ResetColor (-> void)))
(console-write (: System.Console.Write (-> string void))))
(: suit-colour (-> Suit ConsoleColor))
(defun (suit-colour s)
(match s
((or Hearts Diamonds) ConsoleColor.Red)
((or Clubs Spades) ConsoleColor.Blue)))
(: print-card (-> Card void))
(defun (print-card c)
(set-foreground! (suit-colour (record-ref c suit)))
(console-write #"${c} ")
(reset-colour!))
(defun (main)
(vec-for-each print-card (vec-slice (shuffle (full-deck)) 0 5))
(println "")
0)
}|
Each entry in @code{import/extern} specifies a local Bjolang identifier, the fully qualified CLR member, and its signature. For overloaded methods (such as @code{Console.Write}), the compiler determines which overload to bind based on the provided signature. Instance methods accept the target object as their first parameter. Properties and fields can be accessed via @code{#:get} and modified with @code{#:set}:
@codeblock[#:lang "bjolang"]|{
(import/extern
(trim (: System.String.Trim (-> string string)))
(max-int (: System.Int32.MaxValue #:get)))
(trim " hi ") ;; "hi"
max-int ;; 2147483647
}|
.NET's own string, @code{System.String}, is UTF-16 and a different type from @code{string}. A .NET method that takes or returns one, or an array of them, is converted where it is called, so @code{trim} above takes and gives a Bjolang string; a @code{null} from .NET arrives as @code{""}. A .NET string inside another type, such as a @code{(Seq System.String)}, is not converted: @code{string->clr-string} and @code{clr-string->string} do it by hand.
Avoid binding an external method to the name of an existing prelude function. Locally bound names and prelude symbols take precedence over @code{import/extern} declarations, which means an alias named @code{println} would simply be shadowed and ignored.
@subsubsection{Classes: import/class}
The @code{import/class} form maps a .NET class to a Bjolang type, exposing its constructor as a function suffixed with a dot (e.g. @code{StringBuilder.}). CLR enums map similarly, with individual members accessed using dot notation like @code{Type.Member} (seen with @code{ConsoleColor.Red} above).
@codeblock[#:lang "bjolang"]|{
(import/class
(StringBuilder (: System.Text.StringBuilder (-> StringBuilder))))
(def sb (StringBuilder.)) ;; new StringBuilder()
(ignore (.Append sb "hello, "))
(ignore (.Append sb "world"))
(.ToString sb) ;; "hello, world"
(.-Length sb) ;; 12
(.ToUpper "shout") ;; "SHOUT"
}|
A method that @code{string} does not have, like @code{ToUpper}, is called on the string converted to .NET's, so its indices and lengths are .NET's UTF-16 ones and not @code{string-length}'s bytes.
As shown in the example above, you do not need to declare every method up front with @code{import/extern}. Once an object's type is known to the type checker, you can invoke any public method using @code{(.Method obj args ...)} and inspect properties using @code{(.-Property obj)}. Because @code{Append} returns the builder itself and Bjolang enforces explicit handling of return values, discard unused returns intentionally using @code{ignore}.
@subsubsection{When .NET fails}
Idiomatic .NET code reports errors by throwing exceptions. Bjolang provides three distinct ways to translate these exceptions into safe value types:
@read-list{
- Generic @code{try} blocks: Wrap arbitrary code in @code{try}, as introduced in Part 1.
- Exception-mapped constructors: Annotating a constructor with @code{#:exceptions} in @code{import/class} causes the constructor to return a @code{Result} instead of throwing on specified exceptions.
- Output parameter conversion: Common .NET methods adhering to the @code{bool TryParse(..., out T result)} pattern automatically transform into Bjolang functions that return an @code{(Option T)}.
}
@codeblock[#:lang "bjolang"]|{
(import/class
(Uri (: System.Uri (-> string Uri)
#:exceptions (System.UriFormatException))))
(import/extern
(parse-int (: System.Int32.TryParse (-> string (out int) (Option int)))))
(match (Uri. "https://example.com/cards")
((Ok u) (.-Host u)) ;; "example.com"
((Err e) (.-Message e)))
(parse-int "42") ;; (Some 42)
(parse-int "forty-two") ;; None
}|
@subsubsection{Async methods, and blocking calls}
A .NET method returning a @code{Task} or @code{Task<T>} can be imported using the @code{#:async} modifier. Calling an async method from within a bjoroutine suspends the fiber until the task completes without blocking an OS thread, with no explicit @code{await} required:
@codeblock[#:lang "bjolang"]|{
(import/extern
(read-text-async (: System.IO.File.ReadAllTextAsync (-> string string) #:async)))
(defbjo (main)
(println (string-length (read-text-async "deck.bjo")))
0)
}|
Conversely, CLR methods known to perform synchronous, thread-blocking I/O should be tagged with @code{#:blocking}. Calling a blocking method from within a bjoroutine triggers a compiler warning; you can resolve this warning by offloading the call to a dedicated thread pool thread via @code{(sync (blocking (fun () ...)))}.
@subsubsection{Generics}
Generic .NET types are imported by parameterizing them with type variables, and generic methods specify those type variables in their signature:
@codeblock[#:lang "bjolang"]|{
(import/class
((Dict %k %v) (: System.Collections.Generic.Dictionary)))
(import/extern
(dict-try-get (: System.Collections.Generic.Dictionary.TryGetValue
(-> (Dict %k %v) %k (out %v) (Option %v)))))
}|
At the boundary between systems, Bjolang types map cleanly to standard .NET equivalents: a @code{Func<A, B>} corresponds to @code{(-> %a %b)}, @code{IEnumerable<T>} maps to @code{(Seq %a)}, C# tuples map to @code{Tuple}, and native arrays @code{T[]} correspond to @code{(Array %a)}. Note that constructing generic .NET classes dynamically and pattern matching on .NET objects via constructor patterns are not currently supported.
@subsubsection{Culture and formatting}
Bjolang programs execute using the invariant culture by default. For example, floating-point numbers format with a decimal point (@code{2.5}), avoiding regional discrepancies on machines where local culture settings would otherwise produce @code{2,5}. If your application requires localized formatting, you can configure @code{CultureInfo.CurrentCulture} explicitly.
@subsubsection{Calling Bjolang from C#}
Every Bjolang module compiles to a public static class named @code{<module>_Module}, exposing top-level functions defined with @code{defun} as static methods and @code{def} values as static fields. Identifiers with characters invalid in C# are automatically mangled (for example, hyphenated names like @code{full-deck} become @code{fullsubdeck}), while collections remain Bjolang's persistent data structures. You can inspect the emitted C# source code directly by passing the debug flag: @code{bjo build -d}, which generates @code{out.cs}.
@subsection{Macros}
Macros are functions executed by the compiler during the expansion phase. A macro receives an unevaluated syntax tree as input data and returns a new syntax tree that replaces the invocation form. In fact, many constructs in Bjolang that appear to be built-in language keywords—such as @code{cond}, @code{when}, @code{type/derive}, and @code{with-test}—are actually implemented as standard library macros.
@subsubsection{def/macro}
Macros are defined using @code{def/macro}. A macro transformer accepts three parameters: @code{form}, @code{inject}, and @code{compare}. Most macros are written using pattern matching via @code{syntax-match} from the @code{(std syntax-match)} library:
@codeblock[#:lang "bjolang"]|{
(import (std syntax-match))
;; (with-each-card (c deck) body ...) runs the body once for every card.
(def/macro (with-each-card form inject compare)
(syntax-match form
((_ (name deck) body ...)
#'(vec-for-each (fun (,name) ,@body) ,deck))
(bad (syntax-error bad "(with-each-card (name deck) body ...)"))))
}|
The @code{syntax-match} form decomposes syntax by structural shape. A wild-card pattern @code{_} matches the macro name itself, variable names bind matched syntax fragments, and ellipsis @code{...} matches repeated sequences. Templates are constructed using syntax quotation @code{#'(...)}, where comma @code{,x} unquotes an individual syntax value and @code|{,@xs}| splices a list of syntax objects into the form. Malformed invocations can be rejected at compile time using @code{syntax-error}, which highlights the offending source location.
Pattern fragments like @code{((name value) ...)} match lists of pairs and bind @code{name} and @code{value} to lists of syntax elements. Quoted symbols such as @code{'else} match that literal identifier.
Because macro transformers are compiled code executed directly by the compiler while reading subsequent source files, a macro cannot be used in the same module where it is defined:
@codeblock[#:lang "text"]|{
'twice' is a macro defined in this module, and a macro cannot be used where it is defined,
at bad2.bjo:5. Its transformer runs inside the compiler, so it has to be compiled before
whatever uses it is read — which cannot be true of the file it is written in. Move it to a
module of its own and import that. An (include ...) will not do: an included file becomes
part of this one.
}|
For our card game, we place macro definitions in a dedicated module, @code{cardmacros.bjo}. Importing that module with @code{(import "cardmacros.bjo")} automatically makes its macros available. Unlike functions, macros do not require an explicit export list—every macro defined in a module is exported automatically.
The input passed to a macro is a value of type @code{Syntax}, a union type with variants like @code{(SSym s)} for symbols, @code{(SInt text)} for numbers, and @code{(SList items)} for parenthesized forms. Because macros are ordinary Bjolang functions, you can invoke helper functions, inspect @code{Syntax} records directly, and use recursion.
@subsubsection{Hygiene and inject}
Bjolang macros are hygienic by default. Identifiers introduced within a syntax template are automatically renamed to ensure they never collide with local bindings in the calling scope:
@codeblock[#:lang "bjolang"]|{
;; (or-else a b) is a, unless a is 0.
(def/macro (or-else form inject compare)
(syntax-match form
((_ a b) #'(let ((tmp ,a)) (if (= tmp 0) ,b tmp)))))
;; in another module
(def tmp 9)
(or-else 0 tmp) ;; 9, not 0
}|
Here, the template's temporary variable @code{tmp} and the caller's outer variable @code{tmp} are treated as two distinct symbols, allowing the expression to evaluate correctly to @code{9}. Furthermore, free identifiers referenced in the template (such as @code{let}, @code{if}, and @code{=}) always resolve to their prelude definitions, even if the calling scope has shadowed those names locally.
If a macro intentionally needs to introduce an unhygienic binding into the caller's environment, it can explicitly opt out of renaming using @code{(inject 'name)}:
@codeblock[#:lang "bjolang"]|{
;; (aif test then else): in `then`, `it` is what was inside the Some.
(def/macro (aif form inject compare)
(syntax-match form
((_ test then else)
#'(match ,test
((Some ,(inject 'it)) ,then)
(None ,else)))))
(aif (map-try-ref #map(("a" 1)) "a")
(println #"found ${it}")
(println "nothing"))
}|
@subsubsection{Splicing multiple definitions: begin}
A macro returns a single syntax form. If you need a macro to expand into multiple definitions or expressions at the top level, return a @code{(begin ...)} form, whose child elements are spliced directly into the enclosing scope:
@codeblock[#:lang "bjolang"]|{
;; (def/counter name) defines a function that counts how often it is called.
(def/macro (def/counter form inject compare)
(syntax-match form
((_ name)
#'(begin
(def/mutable count 0)
(: ,name (-> int))
(defun (,name)
(set! count (+ count 1))
count)))))
;; in another module
(def/counter next-id)
(next-id) ;; 1
(next-id) ;; 2
}|
In this expansion, the internal counter variable @code{count} is generated hygienically and cannot collide with any @code{count} variable in the importing module, while @code{next-id} takes on the identifier provided by the caller.
@subsubsection{Pattern macros: def/pattern}
The @code{def/pattern} form defines macros intended for use within pattern-matching positions rather than ordinary expressions. Nullary pattern macros can be referenced simply by their bare capitalized name:
@codeblock[#:lang "bjolang"]|{
;; Face matches a jack, a queen or a king.
(def/pattern (Face form inject compare)
(syntax-match form
(_ #'(or Jack Queen King))))
;; in another module
(: court? (-> Card bool))
(defun (court? c)
(match (record-ref c rank)
(Face #t)
(_ #f)))
}|
Pattern macros expand before the compiler performs exhaustiveness checking, meaning the compiler inspects the expanded @code{or} pattern just as if it had been written by hand.
@subsubsection{Hash macros: def/hash-extend}
Literal prefixes like @code{#map(...)} are reader-level hash macros. You can register custom hash literals using @code{def/hash-extend}. For example, we can implement a custom literal syntax for playing cards, converting ranks and suits into their typed constructors:
@codeblock[#:lang "bjolang"]|{
(import (std syntax-match) "deck.bjo")
(re-export Card Rank Suit)
;; A number is a Pip, a letter a court card.
(: rank-syntax (-> Syntax Syntax))
(defun (rank-syntax r)
(syntax-match r
('J #'Jack) ('Q #'Queen) ('K #'King) ('A #'Ace)
(n (match n
((SInt _) #'(Pip ,n))
(_ (syntax-error n "a rank is 2 to 10, J, Q, K or A"))))))
(: suit-syntax (-> Syntax Syntax))
(defun (suit-syntax s)
(syntax-match s
('clubs #'Clubs) ('diamonds #'Diamonds) ('hearts #'Hearts) ('spades #'Spades)
(bad (syntax-error bad "a suit is clubs, diamonds, hearts or spades"))))
;; #card(Q hearts) is (Card (rank Queen) (suit Hearts)).
(def/hash-extend (card form inject compare)
(syntax-match form
((_ r s) #'(Card (rank ,(rank-syntax r)) (suit ,(suit-syntax s))))
(bad (syntax-error bad "#card takes a rank and a suit: #card(Q hearts)"))))
}|
@codeblock[#:lang "bjolang"]|{
(import "deck.bjo" "cardmacros.bjo")
(defun (main)
(println #card(Q hearts)) ;; Q♥
(println #card[10 spades]) ;; 10♠, either bracket works
(println (court? #card(K clubs))) ;; True
(with-each-card (c [#card(A spades) #card(2 hearts)])
(println #"a card: ${c}"))
0)
}|
Invalid literal syntax is flagged with a compile-time error at the point of use:
@codeblock[#:lang "text"]|{
The hash macro 'card' failed at bad.bjo:3: a suit is clubs, diamonds, hearts or spades — in hurts
}|
Notice the @code{re-export} in @code{cardmacros.bjo}: because the macro expansion emits constructor symbols like @code{Card} and @code{Queen}, those types must be resolvable in the module invoking the macro. Re-exporting them ensures that importing @code{cardmacros.bjo} alone brings the required types into scope. Importing @code{deck.bjo} alongside it remains recommended so that the corresponding @code{->str} implementations are also present.
@subsubsection{Literals that elaborate into your unions}
Many embedded domain-specific languages do not require macros at all. In contexts where the type checker expects a specific union type, quoted lists can elaborate directly into union cases. Union cases specify their literal keyword tag using @code{#:tag}:
@codeblock[#:lang "bjolang"]|{
(type (: Action (Union (: Play int #:tag play)
(: Say string #:tag say)
(: Pass #:tag pass))))
(: script (List Action))
(def script '((play 3) (say "your turn") pass (play 1)))
;; = (list (Play 3) (Say "your turn") Pass (Play 1))
(def n 7)
(def (: more (List Action)) '((play ,n) pass))
}|
Because elaboration is handled by the type checker rather than a syntactic macro, misspelled tags or invalid argument counts produce standard type errors:
@codeblock[#:lang "text"]|{
Type Error at elab2.bjo:5: `sya` is not a tag of the union elab2/Action and no case of it
carries a list, so this literal cannot be one. Its tags are play, say or pass.
}|
This elaboration mechanism only applies in positions where the type checker already knows that a specific union type is expected (which is why @code{script} and @code{more} include explicit type signatures). Standard library modules like @code{(std run)} and @code{(std fmt)} use this exact technique for command pipelines and text layouts.
@subsubsection{Computing a value once}
A macro often builds a value out of constants, and that value is then built again every time the expansion is evaluated. When that is expensive, wrap it in @code{please-hoist-i-promise-i-am-not-naughty}: the form is evaluated the first time it is reached, and every later evaluation gets that same value. This is what @code{#rx(...)} does, so a regex is compiled once rather than once per match.
@codeblock[#:lang "bjolang"]|{
;; (tag-set "a" "b" ...) is a HashSet of the strings, built once.
(def/macro (tag-set form inject compare)
(syntax-match form
((_ tag ...)
#'(please-hoist-i-promise-i-am-not-naughty
(list->hashset (list ,@tag))))))
}|
If computing the value throws, nothing is kept, and the next evaluation tries again. The same form written in two places is one value.
The long name is a promise about what the compiler cannot check: that the form has no side effects worth repeating, and that it does not depend on things that change from call to call, such as a parameter or an effect handler. Everything it can check, it does, and a form that breaks a rule is a compile error that lists every reason. The form cannot:
@itemlist{
@item{refer to a local variable or a function parameter;}
@item{read a @code{def} of the module it is in, or any @code{def/mutable};}
@item{use a type variable of the function around it;}
@item{call a bjoroutine, @code{yield} to a sequence, or leave a @code{with-return};}
@item{be void, or produce something that can be changed in place, such as an array, a box or a mutable collection.}
}
It may use literals, functions, and the defs of other modules. A macro that puts its caller's code inside the form passes these rules on to its caller, so keep the form to parts the macro built itself.
@subsubsection{One expansion for both colours: by-colour}
A macro does not know if its expansion will land in a bjoroutine or in ordinary code. When the two need different code, the expansion writes @code{(by-colour (#:sync body ...) (#:bjo body ...))}, and the compiler keeps the body for the colour of the code the expansion is in. This is how @code{with-scope} works in both colours: it ends the scope with a suspending wait in a bjoroutine and with a parking wait in ordinary code.
@codeblock[#:lang "bjolang"]|{
;; (await ev) syncs on ev, in code of either colour.
(def/macro (await form inject compare)
(syntax-match form
((_ ev) #'(by-colour (#:sync (sync/blocking ,ev))
(#:bjo (sync ,ev))))))
}|
The colour is the colour of the innermost of these:
@read-table{
| Where | Body |
|------------------------------------------------------------------------+--------------------|
| @code{defun}, @code{(fun ...)}, @code{(seq ...)}, top-level @code{def} | @code{#:sync} |
| @code{defbjo}, @code{(bjoroutine ...)}, @code{(bjo ...)} | @code{#:bjo} |
| @code{defbjouble} | the body it is in |
| a @code{defun} inside a body | the body around it |
}
A @code{defun} with a @code{by-colour} in it gets a suspending copy, as a @code{defun} that calls @code{file-read-text} does, and a bjoroutine calls that copy. If the @code{#:bjo} body never suspends, the copy is dropped, and the @code{#:sync} body runs everywhere. So write a @code{#:bjo} body that suspends and a @code{#:sync} body that does the same work without suspending.
A lambda applied where it is written, @code{((fun () body))}, is replaced by its body before colours are found. A @code{by-colour} in it has the colour of the code around it.
@subsubsection{Macro limitations}
When designing macros, keep the following constraints in mind:
@read-list{
- Macros cannot be invoked within the module that defines them; they must be imported from a separate module.
- Macros can only appear in expression or definition positions (or in pattern positions when defined with @code{def/pattern}). They cannot be used as loop clauses, let bindings, or type annotations.
- Macros cannot override built-in special forms; a macro named @code{if} will never shadow the compiler primitive.
- Macros cannot inspect types, as macro expansion occurs prior to type checking.
- Macros are not first-class values and cannot be passed as function arguments.
- Macros cannot inject imports into the calling module; any imported dependencies must be declared explicitly in the consuming file.
}
@subsection{Testing and the REPL}
@subsubsection{Tests with (std simpletest)}
Bjolang includes a lightweight test harness in @code{(std simpletest)}. Its core assertions are straightforward: @code{(expect label actual wanted)} checks equality between values, @code{(expect-true label ok)} asserts a boolean condition, and @code{(fail label)} explicitly flags an unreachable branch. Passing checks output @code{ok: label}, while test failures emit descriptive @code{FAILURE:} diagnostics:
@codeblock[#:lang "bjolang"]|{
;; tests/deck-test.bjo
(import (std simpletest) (cardlib deck))
(defun (main)
(expect "a deck has 52 cards" (vec-length (full-deck)) 52)
(expect "an ace is worth 11" (card-points (Card (rank Ace) (suit Spades))) 11)
(expect-true "shuffling keeps the cards" (= (vec-length (shuffle (full-deck))) 52))
0)
}|
@codeblock[#:lang "text"]|{
$ bjo run tests/deck-test.bjo
ok: a deck has 52 cards
ok: an ace is worth 11
ok: shuffling keeps the cards
}|
Whenever an assertion fails, the output reports both the computed value and the expected value: @code{FAILURE: deliberately wrong gave 2 but wanted 3}.
For asynchronous or resource-intensive testing, @code{(with-test "name" body ...)} executes the test body within an isolated structured concurrency scope configured with a default deadline. Any background fibers spawned during the test are automatically awaited, and the test fails if unclosed file ports remain open when the scope exits. Because @code{with-test} forms a concurrency scope, it must be invoked from within a bjoroutine.
You can also mock filesystem interactions using @code{(with-fake-fs ((path contents) ...) body ...)}, which redirects file operations to an in-memory map for the duration of the enclosed block. Here is how we can verify our earlier high-score tracking functions without touching real files on disk:
@codeblock[#:lang "bjolang"]|{
(import (std simpletest) (std effect) "scores.bjo")
(defbjo (main)
(with-test "no file means no scores"
(with-fake-fs ()
(expect "empty" (read-scores) [])))
(with-test "a bad line is skipped"
(with-fake-fs (("scores.txt" "ada 31\nnonsense\nbo 27\n"))
(expect "two scores" (read-scores) [(Tuple "ada" 31) (Tuple "bo" 27)])))
(with-test "a saved score reads back"
(with-fake-fs (("scores.txt" "ada 31\n"))
(save-score "cy" 12)
(expect "appended" (vec-length (read-scores)) 2)))
0)
}|
Because @code{with-fake-fs} is implemented via algebraic effects, test modules using it must also import @code{(std effect)}. In standard project layouts, test files live in @code{tests/} and can be executed individually using @code{bjo run tests/<test-name>.bjo}.
@subsubsection{The REPL}
You can launch an interactive Read-Eval-Print Loop by running @code{bjo repl}. When invoked inside a project directory, the REPL automatically discovers and allows importing any of the project's local modules. At a terminal the REPL edits its own lines and keeps a history across sessions. Pressing Enter while a bracket is still open starts a new line, so a long form can be written and edited over several lines. Submitting an expression evaluates it and prints the result, while entering definitions compiles them and prints the bound names:
@codeblock[#:lang "text"]|{
bjo> (+ 1 2)
3
bjo> (import "deck.bjo")
bjo> (vec-slice (full-deck) 0 3)
[2♣ 3♣ 4♣]
bjo> (defun (double (: x int)) : int (* x 2))
double
bjo> (double 21)
42
}|
Unlike many Lisp REPLs, Bjolang's REPL does not interpret forms. Instead, each entered interaction is compiled on the fly into an in-memory assembly using the exact same compiler pipeline applied to files on disk, ensuring identical semantics between interactive sessions and compiled production builds.
Top-level functions declared in the REPL require explicit type signatures, mirroring the rules for modules. You can supply parameter and return types inline directly on the @code{defun}, or submit a standalone signature on its own line immediately preceding the definition:
@codeblock[#:lang "text"]|{
bjo> (: triple (-> int int))
bjo> (defun (triple x) (* x 3))
triple
}|
Redefining a symbol shadows the previous definition for subsequent interactions; however, any existing code already compiled against the earlier version continues to invoke the original binding:
@codeblock[#:lang "text"]|{
bjo> (defun (f (: x int)) : int (+ x 1))
f
bjo> (defun (g (: x int)) : int (f (f x)))
g
bjo> (defun (f (: x int)) : int (* x 100))
note: f shadows the one from entry 1. Anything already compiled against that one still calls it.
f
bjo> (g 10)
12
}|
The same incremental compilation model applies to trait implementations (@code{impl}): an implementation entered at the prompt is visible to subsequent entries, but cannot retroactively modify calls compiled earlier.
@code{:show name} prints the documentation of a name you can use at the prompt: what it is, where it comes from, how it is called, and the text its author wrote in a @code{(:doc ...)} form. It then lists the other usable names containing the text, in any case, with their modules and one-line summaries. Only names that an entry typed now could use are considered, that is builtins, names your imports bring in, and names defined at the prompt. So @code{:show random-bool} prints nothing until @code{(std random)} is imported:
@codeblock[#:lang "text"]|{
bjo> :show random-bool
bjo> (import (std random))
bjo> :show random-bool
random-bool: function from (std random)
(random-bool #:rng (parameter-ref current-random))
: (-> (#:rng RandomSource) bool)
#t or #f, equally likely.
#:rng The source to draw from.
returns One random bit. They are drawn 64 at a time.
See also: random-chance?
bjo> :show random-u
random-u32 (std random) 32 random bits.
random-u64 (std random) 64 random bits.
}|
Import modifiers apply as they would in an entry: after @code{(import (prefix (std random) "r/"))} the name to ask about is @code{r/random-bool}, and @code{:show} says that it is @code{random-bool} from @code{(std random)}. A name with no documentation is shown with where it comes from and "with no doc".
Tab completes the name at the cursor from the same names, together with the special forms and the names the entry you are typing has already bound. When only one name fits it is inserted, and when several do, the part they share is inserted and pressing Tab again opens a menu showing each name's signature and summary. Inside an import, Tab completes module paths instead: @code{(import (std ra} becomes @code{(import (std random}.
The same completions are available as a command, @code{:complete text}, which prints the names that could be written at the end of @code{text}, one per line followed by where each comes from. It is meant for editors rather than people: an editor can send the entry up to the cursor and offer what comes back.
@codeblock[#:lang "text"]|{
bjo> :complete (random-in
random-int (std random)
}|
Type @code{:help} to view available REPL commands, or press @code{Ctrl-D} (or type @code{:quit}) to exit.
@subsection{Benchmarking}
Sooner or later a program is fast enough everywhere except one place, and the question becomes how fast that place is, and whether a change made it better. Timing it with a stopwatch gives a number, but not one you can trust: the first calls run code the JIT has not optimized yet, a garbage collection can land in the middle of a measurement, and two runs of the same code on a busy laptop differ by more than most improvements do.
@code{(std bench)} measures in a way that holds up against all of that, and says how sure it is. There are three ways in:
@read-list{
- @code{time-it}, for a quick answer at the REPL: one line, measured properly.
- Benchmark files, with @code{def/bench} and @code{(run-benches)}, for the code you care about: a table, a menu, saved runs to compare with later.
- @code{bjo bench FILE --against REF}, for the question "did my change make this slower": the file run against itself at another commit, the two versions measured in turns.
}
All three are measured the same way, which the section "How a benchmark is measured" below describes.
@subsubsection{time-it}
@code{time-it} measures a body where you write it and prints one line. It is in @code{(std bench)}, and @code{(std stopwatch)} has it too:
@codeblock[#:lang "text"]|{
bjo> (import (cardlib deck))
bjo> (import (std bench))
bjo> (time-it (shuffle (full-deck)))
(shuffle (full-deck)): 1.06 µs ± 0.7% (min 1.05 µs), 3.64 KB/call, 67 samples
}|
The line gives the median time per call, its spread (the median absolute deviation, as a percentage of the median), the fastest sample, how much the body allocated per call, and how many samples it took. The body is warmed up, calibrated and sampled just as a benchmark is, with shorter time targets, so it is done in a second or two.
A body that takes more than a second the first time is not run again. That first run is the measurement, and the line says so. Here the body is a slow recursive Fibonacci:
@codeblock[#:lang "text"]|{
(fib 44): 1.46 s (one cold run: a body over a second is run once)
}|
@code{#:times n} and @code{#:warm-up-rounds w} ask for an exact number of runs instead: @code{w} untimed runs, then @code{n} timed ones, with no calibration. Use them when the body has an effect that has to happen a known number of times:
@codeblock[#:lang "text"]|{
bjo> (time-it #:times 1000 (shuffle (full-deck)))
(shuffle (full-deck)): 3.82 µs (min 3.02 µs), 3.64 KB/call, 1000 runs
}|
The median here is more than three times the first one: without a warm-up, the early runs are the JIT's unoptimized code. That is the reason to let @code{time-it} decide how often to run a body.
@subsubsection{A benchmark file}
For code you want to keep measuring, write a benchmark file. Benchmarks belong in @code{benches/} in a project, next to @code{src/} and @code{tests/}. Here is one for the deck from @code{cardlib}, in @code{benches/deck.bjo}:
@codeblock[#:lang "bjolang"]|{
(import (cardlib deck))
(import (std bench))
(def/bench "full-deck" (full-deck))
(def/bench "shuffle"
#:setup (full-deck)
(fun (deck) (shuffle deck)))
(def/bench "hand points"
#:params ((size [5 13 52]))
#:setup (fun (size) (vec-slice (shuffle (full-deck)) 0 size))
(fun (hand) (vec-fold (fun (c total) (+ total (card-points c))) 0 hand)))
;; Sorting changes the deck it sorts, so every call gets a freshly
;; shuffled one, made outside the time.
(def/bench "sort a shuffled deck"
#:setup-every (vec->list (shuffle (full-deck)))
(fun (cards) (list-sort cards)))
(def/bench "deal with vec-slice" #:group "dealing"
#:setup (shuffle (full-deck))
(fun (deck) (vec-slice deck 0 5)))
(def/bench "deal with a loop" #:group "dealing"
#:setup (shuffle (full-deck))
(fun (deck) {vecing (vec-ref deck i) (:for i (range 0 5))}))
(def/bench "a million hands" #:once 3
(loop (:for i (up-from 0 #:to 1000000))
(:acc total (summing (card-points (vec-ref (full-deck) (% i 52)))))
=> total))
(def/bench "deal from a stacked deck" #:skip "the cheat is not written yet"
(full-deck))
(run-benches)
}|
It is laid out just like a test file. @code{def/bench} registers a benchmark where it is written, and @code{(run-benches)} at the end becomes the file's @code{main}. @code{bjo bench} builds every file under @code{benches/} and runs it:
@codeblock[#:lang "text"]|{
$ bjo bench --all
bench: 10 benchmarks, Release build, .NET 10.0.4, X64, 24 cores, gcServer=false
bench: AMD Ryzen 9 5900X 12-Core Processor, Fedora Linux 43
name size median ± min alloc/op gc/1k 0/1/2
full-deck 309 ns 0.3% 307 ns 1.70 KB 0.20/0/0 1 outlier
shuffle 558 ns 0.3% 555 ns 1.94 KB 0.23/0/0 4 outliers
hand points 5 4.67 ns 1.9% 4.52 ns 0 B - 9 outliers
hand points 13 11.3 ns 1.4% 10.7 ns 0 B - 7 outliers
hand points 52 46.2 ns 0.5% 45.5 ns 0 B - 1 outlier
sort a shuffled deck 3.63 µs 0.5% 3.60 µs 3.30 KB 1.00/0/0 9 outliers
deal with vec-slice 19.4 ns 0.2% 19.3 ns 152 B 0.02/0/0 3 outliers
deal with a loop 51.7 ns 0.7% 51.2 ns 656 B 0.08/0/0 4 outliers
a million hands 320 ms - 320 ms 1.62 GB 208/0/0
(324 ms, 320 ms, 320 ms)
deal from a stacked deck skipped: the cheat is not written yet
}|
The first two lines say what the numbers were measured on, because a time is a fact about one machine. A Debug build, an attached debugger and a CPU frequency governor other than @code{performance} add a warning there. @code{bjo bench} always builds in Release, whatever @code{-d} says, since a Debug build measures code the JIT did not optimize.
The columns are:
@read-table{
| Column | What it says |
|----------------------+--------------------------------------------------------------------------------------------------|
| @code{name}, @code{size} | the benchmark, and a column for each of its parameters |
| @code{median} | the median time per call. Not the mean: one collection can move a mean anywhere |
| @code{±} | the median absolute deviation, as a percentage of the median: how much the samples agree |
| @code{min} | the fastest sample, which is what a quiet machine converges to |
| @code{alloc/op} | the bytes allocated per call |
| @code{gc/1k 0/1/2} | the collections of each generation per thousand calls (per call for a @code{#:once} benchmark) |
}
After the columns come the marks. @code{noisy} means a spread above 3%, @code{unsteady} means the warm-up ran out of time before the times settled, and outliers are samples far outside the rest. They are counted and shown, never dropped: a pause that happens once in fifty calls is part of what the call costs.
Each row is printed as soon as its benchmark is measured, so a long run can be read as it goes. At a terminal, a line below the table says where the run is: which benchmark, which phase, the median so far and an estimate of the time left. It is rewritten in place, and is left out when the output goes to a pipe or a file, under @code{CI}, and in an Emacs buffer.
@subsubsection{How a benchmark is measured}
You do not have to know this to use @code{(std bench)}, but it explains its numbers.
@read-list{
- @strong{The body is kept.} The value of the body's last form is written to a sink the JIT cannot remove. A body that computes a value and throws it away is therefore not optimized away, and the sink allocates nothing.
- @strong{Calibration.} The body is run in batches of @code{k} calls, and @code{k} grows until a batch takes 10 ms. One sample is one batch, so the clock's resolution does not matter even for a body of a few nanoseconds.
- @strong{Warm-up.} Batches are run until the medians of the last two windows of five batches agree within 2%, for at least half a second and at most one. The minimum is there because of the JIT. .NET recompiles a hot method into faster code only after a pause with no new compilation, and a body can look steady at the first, slow tier before the faster code arrives. A warm-up that runs out of time leaves the row marked @code{unsteady}.
- @strong{Samples.} Before each sample a collection runs, so that one benchmark's garbage is not collected in another one's time. At least 20 samples are taken, for at least 2 s. While the median's 95% interval is wider than ±1%, it takes more, up to 100 samples or three times the time. A noisy benchmark costs more time, and gets a more precise median for it.
}
Every benchmark of a file runs in the file's one process, one after another, so what one benchmark compiled is there for the next. That is usually what you want: it is how the code runs in a program.
@subsubsection{Setup, parameters, and the rest}
The body of a benchmark should be the thing you want to measure and nothing else. The options of @code{def/bench} are about keeping everything else out of the time:
@read-table{
| Option | What it does |
|------------------------------+--------------------------------------------------------------------------------------------------------|
| @code{#:setup e} | runs @code{e} once, outside the time; the body is then a function of what it made |
| @code{#:setup-every e} | runs @code{e} before every call, outside the time; for a body that uses up its input |
| @code{#:params ((p values) ...)} | a row for each combination of values; the body, or the setup, is a function of the parameters |
| @code{#:once n} | for a body of seconds: @code{n} cold runs, timed one at a time, no warm-up |
| @code{#:group "g"} | a heading in the menu, and what @code{--compare-within} compares within |
| @code{#:skip "why"} | not run; the reason is shown |
}
@code{#:setup} is the one to reach for first. In the example, @code{shuffle} measures shuffling a deck, not making one: the deck is made once, before the warm-up, and every call shuffles that same deck.
@code{#:setup-every} is for a body that changes or uses up what it is given, such as a sort that sorts in place, or a body that consumes a list. Each call gets a fresh input, and each call is then timed on its own, with its setup outside the time and outside the allocation. Timing calls one at a time has a cost, because the clock is read twice per call. A body that takes less than a microsecond is marked, since its time is close to the clock's resolution.
@code{#:params} gives the parameters' names and values, like a @code{let}. The values are a vec of any expressions, evaluated once. There is a row for every combination, in the order written, with the last parameter varying fastest. Without a setup, the body is a function of the parameters. With one, the setup is, and the body gets what the setup made:
@codeblock[#:lang "bjolang"]|{
(def/bench "sort"
#:params ((n [1000 100000])
(order ['sorted 'reversed 'random]))
#:setup (fun (n order) (make-cards n order))
(fun (cards) (list-sort cards)))
}|
Here @code{make-cards} stands for a function of your own that makes @code{n} cards in the given order.
Each combination is a benchmark of its own for everything else: it is warmed up, calibrated, saved and compared on its own. To try a size the file does not list, give it on the command line: @code{--params n=500,5000} replaces the values of @code{n} for that run. A value is read as the parameter's values are typed: an @code{int}, a @code{double}, a string, a symbol or a @code{bool}. Implementing the trait @code{BenchParam} lets other types be read too.
@code{#:once n} is for bodies too slow to run in batches. There is no calibration and no warm-up, just @code{n} runs timed one at a time, the first of them cold. Each time is shown under the row, as for @code{a million hands} above, and the median is the row's.
@subsubsection{Choosing what runs}
Names on the command line choose benchmarks. A benchmark runs when its name contains one of them, and a parameter written the way the table shows it chooses rows:
@codeblock[#:lang "text"]|{
bjo bench benches/deck.bjo shuffle # shuffle, and sort a shuffled deck
bjo bench benches/deck.bjo "hand points" size=52
bjo bench --list # the names, and nothing run
}|
Note that @code{shuffle} also matches @code{sort a shuffled deck}: a name is a part of a name.
With no names, at a terminal, a benchmark file asks what to run:
@codeblock[#:lang "text"]|{
$ bjo bench
Benchmarks in deck.bjo:
#0 full-deck
#1 shuffle
#2 hand points size: 5 13 52
#3 sort a shuffled deck
dealing
#4 deal with vec-slice
#5 deal with a loop
#6 a million hands
#7 deal from a stacked deck
Choose by number (0 2-4), by name (sort), with parameters (2 n=1000),
or * for all. Nothing quits, or repeats the last choice once there is one.
bench> 2 size=5,52
}|
A choice is numbers, ranges of numbers (@code{2-4}), parts of names, or @code{*} for everything, in any mix and separated by spaces. A parameter after a benchmark narrows that benchmark to some of its values: @code{2 size=5,52} runs @code{hand points} at sizes 5 and 52 only. The groups and the parameters' values are shown, so a choice can be made without opening the file.
After a run the menu asks again. Pressing Enter repeats the last choice, so you can change the code in another terminal and run the same benchmarks again with one key. @code{q} or the end of the input leaves.
With several files and no names, @code{bjo bench} makes one menu of all of them, numbered across the files under a heading for each, and runs each file with the part of the choice that is its own. Not at a terminal, when the output is redirected or @code{CI} is set, there is no menu: a run with no names runs everything, as @code{--all} does.
@subsubsection{More numbers}
@code{--detail} adds three lines under each row: the mean, the standard deviation and the quartiles; the samples in the order they were taken; and a histogram of them:
@codeblock[#:lang "text"]|{
$ bjo bench benches/deck.bjo "hand points" --detail
name size median ± min alloc/op gc/1k 0/1/2
hand points 5 6.50 ns 6.6% 5.21 ns 0 B - noisy, 10 outliers
mean 6.69 ns, sd 1.02 ns, q1 6.04 ns, q3 6.90 ns, 100 samples of 1600000 calls
in order ██████▄▂▃▃▃▃▃▂▂▃▁▃▃▃▃▁▂▂▄▂▂▃▃▃▃▂▁▂▄▄▂▂▂▃▃▃▃▃▃▂▂▅▄▁▂▃▃▁▁▃▃▅▂▃
histogram 5.21 ns ▃▂▅▄▄▆█▇▅▂▃▂▂▂ ▁ ▃▂▂▂ 9.64 ns
}|
The "in order" line is the one that explains a strange row. Here the first six samples were slower than the rest, and the spread of 6.6% and the @code{noisy} mark come from those six; the other 94 agree closely. A second run tells whether it was the machine or the code: run again, the same benchmark had a spread of 1.9%. A line that rises from left to right is a body that slows down as it runs, for example because its heap grows. The histogram shows the shape: two humps are two different speeds, not one noisy speed.
@code{--scaling} adds a column to a benchmark with one numeric parameter: how much the time grew from the row before, against how much the parameter grew.
@codeblock[#:lang "text"]|{
$ bjo bench benches/deck.bjo "hand points" --scaling
name size median ± min alloc/op gc/1k 0/1/2 scaling
hand points 5 4.94 ns 5.4% 4.47 ns 0 B - noisy
hand points 13 9.99 ns 2.7% 8.41 ns 0 B - ×2.02 for ×2.60
hand points 52 41.5 ns 0.7% 40.5 ns 0 B - ×4.16 for ×4.00 3 outliers
}|
From 13 to 52 cards the time grew by as much as the hand did, which is what a linear function does. From 5 to 13 it grew less than the hand, because a few nanoseconds of every call are not spent on the cards at all, and with five cards those nanoseconds are a large part of the time. A column that keeps growing faster than the parameter is the sign of something quadratic.
@subsubsection{Which of two ways is faster}
"Is dealing with @code{vec-slice} faster than with a loop, and by how much?" is a question about two benchmarks in the same file. Put them in a @code{#:group}, and run with @code{--compare-within}:
@codeblock[#:lang "text"]|{
$ bjo bench benches/deck.bjo deal --compare-within
name median ± min alloc/op gc/1k 0/1/2
deal with vec-slice 20.0 ns 0.9% 19.0 ns 152 B 0.02/0/0 2 outliers
deal with a loop 47.3 ns 0.6% 46.8 ns 656 B 0.08/0/0 14 outliers
deal from a stacked deck skipped: the cheat is not written yet
bench: within dealing, against deal with vec-slice: 20 rounds in this process, paired (Wilcoxon signed-rank); 1 comparison
bench: seed 11 (--seed repeats the order of the cases)
name deal with vec-slice this change 95% interval p ratio verdict
deal with a loop 19.5 ns 47.0 ns +140.9% +140.0% .. +141.7% <0.001 ×2.41 SLOWER
}|
After the usual table, the benchmarks of each group are measured again in rounds: every round takes one sample of each, in an order shuffled anew each round. Each one is then compared with the first of its group. Taking turns is what makes the comparison fair. If the machine gets slower halfway through, both sides get slower in the same rounds, and the comparison, which pairs the samples of a round, does not move.
The ratio is the number to quote. "A loop is 2.4 times as slow as @code{vec-slice}" is roughly true on the next machine too, where "47 ns" is not. @code{-N 50} takes 50 rounds rather than 20, and @code{--seed 11} repeats the order of an earlier run. The columns of the comparison, and how a verdict is reached, are explained in the next section.
@subsubsection{Saving a run and comparing with it}
@code{--save FILE} keeps a run: every sample of every benchmark, with the facts of the machine it was taken on. @code{--compare FILE} measures again and compares with what was kept. Nothing is saved unless you ask, and nothing is saved anywhere you did not name. Where a baseline lives, and whether it is committed, is up to you.
@codeblock[#:lang "text"]|{
$ bjo bench benches/deck.bjo full-deck shuffle "hand points" --save before.bjodat
...
bench: saved to before.bjodat
}|
The file is bjodat, a benchmark to a line, so a diff of two saved runs shows which benchmarks changed:
@codeblock[#:lang "text"]|{
(bench-run
(version 1)
(saved "2026-10-16T17:25:06Z")
(machine "95c59c6948245e2e")
(header (build "Release") (dotnet ".NET 10.0.4") (arch "X64") (cores "24") ...)
(case (name "full-deck") (labels) (k 60000) (once #f) (bytes 1744.0) (samples [371.62 312.41 ...]))
...)
}|
Now suppose @code{full-deck} is rewritten to collect the cards in a list and turn the list into a vec at the end:
@codeblock[#:lang "bjolang"]|{
(defun (full-deck)
(list->vec
(loop (:for s all-suits)
(:subloop)
(:for r all-ranks)
(:acc deck (listing (Card (rank r) (suit s))))
=> deck)))
}|
The comparison prints the run's own table, and then:
@codeblock[#:lang "text"]|{
$ bjo bench benches/deck.bjo full-deck shuffle "hand points" --compare before.bjodat
...
bench: compared with before.bjodat, saved 2026-10-16T17:25:06Z; unpaired (Mann-Whitney U); 6 comparisons, Holm-corrected
bench: one process each: a difference between processes is not in the p-values
name size was now change 95% interval p verdict
full-deck 311 ns 597 ns +92.1% +91.7% .. +92.5% <0.001 SLOWER
shuffle 623 ns 548 ns -12.0% -12.0% .. -11.8% <0.001 faster
hand points 5 4.58 ns 5.07 ns +10.7% +10.1% .. +11.3% <0.001 SLOWER
hand points 13 10.6 ns 10.2 ns -4.2% -5.0% .. -3.7% <0.001 faster
hand points 52 44.2 ns 44.7 ns +1.0% +0.4% .. +1.4% <0.001 no change (< 2%)
sort a shuffled deck 3.68 µs 3.92 µs +6.8% +6.6% .. +7.0% <0.001 SLOWER
}|
Each benchmark gets four answers:
@read-list{
- @strong{The change} is the ratio of the two medians, as a percentage.
- @strong{The interval} is a bootstrap of that ratio: the samples are resampled 10 000 times, and the middle 95% of the resampled ratios is shown. It makes no assumption about the shape of the timings, which are lopsided and have long tails.
- @strong{The p-value} is the probability of a difference at least this large if nothing had changed, from the Mann–Whitney U test. A run that compares many benchmarks at once will show some differences by chance alone: with 40 benchmarks at a level of 5%, two are expected to look different when nothing changed. So the p-values are corrected for the number of comparisons, by Holm's method, and the corrected value is the one shown and used. @code{--p-raw} shows the uncorrected values beside them.
- @strong{The verdict} is @code{faster} or @code{SLOWER} only when the corrected p is below 0.05 (@code{--alpha}) @em{and} the change is larger than 2% (@code{--threshold}). A change that is real but smaller is @code{no change (< 2%)}: it is there, and it was decided not to matter. Anything else is @code{no change}.
}
The reason for asking for both is that a p-value says how unlikely a difference is, not how large it is: with enough samples, a change of 0.1% has a tiny p.
Now look at the table again. @code{full-deck} is 92% slower, as it should be: that is the change. But @code{shuffle} did not change at all, since its deck is made in its setup, outside the time. Yet it says @code{faster}, by 12%. The rows for @code{hand points} and the sorting moved too, and they never call @code{full-deck} in their time at all.
This is the weakness of comparing with a saved run. The two runs are two processes, and @strong{two processes of the same program can differ by a few percent}, in every benchmark at once. Where the code ends up in memory, which core the process runs on, and what the JIT decided all differ from one process to the next. Within one run the samples agree to a fraction of a percent, so the test is very sure about a difference that is really between the processes. The report says this in its second line. Treat a saved-run comparison as a tool for large changes, like @code{full-deck}'s. For small ones, use @code{--against} with several process pairs, as described next.
A saved run from another machine is still compared, but under a warning above the table, and every verdict is marked @code{(other machine)}: the numbers are shown, but no verdict is to be believed. Differences in the build, the .NET version, the garbage collector's mode or the CPU governor are warned about too.
@code{--fail-slower PCT} makes the run exit with 1 when anything is @code{SLOWER} by more than @code{PCT}, for a CI job that guards a hot path. It does not apply against a run from another machine. A file that cannot be read stops the run with exit code 2 before anything is measured.
@subsubsection{Two versions against each other: @code{--against}}
The real question behind most benchmarking is "did my change make this slower?". @code{--against} answers it in one run. It builds the benchmark file as it is and as it was at a git ref, and measures the two versions in turns:
@codeblock[#:lang "text"]|{
$ bjo bench benches/deck.bjo full-deck shuffle "hand points" --against HEAD~1
bench: 6 benchmarks, Release build, .NET 10.0.4, X64, 24 cores, gcServer=false
bench: AMD Ryzen 9 5900X 12-Core Processor, Fedora Linux 43
bench: A/B against HEAD~1, 20 rounds, paired (Wilcoxon signed-rank); 6 comparisons, Holm-corrected
bench: one process per side: a difference between processes is not in the p-values (--processes P measures it)
bench: seed 42 (--seed repeats the order of the cases)
name size HEAD~1 now change 95% interval p verdict
full-deck 360 ns 649 ns +80.2% +73.9% .. +93.9% <0.001 SLOWER
shuffle 659 ns 616 ns -6.5% -63.1% .. +2.4% 0.097 no change
hand points 5 5.60 ns 5.46 ns -2.4% -5.5% .. +1.5% 0.046 faster
hand points 13 11.5 ns 12.8 ns +11.0% +5.7% .. +17.6% <0.001 SLOWER
hand points 52 47.2 ns 50.3 ns +6.6% +4.7% .. +10.9% <0.001 SLOWER
sort a shuffled deck 4.09 µs 4.14 µs +1.3% -4.5% .. +8.4% 0.50 no change
}|
This is what happens:
@read-list{
- @code{REF} is checked out in a temporary git worktree, which is removed afterwards, also when something fails. It can also be a path: another checkout, or another @code{.bjo} file.
- Both sides are built in Release, by the same compiler. Both programs are started, and @code{bjo bench} drives them over a small protocol on their standard input and output.
- Every benchmark both sides have is warmed up and calibrated on both, with the smaller of the two batch sizes, so that a sample of each is the same number of calls.
- Then come @code{-N} rounds, 20 by default. Each round takes one sample of every benchmark from each side, in an order shuffled from @code{--seed}, which is printed so that a run can be repeated. Which side goes first alternates every round, so that neither side always runs on a cache the other one warmed.
- Each round's two samples are a pair. The p-value is the Wilcoxon signed-rank test on the pairs, and the interval a bootstrap that resamples whole rounds. If the machine slows down during the run, both samples of a round slow down, and the comparison does not move.
}
A benchmark that only one side has is listed, not compared.
Look at the table, though. Rounds cancel a slowdown @em{during} the run, but not the difference between the two processes as a whole, and that difference is still there: @code{hand points} is called @code{SLOWER} by 11% at size 13, and it has not changed. The report says so in its fourth line. This is why there is @code{--processes}:
@codeblock[#:lang "text"]|{
$ bjo bench benches/deck.bjo full-deck shuffle "hand points" --against HEAD~1 -N 60 --processes 6
bench: A/B against HEAD~1, 60 rounds, 6 process pairs (t-test on their log ratios); 6 comparisons, Holm-corrected
bench: seed 42 (--seed repeats the order of the cases)
name size HEAD~1 now change 95% interval p between processes verdict
full-deck 344 ns 614 ns +81.2% +72.4% .. +90.4% <0.001 4.7% SLOWER
shuffle 596 ns 585 ns -2.1% -10.3% .. +6.9% 0.73 8.3% no change
hand points 5 5.55 ns 5.13 ns -9.6% -18.7% .. +0.6% 0.24 10.1% no change
hand points 13 12.1 ns 11.4 ns -7.6% -20.8% .. +7.7% 0.73 14.7% no change
hand points 52 45.2 ns 45.0 ns -3.1% -10.5% .. +4.9% 0.73 7.6% no change
sort a shuffled deck 3.91 µs 4.04 µs +3.6% +0.5% .. +6.7% 0.15 2.8% no change
}|
With @code{--processes 6}, the rounds are spread over six pairs of processes. Each pair gives one ratio of medians, and the test is a t-test on the six ratios, so a difference between processes is now part of what is tested rather than hidden from it. The "between processes" column says how large it was. For these tiny benchmarks it was 3 to 15%, more than any of the "changes" the single pair reported. Now only @code{full-deck} gets a verdict, which is the right answer.
So which to use?
@read-table{
| Question | Use |
|----------------------------------------------------------+-------------------------------------------------------------|
| is it about this fast | @code{time-it}, or a benchmark file |
| which of two ways is faster, and by how much | @code{#:group} and @code{--compare-within}: one process |
| did my change make something a lot slower | @code{--against REF}, or @code{--compare} with a saved run |
| did my change make something a few percent slower | @code{--against REF --processes 6} or more, and more rounds |
}
@code{--against} has a few more options. @code{--against-warmup S} and @code{--against-sample MS} set the warm-up limit and the sample length of both sides. A warm-up much shorter than the default second can mistake the JIT's first tier for steady, and a benchmark that did not get steady is named in a warning. @code{--against-build "CMD"} builds the other side with a command of your own, for a side that is not built the way this one is. @code{--save FILE} keeps both sides' samples: this code's in @code{FILE}, and the other side's beside it, with @code{.against} before the extension. @code{--fail-slower}, @code{--alpha}, @code{--threshold} and @code{--p-raw} work as they do for @code{--compare}.
A CI runner is one machine for the length of one job, which is exactly what @code{--against} needs. A job that guards performance runs @code{bjo bench --against origin/main --processes 6 --fail-slower 5} and fails when something got more than 5% slower.
@subsubsection{The flags}
@read-table{
| Flag | What it does |
|----------------------------+-----------------------------------------------------------------------------|
| @code{NAME} | runs the benchmarks whose names contain it; @code{n=10} chooses rows |
| @code{--all} | runs every benchmark, without the menu |
| @code{--list} | names the benchmarks and their parameters, and runs nothing |
| @code{--quick}, @code{--long} | halves or quadruples the time targets |
| @code{--min-samples N} | takes at least @code{N} samples (20) |
| @code{--max-time S} | lets a noisy benchmark sample for up to @code{S} seconds |
| @code{--params P=V,...} | replaces a parameter's values for the run |
| @code{--detail} | the mean, the deviation, the quartiles, the samples in order, a histogram |
| @code{--scaling} | how the time grows with a numeric parameter |
| @code{--compare-within} | compares each @code{#:group}'s benchmarks with its first, in rounds |
| @code{-N ROUNDS} | the rounds of @code{--compare-within} and @code{--against} (20) |
| @code{--seed N} | repeats the order of an earlier run's rounds |
| @code{--save FILE} | writes every sample of the run to @code{FILE} |
| @code{--compare FILE} | compares with the run saved in @code{FILE} |
| @code{--alpha P} | the significance level of a verdict (0.05) |
| @code{--threshold PCT} | the smallest change a verdict counts (2%) |
| @code{--p-raw} | shows the uncorrected p-values too |
| @code{--fail-slower PCT} | exits with 1 when something is @code{SLOWER} by more than @code{PCT} |
| @code{--against REF} | runs the file against itself at a git ref, or against a path |
| @code{--processes P} | with @code{--against}: @code{P} pairs of processes (1) |
| @code{--against-warmup S} | with @code{--against}: the warm-up limit of both sides (1 s) |
| @code{--against-sample MS} | with @code{--against}: the sample length of both sides (10 ms) |
| @code{--against-build CMD} | with @code{--against}: builds the other side with @code{CMD} |
}
A percentage can be written @code{2} or @code{2%}. A benchmark file is an ordinary program, so @code{bjo run benches/deck.bjo --quick shuffle} runs it just as well. @code{bjo bench} adds the Release build, the files under @code{benches/}, the menu over several files and @code{--against}.
@subsubsection{Driven by another program}
@code{--against} works because a benchmark program can be driven. Started with @code{--serve}, it reads commands on its standard input and answers each with one line of bjodat on its standard output. Below, @code{>} marks what is sent and @code{<} what comes back:
@codeblock[#:lang "text"]|{
> (list)
< (cases (case 0 (name "full-deck") (labels) (group "") (once 0) (skip "")) (case 1 (name "shuffle") (labels) ...) ...)
> (prepare 1 :warmup 1000 :sample 10)
< (prepared 1 :k 20000 :steady #t)
> (sample 1 :k 20000)
< (sample 1 :ns 13663495 :bytes 39680000 :gc (4 0 0))
> (sample 9 :k 1)
< (error "case 9 is not prepared")
> (quit)
}|
@code{prepare} runs a benchmark's setup and warms it up and calibrates it, as a run would. @code{sample} takes one sample of @code{k} calls, and @code{:ns} is the whole sample's time. Anything wrong is answered with @code{(error "why")}, and the session goes on. A served program prints nothing else, so its output is the protocol and nothing more. A script that compares two builds made by two different compilers, which is not something @code{bjo bench} does by itself, can drive both of them this way, or use @code{--against PATH --against-build "CMD"}.
@subsection{Performance}
Bjolang is compiled into pretty straight-forward c#, but without spans or ref structs. It performs pretty similar to c#, except for some small corners of the language that are different.
@subsubsection{Strings}
Strings in bjolang are utf8, as such they do @em{not} support random access. This means we navigate them differently, using string-cursors, which is explained in the part about strings. Navigating a string using cursors is slower than blasting through integer indices, and if your code needs to be really fast, there are some things you can think about:
@read-list{
- Use the provided search functions and the loop form. They work on raw memory, and can thus work around some of the limits that the safety of string-cursor give.
- use @code{string-cursor-next+ref} instead of next and then ref.
- some of the higher order functions are also faster, depending on use case. Benchmark and see.
- Use slices when copying is the bottleneck: an operation ending with @code{/slice} will return a view into the string instead of copying. This will retain the whole string in memory, so use it wisely.
}
@subsubsection{loops}
Most of the time you should use the @code{loop} form. It will be the fastest, and is usually flexible enough to do what you need. The only case is for hot loops (ones that are run often) that are performance sensitive. For those, using the various @code{-fold}, @code{-for-each}, and @code{-iter} functions will be faster for the collections that are tree-based. Those collections are: @code{Vec}, @code{Map}, @code{HashSet}, @code{OrderedMap}, and @code{OrderedMap}. The overhead of loop vs. fold is often something like 3x, with fold being as fast or slightly faster thana @code{foreach} on a C# @code{List<T>}. It does not, however, beat using indexed iteration over the @code{List<T>} due to the @code{foreach} doing some safety checks to detect concurrent mutation.
The absolutely fastest collection to loop over is, unsurprisingly, the Array. In bjolang it should be just as fast as in c# (again, without spans).
The loop form gets turned into a kind of goto soup. For the single level case, it is just as efficient as a c# @code{while} loop, but for really complex cases with many levels and subloops with clauses the JIT in dotnet may struggle. In that case you either live with the clarity and simplicity it gives, or you reach for the glorified gotos of names lets. This is probably not going to be a problem though. I tried to rewrite the worst loop form in bjolang (a 16 line monstrosity with one large multi-level claused subloop and a regular subloop) and there was no difference.
@subsubsection{Choosing a collection}
Bjolang has two pairs of collections that do mostly the same job: @code{List} and @code{Vec}, and @code{Map} and @code{OrderedMap}. Which one you pick matters less for small collections than you think, and more for large ones than you would hope. The tables below come from the benchmarks in @code{bench/collections/} in the bjolang repository (run them with @code{./run.sh}). They were measured on one machine with .NET 10 on Linux, so trust the ratios more than the exact times.
@strong{List or Vec}
@read-table{
| Operation | List | Vec |
|--------------------------------------------+--------------------------+---------------------------|
| memory per int | 32 B | about 7 B |
| first item | 1 ns | 2 ns |
| drop the first item | 1 ns | 21 ns |
| add at the front | 1 ns | 14–200 ns |
| add at the back | copies the whole list | 10 ns |
| last item, length | 1 ns per item | 1–2 ns |
| item at an index | 1 ns per item before it | 1–2 ns |
| read every item (@code{loop}, fold) | 1 ns per item | 0.6 ns per item |
| build with an accumulator, 1000+ items | 1 | 5–8 times faster |
| map, reverse, append, slice, 100+ items | 1 | much faster |
| recursive match that binds the rest | 1 ns per item | 30 ns and 230 B per item |
| match the first items, ignore the rest | 2 ns | 2 ns |
}
Use a @code{List} when the work happens at the front and goes item by item: a stack, a result built with @code{cons}, or a recursive function that matches @code{(Cons h t)} and continues with @code{t}. That is what a list does in constant time, and there it is 3 to 30 times faster than a vec.
Use a @code{Vec} for everything else. It knows its length and last item, it reads any index in a couple of nanoseconds, it is faster to iterate, and it uses about a fifth of the memory. Some things to keep in mind:
@read-list{
- Build vecs with an accumulator (@code{vecing} in @code{loop}, or a vec builder), not with @code{vec-add} in a loop. Each @code{vec-add} copies the tail of the vec, which makes a @code{vec-add} loop 3–4 times slower than a @code{cons} loop.
- Do not bind the rest of a vec in a recursive match. @code{[a rest ...]} slices the vec at every step. @code{[a b _ ...]} ignores the rest and does not slice. To walk a vec, use @code{loop}, a fold or an index.
- @code{list-ref}, @code{length} and the last item of a list read the list from the start. 64 random reads in a list of 100,000 items take 3 ms. In a vec they take 240 ns.
}
@strong{Map or OrderedMap}
@read-table{
| Operation | Map | OrderedMap |
|--------------------------------------+---------------------+------------------------------------|
| lookup, int keys | 1 | 1.6–3 times slower |
| lookup, string keys | 1 | 2–6.5 times slower |
| update, insert, remove | 1 | 1.2–1.9 times slower, 2× the allocation |
| build | 1 | up to 2.9 times slower |
| read every entry, 10,000+ entries | 1 | 2–6 times faster |
| smallest key | reads every entry | 2 ns |
| 64 entries in a key range | reads every entry | about 400 ns at any size |
| every key, in order | collect and sort | about 4 times faster |
}
Use a @code{Map} to look up, update, insert and remove by key. It wins at all of those at every size. The difference is largest for string keys, because a map hashes the key once and an ordered map compares it at every step down the tree.
Use an @code{OrderedMap} when the order of the keys matters: the smallest or largest key, the entries between two keys, or the entries in key order. A map can only answer those by reading every entry. An ordered map is also faster to iterate in full, so a program that walks a large map more often than it looks things up in it may prefer one.
Reference in a new issue View git blame Copy permalink