bjoli/BjoManual
SHA256
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Benchmarks: current numbers, and a service under load

Browse source 
The ring, spawn and choose rows are measured again. Spawn and choose cost
about twice what they did, with the scheduler that replaced the .NET
pool, and several sentences no longer held: an event synced alone costs
nothing over chan-put, a choose allocates 56 bytes, and Hopac's choose
stays ahead with a cancel IVar.

A new section gives bench/service against Go and Hopac: the closed loop,
and the open loop with latency taken from when a request was due.

🤖 Generated with [ECA](https://eca.dev)

Co-Authored-By: eca-agent <git@eca.dev>
This commit is contained in:
Linus Björnstam 2026-10-14 07:51:19 +02:00
parent a4c27bbf7c
commit 6fef5a9c5f
1 changed files with 48 additions and 12 deletions
Download patch file Download diff file Expand all files Collapse all files

60
_02_.Writing-programs.sz
View file

@ -926,15 +926,17 @@ Go is the obvious point of comparison: goroutines and unbuffered channels are th
@read-table{
| Benchmark | Bjolang, @code{sync} | Bjolang, @code{chan-put} | Go | Go, with @code{ctx.Done()} |
|----------------------------------------+----------------------+--------------------------+-----+----------------------------|
| Ring: 1000 fibers, a million messages | 78 | 66 | 115 | 247 |
| The same ring inside a nested scope | 72 | — | — | 248 |
| Spawning a million fibers and waiting | 80 | — | 136 | — |
| An 8-way choose where one branch fires | 154 | 145 | 396 | 449 |
| Ring: 1000 fibers, a million messages | 71 | 75 | 117 | 241 |
| The same ring inside a nested scope | 63 | — | — | 242 |
| Spawning a million fibers and waiting | 226 | — | 190 | — |
| An 8-way choose where one branch fires | 230 | 221 | 403 | 451 |
}
In the choose row only the sender can use @code{chan-put}, since the receiving side has to use channel-receive (and thus allocates). On the ring, syncing an event costs about a sixth of each message. For some reason, this ring inside a nested scope is faster in bjolang, and I haven't figured out why yet.
In the choose row only the sender can use @code{chan-put}, since the receiving side has to use channel-receive (and thus allocates). On the ring, syncing an event costs nothing measurable any more: a send or a receive synced alone does not build an event. For some reason, this ring inside a nested scope is faster in bjolang, and I haven't figured out why yet.
These are small benchmarks on one machine, and they only measure the cost of the concurrency primitives, not of the work between rendezvous. Go is ahead in other ways: a Go channel operation never allocates, where a Bjolang send synced as an event allocates 32 bytes and a @code{choose} 40 (@code{chan-put} and @code{chan-get} allocate nothing). Any Go function can block without being written as a bjoroutine and the Go scheduler preempts a goroutine that computes for too long, where a fiber has to be moved off the pool with @code{spawn/thread}.
The spawn and choose rows are about twice what they were with the scheduler Bjolang used before, which handed every fiber to the .NET thread pool. With it (@code{BJO_SCHEDULER=pool} in the environment, kept as a way back) a spawn costs 105 ns and the choose 113. The scheduler that replaced it runs fibers from run queues of its own, in the shape of Go's, because the pool let work that had waited longest wait longer still under load. That is worth more in a loaded service, as the service benchmark below shows, than these two rows on an idle machine.
These are small benchmarks on one machine, and they only measure the cost of the concurrency primitives, not of the work between rendezvous. Go is ahead in other ways: a Go channel operation never allocates, where a Bjolang @code{choose} allocates 56 bytes and a spawn 136 (a send or a receive synced alone allocates nothing). Any Go function can block without being written as a bjoroutine and the Go scheduler preempts a goroutine that computes for too long, where a fiber has to be moved off the pool with @code{spawn/thread}.
What Bjolang offers that Go does not is in the CML vocabulary rather than in the numbers. A Go @code{select} is a statement, its cases are fixed in the source, and a channel operation cannot be passed to a function, wrapped, or combined with others. In Bjolang an event is a value, which is what made @code{within} above a two-line function that works on any event, and @code{choose}, @code{wrap}, @code{guard} and @code{with-nack} build new events out of old ones. Go has no structured concurrency either. Waiting for goroutines, stopping them, and reporting their failures are a @code{sync.WaitGroup}, a @code{context.Context} passed to every function and checked at every channel operation, and an @code{errgroup}, written by hand each time. A goroutine that nothing waits for leaks without a sound. In Bjolang a scope does all of that by construction: it waits for its fibers, raises their failures when it ends, releases the files they opened, and cancels them through a token that every @code{sync} watches without being asked.
@ -943,14 +945,48 @@ Hopac is the tougher comparison. It is a Concurrent ML library for F#, running o
@read-table{
| Benchmark | Bjolang, @code{sync} | Hopac | Hopac, racing a cancel @code{IVar} |
|-----------------------------------------+----------------------+-------+------------------------------------|
| Ring: 1000 fibers, a million messages | 78 | 43 | 3445 |
| The same ring, one cancel IVar per job | — | — | 180 |
| The same ring inside a nested scope | 72 | — | 3383 |
| Spawning a million fibers and waiting | 80 | 75 | — |
| An 8-way choose where one branch fires | 154 | 116 | 128 |
| Ring: 1000 fibers, a million messages | 71 | 51 | 3487 |
| The same ring, one cancel IVar per job | — | — | 195 |
| The same ring inside a nested scope | 63 | — | 3427 |
| Spawning a million fibers and waiting | 226 | 80 | — |
| An 8-way choose where one branch fires | 230 | 127 | 136 |
}
Without cancellation Hopac is ahead: its ring is almost twice as fast as Bjolang's, its choose takes a quarter less time, and its spawns cost about the same. Bjolang's numbers include watching the scope's token at every operation, which Hopac's plain rows do not do. Once cancellation is added the picture turns. A thousand jobs racing one shared @code{IVar} all take that IVar's lock at every operation, and the ring slows to three and a half microseconds a message. Giving each job an IVar of its own brings it back to 180 ns, at the price of keeping track of a thousand IVars and filling all of them to stop the ring. Hopac also allocates more: 208 bytes a message on the ring and 424 a choose, against Bjolang's 32 and 72. Hopac can run jobs in parallel and wait for all of them, but a job cannot be stopped from the outside, so stopping jobs when one of them fails, or when their owner is done with them, is the same hand-written race against an IVar as in the table.
Without cancellation Hopac is ahead: its ring takes 30% less time than Bjolang's, its choose a little over half, and its spawns about a third. Bjolang's numbers include watching the scope's token at every operation, which Hopac's plain rows do not do. Once cancellation is added, the ring turns. A thousand jobs racing one shared @code{IVar} all take that IVar's lock at every operation, and the ring slows to three and a half microseconds a message. Giving each job an IVar of its own brings it back to 195 ns, at the price of keeping track of a thousand IVars and filling all of them to stop the ring. The choose does not turn: with a cancel IVar Hopac's takes 136 ns, against Bjolang's 230. Hopac allocates more: 208 bytes a message on the ring and 424 a choose, against Bjolang's 0 and 56. Hopac can run jobs in parallel and wait for all of them, but a job cannot be stopped from the outside, so stopping jobs when one of them fails, or when their owner is done with them, is the same hand-written race against an IVar as in the table.
@subsubsection{A service under load}
The rows above time one primitive at a time on an idle machine. @code{bench/service} is closer to a program: a sharded key-value store in all three languages, with 16 shard fibers that each serve a part of the keys, and clients that send gets, puts and four-key multi-gets, each with a fresh reply channel and a one-second deadline raced against the reply. The three programs do the same work and agree on a checksum. @code{bench/service/drive.py} runs them and takes the median of 15 reps.
The two .NET programs run with the server GC, which keeps a few hundred megabytes and collects a few times a rep. Go's collector keeps the heap small and collects about twenty times as often, so Go runs twice: with its default, and with as much memory as the .NET programs took (@code{GOGC=off} and @code{GOMEMLIMIT} at their peak).
In a closed loop, 1000 clients send 1000 requests each, each one as soon as the last is answered, so the machine is always saturated. Wall time, CPU time, p99.9 latency and peak memory:
@read-table{
| Program | all cores: wall | CPU | p99.9 | memory | 4 cores: wall | CPU | p99.9 |
|------------------+-----------------+---------+--------+--------+---------------+---------+--------|
| Go | 400 ms | 5542 ms | 5.8 ms | 27 MB | 479 ms | 1847 ms | 3.4 ms |
| Go, same memory | 259 | 4057 | 3.2 | 402 | 425 | 1680 | 3.1 |
| Hopac | 436 | 8854 | 12.7 | 387 | 294 | 1146 | 3.8 |
| Bjolang | 256 | 5194 | 5.6 | 427 | 411 | 1587 | 4.7 |
}
On all cores Bjolang and Go with the same memory finish together, and Go uses a fifth less CPU and has the shorter tail. On four cores Hopac is clearly the fastest, and Bjolang and Go are close.
In an open loop, 10000 clients send on a fixed schedule at a given rate, so the three are compared at the same load. Latency is taken from when a request was due, so a client that woke late, waited for a core, or slept through a GC pause counts that time too. p99 and p99.9 in milliseconds:
@read-table{
| Load | Go | Go, same memory | Hopac | Bjolang |
|----------------------+-------------+-----------------+-------------+-------------|
| 4 cores, 250k/s | 5.0 / 8.1 | 2.1 / 4.9 | 2.7 / 4.7 | 2.5 / 3.0 |
| 4 cores, 500k/s | 33.1 / 38.0 | 25.0 / 31.1 | 7.3 / 8.7 | 3.0 / 4.2 |
| all cores, 500k/s | 9.2 / 12.3 | 6.6 / 11.0 | 23.5 / 30.5 | 5.0 / 6.7 |
| all cores, 1M/s | 31.1 / 34.8 | 32.2 / 40.0 | 27.1 / 34.5 | 7.4 / 11.7 |
}
A client that is early sleeps until its request is due, in whole milliseconds rounded up, the same in all three programs, so every latency here has up to a millisecond of that in it, and below about 2 ms the numbers say little. Bjolang's p99.9 is the shortest at every load, and at a million requests a second its tail is a quarter to a third of the others'. At 250k/s on four cores the three are close, and Go with the same memory has the shortest p99 there. At 500k/s on all cores Go with the same memory comes near: another run of the same gave Bjolang 3.6 ms and Go 2.1 at p99, so that row is not settled. Hopac keeps all 24 cores busy on all cores at either rate, and why has not been looked into.
The tables in @code{BjolangRuntime/Cml/design.md} in the main repository have every column, including the time a request took once its client had started it.
@subsection{Effects}