From 5e072834047ad1e189829ce65531d92a213655ad86599f29fcf4d674ca2e1881 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Linus=20Bj=C3=B6rnstam?= Date: Sun, 15 Nov 2026 21:59:22 +0100 Subject: [PATCH] Document (std stream), port->stream and run/stream Co-Authored-By: Claude Opus 5.5 --- _01.Language-basics.sz | 2 +- _02_.Writing-programs.sz | 62 ++++++++++++++++++++++++++++++++++++++-- _03.Standard-library.sz | 9 ++++-- modules/std/stream.sz | 3 ++ 4 files changed, 71 insertions(+), 5 deletions(-) create mode 100644 modules/std/stream.sz diff --git a/_01.Language-basics.sz b/_01.Language-basics.sz index 603f162..8697086 100644 --- a/_01.Language-basics.sz +++ b/_01.Language-basics.sz @@ -1530,7 +1530,7 @@ There are three ways to repeat something in bjolang. A named @code{let} (or any @subsubsection{loop} -A @code{loop} is a list of clauses that run in order, once per element. @code{(:for x coll)} walks anything that can be walked: lists, vecs, arrays, maps, strings, ranges and sequences. An accumulator collects a result, and @code{=> expr} says what the loop returns: +A @code{loop} is a list of clauses that run in order, once per element. @code{(:for x coll)} walks anything that can be walked: lists, vecs, arrays, maps, strings, ranges, sequences, and streams, whose values come from work that suspends (Part 2, Concurrency). An accumulator collects a result, and @code{=> expr} says what the loop returns: @codeblock[#:lang "bjolang"]|{ (loop (:for x (list 1 2 3 4 5 6)) diff --git a/_02_.Writing-programs.sz b/_02_.Writing-programs.sz index 4a2a057..033cecf 100644 --- a/_02_.Writing-programs.sz +++ b/_02_.Writing-programs.sz @@ -580,7 +580,7 @@ To avoid leaking resources, you will usually want to use @code{call-with-input-f The variants @code{read-line} and @code{read-char} (without the @code{/opt} suffix) also exist, but throw an end-of-file exception if called at EOF. -Convenience helpers in @code{(std ports)} can ingest an entire port into collections: @code{port->lines}, @code{port->list}, @code{port->vec}, as well as lazy sequences via @code{port->seq} and @code{file->seq}: +Convenience helpers in @code{(std ports)} can ingest an entire port into collections: @code{port->lines}, @code{port->list}, @code{port->vec}, as well as lazy sequences via @code{port->seq} and @code{file->seq}. In a bjoroutine, @code{port->stream} is the lazy one, a @code{Stream} (see Streams, below): @codeblock[#:lang "bjolang"]|{ (import (std ports)) @@ -618,7 +618,7 @@ Prefixing an identifier with a comma (e.g. @code{,who}) splices the variable's v | @code{(run form)} | A running @code{Proc} handle with streaming I/O ports | } -All of these functions return a @code{Result} and execute synchronously, blocking the calling thread until completion. When running inside a bjoroutine, non-blocking asynchronous alternatives are available; refer to the @code{(std run)} module documentation for details. +All of these functions return a @code{Result} and execute synchronously, blocking the calling thread until completion. When running inside a bjoroutine, non-blocking asynchronous alternatives are available: @code{run/stream} gives the output as a @code{Stream} that a loop reads as it comes (see Streams, below), and the @code{(std run)} module documentation has the rest. @subsection{Formatting text} @@ -1208,6 +1208,64 @@ A scope also owns resources. @code{(own! thunk)} registers a release on the inne Do not park a bjoroutine's thread by accident. A scope in an ordinary @code{(fun ...)} that a bjoroutine calls parks the thread of that bjoroutine, and the compiler warns about it. Write the scope in the bjoroutine itself. Or, if the function the lambda is given to takes a @code{-?->} parameter, write the lambda as a @code{(bjoroutine ...)}. +@subsubsection{Streams: loops over work that suspends} + +A @code{Seq} cannot suspend: it is a .NET iterator, and an iterator cannot also await. A @code{Stream}, from @code{(std stream)}, is the sequence for a bjoroutine. It is a driver, @code{(:for x s)}, in a loop or a comprehension. Its producer is a bjoroutine that hands each value to the loop with @code{emit!}, so the producer can wait for anything: + +@codeblock[#:lang "bjolang"]|{ +(import (std stream)) + +(: ticks (-> int (Stream int))) +(defun (ticks n) + (make-stream (bjoroutine (out) + (loop (:for i (range 0 n)) + (:do (sync (timeout (milliseconds 100))) + (println #"tick ${i}") + (emit! out i)))))) + +(defbjo (main) + (println {listing t (:for t (ticks 3))}) + (println {taking 2 t (:for t (ticks 1000))}) + 0) +}| + +@codeblock[#:lang "text"]|{ +tick 0 +tick 1 +tick 2 +(0 1 2) +tick 0 +tick 1 +(0 1) +}| + +A stream is a recipe: holding one runs nothing. Each loop over it starts the producer in a fiber of its own, and the producer computes a value only when the loop asks for it. That is why the second loop prints two ticks and not three. The loop stops the producer however it ends: when the producer returns, at @code{taking} or @code{finding}, at a @code{:finish}, when a driver beside it runs out, or at a raise or a @code{ret}. The stopped producer raises @code{Cancelled} where it waits, so its own releases run. What the producer raises for its own reasons is raised in the loop. @code{{streaming expr clause ...}} writes a producer as a comprehension, @code{{streaming (fetch url) (:for url urls)}}, the @code{seqing} of work that suspends. + +Ports and programs come as streams too. @code{port->stream} in @code{(std ports)} reads one item at a time, so a loop left early leaves the port just after the last item it took. @code{run/stream} in @code{(std run)} runs a program for each loop and kills it when the loop ends: + +@codeblock[#:lang "bjolang"]|{ +(def p (open-input-string "Host: example.com\nAccept: */*\n\nthe body\n")) +(def headers + (loop (:for line (port->stream read-line p)) + (:until (string-empty? line)) + (:acc hs (listing line)) + => hs)) +(println headers) ;; (Host: example.com Accept: */*) +(println (read-all p)) ;; the body + +{taking 2 line (:for line (run/stream '(seq 1 1000000) read-line))} ;; (1 2), and seq is killed +}| + +A channel that something else fills is read with @code{in-chan}. A channel has no close, so @code{in-chan} also takes the event that ends the reading, such as the producer's @code{promise-join}. A send is a rendezvous, so a producer that has returned has handed over every value it sent. This loop does not own the producer: leaving it early leaves the rest on the channel for another reader. + +@codeblock[#:lang "bjolang"]|{ +(def jobs (make-chan)) +(def feeder (bjo (feed jobs 4))) ;; sends 0 1 4 9 +{listing x (:for x (in-chan jobs (promise-join feeder)))} ;; (0 1 4 9) +}| + +In a bjoroutine each step of these loops suspends. In ordinary code it parks the thread, as @code{sync/blocking} does. + @subsubsection{Ctrl-C and SIGTERM} By default, Ctrl-C and SIGTERM end a program at once. No @code{finally} runs, and no scope releases what it owns. @code{with-signal-cancel} from @code{(std os)} is a scope that the first of its signals cancels. The fibers stop at their next wait, the scope releases what it owns, and the form returns @code{(Err s)}, with the signal. When the body returns @code{v}, the form returns @code{(Ok v)}: diff --git a/_03.Standard-library.sz b/_03.Standard-library.sz index 542785b..aa4891c 100644 --- a/_03.Standard-library.sz +++ b/_03.Standard-library.sz @@ -69,7 +69,7 @@ For more details, see @link["modules/std/url.html"]{@code{(std url)}}. @subsection{std/run} -The @code{std/run} module provides a way to run external programs. A form is one thing to run, written as a quoted literal, like @code{'(cat "hej.txt")}. You can connect programs in a pipe with @code{'(pipe (cat "a") (wc -l))}, and redirect input and output to files. Redirections can nest, and you can even splice computed arguments or write stages in Bjolang. +The @code{std/run} module provides a way to run external programs. A form is one thing to run, written as a quoted literal, like @code{'(cat "hej.txt")}. You can connect programs in a pipe with @code{'(pipe (cat "a") (wc -l))}, and redirect input and output to files. Redirections can nest, and you can even splice computed arguments or write stages in Bjolang. @code{run/stream} gives a program's output as a @code{Stream}: each loop over it runs the program, and the program is killed however the loop ends. For more details, see @link["modules/std/run.html"]{@code{(std run)}}. @subsection{std/random} @@ -151,9 +151,14 @@ It also has the facts about the machine and the process: @code{os-description}, And the signals, as union cases such as @code{Interrupt}, @code{Terminate} and @code{Hangup}. @code{with-signal-cancel} is a scope that the first of its signals cancels, so that a program cleans up on Ctrl-C or SIGTERM; a second signal ends the program at once. @code{on-signal} calls a function for each signal that comes, until the scope ends. @code{signal-subscribe} gives signals as values from the event @code{signal-recv}, and signals of one kind that wait are merged into one. For more details, see @link["modules/std/os.html"]{@code{(std os)}}. +@subsection{std/stream} + +Loops over values that suspending work produces. A @code{Stream} is a driver, @code{(:for x s)}, whose producer is a bjoroutine that hands each value to the loop with @code{emit!}. Each loop over it starts the producer, which computes a value only when the loop asks for it, and stops it however the loop ends. @code{make-stream} makes one from a bjoroutine, and @code{{streaming expr clause ...}} from loop clauses. @code{in-chan} reads a channel that something else fills, until an event such as the producer's @code{promise-join} commits. +For more details, see @link["modules/std/stream.html"]{@code{(std stream)}}. + @subsection{std/ports and std/net} -@code{(std ports)} reads and writes bytes on the prelude's ports. The byte reads @code{read-some-evt}, @code{read-bytes-evt} and @code{eof-evt} are CML events: a read that loses its @code{choose} takes nothing, so a read can race a timeout. @code{read-line} and @code{read-bytes} can be used on the same port. The module also drains ports into lists and vecs, streams them as a @code{Seq} or a channel, and has @code{byte-pipe}, a pair of ports with no file and no network, for tests. @code{(std net)} gives a TCP connection as a pair of these ports, so a protocol written for ports also runs over TCP. @code{accept-evt} is an event too: a branch that loses accepts nothing. +@code{(std ports)} reads and writes bytes on the prelude's ports. The byte reads @code{read-some-evt}, @code{read-bytes-evt} and @code{eof-evt} are CML events: a read that loses its @code{choose} takes nothing, so a read can race a timeout. @code{read-line} and @code{read-bytes} can be used on the same port. The module also drains ports into lists and vecs, streams them as a @code{Seq} or, for a bjoroutine, as a @code{Stream} with @code{port->stream}, and has @code{byte-pipe}, a pair of ports with no file and no network, for tests. @code{(std net)} gives a TCP connection as a pair of these ports, so a protocol written for ports also runs over TCP. @code{accept-evt} is an event too: a branch that loses accepts nothing. For more details, see @link["modules/std/ports.html"]{@code{(std ports)}} and @link["modules/std/net.html"]{@code{(std net)}}. @subsection{std/test and std/simpletest} diff --git a/modules/std/stream.sz b/modules/std/stream.sz new file mode 100644 index 0000000..c8334fb --- /dev/null +++ b/modules/std/stream.sz @@ -0,0 +1,3 @@ +(meta (title "(std stream)")) + +@generate-docs[#:module "std stream"]