Document (std text) and (std unicode grapheme)
Module pages for both, entries in the standard library chapter, and sections on grapheme clusters and on text for editing in the language basics. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
93025b7b35
commit
1a2a43bf5a
5 changed files with 85 additions and 0 deletions
|
|
@ -1905,6 +1905,8 @@ A @code{string} is stored as UTF-8, and is always valid UTF-8. A @code{char} is
|
|||
(string-count s) ;; 10: characters, O(n). This is "how long is the text".
|
||||
}|
|
||||
|
||||
There is a third count too, of the characters a reader sees. An "é" can be one code point, or an e followed by a combining accent, and a flag is two code points. @code{string-grapheme-count} counts those; see the section on grapheme clusters below.
|
||||
|
||||
For the same reason there is no @code{string-ref} that takes an index. Finding the n:th character of a UTF-8 string means walking it, so walking is what you do: with a @code{loop}, a fold, or a cursor.
|
||||
|
||||
@codeblock[#:lang "bjolang"]|{
|
||||
|
|
@ -1984,6 +1986,31 @@ The folds walk a string for you, and are usually simpler than a cursor:
|
|||
(count-vowels "Hello World") ;; 3
|
||||
}|
|
||||
|
||||
@subsubsection{Grapheme clusters}
|
||||
|
||||
What a reader takes for one character can be several @code{char}s: a letter and its accents, a flag, an emoji with a skin tone or a family of emoji joined by zero-width joiners, a Hangul syllable written in jamo, and CR LF. Unicode calls these grapheme clusters, and an editor's arrow keys and backspace move by them. @code{(std unicode grapheme)} finds them by Unicode's rules (UAX #29):
|
||||
|
||||
@codeblock[#:lang "bjolang"]|{
|
||||
(import (std unicode grapheme))
|
||||
|
||||
(def s "é🇸🇪\r\n")
|
||||
(string-count s) ;; 6
|
||||
(string-grapheme-count s) ;; 3
|
||||
(string->graphemes s) ;; ["é" "🇸🇪" "\r\n"]
|
||||
|
||||
;; Backspace: the string without its last cluster.
|
||||
(: drop-last-grapheme (-> string string))
|
||||
(defun (drop-last-grapheme s)
|
||||
(if (string-empty? s)
|
||||
s
|
||||
(substring s (string-cursor-start s)
|
||||
(string-cursor-prev-grapheme s (string-cursor-end s)))))
|
||||
|
||||
(drop-last-grapheme "ok 👍🏽") ;; "ok ": the skin tone goes with the thumb
|
||||
}|
|
||||
|
||||
@code{string-cursor-next-grapheme} and @code{string-cursor-prev-grapheme} take a cursor that is on a cluster boundary: the start, the end, or a cursor that a grapheme function returned. @code{string-cursor-grapheme-boundary?} tells whether a cursor that came from elsewhere is on one. A cluster has no limit on its length, since a letter can carry any number of accents, but a loop over a string by clusters takes O(n) time in either direction, so a text written to be slow cannot make it quadratic. @code{(std text)} has the same functions for texts.
|
||||
|
||||
@subsubsection{Slices}
|
||||
|
||||
@code{substring}, @code{string-trim} and @code{string-split} copy the text they answer. A slice is the same text without the copy: a string that shares the bytes of the string it was taken from. Only the forms ending in @code{/slice} make one:
|
||||
|
|
@ -2149,6 +2176,46 @@ The @code{Rx} and @code{RxIn} patterns use a regex inside a @code{match}. @code{
|
|||
|
||||
Define patterns at the top level, as above, rather than writing @code{#rx(...)} inside a clause. Otherwise the regex is recompiled at every call, which will tank performance.
|
||||
|
||||
@subsubsection{Text for editing with (std text)}
|
||||
|
||||
A string is immutable, and every edit copies all of it. For a large text that is edited in many places, such as an editor's buffer, @code{(std text)} has @code{Text}: a balanced tree of UTF-8 chunks, where inserting, deleting, splitting, appending and taking a substring take O(log n) time. An edit shares every chunk it does not change, so the old version stays valid and costs only what differs: undo is keeping the old texts.
|
||||
|
||||
The API is a string's. The functions a string has are there with @code{text-} for @code{string-}, positions are cursors, and the cursor functions take the text and the cursor. @code{text-length} is in bytes, as @code{string-length} is, and @code{text-count} counts characters, in O(1) time here. The tree also counts newlines, so lines are found in O(log n) time.
|
||||
|
||||
@codeblock[#:lang "bjolang"]|{
|
||||
(import (std text))
|
||||
|
||||
(def r (string->text "hello world\nsecond line"))
|
||||
(def c (text-cursor-forward r (text-cursor-start r) 5)) ;; O(log n), however far
|
||||
(def r2 (text-insert r c ","))
|
||||
(text->string r2) ;; "hello, world\nsecond line"
|
||||
(text->string (text-line r2 1)) ;; "second line"
|
||||
(text-cursor-line r2 (option-ref (text-search r2 "second"))) ;; 1
|
||||
(text->string r) ;; "hello world\nsecond line": unchanged
|
||||
}|
|
||||
|
||||
A cursor belongs to one text, and the text an edit returns needs cursors of its own. To keep a position across an edit, such as an editor's caret, keep its index: @code{text-cursor-index} gives it, and @code{text-cursor-forward} from the start of the new text finds it again.
|
||||
|
||||
@codeblock[#:lang "bjolang"]|{
|
||||
(def caret (option-ref (text-search r2 "world")))
|
||||
(def i (text-cursor-index r2 caret)) ;; 7
|
||||
(def r3 (text-insert r2 (text-cursor-start r2) "> "))
|
||||
(def caret3 (text-cursor-forward r3 (text-cursor-start r3) (+ i 2))) ;; still on "world"
|
||||
}|
|
||||
|
||||
A run of edits close together, such as typing, is faster through a @code{TransientText}, which edits in place, takes character indices, and gives a text back with @code{transienttext->text}. A @code{TextBuilder}, or the @code{texting} collector of a loop, builds a text in one pass. And @code{text-cursor-next-grapheme} and @code{text-cursor-prev-grapheme} step by grapheme clusters, as the arrow keys do:
|
||||
|
||||
@codeblock[#:lang "bjolang"]|{
|
||||
(def t (text->transienttext r))
|
||||
(transienttext-insert! t 5 ",")
|
||||
(transienttext-insert! t 6 " my")
|
||||
(text->string (transienttext->text t)) ;; "hello, my world\nsecond line"
|
||||
|
||||
;; Backspace at the end of a text.
|
||||
(def e (string->text "ok 👍🏽"))
|
||||
(text->string (text-delete e (text-cursor-prev-grapheme e (text-cursor-end e)) (text-cursor-end e))) ;; "ok "
|
||||
}|
|
||||
|
||||
@subsection{Traits}
|
||||
|
||||
A trait is a set of functions that a type can implement. It is how bjolang does what other languages do with interfaces, type classes or overloading: @code{=} is a trait method, so is @code{displayed}, and so are @code{length}, @code{fold} and @code{compare}.
|
||||
|
|
|
|||
|
|
@ -47,6 +47,16 @@ For more details, see @link["modules/std/rx.html"]{@code{(std rx)}}.
|
|||
Reading a line by a pattern, after OCaml's @code{Scanf}. The pattern is a quoted list of what the line holds, in order: @code{(scan "3-17" '(long "-" long))} is 3 and 17, and @code{'(int "x" int ": " int ... #:sep " ")} reads @code{"12x5: 1 0 3"} as two ints and a vec of them. There are items for numbers, a character, a word, the rest of the line, a run of given characters and the text up to one, with a width, a skip, a repetition, an option and a choice. The type of the result comes from the pattern, so @code{'(int "," int)} gives a @code{(Tuple int int)}. The macro compiles the pattern into straight code over the line's bytes, and a pattern of numbers and strings allocates nothing. The whole line must match. @code{try-scan} gives an @code{Err} that says at which column it failed, instead of raising. A scan does not search and does not backtrack: use @code{(std rx)} to find text in a line.
|
||||
For more details, see @link["modules/std/scan.html"]{@code{(std scan)}}.
|
||||
|
||||
@subsection{std/text}
|
||||
|
||||
Text that is cheap to edit anywhere, for an editor's buffer or any large text that changes in many places. A @code{Text} is a balanced tree of UTF-8 chunks: inserting, deleting, splitting, appending, taking a substring and finding a line take O(log n) time, and an edit shares everything it does not change, so old versions stay valid for undo. The API is a string's, with @code{text-} for @code{string-}: positions are cursors, the cursor functions take the text and the cursor, and no function takes a byte offset. @code{text-count} is O(1), @code{text-cursor-forward} moves any number of characters in O(log n), and @code{text-cursor-index} gives a cursor's index to carry across an edit. A @code{TransientText} edits in place for a run of edits such as typing, and @code{TextBuilder} and the @code{texting} collector build a text in one pass. @code{text-cursor-next-grapheme} and the others step by grapheme clusters.
|
||||
For more details, see @link["modules/std/text.html"]{@code{(std text)}}.
|
||||
|
||||
@subsection{std/unicode/grapheme}
|
||||
|
||||
Grapheme clusters, the characters a reader sees, by Unicode's rules (UAX #29): a letter and its accents, a flag, an emoji sequence, CR LF. @code{string-cursor-next-grapheme} and @code{string-cursor-prev-grapheme} move a cursor by one, as an editor's arrow keys and backspace do, and @code{string-grapheme-count} and @code{string->graphemes} count and split. A cluster has no limit on its length, but a loop by clusters takes O(n) time in either direction, so long runs of accents or flags cannot make it quadratic. The rules are two state machines over characters, which other containers can drive with their own cursors: @code{(std text)} does, for texts.
|
||||
For more details, see @link["modules/std/unicode/grapheme.html"]{@code{(std unicode grapheme)}}.
|
||||
|
||||
@subsection{std/http}
|
||||
|
||||
The @code{std/http} module allows you to build and send HTTP requests. Sending a request three ways — blocking, suspending, or as an event — is the only thing that differs between the entry points. There is no C# in this module: @code{System.Net.Http} is reached with @code{import/extern}, and the response body is an ordinary @code{TextInputPort}.
|
||||
|
|
|
|||
3
modules/std/text.sz
Normal file
3
modules/std/text.sz
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
(meta (title "(std text)"))
|
||||
|
||||
@generate-docs[#:module "std text"]
|
||||
3
modules/std/unicode/grapheme.sz
Normal file
3
modules/std/unicode/grapheme.sz
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
(meta (title "(std unicode grapheme)"))
|
||||
|
||||
@generate-docs[#:module "std unicode grapheme"]
|
||||
2
modules/std/unicode/index.sz
Normal file
2
modules/std/unicode/index.sz
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
(meta (title "(std unicode)")
|
||||
(sort title))
|
||||
Loading…
Add table
Add a link
Reference in a new issue