Document (std text) and (std unicode grapheme)
Module pages for both, entries in the standard library chapter, and sections on grapheme clusters and on text for editing in the language basics. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
93025b7b35
commit
1a2a43bf5a
5 changed files with 85 additions and 0 deletions
|
|
@ -1905,6 +1905,8 @@ A @code{string} is stored as UTF-8, and is always valid UTF-8. A @code{char} is
|
|||
(string-count s) ;; 10: characters, O(n). This is "how long is the text".
|
||||
}|
|
||||
|
||||
There is a third count too, of the characters a reader sees. An "é" can be one code point, or an e followed by a combining accent, and a flag is two code points. @code{string-grapheme-count} counts those; see the section on grapheme clusters below.
|
||||
|
||||
For the same reason there is no @code{string-ref} that takes an index. Finding the n:th character of a UTF-8 string means walking it, so walking is what you do: with a @code{loop}, a fold, or a cursor.
|
||||
|
||||
@codeblock[#:lang "bjolang"]|{
|
||||
|
|
@ -1984,6 +1986,31 @@ The folds walk a string for you, and are usually simpler than a cursor:
|
|||
(count-vowels "Hello World") ;; 3
|
||||
}|
|
||||
|
||||
@subsubsection{Grapheme clusters}
|
||||
|
||||
What a reader takes for one character can be several @code{char}s: a letter and its accents, a flag, an emoji with a skin tone or a family of emoji joined by zero-width joiners, a Hangul syllable written in jamo, and CR LF. Unicode calls these grapheme clusters, and an editor's arrow keys and backspace move by them. @code{(std unicode grapheme)} finds them by Unicode's rules (UAX #29):
|
||||
|
||||
@codeblock[#:lang "bjolang"]|{
|
||||
(import (std unicode grapheme))
|
||||
|
||||
(def s "é🇸🇪\r\n")
|
||||
(string-count s) ;; 6
|
||||
(string-grapheme-count s) ;; 3
|
||||
(string->graphemes s) ;; ["é" "🇸🇪" "\r\n"]
|
||||
|
||||
;; Backspace: the string without its last cluster.
|
||||
(: drop-last-grapheme (-> string string))
|
||||
(defun (drop-last-grapheme s)
|
||||
(if (string-empty? s)
|
||||
s
|
||||
(substring s (string-cursor-start s)
|
||||
(string-cursor-prev-grapheme s (string-cursor-end s)))))
|
||||
|
||||
(drop-last-grapheme "ok 👍🏽") ;; "ok ": the skin tone goes with the thumb
|
||||
}|
|
||||
|
||||
@code{string-cursor-next-grapheme} and @code{string-cursor-prev-grapheme} take a cursor that is on a cluster boundary: the start, the end, or a cursor that a grapheme function returned. @code{string-cursor-grapheme-boundary?} tells whether a cursor that came from elsewhere is on one. A cluster has no limit on its length, since a letter can carry any number of accents, but a loop over a string by clusters takes O(n) time in either direction, so a text written to be slow cannot make it quadratic. @code{(std text)} has the same functions for texts.
|
||||
|
||||
@subsubsection{Slices}
|
||||
|
||||
@code{substring}, @code{string-trim} and @code{string-split} copy the text they answer. A slice is the same text without the copy: a string that shares the bytes of the string it was taken from. Only the forms ending in @code{/slice} make one:
|
||||
|
|
@ -2149,6 +2176,46 @@ The @code{Rx} and @code{RxIn} patterns use a regex inside a @code{match}. @code{
|
|||
|
||||
Define patterns at the top level, as above, rather than writing @code{#rx(...)} inside a clause. Otherwise the regex is recompiled at every call, which will tank performance.
|
||||
|
||||
@subsubsection{Text for editing with (std text)}
|
||||
|
||||
A string is immutable, and every edit copies all of it. For a large text that is edited in many places, such as an editor's buffer, @code{(std text)} has @code{Text}: a balanced tree of UTF-8 chunks, where inserting, deleting, splitting, appending and taking a substring take O(log n) time. An edit shares every chunk it does not change, so the old version stays valid and costs only what differs: undo is keeping the old texts.
|
||||
|
||||
The API is a string's. The functions a string has are there with @code{text-} for @code{string-}, positions are cursors, and the cursor functions take the text and the cursor. @code{text-length} is in bytes, as @code{string-length} is, and @code{text-count} counts characters, in O(1) time here. The tree also counts newlines, so lines are found in O(log n) time.
|
||||
|
||||
@codeblock[#:lang "bjolang"]|{
|
||||
(import (std text))
|
||||
|
||||
(def r (string->text "hello world\nsecond line"))
|
||||
(def c (text-cursor-forward r (text-cursor-start r) 5)) ;; O(log n), however far
|
||||
(def r2 (text-insert r c ","))
|
||||
(text->string r2) ;; "hello, world\nsecond line"
|
||||
(text->string (text-line r2 1)) ;; "second line"
|
||||
(text-cursor-line r2 (option-ref (text-search r2 "second"))) ;; 1
|
||||
(text->string r) ;; "hello world\nsecond line": unchanged
|
||||
}|
|
||||
|
||||
A cursor belongs to one text, and the text an edit returns needs cursors of its own. To keep a position across an edit, such as an editor's caret, keep its index: @code{text-cursor-index} gives it, and @code{text-cursor-forward} from the start of the new text finds it again.
|
||||
|
||||
@codeblock[#:lang "bjolang"]|{
|
||||
(def caret (option-ref (text-search r2 "world")))
|
||||
(def i (text-cursor-index r2 caret)) ;; 7
|
||||
(def r3 (text-insert r2 (text-cursor-start r2) "> "))
|
||||
(def caret3 (text-cursor-forward r3 (text-cursor-start r3) (+ i 2))) ;; still on "world"
|
||||
}|
|
||||
|
||||
A run of edits close together, such as typing, is faster through a @code{TransientText}, which edits in place, takes character indices, and gives a text back with @code{transienttext->text}. A @code{TextBuilder}, or the @code{texting} collector of a loop, builds a text in one pass. And @code{text-cursor-next-grapheme} and @code{text-cursor-prev-grapheme} step by grapheme clusters, as the arrow keys do:
|
||||
|
||||
@codeblock[#:lang "bjolang"]|{
|
||||
(def t (text->transienttext r))
|
||||
(transienttext-insert! t 5 ",")
|
||||
(transienttext-insert! t 6 " my")
|
||||
(text->string (transienttext->text t)) ;; "hello, my world\nsecond line"
|
||||
|
||||
;; Backspace at the end of a text.
|
||||
(def e (string->text "ok 👍🏽"))
|
||||
(text->string (text-delete e (text-cursor-prev-grapheme e (text-cursor-end e)) (text-cursor-end e))) ;; "ok "
|
||||
}|
|
||||
|
||||
@subsection{Traits}
|
||||
|
||||
A trait is a set of functions that a type can implement. It is how bjolang does what other languages do with interfaces, type classes or overloading: @code{=} is a trait method, so is @code{displayed}, and so are @code{length}, @code{fold} and @code{compare}.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue