textmate-bjolang/Readme.org

* textmate-bjolang — syntax highlighting, as Bjolang values

Code highlighted the way VS Code highlights it: its TextMate grammars and its
colour themes, run by [[https://github.com/danipen/TextMateSharp][TextMateSharp]],
and handed over as Bjolang values — lines of tokens and their scopes, lines of
coloured spans — and as ~(text xml)~ nodes, ready for a page.

#+BEGIN_SRC scheme
(import (textmate-bjolang core))

(def hl (highlighter #:theme 'dark-plus))

(match (find-language hl "c#")
  ((Some cs) (highlight->node cs "var x = 1; // hi"))
  (None ...))
;; (pre (@ (class "tm") (style "color:#D4D4D4;background-color:#1E1E1E"))
;;      (code (@ (class "language-csharp"))
;;            (span (@ (style "color:#569CD6")) "var") " "
;;            (span (@ (style "color:#9CDCFE")) "x") " = "
;;            (span (@ (style "color:#B5CEA8")) "1") "; "
;;            (span (@ (style "color:#6A9955")) "// hi")))
#+END_SRC

~(text xml write)~'s ~node->xml-string~ turns the node into markup.

** What is highlighted

VS Code's own grammars, 64 languages from C to YAML, and Bjolang, whose grammar
is part of this package. A language is found by its id, an alias or an
extension, in any case: ~"csharp"~, ~"C#"~, ~"cs"~ and ~".cs"~ are the same
language, and so are ~"bjolang"~, ~"bjo"~ and ~".bjo"~.

The themes are VS Code's: ~'dark-plus~ ~'light-plus~ ~'dark~ ~'light~ ~'monokai~
~'dimmed-monokai~ ~'one-dark~ ~'atom-one-dark~ ~'atom-one-light~ ~'dracula~
~'solarized-dark~ ~'solarized-light~ ~'quiet-light~ ~'kimbie-dark~
~'tomorrow-night-blue~ ~'abyss~ ~'red~ ~'high-contrast-dark~ ~'high-contrast-light~
~'visual-studio-dark~ ~'visual-studio-light~.

** Inline or classes

A node is coloured one of two ways.

~#:style 'inline~, the default, writes the theme's colours into ~style~
attributes. The page needs nothing else, so it works in a feed, in a mail, or
pasted anywhere.

~#:style 'classes~ writes class names taken from the grammar's scopes, and no
colours at all:

#+BEGIN_SRC scheme
(highlight->node cs "var x = 1; // hi" #:style 'classes)
;; (pre (@ (class "tm"))
;;      (code (@ (class "language-csharp"))
;;            (span (@ (class "tm-storage tm-storage-type")) "var") " "
;;            (span (@ (class "tm-entity tm-entity-name-variable")) "x") " "
;;            (span (@ (class "tm-keyword tm-keyword-operator")) "=") " "
;;            (span (@ (class "tm-constant tm-constant-numeric")) "1")
;;            (span (@ (class "tm-punctuation")) ";") " "
;;            (span (@ (class "tm-comment")) "// hi")))
#+END_SRC

A span has the general class and the particular one, from TextMate's
conventional scope names: ~tm-keyword~ to colour every keyword alike,
~tm-keyword-control~ to set ~if~ and ~return~ apart. The stylesheet is yours to
write, and a light and a dark one are only a media query apart. Or
~highlighter-css~ writes one from a theme:

#+BEGIN_SRC scheme
(highlighter-css (highlighter #:theme 'light-plus) #:selector ".tm")
;; .tm{color:#000000;background-color:#FFFFFF}
;; .tm .tm-comment{color:#008000}
;; .tm .tm-string{color:#A31515}
;; ...
#+END_SRC

It comes close to the theme without being it: a class is one scope, and a few
of a theme's rules look at the scopes around a token, or colour a ~meta.~ scope.

~#:lines? #t~ puts each line in a ~(span (@ (class "line")) ...)~, for line
numbers or a highlighted line.

** The values

#+BEGIN_SRC scheme
(: TmToken (Record (: text string) (: scopes (Vec string))))   ; the outermost scope first
(: TmSpan  (Record (: text string) (: style TmStyle)))
(: TmStyle (Record (: color (Option string))                   ; None: the theme's own
                   (: background (Option string))
                   (: bold bool) (: italic bool) (: underline bool) (: strikethrough bool)))
#+END_SRC

~TmHighlighter~ and ~TmLanguage~ are opaque. Everything is prefixed ~Tm~, so
that the types can stand beside another library's.

Some choices, made once here so that a user of the values need not:

- A line is ended by ~\r\n~, ~\r~ or ~\n~, and a newline at the very end of the
  code ends its last line rather than starting another.
- A span in the theme's default colour, or the editor's, has ~None~: the ~pre~
  around the code has that colour already. (Dark+ does not name a default for
  its tokens, and TextMateSharp calls that black.)
- Whitespace is plain unless something would show on it: a background, or a
  line under or through it.
- Neighbouring spans that look alike are one span.

** The functions

| ~(highlighter #:theme #:grammars #:line-time-limit)~ | the grammars and a theme; build it once                  |
| ~(find-language hl name)~                            | ~(Option TmLanguage)~, by id, alias or extension         |
| ~(language-names hl)~                                | every language's id                                      |
| ~(language-id lang)~                                 | its id                                                   |
| ~(tokenize lang code)~                               | ~(Vec (Vec TmToken))~, a vec a line; no theme            |
| ~(highlight lang code)~                              | ~(Vec (Vec TmSpan))~, coloured by the theme              |
| ~(token-classes token)~                              | the classes ~#:style 'classes~ gives it                  |
| ~(highlight->node lang code #:style #:lines?)~       | a ~pre~ and a ~code~ element around the code             |
| ~(highlight->nodes lang code #:style #:lines?)~      | only the ~code~ element's children                       |
| ~(code->node hl name code #:style #:lines?)~         | the same, named as a Markdown fence names it; plain text when the language is unknown |
| ~(highlighter-css hl #:selector)~                    | a stylesheet for the classes in the theme                |
| ~(highlighter-foreground hl)~, ~-background~         | the editor's colours                                     |

~code->node~ is the one for Markdown: the first word of ~"scheme title=x"~
names the language, and an unknown one is set as plain text, in the same ~pre~.

~#:grammars~ adds grammars of one's own: each a VS Code extension's
~package.json~, or the directory it is in. ~#:line-time-limit~ is how long one
line may take, 500 milliseconds unless it says otherwise; a line that takes
longer is cut there.

** Bjolang's grammar

~grammars/bjolang/~ is laid out as a VS Code extension is: ~package.json~, the
grammar in ~syntaxes/bjolang.tmLanguage.json~ and a language configuration, so
the directory can be one. Every highlighter reads it, from beside the ~src/~
the package's DLL is in.

It reads what Bjolang's reader reads: comments, strings over several lines,
interpolated strings with an expression in each hole, characters, booleans,
numbers, keywords, type variables and quoted symbols. It knows the special
forms, what a definition names and what a signature does, ~->~ and its kin,
capitalised names as types, and ~.Member~ and ~.-Property~.

** Installing it

#+BEGIN_SRC scheme
(depends
  (package (name (textmate-bjolang))
           (version (version-at-least "0.1.0"))
           (source (git (url "https://rikspucko.koketteriet.se/bjoli/textmate-bjolang")))))
#+END_SRC

The package declares TextMateSharp as a NuGet package, and a program using it
declares nothing itself. Building one needs the .NET SDK's ~dotnet~ command,
which restores it; see ~Docs/Projects.org~ in Bjolang, "NuGet packages", for
what that means for where the program runs. TextMateSharp runs its regular
expressions with Oniguruma, a native library, which the package carries for
Linux, macOS and Windows.

** Running the tests

#+BEGIN_SRC sh
bjo run tests/demo.bjo      ;; 27 checks
#+END_SRC

** Licence

MPL-2.0. TextMateSharp, TextMateSharp.Grammars and Onigwrap are MIT, and
Oniguruma is BSD-2-Clause. VS Code's grammars and themes, which
TextMateSharp.Grammars carries, keep their own licences.