• textmate-bjolang — syntax highlighting, as Bjolang values
  • What is highlighted
  • Inline or classes
  • The values
  • The functions
  • Bjolang's grammar
  • Installing it
  • Running the tests
  • Licence

textmate-bjolang — syntax highlighting, as Bjolang values

Code highlighted the way VS Code highlights it: its TextMate grammars and its colour themes, run by TextMateSharp, and handed over as Bjolang values — lines of tokens and their scopes, lines of coloured spans — and as (text xml) nodes, ready for a page.

(import (textmate-bjolang core))

(def hl (highlighter #:theme 'dark-plus))

(match (find-language hl "c#")
  ((Some cs) (highlight->node cs "var x = 1; // hi"))
  (None ...))
;; (pre (@ (class "tm") (style "color:#D4D4D4;background-color:#1E1E1E"))
;;      (code (@ (class "language-csharp"))
;;            (span (@ (style "color:#569CD6")) "var") " "
;;            (span (@ (style "color:#9CDCFE")) "x") " = "
;;            (span (@ (style "color:#B5CEA8")) "1") "; "
;;            (span (@ (style "color:#6A9955")) "// hi")))

(text xml write)'s node->xml-string turns the node into markup.

What is highlighted

VS Code's own grammars, 64 languages from C to YAML, and Bjolang, whose grammar is part of this package. A language is found by its id, an alias or an extension, in any case: "csharp", "C#", "cs" and ".cs" are the same language, and so are "bjolang", "bjo" and ".bjo".

The themes are VS Code's: 'dark-plus 'light-plus 'dark 'light 'monokai 'dimmed-monokai 'one-dark 'atom-one-dark 'atom-one-light 'dracula 'solarized-dark 'solarized-light 'quiet-light 'kimbie-dark 'tomorrow-night-blue 'abyss 'red 'high-contrast-dark 'high-contrast-light 'visual-studio-dark 'visual-studio-light.

Inline or classes

A node is coloured one of two ways.

#:style 'inline, the default, writes the theme's colours into style attributes. The page needs nothing else, so it works in a feed, in a mail, or pasted anywhere.

#:style 'classes writes class names taken from the grammar's scopes, and no colours at all:

(highlight->node cs "var x = 1; // hi" #:style 'classes)
;; (pre (@ (class "tm"))
;;      (code (@ (class "language-csharp"))
;;            (span (@ (class "tm-storage tm-storage-type")) "var") " "
;;            (span (@ (class "tm-entity tm-entity-name-variable")) "x") " "
;;            (span (@ (class "tm-keyword tm-keyword-operator")) "=") " "
;;            (span (@ (class "tm-constant tm-constant-numeric")) "1")
;;            (span (@ (class "tm-punctuation")) ";") " "
;;            (span (@ (class "tm-comment")) "// hi")))

A span has the general class and the particular one, from TextMate's conventional scope names: tm-keyword to colour every keyword alike, tm-keyword-control to set if and return apart. The stylesheet is yours to write, and a light and a dark one are only a media query apart. Or highlighter-css writes one from a theme:

(highlighter-css (highlighter #:theme 'light-plus) #:selector ".tm")
;; .tm{color:#000000;background-color:#FFFFFF}
;; .tm .tm-comment{color:#008000}
;; .tm .tm-string{color:#A31515}
;; ...

It comes close to the theme without being it: a class is one scope, and a few of a theme's rules look at the scopes around a token, or colour a meta. scope.

#:lines? #t puts each line in a (span (@ (class "line")) ...), for line numbers or a highlighted line.

The values

(: TmToken (Record (: text string) (: scopes (Vec string))))   ; the outermost scope first
(: TmSpan  (Record (: text string) (: style TmStyle)))
(: TmStyle (Record (: color (Option string))                   ; None: the theme's own
                   (: background (Option string))
                   (: bold bool) (: italic bool) (: underline bool) (: strikethrough bool)))

TmHighlighter and TmLanguage are opaque. Everything is prefixed Tm, so that the types can stand beside another library's.

Some choices, made once here so that a user of the values need not:

• A line is ended by \r\n, \r or \n, and a newline at the very end of the code ends its last line rather than starting another.
• A span in the theme's default colour, or the editor's, has None: the pre around the code has that colour already. (Dark+ does not name a default for its tokens, and TextMateSharp calls that black.)
• Whitespace is plain unless something would show on it: a background, or a line under or through it.
• Neighbouring spans that look alike are one span.

The functions

(highlighter #:theme #:grammars #:line-time-limit)	the grammars and a theme; build it once
(find-language hl name)	(Option TmLanguage), by id, alias or extension
(language-names hl)	every language's id
(language-id lang)	its id
(tokenize lang code)	(Vec (Vec TmToken)), a vec a line; no theme
(highlight lang code)	(Vec (Vec TmSpan)), coloured by the theme
(token-classes token)	the classes #:style 'classes gives it
(highlight->node lang code #:style #:lines?)	a pre and a code element around the code
(highlight->nodes lang code #:style #:lines?)	only the code element's children
(code->node hl name code #:style #:lines?)	the same, named as a Markdown fence names it; plain text when the language is unknown
(highlighter-css hl #:selector)	a stylesheet for the classes in the theme
(highlighter-foreground hl), -background	the editor's colours

code->node is the one for Markdown: the first word of "scheme title=x" names the language, and an unknown one is set as plain text, in the same pre.

#:grammars adds grammars of one's own: each a VS Code extension's package.json, or the directory it is in. #:line-time-limit is how long one line may take, 500 milliseconds unless it says otherwise; a line that takes longer is cut there.

Bjolang's grammar

grammars/bjolang/ is laid out as a VS Code extension is: package.json, the grammar in syntaxes/bjolang.tmLanguage.json and a language configuration, so the directory can be one. Every highlighter reads it, from beside the src/ the package's DLL is in.

It reads what Bjolang's reader reads: comments, strings over several lines, interpolated strings with an expression in each hole, characters, booleans, numbers, keywords, type variables and quoted symbols. It knows the special forms, what a definition names and what a signature does, -> and its kin, capitalised names as types, and .Member and .-Property.

Installing it

(depends
  (package (name (textmate-bjolang))
           (version (version-at-least "0.1.0"))
           (source (git (url "https://github.com/bjoli/textmate-bjolang")))))

The package declares TextMateSharp as a NuGet package, and a program using it declares nothing itself. Building one needs the .NET SDK's dotnet command, which restores it; see Docs/Projects.org in Bjolang, "NuGet packages", for what that means for where the program runs. TextMateSharp runs its regular expressions with Oniguruma, a native library, which the package carries for Linux, macOS and Windows.

Running the tests

bjo run tests/demo.bjo      ;; 27 checks

Licence

MPL-2.0. TextMateSharp, TextMateSharp.Grammars and Onigwrap are MIT, and Oniguruma is BSD-2-Clause. VS Code's grammars and themes, which TextMateSharp.Grammars carries, keep their own licences.