Custom languages & highlighters

@nextgensoftwares/folio-plugin-code bundles a small headless tokenizer with a dozen grammars. When you need a language it doesn't ship, there are two ways in:

1. Add a grammar with defineLanguage() + registerLanguage(): synchronous, tiny, runs everywhere (browser, worker, Node export).
2. Plug in a whole highlighter (highlight.js, Prism, Shiki, a server, your own) with setHighlighter(). It can answer synchronously or with a promise.

Either way, highlighting happens inside layout, so the screen, the PDF and the DOCX show the same colours, and it never changes line breaks: lines are measured and broken from the plain text first; tokens only split each line into coloured pieces that tile it exactly (unless a token asks for a different font style, e.g. italic, which is drawn at the same advance).

Add a language

import { codePlugin, defineLanguage, PATTERNS as P } from '@nextgensoftwares/folio-plugin-code';

const ini = defineLanguage({
  name: 'ini',
  label: 'INI',
  aliases: ['cfg', 'conf'],
  rules: [
    [/[;#][^\n]*/, 'comment'],
    [/\[[^\]\n]*\]/, 'tag'],               // [section]
    [/[\w.-]+(?=\s*=)/, 'property'],       // key =
    P.dqString,
    P.number,
  ],
  literals: 'true false yes no on off',
});

const code = codePlugin({ languages: [ini] });   // or later: code.registerLanguage(ini)

Rules are tried in order at each position (first match wins); identifiers are then classified by the word lists (keywords, literals, types, builtins). A lazy grammar is { name, label, aliases, load: () => import('./my-lang').then(m => m.default) }. Registering a language after the document is laid out re-renders the blocks that use it.

Plug in a highlighter

A highlighter turns (code, language) into spans with scopes:

import type { Highlighter } from '@nextgensoftwares/folio-plugin-code';

const mine: Highlighter = {
  name: 'my-highlighter',
  languages: [{ name: 'lua', label: 'Lua', aliases: ['luau'] }], // shown in the language menus
  supports: (lang) => lang === 'lua' || lang === 'luau',          // optional, synchronous
  tokenize(code, lang) {                                          // sync, or return a Promise
    return [{ from: 0, to: 5, scope: 'keyword' }];
  },
  theme: { colors: { 'string.special': '#b35900' } },             // optional scope → colour
};

code.setHighlighter(mine);                       // or codePlugin({ highlighter: mine })
code.setHighlighter(mine, { mode: 'override' }); // also take over built-in languages
code.setHighlighter(null);                       // back to the built-ins only

Option (highlighterOptions / 2nd argument)	Default	Meaning
mode	'fallback'	fallback: only languages without a registered grammar; override: every language the highlighter supports
debounce	80	ms an async request waits while the text keeps changing
theme		a ScopeTheme layered over the highlighter's own

Scopes and colours

A span's scope is whatever the highlighter calls it: a highlight.js class (hljs-title.function_), a Prism token type (class-name), a TextMate scope (entity.name.function.ts) or your own word. A list of scopes is tried in order. Folio maps them onto its token types (keyword, string, number, comment, function, type, literal, builtin, operator, punctuation, meta, tag, attribute, property, variable, regexp), so the code theme you picked (GitHub, One, Solarized, Monochrome, light or dark) colours them like built-in tokens.

A ScopeTheme overrides that, matching a scope or any dotted prefix of it:

const theme = {
  types: { 'variable.other.constant': 'literal' }, // scope → token type (null = uncoloured)
  colors: { 'markup.heading': '#0550ae' },          // scope → fixed document colour
  italic: ['emphasis'],                             // scopes drawn italic
};

A span may also carry a color (and italic) directly; Shiki's adapter does that with its theme's colours. Fixed colours are document colours: the PDF and DOCX get them as-is, and on screen the active colour theme (e.g. dark mode) adapts them like any other text colour.

Async highlighters

tokenize may return a promise (a lazily imported library, a worker, a server). Results are cached per (language, text), and nothing waits on them:

• A block is laid out right away. While its answer is pending it borrows the tokens of the nearest earlier version of its text (the unchanged prefix keeps its colours, the unchanged suffix keeps them shifted), else it draws plain.
• While you type, requests are debounced, and a request still waiting for an older version of the same text is dropped, so one keystroke burst costs one call.
• When answers land (coalesced per tick) the plugin bumps that language's generation and the host's subscribeLayout re-lays out only its blocks.
• DOCX export awaits pending answers itself. For a PDF of the current layout, await code.whenHighlighted() before taking the layout (an editor host re-lays out on its own when answers arrive).

Errors (a throwing or rejecting tokenize) go to onError, and the block stays plain.

Adapters

The adapters ship with @nextgensoftwares/folio-plugin-code but import nothing: you pass the library instance, or a loader so it is fetched only when a block needs it. None of these libraries is a dependency of Folio.

highlight.js

import { codePlugin, highlightJs } from '@nextgensoftwares/folio-plugin-code';

// Eager: an instance you already have (menus list its languages).
import hljs from 'highlight.js/lib/common';
codePlugin({ highlighter: highlightJs(hljs) });

// Lazy: loaded with the first block in a language Folio doesn't bundle.
codePlugin({
  highlighter: highlightJs(() => import('highlight.js/lib/common'), {
    languages: [{ name: 'lua', label: 'Lua' }, { name: 'haskell', label: 'Haskell', aliases: ['hs'] }],
  }),
});

It uses highlight.js's public highlight() output; nested spans fall back to their container's meaning when the inner class means nothing to Folio.

Prism

import { prism } from '@nextgensoftwares/folio-plugin-code';

code.setHighlighter(prism(() => import('prismjs'), {
  // Prism grammars are separate modules that register themselves on the global Prism.
  loadLanguage: (lang) => import(`prismjs/components/prism-${lang}.js`),
}));

Shiki

import { shiki } from '@nextgensoftwares/folio-plugin-code';

code.setHighlighter(
  shiki(() => import('shiki').then((s) => s.createHighlighter({ themes: ['github-light'], langs: [] })), {
    theme: 'github-light',
  }),
  { mode: 'override' }, // TextMate grammars for every language, including the bundled ones
);

Shiki answers with its theme's colours, so pick a theme that suits your code box (light or dark). Grammars load on first use with loadLanguage; an id Shiki doesn't know stays plain.

Writing an adapter

Anything with tokenize(code, lang) works. The helpers the adapters are built from are exported too: htmlSpans(html, prefix) (span-per-class HTML output), prismSpans(stream), shikiSpans(lines), resolveTokens(spans, length, themes) (what the plugin does with your spans) and normalizeColor().

Try it

The playground registers an INI grammar (defineLanguage) and an async Lua highlighter (setHighlighter), neither bundled with Folio: open the Code book sample for the ini and lua blocks, or pick INI / Lua in any code block's language chip.