Table of Contents

Unicode text and shaping

Egami.Fonts carries its own Unicode 17.0 data and the Unicode algorithms text shaping builds on: character properties, the bidirectional algorithm and text boundaries, in the Egami.Fonts.Unicode namespace. On top of them, Egami.Fonts.Shaping shapes text with a font's OpenType tables, as HarfBuzz does, with HarfBuzz's script shapers (Arabic, Syriac, N'Ko and Mongolian joining, Hebrew, the nine Indic scripts, Khmer, Myanmar, Thai, Lao, Hangul, and the Universal Shaping Engine for Tibetan, Sinhala, Javanese and the other complex scripts), itemization, font fallback and drawing on the canvas.

The tables are generated from the Unicode Character Database by tools/make-unicode-tables.py and never come from the .NET runtime, so the same text gives the same answer on every runtime and machine. UnicodeData.UnicodeVersion says which version they follow.

Character properties

UnicodeData answers per code point: the general category (as the runtime's UnicodeCategory), the Script and its script extensions, the BidiClass, the mirroring character and BidiPairedBracketType, the JoiningType and JoiningGroup of Arabic and Syriac letters, the canonical combining class, the VerticalOrientation, the EastAsianWidth, the LineBreakClass (UAX #14), the emoji properties and whether a character is default ignorable.

// Egami's own Unicode 17.0 tables: the same answer on every runtime and machine.
Script script = UnicodeData.GetScript('\u0915');                          // Devanagari
ReadOnlySpan<FontTag> tags = script.OpenTypeTags;                         // dev2, then deva
BidiClass bidi = UnicodeData.GetBidiClass('\u0627');                      // ArabicLetter
JoiningType joining = UnicodeData.GetJoiningType('\u0628');               // DualJoining
JoiningGroup group = UnicodeData.GetJoiningGroup('\u0628');               // Beh
VerticalOrientation upright = UnicodeData.GetVerticalOrientation('\u6F22');   // Upright
EastAsianWidth width = UnicodeData.GetEastAsianWidth('\u6F22');               // Wide
bool emoji = UnicodeData.IsEmojiPresentation(0x1F600);                    // true
UnicodeData.TryGetBidiMirroringGlyph('(', out int mirror);                // ')'
BidiPairedBracketType bracket = UnicodeData.GetBidiPairedBracketType('['); // Open
  • A Script has its ISO 15924 Code, its Unicode Name and the OpenType tags fonts use for it, newest first (dev2 then deva). Common, Inherited and Unknown are the shared scripts, and fonts know them as DFLT.
  • Every lookup is a few array reads and allocates nothing. Code points run from 0 to 0x10FFFF; lone surrogates are code points too.

Bidirectional text

Bidi.Resolve runs the Unicode Bidirectional Algorithm (UAX #9) in full: embeddings, overrides and isolates up to depth 125, paired brackets and the weak and neutral rules. It returns a BidiResult with a level per UTF-16 unit and the BidiRuns of equal level; Reorder gives one line's runs from left to right. TextDirection.Auto, the default, takes each paragraph's direction from its first strong character; LeftToRight and RightToLeft force it.

// A Hebrew word inside English text: three runs, the middle one drawn right to left.
string text = "abc \u05D0\u05D1\u05D2 def";
using BidiResult result = Bidi.Resolve(text, TextDirection.Auto);
int paragraph = result.ParagraphLevel;           // 0: the first strong character is Latin
foreach (BidiRun run in result.Reorder(0, result.Length))
{
    Console.WriteLine($"{run.Start}+{run.Length} {(run.IsRightToLeft ? "right to left" : "left to right")}");
}
  • Each paragraph separator ends a paragraph, and each paragraph is resolved on its own.
  • Levels lays each paragraph out as one line. When text is wrapped, call Reorder per line: trailing spaces then take the paragraph's level, as rule L1 asks.
  • The result rents its arrays; dispose it when done.
  • Unicode's BidiTest.txt and BidiCharacterTest.txt pass in full.

Graphemes and words

TextSegmenter.Graphemes walks extended grapheme clusters (UAX #29): what a reader sees as one character, such as a letter with its accents, a Hangul syllable, a Devanagari conjunct or an emoji sequence. TextSegmenter.Words walks the segments between word boundaries with Unicode's default rules. Both return a TextSegment per segment through a GraphemeEnumerator or WordEnumerator that allocates nothing; IsGraphemeBoundary checks one index.

// A family emoji is one grapheme cluster of eight UTF-16 units; "can't" is one word.
string text = "\uD83D\uDC68\u200D\uD83D\uDC69\u200D\uD83D\uDC67 can't stop";
var graphemes = new List<TextSegment>();
foreach (TextSegment grapheme in TextSegmenter.Graphemes(text))
{
    graphemes.Add(grapheme);
}

var words = new List<string>();
foreach (TextSegment word in TextSegmenter.Words(text))
{
    words.Add(text.Substring(word.Start, word.Length));
}

bool inside = TextSegmenter.IsGraphemeBoundary(text, 2); // false: inside the family
  • Words use no dictionary, so Thai, Lao, Khmer and CJK text without spaces is not split into words.
  • Line breaking (UAX #14) arrives with paragraph layout in 2.8.2.
  • Unicode's GraphemeBreakTest.txt and WordBreakTest.txt pass in full.

Shaping a run

TextShaper.ShapeRun shapes one run of text in one font, as HarfBuzz's hb_shape: the font's GSUB substitutions (ligatures, contextual and alternate forms) and GPOS positioning (kerning, marks, cursive attachment) with the default features, normalization against the font's character map, and the fallbacks HarfBuzz uses for fonts that lack them. RunProperties gives the script (taken from the text when null), a BCP 47 language (none by default: the font's default forms, never the machine's culture), the direction, feature settings and the ClusterLevel. The result lands in a reusable GlyphBuffer: glyph ids, the UTF-16 index each glyph's cluster starts at, GlyphFlags and a GlyphPosition per glyph, in visual order and in the font's units.

// One run, one font, as HarfBuzz's hb_shape: ligatures, kerning and marks from the font's GSUB and GPOS.
var font = new Font(face, 24);
var glyphs = new GlyphBuffer();
string text = "office";
TextShaper.ShapeRun(text, 0, text.Length, font, new RunProperties { Script = Script.Latin }, glyphs);

for (int i = 0; i < glyphs.Count; i++)
{
    ushort id = glyphs.GlyphIds[i];
    int cluster = glyphs.Clusters[i];                 // index in text the glyph came from
    GlyphPosition position = glyphs.Positions[i];     // in the font's units (pixels here)
    bool breakable = (glyphs.Flags[i] & GlyphFlags.UnsafeToBreak) == 0;
}

float width = glyphs.Advance;
  • GlyphFlags.UnsafeToBreak marks where breaking the text needs shaping again; line layout uses it to re-shape only there.
  • A run is one script in one direction; TextShaper.Shape splits real text into runs.
  • The engine gives exactly hb-shape's glyphs, clusters, positions and flags on HarfBuzz's own tests and on Egami's corpus.

Shaping text

TextShaper.Shape shapes a whole string: it splits it into lines at hard breaks ( , , , U+0085, U+2028, U+2029), resolves each line's bidi levels, splits the line into runs of one level, one script, one span and one font, shapes each run as ShapeRun does and returns the runs in display order. Nothing wraps; that is line layout's job.

// Whole text: lines at hard breaks, bidi levels, script runs, fallback fonts, runs in display order.
var font = new Font(face, 24);
var options = new ShapingOptions { Fallback = new FontFallback(fonts.Families) };
ShapedText shaped = TextShaper.Shape("Shalom: \u05E9\u05DC\u05D5\u05DD!", font, options);

float x = 0;
foreach (ShapedRun run in shaped.Lines[0].Runs)
{
    GlyphRun glyphs = run.ToGlyphRun(new System.Numerics.Vector2(x, 0), shaped.Text); // what a canvas draws
    x += run.Advance;
}
  • ShapedText.Lines holds a ShapedLine per line, with its ParagraphLevel, its Runs from left to right, its Advance and the highest ascent and lowest descent of its fonts. Each ShapedRun has its Font, BidiLevel, Script, Language, text range and GlyphBuffer; ToGlyphRun turns it into the GlyphRun a canvas draws.
  • ShapingOptions.Direction is Auto by default: each paragraph takes the direction of its first strong character.
  • ShapingOptions.Spans give ranges their own font, language or features; spans must not overlap.
// A styled range: its own font, language or features.
var options = new ShapingOptions
{
    Language = "en",
    Spans = [new TextSpan(6, 6) { Font = font with { Kerning = false }, Features = [FontFeature.SmallCaps] }],
};
ShapedText shaped = TextShaper.Shape("Hello, World", font, options);

Font fallback

A font that does not map a character draws .notdef, unless ShapingOptions.Fallback names fonts to try. Fallback works on whole grapheme clusters, so a letter and its accents, or an emoji sequence, stay in one font: the first font that maps every character of the cluster takes it. The span's font comes first, then the FontFallback.Families (matched to the font's style), then the FontFallback.Resolver. Spaces and punctuation stay in the font before them when it has them, so fallback does not split words. Emoji presentation (a default-emoji character or VS16) prefers a color face (COLR, sbix or CBDT); VS15 prefers an outline face.

// Characters the font lacks: the caller's families first, then installed fonts only when asked for.
FontFallback mine = new(FontCollection.System.Families.Where(f => f.Name is "Segoe UI Emoji" or "Noto Color Emoji"));
FontFallback system = FontFallback.System;      // depends on the machine's fonts
FontFallback none = FontFallback.None;          // the default: missing characters draw as .notdef
  • FontFallback.None, the default, keeps output independent of the machine: only the fonts passed are used.
  • FontFallback.System resolves from the installed fonts through SystemFontFallback: fixed families per script on Windows, macOS and Linux (Segoe UI Emoji, Apple Color Emoji or Noto Color Emoji for emoji; the language picks Japanese, Chinese or Korean faces for Han), then any installed face that maps the cluster. Output then depends on the fonts installed.
  • An IFontFallbackResolver of your own receives the cluster's text, script, language and the font's style.

Features

A FontFeature sets one OpenType feature for the whole run or a range of it: 0 turns a default feature off, 1 turns a feature on, higher values pick an alternate. FontFeature.Parse reads HarfBuzz's syntax (liga=0, +smcp, ss01, aalt=2, kern[3:5]=0); the presets cover common cases.

// HarfBuzz's feature syntax, or the presets.
FontFeature noLigatures = FontFeature.Parse("liga=0");
FontFeature figures = FontFeature.TabularFigures;
FontFeature alternates = FontFeature.StylisticSet(1);
FontFeature someKerning = FontFeature.Parse("kern[0:3]=0"); // off for the first three characters only

var properties = new RunProperties { Features = [noLigatures, figures] };
TextShaper.ShapeRun(text, 0, text.Length, font, properties, glyphs);

Layout tables

Typeface.Layout returns an OpenTypeLayout, which lists the scripts, language systems and features the font's GSUB and GPOS offer. A table whose structure is damaged is ignored whole, with a Font.BadTable diagnostic, as HarfBuzz ignores it; a font that drives shaping past its limits stops with Font.ShapingLimit.

// What the font's layout tables list.
OpenTypeLayout layout = face.Layout;
IReadOnlyList<FontTag> scripts = layout.Scripts;                                // DFLT, cyrl, grek, latn
IReadOnlyList<FontTag> languages = layout.GetLanguageSystems(new FontTag("latn"));
IReadOnlyList<FontTag> features = layout.GetFeatures(new FontTag("latn"));      // liga, kern, ...

Recipes

Short answers to common tasks in this area. Each one compiles and runs in Egami's test run.

Emoji in a caption

// A caption mixing words and emoji: the emoji fall back to the color face, cluster by cluster, so a ZWJ sequence or a flag
// stays whole; one call shapes and draws every run.
string caption = "Launch day \U0001F680 thanks, team \U0001F389";
var fallback = new FontCollection();
fallback.Add(emoji);
var options = new Egami.Fonts.Shaping.ShapingOptions { Fallback = new Egami.Fonts.Shaping.FontFallback(fallback.Families) };
using (RasterCanvas<Rgba8> canvas = image.CreateCanvas())
{
    canvas.DrawText(caption, new Vector2(8, 32), new Font(text, 20), Rgba8.Black, options);
}

Color glyphs draw in color on every canvas; on a raster they are cached as images, like other glyphs. A cluster with emoji presentation prefers a color face from the fallback list; text presentation (VS15) prefers an outline face.

An Arabic caption

// An Arabic caption, right-aligned: the shaper joins the letters and puts the words in right-to-left order, and the measured
// advance places the line's right end at the margin.
using Raster<Rgba8> photo = Raster.Create(640, 120, new Rgba8(40, 60, 80, 255));
var font = new Font(arabic, 32);
string caption = "\u0645\u0631\u062D\u0628\u0627\u0020\u0628\u0627\u0644\u0639\u0627\u0644\u0645";
float width = font.MeasureText(caption).Advance;
using (RasterCanvas<Rgba8> canvas = photo.CreateCanvas())
{
    canvas.DrawText(caption, new Vector2(photo.Width - 24 - width, 80), font, Rgba8.White);
}

The shaper picks each letter's joining form and the bidi algorithm orders the words; the origin is still the line's left end, so right alignment subtracts the measured advance.

A Hindi label with fallback

// A label in Hindi and English with a Latin font: Devanagari clusters (conjuncts, vowel signs, the reph) fall back whole to
// the Devanagari face, and the Indic shaper reorders them.
var families = new FontCollection();
families.Add(devanagari);
var options = new Egami.Fonts.Shaping.ShapingOptions
{
    Language = "hi",
    Fallback = new Egami.Fonts.Shaping.FontFallback(families.Families),
};
using (RasterCanvas<Rgba8> canvas = image.CreateCanvas())
{
    canvas.DrawText("Price: \u0915\u0940\u092E\u0924\u0020\u0915\u094D\u0937\u0947\u0924\u094D\u0930", new Vector2(10, 40), new Font(latin, 22), Rgba8.Black, options);
}

Fallback works on whole grapheme clusters, so a conjunct never splits across two faces. FontFallback.System would find an installed Devanagari face instead, at the cost of output that depends on the machine.

Edit this page