Class TextSegmenter

Namespace

Egami.Fonts.Unicode

Assembly

Egami.Fonts.dll

Text boundaries from UAX #29: extended grapheme clusters (what a reader sees as one character) and words, with Unicode's default rules and no dictionaries.

public static class TextSegmenter

Inheritance

object

TextSegmenter

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Examples

// A family emoji is one grapheme cluster of eight UTF-16 units; "can't" is one word.
string text = "\uD83D\uDC68\u200D\uD83D\uDC69\u200D\uD83D\uDC67 can't stop";
var graphemes = new List<TextSegment>();
foreach (TextSegment grapheme in TextSegmenter.Graphemes(text))
{
    graphemes.Add(grapheme);
}

var words = new List<string>();
foreach (TextSegment word in TextSegmenter.Words(text))
{
    words.Add(text.Substring(word.Start, word.Length));
}

bool inside = TextSegmenter.IsGraphemeBoundary(text, 2); // false: inside the family

Remarks

Unpaired surrogates count as one character each. Enumerators walk the text once and allocate nothing.

Methods

Graphemes(ReadOnlySpan<char>)

Enumerates the extended grapheme clusters of a text.

public static GraphemeEnumerator Graphemes(ReadOnlySpan<char> text)

Parameters

text ReadOnlySpan<char>

The text.

Returns

GraphemeEnumerator

An enumerator of the clusters in order.

IsGraphemeBoundary(ReadOnlySpan<char>, int)

Gets whether an index is an extended grapheme cluster boundary.

public static bool IsGraphemeBoundary(ReadOnlySpan<char> text, int index)

Parameters

text ReadOnlySpan<char>

The text.

index int

An index from 0 to the text's length.

Returns

bool

true at the start and end of the text and between clusters.

Exceptions

ArgumentOutOfRangeException

index is negative or past the end.

Words(ReadOnlySpan<char>)

Enumerates the segments between word boundaries: words, and the spaces, punctuation and symbols between them, each as its own segment.

public static WordEnumerator Words(ReadOnlySpan<char> text)

Parameters

text ReadOnlySpan<char>

The text.

Returns

WordEnumerator

An enumerator of the segments in order.