Exported types and functions

Types

OrthographicSystem

An abstract type for orthographic systems.

TokenCategory

An abstract type for token categories.

LexicalToken

Category of alphabetic tokens.

NumericToken

Category of numeric tokens.

PunctuationToken

Category of punctuation tokens.

Functions

Public functions implemented for all subtypes of OrthographicSystem.

codepoints

Category of punctuation tokens.

tokentypes

Delegate to specific functions based on type’s orthography trait value.

tokentypes(x)

It is an error to invoke the tokentypes function on anything but an orthographic system.

tokentypes(_, x)

Orthographic systems must implement tokentypes.

tokentypes(_, ortho, s)

Implement tokentypes function for SimpleAscii.

tokentypes(ortho)

Implement tokentypes function for WSTokenizer.

tokentypes(ortho)

validcp

True if ch appears in list of all valid characters (codepoints) for this orthography.

validcp(ch, ortho)

ch is a string possibly including more than one Julia Char but representing a single character in the orthographic system ortho.

validstring

True if all chars in s are valid.

validstring(s, ortho)

tokenize

Delegate to specific functions based on type’s orthography trait value.

tokenize(s, x)

It is an error to invoke the tokenize function on anything but an orthographic system.

tokenize(_, s, x)

Orthographic systems must implement tokenize.

tokenize(_, s, ortho)

Tokenize citable node cn using the tokenizer of the given orthographic system.

tokenize(psg, ortho; edition, exemplar)

The return value is a list of pairings of a CitablePassage and a token category. The citable node is citable at the level of the token.

Tokenize corpus c using the tokenizer of the given orthographic system.

tokenize(c, ortho; edition, exemplar)

The return value is a list of pairings of a CitablePassage and a token category. The citable node is citable at the level of the token.

Tokenize document doc using the tokenizer of the given orthographic system.

tokenize(doc, ortho; edition, exemplar)

The return value is a list of pairings of a CitablePassage and a token category. The citable node is citable at the level of the token.

Implement tokenize function for SimpleAscii orthography.

tokenize(s, o)

Implement tokenize function for WSTokenizer orthography.

tokenize(s, o)

Working with text corpora:

corpus_histo

Create an ordered dictionary of text values for tokens in corpus c. Optional parameters let you filter the results to include only tokens of a specified type and normalize the text value of tokens before counting.

corpus_histo(c, ortho; filterby, normalizer)

Other utilities

nfkc

Shorthand function to normalize string s to Unicode form NFKC.

nfkc(s)

Example implementation

SimpleAscii

An orthographic system for a basic alphabetic subset of the ASCII character set.

simpleAscii

Construct a SimpleAscii with correct member values.

simpleAscii()