Crate icu_normalizer 

Normalizing text into Unicode Normalization Forms.

This module is published as its own crate (icu_normalizer) and as part of the icu crate. See the latter for more details on the ICU4X project.

Functionality

The top level of the crate provides normalization of input into the four normalization forms defined in UAX #15: Unicode Normalization Forms: NFC, NFD, NFKC, and NFKD.

Three kinds of contiguous inputs are supported: known-well-formed UTF-8 (&str), potentially-not-well-formed UTF-8, and potentially-not-well-formed UTF-16. Additionally, an iterator over char can be wrapped in a normalizing iterator.

The uts46 module provides the combination of mapping and normalization operations for UTS #46: Unicode IDNA Compatibility Processing. This functionality is not meant to be used by applications directly. Instead, it is meant as a building block for a full implementation of UTS #46, such as the idna crate.

The properties module provides the non-recursive canonical decomposition operation on a per char basis and the canonical compositon operation given two chars. It also provides access to the Canonical Combining Class property. These operations are primarily meant for HarfBuzz, the types CanonicalComposition, CanonicalDecomposition, and CanonicalCombiningClassMap implement the [harfbuzz_traits] if the harfbuzz_traits Cargo feature is enabled.

Notably, this normalizer does not provide the normalization “quick check” that can result in “maybe” in addition to “yes” and “no”. The normalization checks provided by this crate always give a definitive non-“maybe” answer.

Examples

let nfc = icu_normalizer::ComposingNormalizerBorrowed::new_nfc();
assert_eq!(nfc.normalize("a\u{0308}"), "ä");
assert!(nfc.is_normalized("ä"));

let nfd = icu_normalizer::DecomposingNormalizerBorrowed::new_nfd();
assert_eq!(nfd.normalize("ä"), "a\u{0308}");
assert!(!nfd.is_normalized("ä"));

Modules

latin1

Methods for normalizing Latin1 input into a UTF-16 sink.

properties

Access to the Unicode properties or property-based operations that are required for NFC and NFD.

provider

🚧 [Unstable] Data provider struct definitions for this ICU4X component.

uts46

Bundles the part of UTS 46 that makes sense to implement as a normalization.

Macros

ccc 🔒

composing_normalize_to 🔒

decomposing_normalize_to 🔒

normalizer_methods 🔒

Structs

CanonicalCombiningClass 🔒

This type exists as a shim for icu_properties CanonicalCombiningClass when the crate is disabled It should not be exposed to users.

CharacterAndClass 🔒

Pack a char and a CanonicalCombiningClass in 32 bits (the former in the lower 24 bits and the latter in the high 8 bits). The latter can be initialized to 0xFF upon creation, in which case it can be actually set later by calling set_ccc_from_trie_if_not_already_set. This is a micro optimization to avoid the Canonical Combining Class trie lookup when there is only one combining character in a sequence. This type is intentionally non-Copy to get compiler help in making sure that the class is set on the instance on which it is intended to be set and not on a temporary copy.

CharacterAndTrieValue 🔒

Struct for holding together a character and the value looked up for it from the NFD trie in a more explicit way than an anonymous pair. Also holds a flag about the supplementary-trie provenance.

ComposingNormalizer

A normalizer for performing composing normalization.

ComposingNormalizerBorrowed

Borrowed version of a normalizer for performing composing normalization.

Composition

An iterator adaptor that turns an Iterator over char into a lazily-decomposed and then canonically composed char sequence.

CompositionInner 🔒

DecomposingNormalizer

A normalizer for performing decomposing normalization.

DecomposingNormalizerBorrowed

Borrowed version of a normalizer for performing decomposing normalization.

Decomposition

An iterator adaptor that turns an Iterator over char into a lazily-decomposed char sequence.

DecompositionInner 🔒

IsNormalizedSinkStr 🔒

IsNormalizedSinkUtf8 🔒

IsNormalizedSinkUtf16 🔒

Uax15Policy 🔒

Enums

CanonicalCompositionsBorrowed 🔒

CanonicalCompositionsPayload 🔒

CanonicalCompositionsRef 🔒

IgnorableBehavior 🔒

Treatment of the ignorable marker (0xFFFFFFFF) in data.

Constants

BACKWARD_COMBINING_MARKER 🔒

Marker that the first character of the decomposition can combine backwards.

CCC_ABOVE 🔒

CCC_NOT_REORDERED 🔒

EMPTY_CHAR 🔒

EMPTY_U16 🔒

FDFA_MARKER 🔒

Marker value for U+FDFA in NFKD. (Unified with Hangul syllable marker, but they differ by NON_ROUND_TRIP_MARKER.)

HANGUL_L_BASE 🔒

Lead jamo base

HANGUL_L_COUNT 🔒

Lead jamo count

HANGUL_L_TRIE_VAL_BASE 🔒

Trie value base corresponding for L

HANGUL_N_COUNT 🔒

Vowel jamo count times trail jamo count

HANGUL_S_BASE 🔒

Syllable base

HANGUL_S_COUNT 🔒

Syllable count

HANGUL_T_BASE 🔒

Trail jamo base (deliberately off by one to account for the absence of a trail)

HANGUL_T_COUNT 🔒

Trail jamo count (deliberately off by one to account for the absence of a trail)

HANGUL_V_BASE 🔒

Vowel jamo base

HANGUL_V_COUNT 🔒

Vowel jamo count

HIGH_ZEROS_MASK 🔒

Mask for the bits have to be zero for this to be a BMP singleton decomposition, or value baked into the surrogate range.

IGNORABLE_MARKER 🔒

Marker for UTS 46 ignorables.

LOW_ZEROS_MASK 🔒

Mask for the bits have to be zero for this to be a complex decomposition.

NON_ROUND_TRIP_MARKER 🔒

Marker that the decomposition does not round trip via NFC.

Statics

FDFA_NFKD 🔒

The tail (everything after the first character) of the NFKD form U+FDFA as 16-bit units.

Traits

IteratorPolicy 🔒

Functions

ccc_from_trie_value 🔒

Extracts a canonical combining class (possibly zero) from a trie value.

char_from_u16 🔒

Convert a u16 obtained from data provider data to char.

char_from_u32 🔒

Convert a u32 obtained from data provider data to char.

cold_path 🔒

decomposition_starts_with_non_starter 🔒

Checks if a trie value signifies a character whose decomposition starts with a non-starter.

in_inclusive_range 🔒

in_inclusive_range16 🔒

likely 🔒

potential_passthrough_and_cannot_combine_backwards 🔒

See trie-value-format.md

starter_and_decomposes_to_self_except_replacement

See trie-value-format.md

starter_and_decomposes_to_self_impl 🔒

See trie-value-format.md

trie_value_has_ccc 🔒

Checks if a trie value carries a (non-zero) canonical combining class.

trie_value_indicates_non_decomposing_non_starter 🔒

Checks if the trie signifies a non-decomposing non-starter.

trie_value_indicates_special_non_starter_decomposition 🔒

Checks if the trie signifies a special non-starter decomposition.

unlikely 🔒

unwrap_or_gigo 🔒

If opt is Some, unwrap it. If None, panic if debug assertions are enabled and return default if debug assertions are not enabled.

Type Aliases

CombiningBuffer 🔒

CompositionTrie 🔒

Trie 🔒