Crate icu_normalizer

source ·
Expand description

Normalizing text into Unicode Normalization Forms.

This module is published as its own crate (icu_normalizer) and as part of the icu crate. See the latter for more details on the ICU4X project.

§Implementation notes

The normalizer operates on a lazy iterator over Unicode scalar values (Rust char) internally and iterating over guaranteed-valid UTF-8, potentially-invalid UTF-8, and potentially-invalid UTF-16 is a step that doesn’t leak into the normalizer internals. Ill-formed byte sequences are treated as U+FFFD.

The normalizer data layout is not based on the ICU4C design at all. Instead, the normalization data layout is a clean-slate design optimized for the concept of fusing the NFD decomposition into the collator. That is, the decomposing normalizer is a by-product of the collator-motivated data layout.

Notably, the decomposition data structure is optimized for a starter decomposing to itself, which is the most common case, and for a starter decomposing to a starter and a non-starter on the Basic Multilingual Plane. Notably, in this case, the collator makes use of the knowledge that the second character of such a decomposition is a non-starter. Therefore, decomposition into two starters is handled by generic fallback path that looks the decomposition from an array by offset and length instead of baking a BMP starter pair directly into a trie value.

The decompositions into non-starters are hard-coded. At present in Unicode, these appear to be special cases falling into three categories:

  1. Deprecated combining marks.
  2. Particular Tibetan vowel sings.
  3. NFKD only: half-width kana voicing marks.

Hopefully Unicode never adds more decompositions into non-starters (other than a character decomposing to itself), but if it does, a code update is needed instead of a mere data update.

The composing normalizer builds on the decomposing normalizer by performing the canonical composition post-processing per spec. As an optimization, though, the composing normalizer attempts to pass through already-normalized text consisting of starters that never combine backwards and that map to themselves if followed by a character whose decomposition starts with a starter that never combines backwards.

As a difference with ICU4C, the composing normalizer has only the simplest possible passthrough (only one inversion list lookup per character in the best case) and the full decompose-then-canonically-compose behavior, whereas ICU4C has other paths between these extremes. The ICU4X collator doesn’t make use of the FCD concept at all in order to avoid doing the work of checking whether the FCD condition holds.

Re-exports§

Modules§

  • error 🔒
    Normalizer-specific error
  • properties
    Access to the Unicode properties or property-based operations that are required for NFC and NFD.
  • provider
    🚧 [Unstable] Data provider struct definitions for this ICU4X component.
  • uts46
    Bundles the part of UTS 46 that makes sense to implement as a normalization.

Macros§

Structs§

  • CharacterAndClass 🔒
    Pack a char and a CanonicalCombiningClass in 32 bits (the former in the lower 24 bits and the latter in the high 8 bits). The latter can be initialized to 0xFF upon creation, in which case it can be actually set later by calling set_ccc_from_trie_if_not_already_set. This is a micro optimization to avoid the Canonical Combining Class trie lookup when there is only one combining character in a sequence. This type is intentionally non-Copy to get compiler help in making sure that the class is set on the instance on which it is intended to be set and not on a temporary copy.
  • CharacterAndTrieValue 🔒
    Struct for holding together a character and the value looked up for it from the NFD trie in a more explicit way than an anonymous pair. Also holds a flag about the supplementary-trie provenance.
  • ComposingNormalizer
    A normalizer for performing composing normalization.
  • Composition
    An iterator adaptor that turns an Iterator over char into a lazily-decomposed and then canonically composed char sequence.
  • DecomposingNormalizer
    A normalizer for performing decomposing normalization.
  • Decomposition
    An iterator adaptor that turns an Iterator over char into a lazily-decomposed char sequence.
  • IsNormalizedSinkStr 🔒
  • IsNormalizedSinkUtf8 🔒
  • IsNormalizedSinkUtf16 🔒

Enums§

Constants§

Statics§

  • FDFA_NFKD 🔒
    The tail (everything after the first character) of the NFKD form U+FDFA as 16-bit units.

Functions§