Struct WordSegmenter 

pub struct WordSegmenter(WordSegmenter);

An ICU4X word-break segmenter, capable of finding word breakpoints in strings.

Tuple Fields

0: WordSegmenter

Implementations

impl WordSegmenter

pub fn create_auto() -> Box<WordSegmenter>

Construct a WordSegmenter with automatically selecting the best available LSTM or dictionary payload data, using compiled data. This does not assume any content locale.

Note: currently, it uses dictionary for Chinese and Japanese, and LSTM for Burmese, Khmer, Lao, and Thai.

pub fn create_auto_with_content_locale( locale: &Locale, ) -> Result<Box<WordSegmenter>, DataError>

Construct a WordSegmenter with automatically selecting the best available LSTM or dictionary payload data, using compiled data.

Note: currently, it uses dictionary for Chinese and Japanese, and LSTM for Burmese, Khmer, Lao, and Thai.

pub fn create_lstm() -> Box<WordSegmenter>

Construct a WordSegmenter with LSTM payload data for Burmese, Khmer, Lao, and Thai, using compiled data. This does not assume any content locale.

Note: currently, it uses dictionary for Chinese and Japanese, and LSTM for Burmese, Khmer, Lao, and Thai.

pub fn create_lstm_with_content_locale( locale: &Locale, ) -> Result<Box<WordSegmenter>, DataError>

Construct a WordSegmenter with LSTM payload data for Burmese, Khmer, Lao, and Thai, using compiled data.

Note: currently, it uses dictionary for Chinese and Japanese, and LSTM for Burmese, Khmer, Lao, and Thai.

pub fn create_dictionary() -> Box<WordSegmenter>

Construct a WordSegmenter with dictionary payload data for Chinese, Japanese, Burmese, Khmer, Lao, and Thai, using compiled data. This does not assume any content locale.

Note: currently, it uses dictionary for Chinese and Japanese, and dictionary for Burmese, Khmer, Lao, and Thai.

pub fn create_dictionary_with_content_locale( locale: &Locale, ) -> Result<Box<WordSegmenter>, DataError>

Construct a WordSegmenter with dictionary payload data for Chinese, Japanese, Burmese, Khmer, Lao, and Thai, using compiled data.

Note: currently, it uses dictionary for Chinese and Japanese, and dictionary for Burmese, Khmer, Lao, and Thai.

pub fn create_for_non_complex_scripts() -> Box<WordSegmenter>

Construct a WordSegmenter with no support for scripts requiring complex context dependent word breaks (Chinese, Japanese, Burmese, Khmer, Lao, and Thai), using compiled data. This does not assume any content locale.

pub fn create_for_non_complex_scripts_with_content_locale( locale: &Locale, ) -> Result<Box<WordSegmenter>, DataError>

Construct a WordSegmenter with no support for scripts requiring complex context dependent word breaks (Chinese, Japanese, Burmese, Khmer, Lao, and Thai), using compiled data.

pub fn segment_utf8<'a>( &'a self, input: &'a DiplomatStr, ) -> Box<WordBreakIteratorUtf8<'a>>

Segments a string.

Ill-formed input is treated as if errors had been replaced with REPLACEMENT CHARACTERs according to the WHATWG Encoding Standard.

pub fn segment_utf16<'a>( &'a self, input: &'a DiplomatStr16, ) -> Box<WordBreakIteratorUtf16<'a>>

Segments a string.

Ill-formed input is treated as if errors had been replaced with REPLACEMENT CHARACTERs according to the WHATWG Encoding Standard.

pub fn segment_latin1<'a>( &'a self, input: &'a [u8], ) -> Box<WordBreakIteratorLatin1<'a>>

Segments a Latin-1 string.