Struct SentenceSegmenter 

pub struct SentenceSegmenter {
    payload: DataPayload<SegmenterBreakSentenceV1>,
    payload_locale_override: Option<DataPayload<SegmenterBreakSentenceOverrideV1>>,
}

Supports loading sentence break data, and creating sentence break iterators for different string encodings.

Most segmentation methods live on SentenceSegmenterBorrowed, which can be obtained via SentenceSegmenter::new() or SentenceSegmenter::as_borrowed().

Content Locale

You can optionally provide a content locale to the SentenceSegmenter constructor. If you have information on the language of the text being segmented, providing this hint can produce higher-quality results.

If you have a content locale, use SentenceBreakOptions and a constructor begining with new. If you do not have a content locale use SentenceBreakInvariantOptions and a constructor beginning with try_new.

Examples

Segment a string:

use icu::segmenter::SentenceSegmenter;

let segmenter = SentenceSegmenter::new(Default::default());

let breakpoints: Vec<usize> =
    segmenter.segment_str("Hello World").collect();
assert_eq!(&breakpoints, &[0, 11]);

Segment a Latin1 byte string with a content locale:

use icu::locale::langid;
use icu::segmenter::options::SentenceBreakOptions;
use icu::segmenter::SentenceSegmenter;

let mut options = SentenceBreakOptions::default();
let langid = &langid!("en");
options.content_locale = Some(langid);
let segmenter = SentenceSegmenter::try_new(options).unwrap();

let breakpoints: Vec<usize> = segmenter
    .as_borrowed()
    .segment_latin1(b"Hello World")
    .collect();
assert_eq!(&breakpoints, &[0, 11]);

Successive boundaries can be used to retrieve the sentences. In particular, the first boundary is always 0, and the last one is the length of the segmented text in code units.

use itertools::Itertools;
let text = "Ceci tuera cela. Le livre tuera l’édifice.";
let sentences: Vec<&str> = segmenter
    .segment_str(text)
    .tuple_windows()
    .map(|(i, j)| &text[i..j])
    .collect();
assert_eq!(
    &sentences,
    &["Ceci tuera cela. ", "Le livre tuera l’édifice."]
);

Fields

payload: DataPayload<SegmenterBreakSentenceV1>

payload_locale_override: Option<DataPayload<SegmenterBreakSentenceOverrideV1>>

Implementations

impl SentenceSegmenter

pub const fn new( _options: SentenceBreakInvariantOptions, ) -> SentenceSegmenterBorrowed<'static>

Constructs a SentenceSegmenterBorrowed with an invariant locale and compiled data.

✨ Enabled with the compiled_data Cargo feature.

📚 Help choosing a constructor

pub fn try_new(options: SentenceBreakOptions<'_>) -> Result<Self, DataError>

Constructs a SentenceSegmenter for a given options and using compiled data.

✨ Enabled with the compiled_data Cargo feature.

📚 Help choosing a constructor

pub fn try_new_unstable<D>( provider: &D, options: SentenceBreakOptions<'_>, ) -> Result<Self, DataError>

where D: DataProvider<SegmenterBreakSentenceV1> + DataProvider<SegmenterBreakSentenceOverrideV1> + ?Sized,

A version of Self::try_new that uses custom data provided by a DataProvider.

📚 Help choosing a constructor

⚠️ The bounds on provider may change over time, including in SemVer minor releases.

pub fn as_borrowed(&self) -> SentenceSegmenterBorrowed<'_>

Constructs a borrowed version of this type for more efficient querying.

Most useful methods for segmentation are on this type.

Trait Implementations

impl Debug for SentenceSegmenter

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.