Struct icu_segmenter::sentence::SentenceSegmenter
source · pub struct SentenceSegmenter {
payload: DataPayload<SentenceBreakDataV1Marker>,
}
Expand description
Supports loading sentence break data, and creating sentence break iterators for different string encodings.
§Examples
Segment a string:
use icu::segmenter::SentenceSegmenter;
let segmenter = SentenceSegmenter::new();
let breakpoints: Vec<usize> =
segmenter.segment_str("Hello World").collect();
assert_eq!(&breakpoints, &[0, 11]);
Segment a Latin1 byte string:
use icu::segmenter::SentenceSegmenter;
let segmenter = SentenceSegmenter::new();
let breakpoints: Vec<usize> =
segmenter.segment_latin1(b"Hello World").collect();
assert_eq!(&breakpoints, &[0, 11]);
Successive boundaries can be used to retrieve the sentences. In particular, the first boundary is always 0, and the last one is the length of the segmented text in code units.
use itertools::Itertools;
let text = "Ceci tuera cela. Le livre tuera l’édifice.";
let sentences: Vec<&str> = segmenter
.segment_str(text)
.tuple_windows()
.map(|(i, j)| &text[i..j])
.collect();
assert_eq!(
&sentences,
&["Ceci tuera cela. ", "Le livre tuera l’édifice."]
);
Fields§
§payload: DataPayload<SentenceBreakDataV1Marker>
Implementations§
source§impl SentenceSegmenter
impl SentenceSegmenter
sourcepub fn new() -> Self
pub fn new() -> Self
Constructs a SentenceSegmenter
with an invariant locale and compiled data.
✨ Enabled with the compiled_data
Cargo feature.
sourcepub fn try_new_with_any_provider(
provider: &(impl AnyProvider + ?Sized),
) -> Result<Self, SegmenterError>
pub fn try_new_with_any_provider( provider: &(impl AnyProvider + ?Sized), ) -> Result<Self, SegmenterError>
A version of [Self :: new
] that uses custom data provided by an AnyProvider
.
sourcepub fn try_new_unstable<D>(provider: &D) -> Result<Self, SegmenterError>
pub fn try_new_unstable<D>(provider: &D) -> Result<Self, SegmenterError>
A version of Self::new
that uses custom data provided by a DataProvider
.
sourcepub fn segment_str<'l, 's>(
&'l self,
input: &'s str,
) -> SentenceBreakIteratorUtf8<'l, 's>
pub fn segment_str<'l, 's>( &'l self, input: &'s str, ) -> SentenceBreakIteratorUtf8<'l, 's>
Creates a sentence break iterator for an str
(a UTF-8 string).
There are always breakpoints at 0 and the string length, or only at 0 for the empty string.
sourcepub fn segment_utf8<'l, 's>(
&'l self,
input: &'s [u8],
) -> SentenceBreakIteratorPotentiallyIllFormedUtf8<'l, 's>
pub fn segment_utf8<'l, 's>( &'l self, input: &'s [u8], ) -> SentenceBreakIteratorPotentiallyIllFormedUtf8<'l, 's>
Creates a sentence break iterator for a potentially ill-formed UTF8 string
Invalid characters are treated as REPLACEMENT CHARACTER
There are always breakpoints at 0 and the string length, or only at 0 for the empty string.
sourcepub fn segment_latin1<'l, 's>(
&'l self,
input: &'s [u8],
) -> SentenceBreakIteratorLatin1<'l, 's>
pub fn segment_latin1<'l, 's>( &'l self, input: &'s [u8], ) -> SentenceBreakIteratorLatin1<'l, 's>
Creates a sentence break iterator for a Latin-1 (8-bit) string.
There are always breakpoints at 0 and the string length, or only at 0 for the empty string.
sourcepub fn segment_utf16<'l, 's>(
&'l self,
input: &'s [u16],
) -> SentenceBreakIteratorUtf16<'l, 's>
pub fn segment_utf16<'l, 's>( &'l self, input: &'s [u16], ) -> SentenceBreakIteratorUtf16<'l, 's>
Creates a sentence break iterator for a UTF-16 string.
There are always breakpoints at 0 and the string length, or only at 0 for the empty string.