Expand description
Tokenizer states.
This is public for use by the tokenizer tests. Other library users should not have to care about this.
Tokenizer states.
This is public for use by the tokenizer tests. Other library users should not have to care about this.