Module lexer
Source - Lexer 🔒
- Token
- consume_any 🔒
- consume_token 🔒
- Return the token at the start of
input. - is_blankspace 🔒
- Returns whether or not a char is a blankspace (Unicode Pattern_White_Space)
- is_comment_end 🔒
- Returns whether or not a char is a comment end
(Unicode Pattern_White_Space excluding U+0020, U+0009, U+200E and U+200F)
- is_word_part 🔒
- Returns whether or not a char is a word part (Unicode XID_Continue)
- is_word_start 🔒
- Returns whether or not a char is a word start (Unicode XID_Start + ‘_’)
- TokenSpan 🔒