package core:odin/tokenizer
⌘K
Ctrl+K
or
/
Index
Types (7)
Constants (0)
This section is empty.
Variables (2)
Procedures (26)
- advance_rune
- default_error_handler
- digit_val
- error
- init
- is_assignment_operator
- is_digit
- is_keyword
- is_letter
- is_literal
- is_newline
- is_operator
- peek_byte
- pos_compare
- scan
- scan_comment
- scan_escape
- scan_file_tag
- scan_identifier
- scan_number
- scan_raw_string
- scan_rune
- scan_string
- skip_whitespace
- to_string
- token_to_string
Procedure Groups (0)
This section is empty.
Types
Error_Handler ¶
Related Procedures With Parameters
Pos ¶
Pos :: struct { file: string, offset: int, // starting at 0 line: int, // starting at 1 column: int, }
Related Procedures With Parameters
Token ¶
Token :: struct { kind: Token_Kind, text: string, pos: Pos, }
Related Procedures With Parameters
Related Procedures With Returns
Token_Kind ¶
Token_Kind :: enum u32 { Invalid, EOF, Comment, File_Tag, B_Literal_Begin, Ident, // main Integer, // 12345 Float, // 123.45 Imag, // 123.45i Rune, // 'a' String, // "abc" B_Literal_End, B_Operator_Begin, Eq, // = Not, // ! Hash, // # At, // @ Dollar, // $ Pointer, // ^ Question, // ? Add, Sub, // - Mul, // * Quo, // / Mod, // % Mod_Mod, // %% And, // & Or, // | Xor, // ~ And_Not, // &~ Shl, // << Shr, // >> Cmp_And, // && Cmp_Or, // || B_Assign_Op_Begin, Add_Eq, Sub_Eq, // -= Mul_Eq, // *= Quo_Eq, // /= Mod_Eq, // %= Mod_Mod_Eq, // %%= And_Eq, // &= Or_Eq, // |= Xor_Eq, // ~= And_Not_Eq, // &~= Shl_Eq, // <<= Shr_Eq, // >>= Cmp_And_Eq, // &&= Cmp_Or_Eq, // ||= B_Assign_Op_End, Increment, Decrement, // -- Arrow_Right, // -> Undef, // --- B_Comparison_Begin, Cmp_Eq, // == Not_Eq, // != Lt, // < Gt, // > Lt_Eq, // <= Gt_Eq, // >= B_Comparison_End, Open_Paren, // ( Close_Paren, // ) Open_Bracket, // [ Close_Bracket, // ] Open_Brace, // { Close_Brace, // } Colon, // : Semicolon, // ; Period, // . Comma, // , Ellipsis, // .. Range_Half, // ..< Range_Full, // ..= B_Operator_End, B_Keyword_Begin, Import, // import Foreign, // foreign Package, // package Typeid, // typeid When, // when Where, // where If, // if Else, // else For, // for Switch, // switch In, // in Not_In, // not_in Do, // do Case, // case Break, // break Continue, // continue Fallthrough, // fallthrough Defer, // defer Return, // return Proc, // proc Struct, // struct Union, // union Enum, // enum Bit_Set, // bit_set Bit_Field, // bit_field Map, // map Dynamic, // dynamic Auto_Cast, // auto_cast Cast, // cast Transmute, // transmute Distinct, // distinct Using, // using Context, // context Or_Else, // or_else Or_Return, // or_return Or_Break, // or_break Or_Continue, // or_continue Asm, // asm Inline, // inline No_Inline, // no_inline Matrix, // matrix B_Keyword_End, COUNT, B_Custom_Keyword_Begin = 120, }
Related Procedures With Parameters
Related Procedures With Returns
Tokenizer ¶
Tokenizer :: struct { // Immutable data path: string, src: string, err: Error_Handler, flags: Flags, // Tokenizing state ch: rune, offset: int, read_offset: int, line_offset: int, line_count: int, insert_semicolon: bool, // Mutable data error_count: int, }
Related Procedures With Parameters
Constants
This section is empty.
Variables
custom_keyword_tokens ¶
custom_keyword_tokens: []string
Procedures
advance_rune ¶
advance_rune :: proc(t: ^Tokenizer) {…}
init ¶
init :: proc(t: ^Tokenizer, src: string, path: string, err: Error_Handler = default_error_handler) {…}
is_assignment_operator ¶
is_assignment_operator :: proc(kind: Token_Kind) -> bool {…}
is_keyword ¶
is_keyword :: proc(kind: Token_Kind) -> bool {…}
is_literal ¶
is_literal :: proc(kind: Token_Kind) -> bool {…}
is_operator ¶
is_operator :: proc(kind: Token_Kind) -> bool {…}
scan_number ¶
scan_number :: proc(t: ^Tokenizer, seen_decimal_point: bool) -> (Token_Kind, string) {…}
skip_whitespace ¶
skip_whitespace :: proc(t: ^Tokenizer) {…}
to_string ¶
to_string :: proc(kind: Token_Kind) -> string {…}
Procedure Groups
This section is empty.
Source Files
Generation Information
Generated with odin version dev-2024-12 (vendor "odin") Windows_amd64 @ 2024-12-06 21:12:12.670898500 +0000 UTC