1HCT::Lang::Token(3)   User Contributed Perl Documentation  HCT::Lang::Token(3)
2
3
4

NAME

6       HCT::Lang::Token - Definition of tokens.
7

DESCRIPTION

9       A token is a categorized block of text. The block of text corresponding
10       to the token is known as a lexeme.  A lexical analyzer divides a
11       sequence of characters into tokens (tokenization) and categorizes them
12       according to function, giving them meaning. A token can look like
13       anything; it just needs to be a useful part of the structured text.
14
15       The object of token created by class "HCT::Lang::Token" has such
16       features: name, type and list of patterns. The lexemes will be gotten
17       by lexer by using patterns.
18
19       The token name is an unique name that will be returned by lexer to
20       parser.
21
22       The patterns are a list of patterns, that will be used for definition
23       of the token name and token lexeme. All patterns described by regular
24       expressions.
25
26       The token type is a number, that means position/preority, where the
27       token have to be used. For instance, tokens can be divided on keywords,
28       operators, numbers and identifiers, and the order in which lexer should
29       use them will be: operators, numbers, keywords and identifiers.
30       According to this the order can be: operators - 3, numbers - 2,
31       keywrods - 1, identifiers - 0.
32
33   FUNCTIONS
34       The functions above will be exported by default.
35
36       EMPTY_TOKEN
37           When the lexer reach the end of input, it returns the '' empty
38           token with an "undef" value.
39
40   CONSTRUCTOR
41       new ()
42           Creates a new "HCT::Language::Token" object.
43
44   METHODS
45       get_patterns ()
46           Returns a list of token patterns.
47
48       get_name ()
49           Returns token name.
50
51       get_type ()
52           Returns token type.
53
54
55
56perl v5.34.0                      2021-07-22               HCT::Lang::Token(3)