1RE2C(1) RE2C(1)
2
6 re2c - compile regular expressions to code
7
9 re2c [OPTIONS] INPUT [-o OUTPUT]
10 re2go [OPTIONS] INPUT [-o OUTPUT]
12
14 re2c is a tool for generating fast lexical analyzers for C, C++ and Go.
15 Note: This manual includes examples for Go, but it refers to re2c
17 (rather than re2go) as the name of the program in general.
18
20 A re2c program consists of normal code intermixed with re2c blocks and
21 directives. Each re2c block may contain definitions, configurations
22 and rules. Definitions are of the form name = regexp; where name is
23 an identifier that consists of letters, digits and underscores, and
24 regexp is a regular expression. Regular expressions may contain other
25 definitions, but recursion is not allowed and each name should be
26 defined before used. Configurations are of the form re2c:config =
27 value; where config is the configuration descriptor and value can be a
28 number, a string or a special word. Rules consist of a regular expres‐
29 sion followed by a semantic action (a block of code enclosed in curly
30 braces { and }, or a raw one line of code preceded with := and ended
31 with a newline that is not followed by a whitespace). If the input
32 matches the regular expression, the associated semantic action is exe‐
33 cuted. If multiple rules match, the longest match takes precedence.
34 If multiple rules match the same string, the earlier rule takes prece‐
35 dence. There are two special rules: default rule * and EOF rule $.
36 Default rule should always be defined, it has the lowest priority
37 regardless of its place and matches any code unit (not necessarily a
38 valid character, see encoding support). EOF rule matches the end of
39 input, it should be defined if the corresponding EOF handling method is
40 used. If start conditions are used, rules have more complex syntax.
41 All rules of a single block are compiled into a deterministic
42 finite-state automaton (DFA) and encoded in the form of a program in
43 the target language. The generated code interfaces with the outer pro‐
44 gram by the means of a few user-defined primitives (see the program
45 interface section). Reusable blocks allow sharing rules, definitions
46 and configurations between different blocks.
47
49 Input file
50 //go:generate re2go $INPUT -o $OUTPUT -i
51 package main //
52 //
53 func lex(str string) int { // Go code
54 var cursor int //
55 /*!re2c // start of re2c block
57 re2c:define:YYCTYPE = byte; // configuration
58 re2c:define:YYPEEK = "str[cursor]"; // configuration
59 re2c:define:YYSKIP = "cursor += 1"; // configuration
60 re2c:yyfill:enable = 0; // configuration
61 re2c:flags:nested-ifs = 1; // configuration
62 //
63 number = [1-9][0-9]*; // named definition
64 //
65 number { return 0; } // normal rule
66 * { return 1; } // default rule
67 */
68 } //
69 //
70 func main() { //
71 if lex("1234\x00") != 0 { // Go code
72 panic("failed!") //
73 } //
74 } //
75 Output file
78 // Code generated by re2c, DO NOT EDIT.
79 //go:generate re2go $INPUT -o $OUTPUT -i
80 package main //
81 //
82 func lex(str string) int { // Go code
83 var cursor int //
84 {
87 var yych byte
88 yych = str[cursor]
89 if (yych <= '0') {
90 goto yy2
91 }
92 if (yych <= '9') {
93 goto yy4
94 }
95 yy2:
96 cursor += 1
97 { return 1; }
98 yy4:
99 cursor += 1
100 yych = str[cursor]
101 if (yych <= '/') {
102 goto yy6
103 }
104 if (yych <= '9') {
105 goto yy4
106 }
107 yy6:
108 { return 0; }
109 }
110 } //
112 //
113 func main() { //
114 if lex("1234\x00") != 0 { // Go code
115 panic("failed!") //
116 } //
117 } //
118
121 -? -h --help
122 Show help message.
123 -1 --single-pass
125 Deprecated. Does nothing (single pass is the default now).
126 -8 --utf-8
128 Generate a lexer that reads input in UTF-8 encoding. re2c
129 assumes that character range is 0 -- 0x10FFFF and character size
130 is 1 byte.
131 -b --bit-vectors
133 Optimize conditional jumps using bit masks. Implies -s.
134 -c --conditions --start-conditions
136 Enable support of Flex-like "conditions": multiple interrelated
137 lexers within one block. Option --start-conditions is a legacy
138 alias; use --conditions instead.
139 --case-insensitive
141 Treat single-quoted and double-quoted strings as case-insensi‐
142 tive.
143 --case-inverted
145 Invert the meaning of single-quoted and double-quoted strings:
146 treat single-quoted strings as case-sensitive and double-quoted
147 strings as case-insensitive.
148 --case-ranges
150 Collapse consecutive cases in a switch statements into a range
151 of the form case low ... high:. This syntax is an extension of
152 the C/C++ language, supported by compilers like GCC, Clang and
153 Tcc. The main advantage over using single cases is smaller gen‐
154 erated C code and faster generation time, although for some com‐
155 pilers like Tcc it also results in smaller binary size. This
156 option doesn't work for the Go backend.
157 -e --ecb
159 Generate a lexer that reads input in EBCDIC encoding. re2c
160 assumes that character range is 0 -- 0xFF an character size is 1
161 byte.
162 --empty-class <match-empty | match-none | error>
164 Define the way re2c treats empty character classes. With
165 match-empty (the default) empty class matches empty input (which
166 is illogical, but backwards-compatible). With``match-none``
167 empty class always fails to match. With error empty class
168 raises a compilation error.
169 --encoding-policy <fail | substitute | ignore>
171 Define the way re2c treats Unicode surrogates. With fail re2c
172 aborts with an error when a surrogate is encountered. With sub‐
173 stitute re2c silently replaces surrogates with the error code
174 point 0xFFFD. With ignore (the default) re2c treats surrogates
175 as normal code points. The Unicode standard says that standalone
176 surrogates are invalid, but real-world libraries and programs
177 behave in different ways.
178 -f --storable-state
180 Generate a lexer which can store its inner state. This is use‐
181 ful in push-model lexers which are stopped by an outer program
182 when there is not enough input, and then resumed when more input
183 becomes available. In this mode users should additionally define
184 YYGETSTATE() and YYSETSTATE(state) macros and variables yych,
185 yyaccept and state as part of the lexer state.
186 -F --flex-syntax
188 Partial support for Flex syntax: in this mode named definitions
189 don't need the equal sign and the terminating semicolon, and
190 when used they must be surrounded by curly braces. Names without
191 curly braces are treated as double-quoted strings.
192 -g --computed-gotos
194 Optimize conditional jumps using non-standard "computed goto"
195 extension (which must be supported by the compiler). re2c gener‐
196 ates jump tables only in complex cases with a lot of conditional
197 branches. Complexity threshold can be configured with
198 cgoto:threshold configuration. This option implies -b. This
199 option doesn't work for the Go backend.
200 -I PATH
202 Add PATH to the list of locations which are used when searching
203 for include files. This option is useful in combination with
204 /*!include:re2c ... */ directive. Re2c looks for FILE in the
205 directory of including file and in the list of include paths
206 specified by -I option.
207 -i --no-debug-info
209 Do not output #line information. This is useful when the gener‐
210 ated code is tracked by some version control system or IDE.
211 --input <default | custom>
213 Specify the API used by the generated code to interface with
214 used-defined code. Option default is the C API based on pointer
215 arithmetic (it is the default for the C backend). Option custom
216 is the generic API (it is the default for the Go backend).
217 --input-encoding <ascii | utf8>
219 Specify the way re2c parses regular expressions. With ascii
220 (the default) re2c handles input as ASCII-encoded: any sequence
221 of code units is a sequence of standalone 1-byte characters.
222 With utf8 re2c handles input as UTF8-encoded and recognizes
223 multibyte characters.
224 --lang <c | go>
226 Specify the output language. Supported languages are C and Go
227 (the default is C).
228 --location-format <gnu | msvc>
230 Specify location format in messages. With gnu locations are
231 printed as 'filename:line:column: ...'. With msvc locations are
232 printed as 'filename(line,column) ...'. Default is gnu.
233 --no-generation-date
235 Suppress date output in the generated file.
236 --no-version
238 Suppress version output in the generated file.
239 -o OUTPUT --output=OUTPUT
241 Specify the OUTPUT file.
242 -P --posix-captures
244 Enable submatch extraction with POSIX-style capturing groups.
245 -r --reusable
247 Allows reuse of re2c rules with /*!rules:re2c */ and /*!use:re2c
248 */ blocks. Exactly one rules-block must be present. The rules
249 are saved and used by every use-block that follows, which may
250 add its own rules and configurations.
251 -S --skeleton
253 Ignore user-defined interface code and generate a self-contained
254 "skeleton" program. Additionally, generate input files with
255 strings derived from the regular grammar and compressed match
256 results that are used to verify "skeleton" behavior on all
257 inputs. This option is useful for finding bugs in optimizations
258 and code generation. This option doesn't work for the Go back‐
259 end.
260 -s --nested-ifs
262 Use nested if statements instead of switch statements in condi‐
263 tional jumps. This usually results in more efficient code with
264 non-optimizing compilers.
265 -T --tags
267 Enable submatch extraction with tags.
268 -t HEADER --type-header=HEADER
270 Generate a HEADER file that contains enum with condition names.
271 Requires -c option.
272 -u --unicode
274 Generate a lexer that reads UTF32-encoded input. Re2c assumes
275 that character range is 0 -- 0x10FFFF and character size is 4
276 bytes. This option implies -s.
277 -V --vernum
279 Show version information in MMmmpp format (major, minor, patch).
280 --verbose
282 Output a short message in case of success.
283 -v --version
285 Show version information.
286 -w --wide-chars
288 Generate a lexer that reads UCS2-encoded input. Re2c assumes
289 that character range is 0 -- 0xFFFF and character size is 2
290 bytes. This option implies -s.
291 -x --utf-16
293 Generate a lexer that reads UTF16-encoded input. Re2c assumes
294 that character range is 0 -- 0x10FFFF and character size is 2
295 bytes. This option implies -s.
296 Debug options
298 -D --emit-dot
299 Instead of normal output generate lexer graph in .dot format.
300 The output can be converted to an image with the help of
301 Graphviz (e.g. something like dot -Tpng -odfa.png dfa.dot).
302 -d --debug-output
304 Emit YYDEBUG in the generated code. YYDEBUG should be defined
305 by the user in the form of a void function with two parameters:
306 state (lexer state or -1) and symbol (current input symbol of
307 type YYCTYPE).
308 --dump-adfa
310 Debug option: output DFA after tunneling (in .dot format).
311 --dump-cfg
313 Debug option: output control flow graph of tag variables (in
314 .dot format).
315 --dump-closure-stats
317 Debug option: output statistics on the number of states in clo‐
318 sure.
319 --dump-dfa-det
321 Debug option: output DFA immediately after determinization (in
322 .dot format).
323 --dump-dfa-min
325 Debug option: output DFA after minimization (in .dot format).
326 --dump-dfa-tagopt
328 Debug option: output DFA after tag optimizations (in .dot for‐
329 mat).
330 --dump-dfa-tree
332 Debug option: output DFA under construction with states repre‐
333 sented as tag history trees (in .dot format).
334 --dump-dfa-raw
336 Debug option: output DFA under construction with expanded
337 state-sets (in .dot format).
338 --dump-interf
340 Debug option: output interference table produced by liveness
341 analysis of tag variables.
342 --dump-nfa
344 Debug option: output NFA (in .dot format).
345 Internal options
347 --dfa-minimization <moore | table>
348 Internal option: DFA minimization algorithm used by re2c. The
349 moore option is the Moore algorithm (it is the default). The ta‐
350 ble option is the "table filling" algorithm. Both algorithms
351 should produce the same DFA up to states relabeling; table fill‐
352 ing is simpler and much slower and serves as a reference imple‐
353 mentation.
354 --eager-skip
356 Internal option: make the generated lexer advance the input
357 position eagerly -- immediately after reading the input symbol.
358 This changes the default behavior when the input position is
359 advanced lazily -- after transition to the next state. This
360 option is implied by --no-lookahead.
361 --no-lookahead
363 Internal option: use TDFA(0) instead of TDFA(1). This option
364 has effect only with --tags or --posix-captures options.
365 --no-optimize-tags
367 Internal optionL: suppress optimization of tag variables (useful
368 for debugging).
369 --posix-closure <gor1 | gtop>
371 Internal option: specify shortest-path algorithm used for the
372 construction of epsilon-closure with POSIX disambiguation seman‐
373 tics: gor1 (the default) stands for Goldberg-Radzik algorithm,
374 and gtop stands for "global topological order" algorithm.
375 --posix-prectable <complex | naive>
377 Internal option: specify the algorithm used to compute POSIX
378 precedence table. The complex algorithm computes precedence ta‐
379 ble in one traversal of tag history tree and has quadratic com‐
380 plexity in the number of TNFA states; it is the default. The
381 naive algorithm has worst-case cubic complexity in the number of
382 TNFA states, but it is much simpler than complex and may be
383 slightly faster in non-pathological cases.
384 --stadfa
386 Internal option: use staDFA algorithm for submatch extraction.
387 The main difference with TDFA is that tag operations in staDFA
388 are placed in states, not on transitions.
389 Warnings
391 -W Turn on all warnings.
392 -Werror
394 Turn warnings into errors. Note that this option alone doesn't
395 turn on any warnings; it only affects those warnings that have
396 been turned on so far or will be turned on later.
397 -W<warning>
399 Turn on warning.
400 -Wno-<warning>
402 Turn off warning.
403 -Werror-<warning>
405 Turn on warning and treat it as an error (this implies -W<warn‐
406 ing>).
407 -Wno-error-<warning>
409 Don't treat this particular warning as an error. This doesn't
410 turn off the warning itself.
411 -Wcondition-order
413 Warn if the generated program makes implicit assumptions about
414 condition numbering. One should use either the -t, --type-header
415 option or the /*!types:re2c*/ directive to generate a mapping of
416 condition names to numbers and then use the autogenerated condi‐
417 tion names.
418 -Wempty-character-class
420 Warn if a regular expression contains an empty character class.
421 Trying to match an empty character class makes no sense: it
422 should always fail. However, for backwards compatibility rea‐
423 sons re2c allows empty character classes and treats them as
424 empty strings. Use the --empty-class option to change the
425 default behavior.
426 -Wmatch-empty-string
428 Warn if a rule is nullable (matches an empty string). If the
429 lexer runs in a loop and the empty match is unintentional, the
430 lexer may unexpectedly hang in an infinite loop.
431 -Wswapped-range
433 Warn if the lower bound of a range is greater than its upper
434 bound. The default behavior is to silently swap the range
435 bounds.
436 -Wundefined-control-flow
438 Warn if some input strings cause undefined control flow in the
439 lexer (the faulty patterns are reported). This is the most dan‐
440 gerous and most common mistake. It can be easily fixed by adding
441 the default rule * which has the lowest priority, matches any
442 code unit, and consumes exactly one code unit.
443 -Wunreachable-rules
445 Warn about rules that are shadowed by other rules and will never
446 match.
447 -Wuseless-escape
449 Warn if a symbol is escaped when it shouldn't be. By default,
450 re2c silently ignores such escapes, but this may as well indi‐
451 cate a typo or an error in the escape sequence.
452 -Wnondeterministic-tags
454 Warn if a tag has n-th degree of nondeterminism, where n is
455 greater than 1.
456 -Wsentinel-in-midrule
458 Warn if the sentinel symbol occurs in the middle of a rule ---
459 this may cause reads past the end of buffer, crashes or memory
460 corruption in the generated lexer. This warning is only applica‐
461 ble if the sentinel method of checking for the end of input is
462 used. It is set to an error if re2c:sentinel configuration is
463 used.
464
466 Re2c has a flexible interface that gives the user both the freedom and
467 the responsibility to define how the generated code interacts with the
468 outer program. There are two major options:
469 · Pointer API. It is also called "default API", since it was histori‐
471 cally the first, and for a long time the only one. This is a more
472 restricted API based on C pointer arithmetics. It consists of
473 pointer-like primitives YYCURSOR, YYMARKER, YYCTXMARKER and YYLIMIT,
474 which are normally defined as pointers of type YYCTYPE*. Pointer API
475 is enabled by default for the C backend, and it cannot be used with
476 other backends that do not have pointer arithmetics.
477 · Generic API. This is a less restricted API that does not assume
481 pointer semantics. It consists of primitives YYPEEK, YYSKIP,
482 YYBACKUP, YYBACKUPCTX, YYSTAGP, YYSTAGN, YYMTAGP, YYMTAGN, YYRESTORE,
483 YYRESTORECTX, YYRESTORETAG, YYSHIFT, YYSHIFTSTAG, YYSHIFTMTAG and
484 YYLESSTHAN. For the C backend generic API is enabled with --input
485 custom option or re2c:flags:input = custom; configuration; for the Go
486 backend it is enabled by default. Generic API was added in version
487 0.14. It is intentionally designed to give the user as much freedom
488 as possible in redefining the input model and the semantics of dif‐
489 ferent actions performed by the generated code. As an example, one
490 can override YYPEEK to check for the end of input before reading the
491 input character, or do some logging, etc.
492 Generic API has two styles:
494 · Function-like. This style is enabled with re2c:api:style = func‐
496 tions; configuration, and it is the default for C backend. In this
497 style API primitives should be defined as functions or macros with
498 parentheses, accepting the necessary arguments. For example, in C the
499 default pointer API can be defined in function-like style generic API
500 as follows:
501 #define YYPEEK() *YYCURSOR
503 #define YYSKIP() ++YYCURSOR
504 #define YYBACKUP() YYMARKER = YYCURSOR
505 #define YYBACKUPCTX() YYCTXMARKER = YYCURSOR
506 #define YYRESTORE() YYCURSOR = YYMARKER
507 #define YYRESTORECTX() YYCURSOR = YYCTXMARKER
508 #define YYRESTORETAG(tag) YYCURSOR = tag
509 #define YYLESSTHAN(len) YYLIMIT - YYCURSOR < len
510 #define YYSTAGP(tag) tag = YYCURSOR
511 #define YYSTAGN(tag) tag = NULL
512 #define YYSHIFT(shift) YYCURSOR += shift
513 #define YYSHIFTSTAG(tag, shift) tag += shift
514 · Free-form. This style is enabled with re2c:api:style = free-form;
518 configuration, and it is the default for Go backend. In this style
519 API primitives can be defined as free-form pieces of code, and
520 instead of arguments they have interpolated variables of the form
521 @@{name}, or optionally just @@ if there is only one argument. The @@
522 text is called "sigil". It can be redefined to any other text with
523 re2c:api:sigil configuration. For example, the default pointer API
524 can be defined in free-form style generic API as follows:
525 re2c:define:YYPEEK = "*YYCURSOR";
527 re2c:define:YYSKIP = "++YYCURSOR";
528 re2c:define:YYBACKUP = "YYMARKER = YYCURSOR";
529 re2c:define:YYBACKUPCTX = "YYCTXMARKER = YYCURSOR";
530 re2c:define:YYRESTORE = "YYCURSOR = YYMARKER";
531 re2c:define:YYRESTORECTX = "YYCURSOR = YYCTXMARKER";
532 re2c:define:YYRESTORETAG = "YYCURSOR = ${tag}";
533 re2c:define:YYLESSTHAN = "YYLIMIT - YYCURSOR < @@{len}";
534 re2c:define:YYSTAGP = "@@{tag} = YYCURSOR";
535 re2c:define:YYSTAGN = "@@{tag} = NULL";
536 re2c:define:YYSHIFT = "YYCURSOR += @@{shift}";
537 re2c:define:YYSHIFTSTAG = "@@{tag} += @@{shift}";
538 API primitives
540 Here is a list of API primitives that may be used by the generated code
541 in order to interface with the outer program. Which primitives are
542 needed depends on multiple factors, including the complexity of regular
543 expressions, input representation, buffering, the use of various fea‐
544 tures and so on. All the necessary primitives should be defined by the
545 user in the form of macros, functions, variables, free-form pieces of
546 code or any other suitable form. Re2c does not (and cannot) check the
547 definitions, so if anything is missing or defined incorrectly the gen‐
548 erated code will not compile.
549 YYCTYPE
551 The type of the input characters (code units). For ASCII,
552 EBCDIC and UTF-8 encodings it should be 1-byte unsigned integer.
553 For UTF-16 or UCS-2 it should be 2-byte unsigned integer. For
554 UTF-32 it should be 4-byte unsigned integer.
555 YYCURSOR
557 A pointer-like l-value that stores the current input position
558 (usually a pointer of type YYCTYPE*). Initially YYCURSOR should
559 point to the first input character. It is advanced by the gener‐
560 ated code. When a rule matches, YYCURSOR points to the one
561 after the last matched character. It is used only in the default
562 C API.
563 YYLIMIT
565 A pointer-like r-value that stores the end of input position
566 (usually a pointer of type YYCTYPE*). Initially YYLIMIT should
567 point to the one after the last available input character. It is
568 not changed by the generated code. Lexer compares YYCURSOR to
569 YYLIMIT in order to determine if there is enough input charac‐
570 ters left. YYLIMIT is used only in the default C API.
571 YYMARKER
573 A pointer-like l-value (usually a pointer of type YYCTYPE*) that
574 stores the position of the latest matched rule. It is used to
575 restores YYCURSOR position if the longer match fails and lexer
576 needs to rollback. Initialization is not needed. YYMARKER is
577 used only in the default C API.
578 YYCTXMARKER
580 A pointer-like l-value that stores the position of the trailing
581 context (usually a pointer of type YYCTYPE*). No initialization
582 is needed. It is used only in the default C API, and only with
583 the lookahead operator /.
584 YYFILL API primitive with one argument len. The meaning of YYFILL is
586 to provide at least len more input characters or fail. If EOF
587 rule is used, YYFILL should always return to the calling func‐
588 tion; the return value should be zero on success and non-zero on
589 failure. If EOF rule is not used, YYFILL return value is ignored
590 and it should not return on failure. Maximal value of len is
591 YYMAXFILL, which can be generated with /*!max:re2c*/ directive.
592 The definition of YYFILL can be either function-like or
593 free-form depending on the API style (see re2c:api:style and
594 re2c:define:YYFILL:naked).
595 YYMAXFILL
597 An integral constant equal to the maximal value of YYFILL argu‐
598 ment. It can be generated with /*!max:re2c*/ directive.
599 YYLESSTHAN
601 A generic API primitive with one argument len. It should be
602 defined as an r-value of boolean type that equals true if and
603 only if there is less than len input characters left. The defi‐
604 nition can be either function-like or free-form depending on the
605 API style (see re2c:api:style).
606 YYPEEK A generic API primitive with no arguments. It should be defined
608 as an r-value of type YYCTYPE that is equal to the character at
609 the current input position. The definition can be either func‐
610 tion-like or free-form depending on the API style (see
611 re2c:api:style).
612 YYSKIP A generic API primitive with no arguments. The meaning of
614 YYSKIP is to advance the current input position by one charac‐
615 ter. The definition can be either function-like or free-form
616 depending on the API style (see re2c:api:style).
617 YYBACKUP
619 A generic API primitive with no arguments. The meaning of
620 YYBACKUP is to save the current input position, which is later
621 restored with YYRESTORE. The definition should be either func‐
622 tion-like or free-form depending on the API style (see
623 re2c:api:style).
624 YYRESTORE
626 A generic API primitive with no arguments. The meaning of YYRE‐
627 STORE is to restore the current input position to the value
628 saved by YYBACKUP. The definition should be either func‐
629 tion-like or free-form depending on the API style (see
630 re2c:api:style).
631 YYBACKUPCTX
633 A generic API primitive with zero arguments. The meaning of
634 YYBACKUPCTX is to save the current input position as the posi‐
635 tion of the trailing context, which is later restored by YYRE‐
636 STORECTX. The definition should be either function-like or
637 free-form depending on the API style (see re2c:api:style).
638 YYRESTORECTX
640 A generic API primitive with no arguments. The meaning of YYRE‐
641 STORECTX is to restore the trailing context position saved with
642 YYBACKUPCTX. The definition should be either function-like or
643 free-form depending on the API style (see re2c:api:style).
644 YYRESTORETAG
646 A generic API primitive with one argument tag. The meaning of
647 YYRESTORETAG is to restore the trailing context position to the
648 value of tag. The definition should be either function-like or
649 free-form depending on the API style (see re2c:api:style).
650 YYSTAGP
652 A generic API primitive with one argument tag. The meaning of
653 YYSTAGP is to set tag value to the current input position. The
654 definition should be either function-like or free-form depending
655 on the API style (see re2c:api:style).
656 YYSTAGN
658 A generic API primitive with one argument tag. The meaning of
659 YYSTAGP is to set tag value to null (or some default value). The
660 definition should be either function-like or free-form depending
661 on the API style (see re2c:api:style).
662 YYMTAGP
664 A generic API primitive with one argument tag. The meaning of
665 YYMTAGP is to append the current position to the history of tag.
666 The definition should be either function-like or free-form
667 depending on the API style (see re2c:api:style).
668 YYMTAGN
670 A generic API primitive with one argument tag. The meaning of
671 YYMTAGN is to append null (or some other default) value to the
672 history of tag. The definition can be either function-like or
673 free-form depending on the API style (see re2c:api:style).
674 YYSHIFT
676 A generic API primitive with one argument shift. The meaning of
677 YYSHIFT is to shift the current input position by shift charac‐
678 ters (the shift value may be negative). The definition can be
679 either function-like or free-form depending on the API style
680 (see re2c:api:style).
681 YYSHIFTSTAG
683 A generic API primitive with two arguments, tag and shift. The
684 meaning of YYSHIFTSTAG is to shift tag by shift characters (the
685 shift value may be negative). The definition can be either
686 function-like or free-form depending on the API style (see
687 re2c:api:style).
688 YYSHIFTMTAG
690 A generic API primitive with two arguments, tag and shift. The
691 meaning of YYSHIFTMTAG is to shift the latest value in the his‐
692 tory of tag by shift characters (the shift value may be nega‐
693 tive). The definition should be either function-like or
694 free-form depending on the API style (see re2c:api:style).
695 YYMAXNMATCH
697 An integral constant equal to the maximal number of POSIX cap‐
698 turing groups in a rule. It is generated with /*!maxn‐
699 match:re2c*/ directive.
700 YYCONDTYPE
702 The type of the condition enum. It should be generated either
703 with /*!types:re2c*/ directive or -t --type-header option.
704 YYGETCONDITION
706 An API primitive with zero arguments. It should be defined as
707 an r-value of type YYCONDTYPE that is equal to the current con‐
708 dition identifier. The definition can be either function-like or
709 free-form depending on the API style (see re2c:api:style and
710 re2c:define:YYGETCONDITION:naked).
711 YYSETCONDITION
713 An API primitive with one argument cond. The meaning of YYSET‐
714 CONDITION is to set the current condition identifier to cond.
715 The definition should be either function-like or free-form
716 depending on the API style (see re2c:api:style and
717 re2c:define:YYSETCONDITION@cond).
718 YYGETSTATE
720 An API primitive with zero arguments. It should be defined as
721 an r-value of integer type that is equal to the current lexer
722 state. Should be initialized to -1. The definition can be either
723 function-like or free-form depending on the API style (see
724 re2c:api:style and re2c:define:YYGETSTATE:naked).
725 YYSETSTATE
727 An API primitive with one argument state. The meaning of YYSET‐
728 STATE is to set the current lexer state to state. The defini‐
729 tion should be either function-like or free-form depending on
730 the API style (see re2c:api:style and re2c:define:YYSET‐
731 STATE@state).
732 YYDEBUG
734 A debug API primitive with two arguments. It can be used to
735 debug the generated code (with -d --debug-output option). YYDE‐
736 BUG should return no value and accept two arguments: state
737 (either a DFA state index or -1) and symbol (the current input
738 symbol).
739 yych An l-value of type YYCTYPE that stores the current input charac‐
741 ter. User definition is necessary only with -f --storable-state
742 option.
743 yyaccept
745 An l-value of unsigned integral type that stores the number of
746 the latest matched rule. User definition is necessary only with
747 -f --storable-state option.
748 yynmatch
750 An l-value of unsigned integral type that stores the number of
751 POSIX capturing groups in the matched rule. Used only with -P
752 --posix-captures option.
753 yypmatch
755 An array of l-values that are used to hold the tag values corre‐
756 sponding to the capturing parentheses in the matching rule.
757 Array length must be at least yynmatch * 2 (usually YYMAXNMATCH
758 * 2 is a good choice). Used only with -P --posix-captures
759 option.
760 Directives
762 Below is the list of all directives provided by re2c (in no particular
763 order). More information on each directive can be found in the related
764 sections.
765 /*!re2c ... */
767 A standard re2c block.
768 %{ ... %}
770 A standard re2c block in -F --flex-support mode.
771 /*!rules:re2c ... */
773 A reusable re2c block (requires -r --reuse option).
774 /*!use:re2c ... */
776 A block that reuses previous rules-block specified with
777 /*!rules:re2c ... */ (requires -r --reuse option).
778 /*!ignore:re2c ... */
780 A block which contents are ignored and cut off from the output
781 file.
782 /*!max:re2c*/
784 This directive is substituted with the macro-definition of
785 YYMAXFILL.
786 /*!maxnmatch:re2c*/
788 This directive is substituted with the macro-definition of
789 YYMAXNMATCH (requires -P --posix-captures option).
790 /*!getstate:re2c*/
792 This directive is substituted with conditional dispatch on lexer
793 state (requires -f --storable-state option).
794 /*!types:re2c ... */
796 This directive is substituted with the definition of condition
797 enum (requires -c --conditions option).
798 /*!stags:re2c ... */, /*!mtags:re2c ... */
800 These directives allow one to specify a template piece of code
801 that is expanded for each s-tag/m-tag variable generated by
802 re2c. This block has two optional configurations: format = "@@";
803 (specifies the template where @@ is substituted with the name of
804 each tag variable), and separator = ""; (specifies the piece of
805 code used to join the generated pieces for different tag vari‐
806 ables).
807 /*!include:re2c FILE */
809 This directive allows one to include FILE (in the same sense as
810 #include directive in C/C++).
811 /*!header:re2c:on*/
813 This directive marks the start of header file. Everything after
814 it and up to the following /*!header:re2c:off*/ directive is
815 processed by re2c and written to the header file specified with
816 -t --type-header option.
817 /*!header:re2c:off*/
819 This directive marks the end of header file started with
820 /*!header:re2c:on*/.
821 Configurations
823 re2c:flags:t, re2c:flags:type-header
824 Specify the name of the generated header file relative to the
825 directory of the output file. (Same as -t, --type-header com‐
826 mand-line option except that the filepath is relative.)
827 re2c:flags:input
829 Same as --input command-line option.
830 re2c:api:style
832 Allows one to specify the style of generic API. Possible values
833 are functions and free-form. With functions style (the default
834 for the C backend) API primitives behave like functions, and
835 re2c generates parentheses with an argument list after the name
836 of each primitive. With free-form style (the default for the Go
837 backend) re2c treats API definitions as interpolated strings and
838 substitutes argument placeholders with the actual argument val‐
839 ues. This option can be overridden by options for individual
840 API primitives, e.g. re2c:define:YYFILL:naked for YYFILL.
841 re2c:api:sigil
843 Allows one to specify the "sigil" symbol (or string) that is
844 used to recognize argument placeholders in the definitions of
845 generic API primitives. The default value is @@. Placeholders
846 start with sigil, followed by the argument name in curly braces.
847 For example, if sigil is set to $, then placeholders will have
848 the form ${name}. Single-argument APIs may use shorthand nota‐
849 tion without the name in braces. This option can be overridden
850 by options for individual API primitives, e.g.
851 re2c:define:YYFILL@len for YYFILL.
852 re2c:define:YYCTYPE
854 Defines YYCTYPE (see the user interface section).
855 re2c:define:YYCURSOR
857 Defines C API primitive YYCURSOR (see the user interface sec‐
858 tion).
859 re2c:define:YYLIMIT
861 Defines C API primitive YYLIMIT (see the user interface sec‐
862 tion).
863 re2c:define:YYMARKER
865 Defines C API primitive YYMARKER (see the user interface sec‐
866 tion).
867 re2c:define:YYCTXMARKER
869 Defines C API primitive YYCTXMARKER (see the user interface sec‐
870 tion).
871 re2c:define:YYFILL
873 Defines API primitive YYFILL (see the user interface section).
874 re2c:define:YYFILL@len
876 Specifies the sigil used for argument substitution in YYFILL
877 definition. Defaults to @@. Overrides the more generic
878 re2c:api:sigil configuration.
879 re2c:define:YYFILL:naked
881 Allows one to override re2c:api:style for YYFILL. Value 0 cor‐
882 responds to free-form API style.
883 re2c:yyfill:enable
885 Defaults to 1 (YYFILL is enabled). Set this to zero to suppress
886 the generation of YYFILL. Use warnings (-W option) and re2c:sen‐
887 tinel configuration to verify that the generated lexer cannot
888 read past the end of input, as this might introduce severe secu‐
889 rity issues to your programs.
890 re2c:yyfill:parameter
892 Controls the argument in the parentheses that follow YYFILL.
893 Defaults to 1, which means that the argument is generated. If
894 zero, the argument is omitted. Can be overridden with
895 re2c:define:YYFILL:naked or re2c:api:style.
896 re2c:eof
898 Specifies the sentinel symbol used with EOF rule $ to check for
899 the end of input in the generated lexer. The default value is -1
900 (EOF rule is not used). Other possible values include all valid
901 code units. Only decimal numbers are recognized.
902 re2c:sentinel
904 Specifies the sentinel symbol used with the sentinel method of
905 checking for the end of input in the generated lexer (the case
906 when bounds checking is disabled with re2c:yyfill:enable = 0;
907 and EOF rule $ is not used). This configuration does not affect
908 code generation. It is used by re2c to verify that the sentinel
909 symbol is not allowed in the middle of the rule, and prevent
910 possible reads past the end of buffer in the generated lexer.
911 The default value is -1 (re2c assumes that the sentinel symbol
912 is 0, which is the most common case). Other possible values
913 include all valid code units. Only decimal numbers are recog‐
914 nized.
915 re2c:define:YYLESSTHAN
917 Defines generic API primitive YYLESSTHAN (see the user interface
918 section).
919 re2c:yyfill:check
921 Setting this to zero allows to suppress the generation of YYFILL
922 check (YYLESSTHAN in generic API of YYLIMIT-based comparison in
923 default C API). This configuration is useful when the necessary
924 input is always available. it defaults to 1 (the check is gener‐
925 ated).
926 re2c:label:yyFillLabel
928 Allows one to change the prefix of YYFILL labels (used with EOF
929 rule or with storable states).
930 re2c:define:YYPEEK
932 Defines generic API primitive YYPEEK (see the user interface
933 section).
934 re2c:define:YYSKIP
936 Defines generic API primitive YYSKIP (see the user interface
937 section).
938 re2c:define:YYBACKUP
940 Defines generic API primitive YYBACKUP (see the user interface
941 section).
942 re2c:define:YYBACKUPCTX
944 Defines generic API primitive YYBACKUPCTX (see the user inter‐
945 face section).
946 re2c:define:YYRESTORE
948 Defines generic API primitive YYRESTORE (see the user interface
949 section).
950 re2c:define:YYRESTORECTX
952 Defines generic API primitive YYRESTORECTX (see the user inter‐
953 face section).
954 re2c:define:YYRESTORETAG
956 Defines generic API primitive YYRESTORETAG (see the user inter‐
957 face section).
958 re2c:define:YYSHIFT
960 Defines generic API primitive YYSHIFT (see the user interface
961 section).
962 re2c:define:YYSHIFTMTAG
964 Defines generic API primitive YYSHIFTMTAG (see the user inter‐
965 face section).
966 re2c:define:YYSHIFTSTAG
968 Defines generic API primitive YYSHIFTSTAG (see the user inter‐
969 face section).
970 re2c:define:YYSTAGN
972 Defines generic API primitive YYSTAGN (see the user interface
973 section).
974 re2c:define:YYSTAGP
976 Defines generic API primitive YYSTAGP (see the user interface
977 section).
978 re2c:define:YYMTAGN
980 Defines generic API primitive YYMTAGN (see the user interface
981 section).
982 re2c:define:YYMTAGP
984 Defines generic API primitive YYMTAGP (see the user interface
985 section).
986 re2c:flags:T, re2c:flags:tags
988 Same as -T --tags command-line option.
989 re2c:flags:P, re2c:flags:posix-captures
991 Same as -P --posix-captures command-line option.
992 re2c:tags:expression
994 Allows one to customize the way re2c addresses tag variables.
995 By default re2c generates expressions of the form yyt<N>. This
996 might be inconvenient, for example if tag variables are defined
997 as fields in a struct. Re2c recognizes placeholder of the form
998 @@{tag} or @@ and replaces it with the actual tag name. Sigil
999 @@ can be redefined with re2c:api:sigil configuration. For
1000 example, setting re2c:tags:expression = "p->@@"; results in
1001 expressions of the form p->yyt<N> in the generated code.
1002 re2c:tags:prefix
1004 Allows one to override the prefix of tag variables (defaults to
1005 yyt).
1006 re2c:flags:lookahead
1008 Same as inverted --no-lookahead command-line option.
1009 re2c:flags:optimize-tags
1011 Same as inverted --no-optimize-tags command-line option.
1012 re2c:define:YYCONDTYPE
1014 Defines YYCONDTYPE (see the user interface section).
1015 re2c:define:YYGETCONDITION
1017 Defines API primitive YYGETCONDITION (see the user interface
1018 section).
1019 re2c:define:YYGETCONDITION:naked
1021 Allows one to override re2c:api:style for YYGETCONDITION. Value
1022 0 corresponds to free-form API style.
1023 re2c:define:YYSETCONDITION
1025 Defines API primitive YYSETCONDITION (see the user interface
1026 section).
1027 re2c:define:YYSETCONDITION@cond
1029 Specifies the sigil used for argument substitution in YYSETCON‐
1030 DITION definition. The default value is @@. Overrides the more
1031 generic re2c:api:sigil configuration.
1032 re2c:define:YYSETCONDITION:naked
1034 Allows one to override re2c:api:style for YYSETCONDITION. Value
1035 0 corresponds to free-form API style.
1036 re2c:cond:goto
1038 Allows one to customize the goto statements used with the short‐
1039 cut :=> rules in conditions. The default value is goto @@;.
1040 Placeholders are substituted with condition name (see
1041 re2c:api;sigil and re2c:cond:goto@cond).
1042 re2c:cond:goto@cond
1044 Specifies the sigil used for argument substitution in
1045 re2c:cond:goto definition. The default value is @@. Overrides
1046 the more generic re2c:api:sigil configuration.
1047 re2c:cond:divider
1049 Defines the divider for condition blocks. The default value is
1050 /* *********************************** */. Placeholders are
1051 substituted with condition name (see re2c:api;sigil and
1052 re2c:cond:divider@cond).
1053 re2c:cond:divider@cond
1055 Specifies the sigil used for argument substitution in
1056 re2c:cond:divider definition. The default value is @@. Over‐
1057 rides the more generic re2c:api:sigil configuration.
1058 re2c:condprefix
1060 Specifies the prefix used for condition labels. The default
1061 value is yyc_.
1062 re2c:condenumprefix
1064 Specifies the prefix used for condition identifiers. The
1065 default value is yyc.
1066 re2c:define:YYGETSTATE
1068 Defines API primitive YYGETSTATE (see the user interface sec‐
1069 tion).
1070 re2c:define:YYGETSTATE:naked
1072 Allows one to override re2c:api:style for YYGETSTATE. Value 0
1073 corresponds to free-form API style.
1074 re2c:define:YYSETSTATE
1076 Defines API primitive YYSETSTATE (see the user interface sec‐
1077 tion).
1078 re2c:define:YYSETSTATE@state
1080 Specifies the sigil used for argument substitution in YYSETSTATE
1081 definition. The default value is @@. Overrides the more generic
1082 re2c:api:sigil configuration.
1083 re2c:define:YYSETSTATE:naked
1085 Allows one to override re2c:api:style for YYSETSTATE. Value 0
1086 corresponds to free-form API style.
1087 re2c:state:abort
1089 If set to a positive integer value, changes the form of the
1090 YYGETSTATE switch: instead of using default case to jump to the
1091 beginning of the lexer block, a -1 case is used, and the default
1092 case aborts the program.
1093 re2c:state:nextlabel
1095 With storable states, allows to control if the YYGETSTATE block
1096 is followed by a yyNext label (the default value is zero, which
1097 corresponds to no label). Instead of using yyNext it is possible
1098 to use re2c:startlabel to force the generation of a specific
1099 start label. Instead of using labels it is often more conve‐
1100 nient to generate YYGETSTATE code using /*!getstate:re2c*/.
1101 re2c:label:yyNext
1103 Allows one to change the name of the yyNext label.
1104 re2c:startlabel
1106 Controls the generation of start label for the next lexer block.
1107 The default value is zero, which means that the start label is
1108 generated only if it is used. An integer value greater than zero
1109 forces the generation of start label even if it is unused by the
1110 lexer. A string value also forces start label generation and
1111 sets the label name to the specified string. This configuration
1112 applies only to the current block (it is reset to default for
1113 the next block).
1114 re2c:flags:s, re2c:flags:nested-ifs
1116 Same as -s --nested-ifs command-line option.
1117 re2c:flags:b, re2c:flags:bit-vectors
1119 Same as -b --bit-vectors command-line option.
1120 re2c:variable:yybm
1122 Overrides the name of the yybm variable.
1123 re2c:yybm:hex
1125 Defaults to zero (a decimal bitmap table is generated). If set
1126 to nonzero, a hexadecimal table is generated.
1127 re2c:flags:g, re2c:flags:computed-gotos
1129 Same as -g --computed-gotos command-line option.
1130 re2c:cgoto:threshold
1132 With -g --computed-gotos option this value specifies the com‐
1133 plexity threshold that triggers the generation of jump tables
1134 instead of nested if statements and bitmaps. The default value
1135 is 9.
1136 re2c:flags:case-ranges
1138 Same as --case-ranges command-line option.
1139 re2c:flags:e, re2c:flags:ecb
1141 Same as -e --ecb command-line option.
1142 re2c:flags:8, re2c:flags:utf-8
1144 Same as -8 --utf-8 command-line option.
1145 re2c:flags:w, re2c:flags:wide-chars
1147 Same as -w --wide-chars command-line option.
1148 re2c:flags:x, re2c:flags:utf-16
1150 Same as -x --utf-16 command-line option.
1151 re2c:flags:u, re2c:flags:unicode
1153 Same as -u --unicode command-line option.
1154 re2c:flags:encoding-policy
1156 Same as --encoding-policy command-line option.
1157 re2c:flags:empty-class
1159 Same as --empty-class command-line option.
1160 re2c:flags:case-insensitive
1162 Same as --case-insensitive command-line option.
1163 re2c:flags:case-inverted
1165 Same as --case-inverted command-line option.
1166 re2c:flags:i, re2c:flags:no-debug-info
1168 Same as -i --no-debug-info command-line option.
1169 re2c:indent:string
1171 Specifies the string to use for indentation. The default value
1172 is "\t". Indent string should contain only whitespace charac‐
1173 ters. To disable indentation entirely, set this configuration
1174 to empty string "".
1175 re2c:indent:top
1177 Specifies the minimum amount of indentation to use. The default
1178 value is zero. The value should be a non-negative integer num‐
1179 ber.
1180 re2c:labelprefix
1182 Allows one to change the prefix of DFA state labels. The
1183 default value is yy.
1184 re2c:yych:emit
1186 Set this to zero to suppress the generation of yych definition.
1187 Defaults to 1 (the definition is generated).
1188 re2c:variable:yych
1190 Overrides the name of the yych variable.
1191 re2c:yych:conversion
1193 If set to nonzero, re2c automatically generates a cast to YYC‐
1194 TYPE every time yych is read. Defaults to zero (no cast).
1195 re2c:variable:yyaccept
1197 Overrides the name of the yyaccept variable.
1198 re2c:variable:yytarget
1200 Overrides the name of the yytarget variable.
1201 re2c:variable:yystable
1203 Deprecated.
1204 re2c:variable:yyctable
1206 When both -c --conditions and -g --computed-gotos are active,
1207 re2c will use this variable to generate a static jump table for
1208 YYGETCONDITION.
1209 re2c:define:YYDEBUG
1211 Defines YYDEBUG (see the user interface section).
1212 re2c:flags:d, re2c:flags:debug-output
1214 Same as -d --debug-output command-line option.
1215 re2c:flags:dfa-minimization
1217 Same as --dfa-minimization command-line option.
1218 re2c:flags:eager-skip
1220 Same as --eager-skip command-line option.
1221
1223 re2c uses the following syntax for regular expressions:
1224 · "foo" case-sensitive string literal
1226 · 'foo' case-insensitive string literal
1228 · [a-xyz], [^a-xyz] character class (possibly negated)
1230 · . any character except newline
1232 · R \ S difference of character classes R and S
1234 · R* zero or more occurrences of R
1236 · R+ one or more occurrences of R
1238 · R? optional R
1240 · R{n} repetition of R exactly n times
1242 · R{n,} repetition of R at least n times
1244 · R{n,m} repetition of R from n to m times
1246 · (R) just R; parentheses are used to override precedence or for
1248 POSIX-style submatch
1249 · R S concatenation: R followed by S
1251 · R | S alternative: R or S
1253 · R / S lookahead: R followed by S, but S is not consumed
1255 · name the regular expression defined as name (or literal string "name"
1257 in Flex compatibility mode)
1258 · {name} the regular expression defined as name in Flex compatibility
1260 mode
1261 · @stag an s-tag: saves the last input position at which @stag matches
1263 in a variable named stag
1264 · #mtag an m-tag: saves all input positions at which #mtag matches in a
1266 variable named mtag
1267 Character classes and string literals may contain the following escape
1269 sequences: \a, \b, \f, \n, \r, \t, \v, \\, octal escapes \ooo and hexa‐
1270 decimal escapes \xhh, \uhhhh and \Uhhhhhhhh.
1271
1273 Re2c provides a number of ways to handle end-of-input situation. Which
1274 way to use depends on the complexity of regular expressions, perfor‐
1275 mance considerations, the need for input buffering and various other
1276 factors. EOF handling is probably the most complex part of re2c user
1277 interface --- it definitely requires a bit of understanding of how the
1278 generated lexer works. But in return is allows the user to customize
1279 lexer for a particular environment and avoid the unnecessary overhead
1280 of generic methods when a simpler method is sufficient. Roughly speak‐
1281 ing, there are four main methods:
1282 · using sentinel symbol (simple and efficient, but limited)
1284 · bounds checking with padding (generic, but complex)
1286 · EOF rule: a combination of sentinel symbol and bounds checking
1288 (generic and simple, can be more or less efficient than bounds check‐
1289 ing with padding depending on the grammar)
1290 · using generic API (user-defined, so may be incorrect ;])
1292 Using sentinel symbol
1294 This is the simplest and the most efficient method. It is applicable in
1295 cases when the input is small enough to fit into a continuous memory
1296 buffer and there is a natural "sentinel" symbol --- a code unit that is
1297 not allowed by any of the regular expressions in grammar (except possi‐
1298 bly as a terminating character). Sentinel symbol never appears in
1299 well-formed input, therefore it can be appended at the end of input and
1300 used as a stop signal by the lexer. A good example of such input is a
1301 null-terminated C-string, provided that the grammar does not allow NULL
1302 in the middle of lexemes. Sentinel method is very efficient, because
1303 the lexer does not need to perform any additional checks for the end of
1304 input --- it comes naturally as a part of processing the next charac‐
1305 ter. It is very important that the sentinel symbol is not allowed in
1306 the middle of the rule --- otherwise on some inputs the lexer may read
1307 past the end of buffer and crash or cause memory corruption. Re2c veri‐
1308 fies this automatically. Use re2c:sentinel configuration to specify
1309 which sentinel symbol is used.
1310 Below is an example of using sentinel method. Configuration
1312 re2c:yyfill:enable = 0; suppresses generation of end-of-input checks
1313 and YYFILL calls.
1314 //go:generate re2go $INPUT -o $OUTPUT
1316 package main
1317 import "testing"
1319 // expect a null-terminated string
1321 func lex(str string) int {
1322 var cursor int
1323 count := 0
1324 loop:
1325 /*!re2c
1326 re2c:yyfill:enable = 0;
1327 re2c:define:YYCTYPE = byte;
1328 re2c:define:YYPEEK = "str[cursor]";
1329 re2c:define:YYSKIP = "cursor += 1";
1330 * { return -1 }
1332 [\x00] { return count }
1333 [a-z]+ { count += 1; goto loop }
1334 [ ]+ { goto loop }
1335 */
1336 }
1337 func TestLex(t *testing.T) {
1339 var tests = []struct {
1340 res int
1341 str string
1342 }{
1343 {0, "\000"},
1344 {3, "one two three\000"},
1345 {-1, "f0ur\000"},
1346 }
1347 for _, x := range tests {
1349 t.Run(x.str, func(t *testing.T) {
1350 res := lex(x.str)
1351 if res != x.res {
1352 t.Errorf("got %d, want %d", res, x.res)
1353 }
1354 })
1355 }
1356 }
1357 Bounds checking with padding
1360 Bounds checking is a generic method: it can be used with any input
1361 grammar. The basic idea is simple: we need to check for the end of
1362 input before reading the next input character. However, if implemented
1363 in a straightforward way, this would be quite inefficient: checking on
1364 each input character would cause a major slowdown. Re2c avoids slowdown
1365 by generating checks only in certain key states of the lexer, and let‐
1366 ting it run without checks in-between the key states. More precisely,
1367 re2c computes strongly connected components (SCCs) of the underlying
1368 DFA (which roughly correspond to loops), and generates only a few
1369 checks per each SCC (usually just one, but in general enough to make
1370 the SCC acyclic). The check is of the form (YYLIMIT - YYCURSOR) < n,
1371 where n is the maximal length of a simple path in the corresponding
1372 SCC. If this condiiton is true, the lexer calls YYFILL(n), which must
1373 either supply at least n input characters, or do not return. When the
1374 lexer continues after the check, it is certain that the next n charac‐
1375 ters can be read safely without checks.
1376 This approach reduces the number of checks significantly (and makes the
1378 lexer much faster as a result), but it has a downside. Since the lexer
1379 checks for multiple characters at once, it may end up in a situation
1380 when there are a few remaining input characters (less than n) corre‐
1381 sponding to a short path in the SCC, but the lexer cannot proceed
1382 because of the check, and YYFILL cannot supply more character because
1383 it is the end of input. To solve this problem, re2c requires that addi‐
1384 tional padding consisting of fake characters is appended at the end of
1385 input. The length of padding should be YYMAXFILL, which equals to the
1386 maximum n parameter to YYFILL and must be generated by re2c using
1387 /*!max:re2c*/ directive. The fake characters should not form a valid
1388 lexeme suffix, otherwise the lexer may be fooled into matching a fake
1389 lexeme. Usually it's a good idea to use NULL characters for padding.
1390 Below is an example of using bounds checking with padding. Note that
1392 the grammar rule for single-quoted strings allows arbitrary symbols in
1393 the middle of lexeme, so there is no natural sentinel in the grammar.
1394 Strings like "aha\0ha" are perfectly valid, but ill-formed strings like
1395 "aha\0 are also possible and shouldn’t crash the lexer. In this example
1396 we do not use buffer refilling, therefore YYFILL definition simply
1397 returns an error. Note that YYFILL will only be called after the lexer
1398 reaches padding, because only then will the check condition be satis‐
1399 fied.
1400 //go:generate re2go $INPUT -o $OUTPUT
1402 package main
1403 import (
1405 "strings"
1406 "testing"
1407 )
1408 /*!max:re2c*/
1410 // Expects YYMAXFILL-padded string.
1412 func lex(str string) int {
1413 var cursor int
1414 limit := len(str)
1415 count := 0
1416 loop:
1417 /*!re2c
1418 re2c:define:YYCTYPE = byte;
1419 re2c:define:YYPEEK = "str[cursor]";
1420 re2c:define:YYSKIP = "cursor += 1";
1421 re2c:define:YYLESSTHAN = "limit - cursor < @@{len}";
1422 re2c:define:YYFILL = "return -1";
1423 * { return -1 }
1425 [\x00] { return count }
1426 ['] ([^'\\] | [\\][^])* ['] { count += 1; goto loop }
1427 [ ]+ { goto loop }
1428 */
1429 }
1430 // Pad string with YYMAXFILL zeroes at the end.
1432 func pad(str string) string {
1433 return str + strings.Repeat("\000", YYMAXFILL)
1434 }
1435 func TestLex(t *testing.T) {
1437 var tests = []struct {
1438 res int
1439 str string
1440 }{
1441 {0, ""},
1442 {3, "'qu\000tes' 'are' 'fine: \\'' "},
1443 {-1, "'unterminated\\'"},
1444 }
1445 for _, x := range tests {
1447 t.Run(x.str, func(t *testing.T) {
1448 res := lex(pad(x.str))
1449 if res != x.res {
1450 t.Errorf("got %d, want %d", res, x.res)
1451 }
1452 })
1453 }
1454 }
1455 EOF rule
1458 EOF rule $ was introduced in version 1.2. It is a hybrid approach that
1459 tries to take the best of both worlds: simplicity and efficiency of the
1460 sentinel method combined with the generality of bounds-checking method.
1461 The idea is to appoint an arbitrary symbol to be the sentinel, and only
1462 perform further bounds checking if the sentinel symbol matches (more
1463 precisely, if the symbol class that contains it matches). The check is
1464 of the form YYLIMIT <= YYCURSOR. If this condition is not satisfied,
1465 then the sentinel is just an ordinary input character and the lexer
1466 continues. Otherwise this is a real sentinel, and the lexer calls
1467 YYFILL(). If YYFILL returns zero, the lexer assumes that it has more
1468 input and tries to re-match. Otherwise YYFILL returns non-zero and the
1469 lexer knows that it has reached the end of input. At this point there
1470 are three possibilities. First, it might have already matched a shorter
1471 lexeme --- in this case it just rolls back to the last accepting state.
1472 Second, it might have consumed some characters, but failed to match ---
1473 in this case it falls back to default rule *. Finally, it might be in
1474 the initial state --- in this (and only this!) case it matches EOF rule
1475 $.
1476 Below is an example of using EOF rule. Configuration re2c:yyfill:enable
1478 = 0; suppresses generation of YYFILL calls (but not the bounds checks).
1479 //go:generate re2go $INPUT -o $OUTPUT
1481 package main
1482 import "testing"
1484 // Expects a null-terminated string.
1486 func lex(str string) int {
1487 var cursor, marker int
1488 limit := len(str) - 1 // limit points at the terminating null
1489 count := 0
1490 loop:
1491 /*!re2c
1492 re2c:yyfill:enable = 0;
1493 re2c:eof = 0;
1494 re2c:define:YYCTYPE = byte;
1495 re2c:define:YYPEEK = "str[cursor]";
1496 re2c:define:YYSKIP = "cursor += 1";
1497 re2c:define:YYBACKUP = "marker = cursor";
1498 re2c:define:YYRESTORE = "cursor = marker";
1499 re2c:define:YYLESSTHAN = "limit <= cursor";
1500 * { return -1 }
1502 $ { return count }
1503 ['] ([^'\\] | [\\][^])* ['] { count += 1; goto loop }
1504 [ ]+ { goto loop }
1505 */
1506 }
1507 func TestLex(t *testing.T) {
1509 var tests = []struct {
1510 res int
1511 str string
1512 }{
1513 {0, "\000"},
1514 {3, "'qu\000tes' 'are' 'fine: \\'' \000"},
1515 {-1, "'unterminated\\'\000"},
1516 }
1517 for _, x := range tests {
1519 t.Run(x.str, func(t *testing.T) {
1520 res := lex(x.str)
1521 if res != x.res {
1522 t.Errorf("got %d, want %d", res, x.res)
1523 }
1524 })
1525 }
1526 }
1527 Using generic API
1530 Generic API can be used with any of the above methods. It also allows
1531 one to use a user-defined method by placing EOF checks in one of the
1532 basic primitives. Usually this is either YYSKIP (the check is per‐
1533 formed when advancing to the next input character), or YYPEEK (the
1534 check is performed when reading the next input character). The result‐
1535 ing methods are inefficient, as they check on each input character.
1536 However, they can be useful in cases when the input cannot be buffered
1537 or padded and does not contain a sentinel character at the end. One
1538 should be cautious when using such ad-hoc methods, as it is easy to
1539 overlook some corner cases and come up with a method that only par‐
1540 tially works. Also it should be noted that not everything can be
1541 expressed via generic API: for example, it is impossible to reimplement
1542 the way EOF rule works (in particular, it is impossible to re-match the
1543 character after successful YYFILL).
1544 Below is an example of using YYSKIP to perform bounds checking without
1546 padding. YYFILL generation is suppressed using re2c:yyfill:enable = 0;
1547 configuration. Note that if the grammar was more complex, this method
1548 might not work in case when two rules overlap and EOF check fails after
1549 a shorter lexeme has already been matched (as it happens in our exam‐
1550 ple, there are no overlapping rules).
1551 //go:generate re2go $INPUT -o $OUTPUT
1553 package main
1554 import "testing"
1556 // Returns "fake" terminating null if cursor has reached limit.
1558 func peek(str string, cursor int, limit int) byte {
1559 if cursor >= limit {
1560 return 0 // fake null
1561 } else {
1562 return str[cursor]
1563 }
1564 }
1565 // Expects a string without terminating null.
1567 func lex(str string) int {
1568 var cursor, marker int
1569 limit := len(str)
1570 count := 0
1571 loop:
1572 /*!re2c
1573 re2c:yyfill:enable = 0;
1574 re2c:eof = 0;
1575 re2c:define:YYCTYPE = byte;
1576 re2c:define:YYLESSTHAN = "cursor >= limit";
1577 re2c:define:YYPEEK = "peek(str, cursor, limit)";
1578 re2c:define:YYSKIP = "cursor += 1";
1579 re2c:define:YYBACKUP = "marker = cursor";
1580 re2c:define:YYRESTORE = "cursor = marker";
1581 * { return -1 }
1583 $ { return count }
1584 ['] ([^'\\] | [\\][^])* ['] { count += 1; goto loop }
1585 [ ]+ { goto loop }
1586 */
1587 }
1588 func TestLex(t *testing.T) {
1590 var tests = []struct {
1591 res int
1592 str string
1593 }{
1594 {0, ""},
1595 {3, "'qu\000tes' 'are' 'fine: \\'' "},
1596 {-1, "'unterminated\\'"},
1597 }
1598 for _, x := range tests {
1600 t.Run(x.str, func(t *testing.T) {
1601 res := lex(x.str)
1602 if res != x.res {
1603 t.Errorf("got %d, want %d", res, x.res)
1604 }
1605 })
1606 }
1607 }
1608
1611 The need for buffering arises when the input cannot be mapped in memory
1612 all at once: either it is too large, or it comes in a streaming fashion
1613 (like reading from a socket). The usual technique in such cases is to
1614 allocate a fixed-sized memory buffer and process input in chunks that
1615 fit into the buffer. When the current chunk is processed, it is moved
1616 out and new data is moved in. In practice it is somewhat more complex,
1617 because lexer state consists not of a single input position, but a set
1618 of interrelated posiitons:
1619 · cursor: the next input character to be read (YYCURSOR in default API
1621 or YYSKIP/YYPEEK in generic API)
1622 · limit: the position after the last available input character (YYLIMIT
1624 in default API, implicitly handled by YYLESSTHAN in generic API)
1625 · marker: the position of the most recent match, if any (YYMARKER in
1627 default API or YYBACKUP/YYRESTORE in generic API)
1628 · token: the start of the current lexeme (implicit in re2c API, as it
1630 is not needed for the normal lexer operation and can be defined and
1631 updated by the user)
1632 · context marker: the position of the trailing context (YYCTXMARKER in
1634 default API or YYBACKUPCTX/YYRESTORECTX in generic API)
1635 · tag variables: submatch positions (defined with /*!stags:re2c*/ and
1637 /*!mtags:re2c*/ directives and YYSTAGP/YYSTAGN/YYMTAGP/YYMTAGN in
1638 generic API)
1639 Not all these are used in every case, but if used, they must be updated
1641 by YYFILL. All active positions are contained in the segment between
1642 token and cursor, therefore everything between buffer start and token
1643 can be discarded, the segment from token and up to limit should be
1644 moved to the beginning of buffer, and the free space at the end of buf‐
1645 fer should be filled with new data. In order to avoid frequent YYFILL
1646 calls it is best to fill in as many input characters as possible (even
1647 though fewer characters might suffice to resume the lexer). The details
1648 of YYFILL implementation are slightly different depending on which EOF
1649 handling method is used: the case of EOF rule is somewhat simpler than
1650 the case of bounds-checking with padding. Also note that if -f
1651 --storable-state option is used, YYFILL has slightly different seman‐
1652 tics (desrbed in the section about storable state).
1653 YYFILL with EOF rule
1655 If EOF rule is used, YYFILL is a function-like primitive that accepts
1656 no arguments and returns a value which is checked against zero. YYFILL
1657 invocation is triggered by condition YYLIMIT <= YYCURSOR in default API
1658 and YYLESSTHAN() in generic API. A non-zero return value means that
1659 YYFILL has failed. A successful YYFILL call must supply at least one
1660 character and adjust input positions accordingly. Limit must always be
1661 set to one after the last input position in buffer, and the character
1662 at the limit position must be the sentinel symbol specified by re2c:eof
1663 configuration. The pictures below show the relative locations of input
1664 positions in buffer before and after YYFILL call (sentinel symbol is
1665 marked with #, and the second picture shows the case when there is not
1666 enough input to fill the whole buffer).
1667 <-- shift -->
1669 >-A------------B---------C-------------D#-----------E->
1670 buffer token marker limit,
1671 cursor
1672 >-A------------B---------C-------------D------------E#->
1673 buffer, marker cursor limit
1674 token
1675 <-- shift -->
1677 >-A------------B---------C-------------D#--E (EOF)
1678 buffer token marker limit,
1679 cursor
1680 >-A------------B---------C-------------D---E#........
1681 buffer, marker cursor limit
1682 token
1683 Here is an example of a program that reads input file input.txt in
1685 chunks of 4096 bytes and uses EOF rule.
1686 //go:generate re2go $INPUT -o $OUTPUT
1688 package main
1689 import (
1691 "os"
1692 "testing"
1693 )
1694 // Intentionally small to trigger buffer refill.
1696 const SIZE int = 16
1697 type Input struct {
1699 file *os.File
1700 data []byte
1701 cursor int
1702 marker int
1703 token int
1704 limit int
1705 eof bool
1706 }
1707 func fill(in *Input) int {
1709 // If nothing can be read, fail.
1710 if in.eof {
1711 return 1
1712 }
1713 // Check if at least some space can be freed.
1715 if in.token == 0 {
1716 // In real life can reallocate a larger buffer.
1717 panic("fill error: lexeme too long")
1718 }
1719 // Discard everything up to the start of the current lexeme,
1721 // shift buffer contents and adjust offsets.
1722 copy(in.data[0:], in.data[in.token:in.limit])
1723 in.cursor -= in.token
1724 in.marker -= in.token
1725 in.limit -= in.token
1726 in.token = 0
1727 // Read new data (as much as possible to fill the buffer).
1729 n, _ := in.file.Read(in.data[in.limit:SIZE])
1730 in.limit += n
1731 in.data[in.limit] = 0
1732 // If read less than expected, this is the end of input.
1734 in.eof = in.limit < SIZE
1735 // If nothing has been read, fail.
1737 if n == 0 {
1738 return 1
1739 }
1740 return 0
1742 }
1743 func lex(in *Input) int {
1745 count := 0
1746 loop:
1747 in.token = in.cursor
1748 /*!re2c
1749 re2c:eof = 0;
1750 re2c:define:YYCTYPE = byte;
1751 re2c:define:YYPEEK = "in.data[in.cursor]";
1752 re2c:define:YYSKIP = "in.cursor += 1";
1753 re2c:define:YYBACKUP = "in.marker = in.cursor";
1754 re2c:define:YYRESTORE = "in.cursor = in.marker";
1755 re2c:define:YYLESSTHAN = "in.limit <= in.cursor";
1756 re2c:define:YYFILL = "fill(in) == 0";
1757 * { return -1 }
1759 $ { return count }
1760 ['] ([^'\\] | [\\][^])* ['] { count += 1; goto loop }
1761 [ ]+ { goto loop }
1762 */
1763 }
1764 // Prepare a file with the input text and run the lexer.
1766 func test(data string) (result int) {
1767 tmpfile := "input.txt"
1768 f, _ := os.Create(tmpfile)
1770 f.WriteString(data)
1771 f.Seek(0, 0)
1772 defer func() {
1774 if r := recover(); r != nil {
1775 result = -2
1776 }
1777 f.Close()
1778 os.Remove(tmpfile)
1779 }()
1780 in := &Input{
1782 file: f,
1783 data: make([]byte, SIZE+1),
1784 cursor: SIZE,
1785 marker: SIZE,
1786 token: SIZE,
1787 limit: SIZE,
1788 eof: false,
1789 }
1790 return lex(in)
1792 }
1793 func TestLex(t *testing.T) {
1795 var tests = []struct {
1796 res int
1797 str string
1798 }{
1799 {0, ""},
1800 {2, "'one' 'two'"},
1801 {3, "'qu\000tes' 'are' 'fine: \\'' "},
1802 {-1, "'unterminated\\'"},
1803 {-2, "'loooooooooooong'"},
1804 }
1805 for _, x := range tests {
1807 t.Run(x.str, func(t *testing.T) {
1808 res := test(x.str)
1809 if res != x.res {
1810 t.Errorf("got %d, want %d", res, x.res)
1811 }
1812 })
1813 }
1814 }
1815 YYFILL with padding
1818 In the default case (when EOF rule is not used) YYFILL is a func‐
1819 tion-like primitive that accepts a single argument and does not return
1820 any value. YYFILL invocation is triggered by condition (YYLIMIT -
1821 YYCURSOR) < n in default API and YYLESSTHAN(n) in generic API. The
1822 argument passed to YYFILL is the minimal number of characters that must
1823 be supplied. If it fails to do so, YYFILL must not return to the lexer
1824 (for that reason it is best implemented as a macro that returns from
1825 the calling function on failure). In case of a successful YYFILL invo‐
1826 cation the limit position must be set either to one after the last
1827 input position in buffer, or to the end of YYMAXFILL padding (in case
1828 YYFILL has successfully read at least n characters, but not enough to
1829 fill the entire buffer). The pictures below show the relative locations
1830 of input positions in buffer before and after YYFILL invocation (YYMAX‐
1831 FILL padding on the second picture is marked with # symbols).
1832 <-- shift --> <-- need -->
1834 >-A------------B---------C-----D-------E---F--------G->
1835 buffer token marker cursor limit
1836 >-A------------B---------C-----D-------E---F--------G->
1838 buffer, marker cursor limit
1839 token
1840 <-- shift --> <-- need -->
1842 >-A------------B---------C-----D-------E-F (EOF)
1843 buffer token marker cursor limit
1844 >-A------------B---------C-----D-------E-F###############
1846 buffer, marker cursor limit
1847 token <- YYMAXFILL ->
1848 Here is an example of a program that reads input file input.txt in
1850 chunks of 4096 bytes and uses bounds-checking with padding.
1851 //go:generate re2go $INPUT -o $OUTPUT
1853 package main
1854 import (
1856 "fmt"
1857 "os"
1858 "testing"
1859 )
1860 /*!max:re2c*/
1862 // Intentionally small to trigger buffer refill.
1864 const SIZE int = 16
1865 type Input struct {
1867 file *os.File
1868 data []byte
1869 cursor int
1870 marker int
1871 token int
1872 limit int
1873 eof bool
1874 }
1875 func fill(in *Input, need int) int {
1877 // End of input has already been reached, nothing to do.
1878 if in.eof {
1879 return -1 // Error: unexpected EOF
1880 }
1881 // Check if after moving the current lexeme to the beginning
1883 // of buffer there will be enough free space.
1884 if SIZE-(in.cursor-in.token) < need {
1885 return -2 // Error: lexeme too long
1886 }
1887 // Discard everything up to the start of the current lexeme,
1889 // shift buffer contents and adjust offsets.
1890 copy(in.data[0:], in.data[in.token:in.limit])
1891 in.cursor -= in.token
1892 in.marker -= in.token
1893 in.limit -= in.token
1894 in.token = 0
1895 // Read new data (as much as possible to fill the buffer).
1897 n, _ := in.file.Read(in.data[in.limit:SIZE])
1898 in.limit += n
1899 // If read less than expected, this is the end of input.
1901 in.eof = in.limit < SIZE
1902 // If end of input, add padding so that the lexer can read
1904 // the remaining characters at the end of buffer.
1905 if in.eof {
1906 for i := 0; i < YYMAXFILL; i += 1 {
1907 in.data[in.limit+i] = 0
1908 }
1909 in.limit += YYMAXFILL
1910 }
1911 return 0
1913 }
1914 func lex(in *Input) int {
1916 count := 0
1917 loop:
1918 in.token = in.cursor
1919 /*!re2c
1920 re2c:define:YYCTYPE = byte;
1921 re2c:define:YYPEEK = "in.data[in.cursor]";
1922 re2c:define:YYSKIP = "in.cursor += 1";
1923 re2c:define:YYBACKUP = "in.marker = in.cursor";
1924 re2c:define:YYRESTORE = "in.cursor = in.marker";
1925 re2c:define:YYLESSTHAN = "in.limit-in.cursor < @@{len}";
1926 re2c:define:YYFILL = "if r := fill(in, @@{len}); r != 0 { return r }";
1927 * { return -1 }
1929 [\x00] { return count }
1930 ['] ([^'\\] | [\\][^])* ['] { count += 1; goto loop }
1931 [ ]+ { goto loop }
1932 */
1933 }
1934 // Prepare a file with the input text and run the lexer.
1936 func test(data string) (result int) {
1937 tmpfile := "input.txt"
1938 f, _ := os.Create(tmpfile)
1940 f.WriteString(data)
1941 f.Seek(0, 0)
1942 defer func() {
1944 if r := recover(); r != nil {
1945 fmt.Println(r)
1946 result = -2
1947 }
1948 f.Close()
1949 os.Remove(tmpfile)
1950 }()
1951 in := &Input{
1953 file: f,
1954 data: make([]byte, SIZE+YYMAXFILL),
1955 cursor: SIZE,
1956 marker: SIZE,
1957 token: SIZE,
1958 limit: SIZE,
1959 eof: false,
1960 }
1961 return lex(in)
1963 }
1964 func TestLex(t *testing.T) {
1966 var tests = []struct {
1967 res int
1968 str string
1969 }{
1970 {0, ""},
1971 {2, "'one' 'two'"},
1972 {3, "'qu\000tes' 'are' 'fine: \\'' "},
1973 {-1, "'unterminated\\'"},
1974 {-2, "'loooooooooooong'"},
1975 }
1976 for _, x := range tests {
1978 t.Run(x.str, func(t *testing.T) {
1979 res := test(x.str)
1980 if res != x.res {
1981 t.Errorf("got %d, want %d", res, x.res)
1982 }
1983 })
1984 }
1985 }
1986
1989 Re2c allows one to include other files using directive /*!include:re2c
1990 FILE */, where FILE is the name of file to be included. Re2c looks for
1991 included files in the directory of the including file and in include
1992 locations, which can be specified with -I option. Re2c include direc‐
1993 tive works in the same way as C/C++ #include: the contents of FILE are
1994 copy-pasted verbatim in place of the directive. Include files may have
1995 further includes of their own. Re2c provides some predefined include
1996 files that can be found in the include/ subdirectory of the project.
1997 These files contain definitions that can be useful to other projects
1998 (such as Unicode categories) and form something like a standard library
1999 for re2c. Here is an example:
2000 Include file (definitions.go)
2002 const (
2003 ResultOk = iota
2004 ResultFail
2005 )
2006 /*!re2c
2008 number = [1-9][0-9]*;
2009 */
2010 Input file
2013 //go:generate re2go -c $INPUT -o $OUTPUT -i
2014 package main
2015 import "testing"
2017 /*!include:re2c "definitions.go" */
2018 func lex(str string) int {
2020 var cursor int
2021 /*!re2c
2022 re2c:yyfill:enable = 0;
2023 re2c:define:YYCTYPE = byte;
2024 re2c:define:YYPEEK = "str[cursor]";
2025 re2c:define:YYSKIP = "cursor += 1";
2026 number { return ResultOk }
2028 * { return ResultFail }
2029 */
2030 }
2031 func TestLex(t *testing.T) {
2033 if lex("123\000") != ResultOk {
2034 t.Errorf("error")
2035 }
2036 }
2037
2040 Re2c allows one to generate header file from the input .re file using
2041 option -t, --type-header or configuration re2c:flags:type-header and
2042 directives /*!header:re2c:on*/ and /*!header:re2c:off*/. The first
2043 directive marks the beginning of header file, and the second directive
2044 marks the end of it. Everything between these directives is processed
2045 by re2c, and the generated code is written to the file specified by the
2046 -t --type-header option (or stdout if this option was not used). Auto‐
2047 generated header file may be needed in cases when re2c is used to gen‐
2048 erate definitions of constants, variables and structs that must be vis‐
2049 ible from other translation units.
2050 Here is an example of generating a header file that contains definition
2052 of the lexer state with tag variables (the number variables depends on
2053 the regular grammar and is unknown to the programmer).
2054 Input file
2056 //go:generate re2go $INPUT -o $OUTPUT -i --type-header src/lexer/lexer.go
2057 package main
2058 import (
2060 "lexer" // generated by re2c
2061 "testing"
2062 )
2063 /*!header:re2c:on*/
2065 package lexer
2066 type State struct {
2068 Data string
2069 Cur, Mar, /*!stags:re2c format="@@{tag}"; separator=", "; */ int
2070 }
2071 /*!header:re2c:off*/
2072 func lex(st *lexer.State) int {
2074 /*!re2c
2075 re2c:flags:type-header = "src/lexer/lexer.go";
2076 re2c:yyfill:enable = 0;
2077 re2c:flags:tags = 1;
2078 re2c:define:YYCTYPE = byte;
2079 re2c:define:YYPEEK = "st.Data[st.Cur]";
2080 re2c:define:YYSKIP = "st.Cur++";
2081 re2c:define:YYBACKUP = "st.Mar = st.Cur";
2082 re2c:define:YYRESTORE = "st.Cur = st.Mar";
2083 re2c:define:YYRESTORETAG = "st.Cur = @@{tag}";
2084 re2c:define:YYSTAGP = "@@{tag} = st.Cur";
2085 re2c:tags:expression = "st.@@{tag}";
2086 re2c:tags:prefix = "Tag";
2087 [x]{1,4} / [x]{3,5} { return 0 } // ambiguous trailing context
2089 * { return 1 }
2090 */
2091 }
2092 func TestLex(t *testing.T) {
2094 st := &lexer.State{
2095 Data: "xxxxxxxx\x00",
2096 }
2097 if !(lex(st) == 0 && st.Cur == 4) {
2098 t.Error("failed")
2099 }
2100 }
2101 Header file
2104 // Code generated by re2c, DO NOT EDIT.
2105 package lexer
2107 type State struct {
2109 Data string
2110 Cur, Mar, Tag1, Tag2, Tag3 int
2111 }
2112
2115 Re2c has two options for submatch extraction.
2116 The first option is -T --tags. With this option one can use standalone
2118 tags of the form @stag and #mtag, where stag and mtag are arbitrary
2119 used-defined names. Tags can be used anywhere inside of a regular
2120 expression; semantically they are just position markers. Tags of the
2121 form @stag are called s-tags: they denote a single submatch value (the
2122 last input position where this tag matched). Tags of the form #mtag are
2123 called m-tags: they denote multiple submatch values (the whole history
2124 of repetitions of this tag). All tags should be defined by the user as
2125 variables with the corresponding names. With standalone tags re2c uses
2126 leftmost greedy disambiguation: submatch positions correspond to the
2127 leftmost matching path through the regular expression.
2128 The second option is -P --posix-captures: it enables POSIX-compliant
2130 capturing groups. In this mode parentheses in regular expressions
2131 denote the beginning and the end of capturing groups; the whole regular
2132 expression is group number zero. The number of groups for the matching
2133 rule is stored in a variable yynmatch, and submatch results are stored
2134 in yypmatch array. Both yynmatch and yypmatch should be defined by the
2135 user, and yypmatch size must be at least [yynmatch * 2]. Re2c provides
2136 a directive /*!maxnmatch:re2c*/ that defines YYMAXNMATCH: a constant
2137 equal to the maximal value of yynmatch among all rules. Note that re2c
2138 implements POSIX-compliant disambiguation: each subexpression matches
2139 as long as possible, and subexpressions that start earlier in regular
2140 expression have priority over those starting later. Capturing groups
2141 are translated into s-tags under the hood, therefore we use the word
2142 "tag" to describe them as well.
2143 With both -P --posix-captures and T --tags options re2c uses efficient
2145 submatch extraction algorithm described in the Tagged Deterministic
2146 Finite Automata with Lookahead paper. The overhead on submatch extrac‐
2147 tion in the generated lexer grows with the number of tags --- if this
2148 number is moderate, the overhead is barely noticeable. In the lexer
2149 tags are implemented using a number of tag variables generated by re2c.
2150 There is no one-to-one correspondence between tag variables and tags: a
2151 single variable may be reused for different tags, and one tag may
2152 require multiple variables to hold all its ambiguous values. Eventually
2153 ambiguity is resolved, and only one final variable per tag survives.
2154 When a rule matches, all its tags are set to the values of the corre‐
2155 sponding tag variables. The exact number of tag variables is unknown
2156 to the user; this number is determined by re2c. However, tag variables
2157 should be defined by the user as a part of the lexer state and updated
2158 by YYFILL, therefore re2c provides directives /*!stags:re2c*/ and
2159 /*!mtags:re2c*/ that can be used to declare, initialize and manipulate
2160 tag variables. These directives have two optional configurations: for‐
2161 mat = "@@"; (specifies the template where @@ is substituted with the
2162 name of each tag variable), and separator = ""; (specifies the piece of
2163 code used to join the generated pieces for different tag variables).
2164 S-tags support the following operations:
2166 · save input position to an s-tag: t = YYCURSOR with default API or a
2168 user-defined operation YYSTAGP(t) with generic API
2169 · save default value to an s-tag: t = NULL with default API or a
2171 user-defined operation YYSTAGN(t) with generic API
2172 · copy one s-tag to another: t1 = t2
2174 M-tags support the following operations:
2176 · append input position to an m-tag: a user-defined operation YYM‐
2178 TAGP(t) with both default and generic API
2179 · append default value to an m-tag: a user-defined operation YYMTAGN(t)
2181 with both default and generic API
2182 · copy one m-tag to another: t1 = t2
2184 S-tags can be implemented as scalar values (pointers or offsets).
2186 M-tags need a more complex representation, as they need to store a
2187 sequence of tag values. The most naive and inefficient representation
2188 of an m-tag is a list (array, vector) of tag values; a more efficient
2189 representation is to store all m-tags in a prefix-tree represented as
2190 array of nodes (v, p), where v is tag value and p is a pointer to par‐
2191 ent node.
2192 Here is an example of using s-tags to parse an IPv4 address.
2194 //go:generate re2go $INPUT -o $OUTPUT
2196 package main
2197 import (
2199 "errors"
2200 "testing"
2201 )
2202 var eBadIP error = errors.New("bad IP")
2204 func lex(str string) (int, error) {
2206 var cursor, marker, o1, o2, o3, o4 int
2207 /*!stags:re2c format = 'var @@ int'; separator = "\n\t"; */
2208 num := func(pos int, end int) int {
2210 n := 0
2211 for ; pos < end; pos++ {
2212 n = n*10 + int(str[pos]-'0')
2213 }
2214 return n
2215 }
2216 /*!re2c
2218 re2c:flags:tags = 1;
2219 re2c:yyfill:enable = 0;
2220 re2c:define:YYCTYPE = byte;
2221 re2c:define:YYPEEK = "str[cursor]";
2222 re2c:define:YYSKIP = "cursor += 1";
2223 re2c:define:YYBACKUP = "marker = cursor";
2224 re2c:define:YYRESTORE = "cursor = marker";
2225 re2c:define:YYSTAGP = "@@{tag} = cursor";
2226 re2c:define:YYSTAGN = "@@{tag} = -1";
2227 octet = [0-9] | [1-9][0-9] | [1][0-9][0-9] | [2][0-4][0-9] | [2][5][0-5];
2229 dot = [.];
2230 end = [\x00];
2231 @o1 octet dot @o2 octet dot @o3 octet dot @o4 octet end {
2233 return num(o4, cursor-1)+
2234 (num(o3, o4-1) << 8)+
2235 (num(o2, o3-1) << 16)+
2236 (num(o1, o2-1) << 24), nil
2237 }
2238 * { return 0, eBadIP }
2239 */
2240 }
2241 func TestLex(t *testing.T) {
2243 var tests = []struct {
2244 str string
2245 res int
2246 err error
2247 }{
2248 {"1.2.3.4\000", 0x01020304, nil},
2249 {"127.0.0.1\000", 0x7f000001, nil},
2250 {"255.255.255.255\000", 0xffffffff, nil},
2251 {"1.2.3.\000", 0, eBadIP},
2252 {"1.2.3.256\000", 0, eBadIP},
2253 }
2254 for _, x := range tests {
2256 t.Run(x.str, func(t *testing.T) {
2257 res, err := lex(x.str)
2258 if !(res == x.res && err == x.err) {
2259 t.Errorf("got %d, want %d", res, x.res)
2260 }
2261 })
2262 }
2263 }
2264 Here is an example of using POSIX capturing groups to parse an IPv4
2267 address.
2268 //go:generate re2go $INPUT -o $OUTPUT
2270 package main
2271 import (
2273 "errors"
2274 "testing"
2275 )
2276 /*!maxnmatch:re2c*/
2278 var eBadIP error = errors.New("bad IP")
2280 func lex(str string) (int, error) {
2282 var cursor, marker, yynmatch int
2283 yypmatch := make([]int, YYMAXNMATCH*2)
2284 /*!stags:re2c format = 'var @@ int'; separator = "\n\t"; */
2285 num := func(pos int, end int) int {
2287 n := 0
2288 for ; pos < end; pos++ {
2289 n = n*10 + int(str[pos]-'0')
2290 }
2291 return n
2292 }
2293 /*!re2c
2295 re2c:flags:posix-captures = 1;
2296 re2c:yyfill:enable = 0;
2297 re2c:define:YYCTYPE = byte;
2298 re2c:define:YYPEEK = "str[cursor]";
2299 re2c:define:YYSKIP = "cursor += 1";
2300 re2c:define:YYBACKUP = "marker = cursor";
2301 re2c:define:YYRESTORE = "cursor = marker";
2302 re2c:define:YYSTAGP = "@@{tag} = cursor";
2303 re2c:define:YYSTAGN = "@@{tag} = -1";
2304 re2c:define:YYSHIFTSTAG = "@@{tag} += @@{shift}";
2305 octet = [0-9] | [1-9][0-9] | [1][0-9][0-9] | [2][0-4][0-9] | [2][5][0-5];
2307 dot = [.];
2308 end = [\x00];
2309 (octet) dot (octet) dot (octet) dot (octet) end {
2311 if yynmatch != 5 {
2312 panic("expected 5 submatch groups")
2313 }
2314 return num(yypmatch[8], yypmatch[9])+
2315 (num(yypmatch[6], yypmatch[7]) << 8)+
2316 (num(yypmatch[4], yypmatch[5]) << 16)+
2317 (num(yypmatch[2], yypmatch[3]) << 24), nil
2318 }
2319 * { return 0, eBadIP }
2320 */
2321 }
2322 func TestLex(t *testing.T) {
2324 var tests = []struct {
2325 str string
2326 res int
2327 err error
2328 }{
2329 {"1.2.3.4\000", 0x01020304, nil},
2330 {"127.0.0.1\000", 0x7f000001, nil},
2331 {"255.255.255.255\000", 0xffffffff, nil},
2332 {"1.2.3.\000", 0, eBadIP},
2333 {"1.2.3.256\000", 0, eBadIP},
2334 }
2335 for _, x := range tests {
2337 t.Run(x.str, func(t *testing.T) {
2338 res, err := lex(x.str)
2339 if !(res == x.res && err == x.err) {
2340 t.Errorf("got %d, want %d", res, x.res)
2341 }
2342 })
2343 }
2344 }
2345 Here is an example of using m-tags to parse a semicolon-separated
2348 sequence of words (C++). Tag variables are stored in a tree that is
2349 packed in a vector.
2350 //go:generate re2go $INPUT -o $OUTPUT
2352 package main
2353 import (
2355 "reflect"
2356 "testing"
2357 )
2358 const (
2360 mtagRoot int = -1
2361 mtagNil int = -2
2362 )
2363 type mtagElem struct {
2365 val int
2366 pred int
2367 }
2368 type mtagTrie = []mtagElem
2370 func createTrie(capacity int) mtagTrie {
2372 return make([]mtagElem, 0, capacity)
2373 }
2374 func mtag(trie *mtagTrie, tag int, val int) int {
2376 *trie = append(*trie, mtagElem{val, tag})
2377 return len(*trie) - 1
2378 }
2379 // Recursively unwind both tag histories and consruct submatches.
2381 func unwind(trie mtagTrie, x int, y int, str string) []string {
2382 if x == mtagRoot && y == mtagRoot {
2383 return []string{}
2384 } else if x == mtagRoot || y == mtagRoot {
2385 panic("tag histories have different length")
2386 } else {
2387 xval := trie[x].val
2388 yval := trie[y].val
2389 ss := unwind(trie, trie[x].pred, trie[y].pred, str)
2390 // Either both tags should be nil, or none of them.
2392 if xval == mtagNil && yval == mtagNil {
2393 return ss
2394 } else if xval == mtagNil || yval == mtagNil {
2395 panic("tag histories positive/negative tag mismatch")
2396 } else {
2397 s := str[xval:yval]
2398 return append(ss, s)
2399 }
2400 }
2401 }
2402 func lex(str string) []string {
2404 var cursor, marker int
2405 trie := createTrie(256)
2406 x := mtagRoot
2407 y := mtagRoot
2408 /*!mtags:re2c format = "@@ := mtagRoot"; separator = "\n\t"; */
2409 /*!re2c
2411 re2c:flags:tags = 1;
2412 re2c:yyfill:enable = 0;
2413 re2c:define:YYCTYPE = byte;
2414 re2c:define:YYPEEK = "str[cursor]";
2415 re2c:define:YYSKIP = "cursor += 1";
2416 re2c:define:YYBACKUP = "marker = cursor";
2417 re2c:define:YYRESTORE = "cursor = marker";
2418 re2c:define:YYMTAGP = "@@{tag} = mtag(&trie, @@{tag}, cursor)";
2419 re2c:define:YYMTAGN = "@@{tag} = mtag(&trie, @@{tag}, mtagNil)";
2420 end = [\x00];
2422 (#x [a-z]+ #y [;])* end { return unwind(trie, x, y, str) }
2424 * { return nil }
2425 */
2426 }
2427 func TestLex(t *testing.T) {
2429 var tests = []struct {
2430 str string
2431 res []string
2432 }{
2433 {"\000", []string{}},
2434 {"one;two;three;\000", []string{"one", "two", "three"}},
2435 {"one;two\000", nil},
2436 }
2437 for _, x := range tests {
2439 t.Run(x.str, func(t *testing.T) {
2440 res := lex(x.str)
2441 if !reflect.DeepEqual(res, x.res) {
2442 t.Errorf("got %v, want %v", res, x.res)
2443 }
2444 })
2445 }
2446 }
2447
2450 With -f --storable-state option re2c generates a lexer that can store
2451 its current state, return to the caller, and later resume operations
2452 exactly where it left off. The default mode of operation in re2c is a
2453 "pull" model, in which the lexer "pulls" more input whenever it needs
2454 it. This may be unacceptable in cases when the input becomes available
2455 piece by piece (for example, if the lexer is invoked by the parser, or
2456 if the lexer program communicates via a socket protocol with some other
2457 program that must wait for a reply from the lexer before it transmits
2458 the next message). Storable state feature is intended exactly for such
2459 cases: it allows one to generate lexers that work in a "push" model.
2460 When the lexer needs more input, it stores its state and returns to the
2461 caller. Later, when more input becomes available, the caller resumes
2462 the lexer exactly where it stopped. There are a few changes necessary
2463 compared to the "pull" model:
2464 · Define YYSETSTATE() and YYGETSTATE(state) promitives.
2466 · Define yych, yyaccept and state variables as a part of persistent
2468 lexer state. The state variable should be initialized to -1.
2469 · YYFILL should return to the outer program instead of trying to supply
2471 more input. Return code should indicate that lexer needs more input.
2472 · The outer program should recognize situations when lexer needs more
2474 input and respond appropriately.
2475 · Use /*!getstate:re2c*/ directive if it is necessary to execute any
2477 code before entering the lexer.
2478 · Use configurations state:abort and state:nextlabel to further tweak
2480 the generated code.
2481 Here is an example of a "push"-model lexer that reads input from stdin
2483 and expects a sequence of words separated by spaces and newlines. The
2484 lexer loops forever, waiting for more input. It can be terminated by
2485 sending a special EOF token --- a word "stop", in which case the lexer
2486 terminates successfully and prints the number of words it has seen.
2487 Abnormal termination happens in case of a syntax error, premature end
2488 of input (without the "stop" word) or in case the buffer is too small
2489 to hold a lexeme (for example, if one of the words exceeds buffer
2490 size). Premature end of input happens in case the lexer fails to read
2491 any input while being in the initial state --- this is the only case
2492 when EOF rule matches. Note that the lexer may call YYFILL twice before
2493 terminating (and thus require hitting Ctrl+D a few times). First time
2494 YYFILL is called when the lexer expects continuation of the current
2495 greedy lexeme (either a word or a whitespace sequence). If YYFILL
2496 fails, the lexer knows that it has reached the end of the current lex‐
2497 eme and executes the corresponding semantic action. The action jumps to
2498 the beginning of the loop, the lexer enters the initial state and calls
2499 YYFILL once more. If it fails, the lexer matches EOF rule. (Alterna‐
2500 tively EOF rule can be used for termination instead of a special EOF
2501 lexeme.)
2502 Example
2504 //go:generate re2go -f $INPUT -o $OUTPUT
2505 package main
2506 import (
2508 "fmt"
2509 "os"
2510 "testing"
2511 )
2512 // Intentionally small to trigger buffer refill.
2514 const SIZE int = 16
2515 type Input struct {
2517 file *os.File
2518 data []byte
2519 cursor int
2520 marker int
2521 token int
2522 limit int
2523 state int
2524 yyaccept int
2525 }
2526 const (
2528 lexEnd = iota
2529 lexReady
2530 lexWaitingForInput
2531 lexPacketBroken
2532 lexPacketTooBig
2533 lexCountMismatch
2534 )
2535 func fill(in *Input) int {
2537 if in.token == 0 {
2538 // Error: no space can be freed.
2539 // In real life can reallocate a larger buffer.
2540 return lexPacketTooBig
2541 }
2542 // Discard everything up to the start of the current lexeme,
2544 // shift buffer contents and adjust offsets.
2545 copy(in.data[0:], in.data[in.token:in.limit])
2546 in.cursor -= in.token
2547 in.marker -= in.token
2548 in.limit -= in.token
2549 in.token = 0
2550 // Read new data (as much as possible to fill the buffer).
2552 n, _ := in.file.Read(in.data[in.limit:SIZE])
2553 in.limit += n
2554 in.data[in.limit] = 0 // append sentinel symbol
2555 return lexReady
2557 }
2558 func lex(in *Input, recv *int) int {
2560 var yych byte
2561 /*!getstate:re2c*/
2562 loop:
2563 in.token = in.cursor
2564 /*!re2c
2565 re2c:eof = 0;
2566 re2c:define:YYPEEK = "in.data[in.cursor]";
2567 re2c:define:YYSKIP = "in.cursor += 1";
2568 re2c:define:YYBACKUP = "in.marker = in.cursor";
2569 re2c:define:YYRESTORE = "in.cursor = in.marker";
2570 re2c:define:YYLESSTHAN = "in.limit <= in.cursor";
2571 re2c:define:YYFILL = "return lexWaitingForInput";
2572 re2c:define:YYGETSTATE = "in.state";
2573 re2c:define:YYSETSTATE = "in.state = @@{state}";
2574 packet = [a-z]+[;];
2576 * { return lexPacketBroken }
2578 $ { return lexEnd }
2579 packet { *recv = *recv + 1; goto loop }
2580 */
2581 }
2582 func test(packets []string) int {
2584 fname := "pipe"
2585 fw, _ := os.Create(fname);
2586 fr, _ := os.Open(fname);
2587 in := &Input{
2589 file: fr,
2590 data: make([]byte, SIZE+1),
2591 cursor: SIZE,
2592 marker: SIZE,
2593 token: SIZE,
2594 limit: SIZE,
2595 state: -1,
2596 }
2597 // data is zero-initialized, no need to write sentinel
2598 var status int
2600 send := 0
2601 recv := 0
2602 loop:
2603 for {
2604 status = lex(in, &recv)
2605 if status == lexEnd {
2606 if send != recv {
2607 status = lexCountMismatch
2608 }
2609 break loop
2610 } else if status == lexWaitingForInput {
2611 if send < len(packets) {
2612 fw.WriteString(packets[send])
2613 send += 1
2614 }
2615 status = fill(in)
2616 if status != lexReady {
2617 break loop
2618 }
2619 } else if status == lexPacketBroken {
2620 break loop
2621 } else {
2622 panic("unexpected status")
2623 }
2624 }
2625 fr.Close()
2627 fw.Close()
2628 os.Remove(fname)
2629 return status
2631 }
2632 func TestLex(t *testing.T) {
2634 var tests = []struct {
2635 status int
2636 packets []string
2637 }{
2638 {lexEnd, []string{}},
2639 {lexEnd, []string{"zero;", "one;", "two;", "three;", "four;"}},
2640 {lexPacketBroken, []string{"??;"}},
2641 {lexPacketTooBig, []string{"looooooooooooong;"}},
2642 }
2643 for i, x := range tests {
2645 t.Run(fmt.Sprintf("%d", i), func(t *testing.T) {
2646 status := test(x.packets)
2647 if status != x.status {
2648 t.Errorf("got %d, want %d", status, x.status)
2649 }
2650 })
2651 }
2652 }
2653
2656 Reuse mode is enabled with the -r --reusable option. In this mode re2c
2657 allows one to reuse definitions, configurations and rules specified by
2658 a /*!rules:re2c*/ block in subsequent /*!use:re2c*/ blocks. As of
2659 re2c-1.2 it is possible to mix such blocks with normal /*!re2c*/
2660 blocks; prior to that re2c expects a single rules-block followed by
2661 use-blocks (normal blocks are disallowed). Use-blocks can have addi‐
2662 tional definitions, configurations and rules: they are merged to those
2663 specified by the rules-block. A very common use case for -r --reusable
2664 option is a lexer that supports multiple input encodings: lexer rules
2665 are defined once and reused multiple times with encoding-specific con‐
2666 figurations, such as re2c:flags:utf-8.
2667 Below is an example of a multi-encoding lexer: it reads a phrase with
2669 Unicode math symbols and accepts input either in UTF8 or in UT32. Note
2670 that the --input-encoding utf8 option allows us to write UTF8-encoded
2671 symbols in the regular expressions; without this option re2c would
2672 parse them as a plain ASCII byte sequnce (and we would have to use
2673 hexadecimal escape sequences).
2674 Example
2676 //go:generate re2go $INPUT -o $OUTPUT -r --input-encoding utf8
2677 package main
2678 import "testing"
2680 /*!rules:re2c
2682 re2c:yyfill:enable = 0;
2683 re2c:define:YYPEEK = "str[cursor]";
2684 re2c:define:YYSKIP = "cursor += 1";
2685 re2c:define:YYBACKUP = "marker = cursor";
2686 re2c:define:YYRESTORE = "cursor = marker";
2687 "∀x ∃y: p(x, y)" { return 0; }
2689 * { return 1; }
2690 */
2691 func lexUTF8(str []uint8) int {
2693 var cursor, marker int
2694 /*!use:re2c
2695 re2c:flags:8 = 1;
2696 re2c:define:YYCTYPE = uint8;
2697 */
2698 }
2699 func lexUTF32(str []uint32) int {
2701 var cursor, marker int
2702 /*!use:re2c
2703 re2c:flags:u = 1;
2704 re2c:define:YYCTYPE = uint32;
2705 */
2706 }
2707 func TestLexUTF8(t *testing.T) {
2709 s_utf8 := []uint8{
2710 0xe2, 0x88, 0x80, 0x78, 0x20, 0xe2, 0x88, 0x83, 0x79,
2711 0x3a, 0x20, 0x70, 0x28, 0x78, 0x2c, 0x20, 0x79, 0x29};
2712 if lexUTF8(s_utf8) != 0 {
2714 t.Errorf("utf8 failed")
2715 }
2716 }
2717 func TestLexUTF32(t *testing.T) {
2719 s_utf32 := []uint32{
2720 0x00002200, 0x00000078, 0x00000020, 0x00002203, 0x00000079,
2721 0x0000003a, 0x00000020, 0x00000070, 0x00000028, 0x00000078,
2722 0x0000002c, 0x00000020, 0x00000079, 0x00000029};
2723 if lexUTF32(s_utf32) != 0 {
2725 t.Errorf("utf32 failed")
2726 }
2727 }
2728
2731 re2c supports the following encodings: ASCII (default), EBCDIC (-e),
2732 UCS-2 (-w), UTF-16 (-x), UTF-32 (-u) and UTF-8 (-8). See also inplace
2733 configuration re2c:flags.
2734 The following concepts should be clarified when talking about encod‐
2736 ings. A code point is an abstract number that represents a single sym‐
2737 bol. A code unit is the smallest unit of memory, which is used in the
2738 encoded text (it corresponds to one character in the input stream). One
2739 or more code units may be needed to represent a single code point,
2740 depending on the encoding. In a fixed-length encoding, each code point
2741 is represented with an equal number of code units. In variable-length
2742 encodings, different code points can be represented with different num‐
2743 ber of code units.
2744 · ASCII is a fixed-length encoding. Its code space includes 0x100 code
2746 points, from 0 to 0xFF. A code point is represented with exactly one
2747 1-byte code unit, which has the same value as the code point. The
2748 size of YYCTYPE must be 1 byte.
2749 · EBCDIC is a fixed-length encoding. Its code space includes 0x100 code
2751 points, from 0 to 0xFF. A code point is represented with exactly one
2752 1-byte code unit, which has the same value as the code point. The
2753 size of YYCTYPE must be 1 byte.
2754 · UCS-2 is a fixed-length encoding. Its code space includes 0x10000
2756 code points, from 0 to 0xFFFF. One code point is represented with
2757 exactly one 2-byte code unit, which has the same value as the code
2758 point. The size of YYCTYPE must be 2 bytes.
2759 · UTF-16 is a variable-length encoding. Its code space includes all
2761 Unicode code points, from 0 to 0xD7FF and from 0xE000 to 0x10FFFF.
2762 One code point is represented with one or two 2-byte code units. The
2763 size of YYCTYPE must be 2 bytes.
2764 · UTF-32 is a fixed-length encoding. Its code space includes all Uni‐
2766 code code points, from 0 to 0xD7FF and from 0xE000 to 0x10FFFF. One
2767 code point is represented with exactly one 4-byte code unit. The size
2768 of YYCTYPE must be 4 bytes.
2769 · UTF-8 is a variable-length encoding. Its code space includes all Uni‐
2771 code code points, from 0 to 0xD7FF and from 0xE000 to 0x10FFFF. One
2772 code point is represented with a sequence of one, two, three, or four
2773 1-byte code units. The size of YYCTYPE must be 1 byte.
2774 In Unicode, values from range 0xD800 to 0xDFFF (surrogates) are not
2776 valid Unicode code points. Any encoded sequence of code units that
2777 would map to Unicode code points in the range 0xD800-0xDFFF, is
2778 ill-formed. The user can control how re2c treats such ill-formed
2779 sequences with the --encoding-policy <policy> switch.
2780 For some encodings, there are code units that never occur in a valid
2782 encoded stream (e.g., 0xFF byte in UTF-8). If the generated scanner
2783 must check for invalid input, the only correct way to do so is to use
2784 the default rule (*). Note that the full range rule ([^]) won't catch
2785 invalid code units when a variable-length encoding is used ([^] means
2786 "any valid code point", whereas the default rule (*) means "any possi‐
2787 ble code unit").
2788
2790 Conditions are enabled with -c --conditions. This option allows one to
2791 encode multiple interrelated lexers within the same re2c block.
2792 Each lexer corresponds to a single condition. It starts with a label
2794 of the form yyc_name, where name is condition name and yyc prefix can
2795 be adjusted with configuration re2c:condprefix. Different lexers are
2796 separated with a comment /* *********************************** */
2797 which can be adjusted with configuration re2c:cond:divider.
2798 Furthermore, each condition has a unique identifier of the form yyc‐
2800 name, where name is condition name and yyc prefix can be adjusted with
2801 configuration re2c:condenumprefix. Identifiers have the type YYCOND‐
2802 TYPE and should be generated with /*!types:re2c*/ directive or -t
2803 --type-header option. Users shouldn't define these identifiers manu‐
2804 ally, as the order of conditions is not specified.
2805 Before all conditions re2c generates entry code that checks the current
2807 condition identifier and transfers control flow to the start label of
2808 the active condition. After matching some rule of this condition,
2809 lexer may either transfer control flow back to the entry code (after
2810 executing the associated action and optionally setting another condi‐
2811 tion with =>), or use :=> shortcut and transition directly to the start
2812 label of another condition (skipping the action and the entry code).
2813 Configuration re2c:cond:goto allows one to change the default behavior.
2814 Syntactically each rule must be preceded with a list of comma-separated
2816 condition names or a wildcard * enclosed in angle brackets < and >.
2817 Wildcard means "any condition" and is semantically equivalent to list‐
2818 ing all condition names. Here regexp is a regular expression, default
2819 refers to the default rule *, and action is a block of code.
2820 · <conditions-or-wildcard> regexp-or-default action
2822 · <conditions-or-wildcard> regexp-or-default => condition action
2824 · <conditions-or-wildcard> regexp-or-default :=> condition
2826 Rules with an exclamation mark ! in front of condition list have a spe‐
2828 cial meaning: they have no regular expression, and the associated
2829 action is merged as an entry code to actions of normal rules. This
2830 might be a convenient place to peform a routine task that is common to
2831 all rules.
2832 · <!conditions-or-wildcard> action
2834 Another special form of rules with an empty condition list <> and no
2836 regular expression allows one to specify an "entry condition" that can
2837 be used to execute code before entering the lexer. It is semantically
2838 equivalent to a condition with number zero, name 0 and an empty regular
2839 expression.
2840 · <> action
2842 · <> => condition action
2844 · <> :=> condition
2846 Example
2848 //go:generate re2go -c $INPUT -o $OUTPUT -i
2849 package main
2850 import (
2852 "errors"
2853 "testing"
2854 )
2855 var (
2857 eSyntax = errors.New("syntax error")
2858 eOverflow = errors.New("overflow error")
2859 )
2860 /*!types:re2c*/
2862 const u32Limit uint64 = 1<<32
2864 func parse_u32(str string) (uint32, error) {
2866 var cursor, marker int
2867 result := uint64(0)
2868 cond := yycinit
2869 add_digit := func(base uint64, offset byte) {
2871 result = result * base + uint64(str[cursor-1] - offset)
2872 if result >= u32Limit {
2873 result = u32Limit
2874 }
2875 }
2876 /*!re2c
2878 re2c:yyfill:enable = 0;
2879 re2c:define:YYCTYPE = byte;
2880 re2c:define:YYPEEK = "str[cursor]";
2881 re2c:define:YYSKIP = "cursor += 1";
2882 re2c:define:YYSHIFT = "cursor += @@{shift}";
2883 re2c:define:YYBACKUP = "marker = cursor";
2884 re2c:define:YYRESTORE = "cursor = marker";
2885 re2c:define:YYGETCONDITION = "cond";
2886 re2c:define:YYSETCONDITION = "cond = @@";
2887 <*> * { return 0, eSyntax }
2889 <init> '0b' / [01] :=> bin
2891 <init> "0" :=> oct
2892 <init> "" / [1-9] :=> dec
2893 <init> '0x' / [0-9a-fA-F] :=> hex
2894 <bin, oct, dec, hex> "\x00" {
2896 if result < u32Limit {
2897 return uint32(result), nil
2898 } else {
2899 return 0, eOverflow
2900 }
2901 }
2902 <bin> [01] { add_digit(2, '0'); goto yyc_bin }
2904 <oct> [0-7] { add_digit(8, '0'); goto yyc_oct }
2905 <dec> [0-9] { add_digit(10, '0'); goto yyc_dec }
2906 <hex> [0-9] { add_digit(16, '0'); goto yyc_hex }
2907 <hex> [a-f] { add_digit(16, 'a'-10); goto yyc_hex }
2908 <hex> [A-F] { add_digit(16, 'A'-10); goto yyc_hex }
2909 */
2910 }
2911 func TestLex(t *testing.T) {
2913 var tests = []struct {
2914 num uint32
2915 str string
2916 err error
2917 }{
2918 {1234567890, "1234567890\000", nil},
2919 {13, "0b1101\000", nil},
2920 {0x7fe, "0x007Fe\000", nil},
2921 {0644, "0644\000", nil},
2922 {0, "9999999999\000", eOverflow},
2923 {0, "123??\000", eSyntax},
2924 }
2925 for _, x := range tests {
2927 t.Run(x.str, func(t *testing.T) {
2928 num, err := parse_u32(x.str)
2929 if !(num == x.num && err == x.err) {
2930 t.Errorf("got %d, want %d", num, x.num)
2931 }
2932 })
2933 }
2934 }
2935
2938 With the -S, --skeleton option, re2c ignores all non-re2c code and gen‐
2939 erates a self-contained C program that can be further compiled and exe‐
2940 cuted. The program consists of lexer code and input data. For each con‐
2941 structed DFA (block or condition) re2c generates a standalone lexer and
2942 two files: an .input file with strings derived from the DFA and a .keys
2943 file with expected match results. The program runs each lexer on the
2944 corresponding .input file and compares results with the expectations.
2945 Skeleton programs are very useful for a number of reasons:
2946 · They can check correctness of various re2c optimizations (the data is
2948 generated early in the process, before any DFA transformations have
2949 taken place).
2950 · Generating a set of input data with good coverage may be useful for
2952 both testing and benchmarking.
2953 · Generating self-contained executable programs allows one to get mini‐
2955 mized test cases (the original code may be large or have a lot of
2956 dependencies).
2957 The difficulty with generating input data is that for all but the most
2959 trivial cases the number of possible input strings is too large (even
2960 if the string length is limited). Re2c solves this difficulty by gener‐
2961 ating sufficiently many strings to cover almost all DFA transitions. It
2962 uses the following algorithm. First, it constructs a skeleton of the
2963 DFA. For encodings with 1-byte code unit size (such as ASCII, UTF-8 and
2964 EBCDIC) skeleton is just an exact copy of the original DFA. For encod‐
2965 ings with multibyte code units skeleton is a copy of DFA with certain
2966 transitions omitted: namely, re2c takes at most 256 code units for each
2967 disjoint continuous range that corresponds to a DFA transition. The
2968 chosen values are evenly distributed and include range bounds. Instead
2969 of trying to cover all possible paths in the skeleton (which is infea‐
2970 sible) re2c generates sufficiently many paths to cover all skeleton
2971 transitions, and thus trigger the corresponding conditional jumps in
2972 the lexer. The algorithm implementation is limited by ~1Gb of transi‐
2973 tions and consumes constant amount of memory (re2c writes data to file
2974 as soon as it is generated).
2975
2977 With the -D, --emit-dot option, re2c does not generate code. Instead,
2978 it dumps the generated DFA in DOT format. One can convert this dump to
2979 an image of the DFA using Graphviz or another library. Note that this
2980 option shows the final DFA after it has gone through a number of opti‐
2981 mizations and transformations. Earlier stages can be dumped with vari‐
2982 ous debug options, such as --dump-nfa, --dump-dfa-raw etc. (see the
2983 full list of options).
2984
2986 You can find more information about re2c at the official website:
2987 http://re2c.org. Similar programs are flex(1), lex(1), quex(‐
2988 http://quex.sourceforge.net).
2989
2991 Re2c was originaly written by Peter Bumbulis in 1993. Since then it
2992 has been developed and maintained by multiple volunteers; mots notably,
2993 Brain Young, Marcus Boerger, Dan Nuffer and Ulya Trofimovich.
2994 RE2C(1)