1FLEX(1) General Commands Manual FLEX(1)
2
6 flex - fast lexical analyzer generator
7
9 flex [-bcdfhilnpstvwBFILTV78+? -C[aefFmr] -ooutput -Pprefix -Sskeleton]
10 [--help --version] [filename ...]
11
13 This manual describes flex, a tool for generating programs that perform
14 pattern-matching on text. The manual includes both tutorial and refer‐
15 ence sections:
16 Description
18 a brief overview of the tool
19 Some Simple Examples
21 Format Of The Input File
23 Patterns
25 the extended regular expressions used by flex
26 How The Input Is Matched
28 the rules for determining what has been matched
29 Actions
31 how to specify what to do when a pattern is matched
32 The Generated Scanner
34 details regarding the scanner that flex produces;
35 how to control the input source
36 Start Conditions
38 introducing context into your scanners, and
39 managing "mini-scanners"
40 Multiple Input Buffers
42 how to manipulate multiple input sources; how to
43 scan from strings instead of files
44 End-of-file Rules
46 special rules for matching the end of the input
47 Miscellaneous Macros
49 a summary of macros available to the actions
50 Values Available To The User
52 a summary of values available to the actions
53 Interfacing With Yacc
55 connecting flex scanners together with yacc parsers
56 Options
58 flex command-line options, and the "%option"
59 directive
60 Performance Considerations
62 how to make your scanner go as fast as possible
63 Generating C++ Scanners
65 the (experimental) facility for generating C++
66 scanner classes
67 Incompatibilities With Lex And POSIX
69 how flex differs from AT&T lex and the POSIX lex
70 standard
71 Diagnostics
73 those error messages produced by flex (or scanners
74 it generates) whose meanings might not be apparent
75 Files
77 files used by flex
78 Deficiencies / Bugs
80 known problems with flex
81 See Also
83 other documentation, related tools
84 Author
86 includes contact information
87
90 flex is a tool for generating scanners: programs which recognized lexi‐
91 cal patterns in text. flex reads the given input files, or its stan‐
92 dard input if no file names are given, for a description of a scanner
93 to generate. The description is in the form of pairs of regular
94 expressions and C code, called rules. flex generates as output a C
95 source file, lex.yy.c, which defines a routine yylex(). This file is
96 compiled and linked with the -lfl library to produce an executable.
97 When the executable is run, it analyzes its input for occurrences of
98 the regular expressions. Whenever it finds one, it executes the corre‐
99 sponding C code.
100
102 First some simple examples to get the flavor of how one uses flex. The
103 following flex input specifies a scanner which whenever it encounters
104 the string "username" will replace it with the user's login name:
105 %%
107 username printf( "%s", getlogin() );
108 By default, any text not matched by a flex scanner is copied to the
110 output, so the net effect of this scanner is to copy its input file to
111 its output with each occurrence of "username" expanded. In this input,
112 there is just one rule. "username" is the pattern and the "printf" is
113 the action. The "%%" marks the beginning of the rules.
114 Here's another simple example:
116 int num_lines = 0, num_chars = 0;
118 %%
120 \n ++num_lines; ++num_chars;
121 . ++num_chars;
122 %%
124 main()
125 {
126 yylex();
127 printf( "# of lines = %d, # of chars = %d\n",
128 num_lines, num_chars );
129 }
130 This scanner counts the number of characters and the number of lines in
132 its input (it produces no output other than the final report on the
133 counts). The first line declares two globals, "num_lines" and
134 "num_chars", which are accessible both inside yylex() and in the main()
135 routine declared after the second "%%". There are two rules, one which
136 matches a newline ("\n") and increments both the line count and the
137 character count, and one which matches any character other than a new‐
138 line (indicated by the "." regular expression).
139 A somewhat more complicated example:
141 /* scanner for a toy Pascal-like language */
143 %{
145 /* need this for the call to atof() below */
146 #include <math.h>
147 %}
148 DIGIT [0-9]
150 ID [a-z][a-z0-9]*
151 %%
153 {DIGIT}+ {
155 printf( "An integer: %s (%d)\n", yytext,
156 atoi( yytext ) );
157 }
158 {DIGIT}+"."{DIGIT}* {
160 printf( "A float: %s (%g)\n", yytext,
161 atof( yytext ) );
162 }
163 if|then|begin|end|procedure|function {
165 printf( "A keyword: %s\n", yytext );
166 }
167 {ID} printf( "An identifier: %s\n", yytext );
169 "+"|"-"|"*"|"/" printf( "An operator: %s\n", yytext );
171 "{"[^}\n]*"}" /* eat up one-line comments */
173 [ \t\n]+ /* eat up whitespace */
175 . printf( "Unrecognized character: %s\n", yytext );
177 %%
179 main( argc, argv )
181 int argc;
182 char **argv;
183 {
184 ++argv, --argc; /* skip over program name */
185 if ( argc > 0 )
186 yyin = fopen( argv[0], "r" );
187 else
188 yyin = stdin;
189 yylex();
191 }
192 This is the beginnings of a simple scanner for a language like Pascal.
194 It identifies different types of tokens and reports on what it has
195 seen.
196 The details of this example will be explained in the following sec‐
198 tions.
199
201 The flex input file consists of three sections, separated by a line
202 with just %% in it:
203 definitions
205 %%
206 rules
207 %%
208 user code
209 The definitions section contains declarations of simple name defini‐
211 tions to simplify the scanner specification, and declarations of start
212 conditions, which are explained in a later section.
213 Name definitions have the form:
215 name definition
217 The "name" is a word beginning with a letter or an underscore ('_')
219 followed by zero or more letters, digits, '_', or '-' (dash). The def‐
220 inition is taken to begin at the first non-white-space character fol‐
221 lowing the name and continuing to the end of the line. The definition
222 can subsequently be referred to using "{name}", which will expand to
223 "(definition)". For example,
224 DIGIT [0-9]
226 ID [a-z][a-z0-9]*
227 defines "DIGIT" to be a regular expression which matches a single
229 digit, and "ID" to be a regular expression which matches a letter fol‐
230 lowed by zero-or-more letters-or-digits. A subsequent reference to
231 {DIGIT}+"."{DIGIT}*
233 is identical to
235 ([0-9])+"."([0-9])*
237 and matches one-or-more digits followed by a '.' followed by zero-or-
239 more digits.
240 The rules section of the flex input contains a series of rules of the
242 form:
243 pattern action
245 where the pattern must be unindented and the action must begin on the
247 same line.
248 See below for a further description of patterns and actions.
250 Finally, the user code section is simply copied to lex.yy.c verbatim.
252 It is used for companion routines which call or are called by the scan‐
253 ner. The presence of this section is optional; if it is missing, the
254 second %% in the input file may be skipped, too.
255 In the definitions and rules sections, any indented text or text
257 enclosed in %{ and %} is copied verbatim to the output (with the %{}'s
258 removed). The %{}'s must appear unindented on lines by themselves.
259 In the rules section, any indented or %{} text appearing before the
261 first rule may be used to declare variables which are local to the
262 scanning routine and (after the declarations) code which is to be exe‐
263 cuted whenever the scanning routine is entered. Other indented or %{}
264 text in the rule section is still copied to the output, but its meaning
265 is not well-defined and it may well cause compile-time errors (this
266 feature is present for POSIX compliance; see below for other such fea‐
267 tures).
268 In the definitions section (but not in the rules section), an unin‐
270 dented comment (i.e., a line beginning with "/*") is also copied verba‐
271 tim to the output up to the next "*/".
272
274 The patterns in the input are written using an extended set of regular
275 expressions. These are:
276 x match the character 'x'
278 . any character (byte) except newline
279 [xyz] a "character class"; in this case, the pattern
280 matches either an 'x', a 'y', or a 'z'
281 [abj-oZ] a "character class" with a range in it; matches
282 an 'a', a 'b', any letter from 'j' through 'o',
283 or a 'Z'
284 [^A-Z] a "negated character class", i.e., any character
285 but those in the class. In this case, any
286 character EXCEPT an uppercase letter.
287 [^A-Z\n] any character EXCEPT an uppercase letter or
288 a newline
289 r* zero or more r's, where r is any regular expression
290 r+ one or more r's
291 r? zero or one r's (that is, "an optional r")
292 r{2,5} anywhere from two to five r's
293 r{2,} two or more r's
294 r{4} exactly 4 r's
295 {name} the expansion of the "name" definition
296 (see above)
297 "[xyz]\"foo"
298 the literal string: [xyz]"foo
299 \X if X is an 'a', 'b', 'f', 'n', 'r', 't', or 'v',
300 then the ANSI-C interpretation of \x.
301 Otherwise, a literal 'X' (used to escape
302 operators such as '*')
303 \0 a NUL character (ASCII code 0)
304 \123 the character with octal value 123
305 \x2a the character with hexadecimal value 2a
306 (r) match an r; parentheses are used to override
307 precedence (see below)
308 rs the regular expression r followed by the
311 regular expression s; called "concatenation"
312 r|s either an r or an s
315 r/s an r but only if it is followed by an s. The
318 text matched by s is included when determining
319 whether this rule is the "longest match",
320 but is then returned to the input before
321 the action is executed. So the action only
322 sees the text matched by r. This type
323 of pattern is called trailing context".
324 (There are some combinations of r/s that flex
325 cannot match correctly; see notes in the
326 Deficiencies / Bugs section below regarding
327 "dangerous trailing context".)
328 ^r an r, but only at the beginning of a line (i.e.,
329 which just starting to scan, or right after a
330 newline has been scanned).
331 r$ an r, but only at the end of a line (i.e., just
332 before a newline). Equivalent to "r/\n".
333 Note that flex's notion of "newline" is exactly
335 whatever the C compiler used to compile flex
336 interprets '\n' as; in particular, on some DOS
337 systems you must either filter out \r's in the
338 input yourself, or explicitly use r/\r\n for "r$".
339 <s>r an r, but only in start condition s (see
342 below for discussion of start conditions)
343 <s1,s2,s3>r
344 same, but in any of start conditions s1,
345 s2, or s3
346 <*>r an r in any start condition, even an exclusive one.
347 <<EOF>> an end-of-file
350 <s1,s2><<EOF>>
351 an end-of-file when in start condition s1 or s2
352 Note that inside of a character class, all regular expression operators
354 lose their special meaning except escape ('\') and the character class
355 operators, '-', ']', and, at the beginning of the class, '^'.
356 The regular expressions listed above are grouped according to prece‐
358 dence, from highest precedence at the top to lowest at the bottom.
359 Those grouped together have equal precedence. For example,
360 foo|bar*
362 is the same as
364 (foo)|(ba(r*))
366 since the '*' operator has higher precedence than concatenation, and
368 concatenation higher than alternation ('|'). This pattern therefore
369 matches either the string "foo" or the string "ba" followed by zero-or-
370 more r's. To match "foo" or zero-or-more "bar"'s, use:
371 foo|(bar)*
373 and to match zero-or-more "foo"'s-or-"bar"'s:
375 (foo|bar)*
377 In addition to characters and ranges of characters, character classes
380 can also contain character class expressions. These are expressions
381 enclosed inside [: and :] delimiters (which themselves must appear
382 between the '[' and ']' of the character class; other elements may
383 occur inside the character class, too). The valid expressions are:
384 [:alnum:] [:alpha:] [:blank:]
386 [:cntrl:] [:digit:] [:graph:]
387 [:lower:] [:print:] [:punct:]
388 [:space:] [:upper:] [:xdigit:]
389 These expressions all designate a set of characters equivalent to the
391 corresponding standard C isXXX function. For example, [:alnum:] desig‐
392 nates those characters for which isalnum() returns true - i.e., any
393 alphabetic or numeric. Some systems don't provide isblank(), so flex
394 defines [:blank:] as a blank or a tab.
395 For example, the following character classes are all equivalent:
397 [[:alnum:]]
399 [[:alpha:][:digit:]
400 [[:alpha:]0-9]
401 [a-zA-Z0-9]
402 If your scanner is case-insensitive (the -i flag), then [:upper:] and
404 [:lower:] are equivalent to [:alpha:].
405 Some notes on patterns:
407 - A negated character class such as the example "[^A-Z]" above
409 will match a newline unless "\n" (or an equivalent escape
410 sequence) is one of the characters explicitly present in the
411 negated character class (e.g., "[^A-Z\n]"). This is unlike how
412 many other regular expression tools treat negated character
413 classes, but unfortunately the inconsistency is historically
414 entrenched. Matching newlines means that a pattern like [^"]*
415 can match the entire input unless there's another quote in the
416 input.
417 - A rule can have at most one instance of trailing context (the
419 '/' operator or the '$' operator). The start condition, '^',
420 and "<<EOF>>" patterns can only occur at the beginning of a pat‐
421 tern, and, as well as with '/' and '$', cannot be grouped inside
422 parentheses. A '^' which does not occur at the beginning of a
423 rule or a '$' which does not occur at the end of a rule loses
424 its special properties and is treated as a normal character.
425 The following are illegal:
427 foo/bar$
429 <sc1>foo<sc2>bar
430 Note that the first of these, can be written "foo/bar\n".
432 The following will result in '$' or '^' being treated as a nor‐
434 mal character:
435 foo|(bar$)
437 foo|^bar
438 If what's wanted is a "foo" or a bar-followed-by-a-newline, the
440 following could be used (the special '|' action is explained
441 below):
442 foo |
444 bar$ /* action goes here */
445 A similar trick will work for matching a foo or a bar-at-the-
447 beginning-of-a-line.
448
450 When the generated scanner is run, it analyzes its input looking for
451 strings which match any of its patterns. If it finds more than one
452 match, it takes the one matching the most text (for trailing context
453 rules, this includes the length of the trailing part, even though it
454 will then be returned to the input). If it finds two or more matches
455 of the same length, the rule listed first in the flex input file is
456 chosen.
457 Once the match is determined, the text corresponding to the match
459 (called the token) is made available in the global character pointer
460 yytext, and its length in the global integer yyleng. The action corre‐
461 sponding to the matched pattern is then executed (a more detailed
462 description of actions follows), and then the remaining input is
463 scanned for another match.
464 If no match is found, then the default rule is executed: the next char‐
466 acter in the input is considered matched and copied to the standard
467 output. Thus, the simplest legal flex input is:
468 %%
470 which generates a scanner that simply copies its input (one character
472 at a time) to its output.
473 Note that yytext can be defined in two different ways: either as a
475 character pointer or as a character array. You can control which defi‐
476 nition flex uses by including one of the special directives %pointer or
477 %array in the first (definitions) section of your flex input. The
478 default is %pointer, unless you use the -l lex compatibility option, in
479 which case yytext will be an array. The advantage of using %pointer is
480 substantially faster scanning and no buffer overflow when matching very
481 large tokens (unless you run out of dynamic memory). The disadvantage
482 is that you are restricted in how your actions can modify yytext (see
483 the next section), and calls to the unput() function destroys the
484 present contents of yytext, which can be a considerable porting
485 headache when moving between different lex versions.
486 The advantage of %array is that you can then modify yytext to your
488 heart's content, and calls to unput() do not destroy yytext (see
489 below). Furthermore, existing lex programs sometimes access yytext
490 externally using declarations of the form:
491 extern char yytext[];
492 This definition is erroneous when used with %pointer, but correct for
493 %array.
494 %array defines yytext to be an array of YYLMAX characters, which
496 defaults to a fairly large value. You can change the size by simply
497 #define'ing YYLMAX to a different value in the first section of your
498 flex input. As mentioned above, with %pointer yytext grows dynamically
499 to accommodate large tokens. While this means your %pointer scanner
500 can accommodate very large tokens (such as matching entire blocks of
501 comments), bear in mind that each time the scanner must resize yytext
502 it also must rescan the entire token from the beginning, so matching
503 such tokens can prove slow. yytext presently does not dynamically grow
504 if a call to unput() results in too much text being pushed back;
505 instead, a run-time error results.
506 Also note that you cannot use %array with C++ scanner classes (the c++
508 option; see below).
509
511 Each pattern in a rule has a corresponding action, which can be any
512 arbitrary C statement. The pattern ends at the first non-escaped
513 whitespace character; the remainder of the line is its action. If the
514 action is empty, then when the pattern is matched the input token is
515 simply discarded. For example, here is the specification for a program
516 which deletes all occurrences of "zap me" from its input:
517 %%
519 "zap me"
520 (It will copy all other characters in the input to the output since
522 they will be matched by the default rule.)
523 Here is a program which compresses multiple blanks and tabs down to a
525 single blank, and throws away whitespace found at the end of a line:
526 %%
528 [ \t]+ putchar( ' ' );
529 [ \t]+$ /* ignore this token */
530 If the action contains a '{', then the action spans till the balancing
533 '}' is found, and the action may cross multiple lines. flex knows
534 about C strings and comments and won't be fooled by braces found within
535 them, but also allows actions to begin with %{ and will consider the
536 action to be all the text up to the next %} (regardless of ordinary
537 braces inside the action).
538 An action consisting solely of a vertical bar ('|') means "same as the
540 action for the next rule." See below for an illustration.
541 Actions can include arbitrary C code, including return statements to
543 return a value to whatever routine called yylex(). Each time yylex()
544 is called it continues processing tokens from where it last left off
545 until it either reaches the end of the file or executes a return.
546 Actions are free to modify yytext except for lengthening it (adding
548 characters to its end--these will overwrite later characters in the
549 input stream). This however does not apply when using %array (see
550 above); in that case, yytext may be freely modified in any way.
551 Actions are free to modify yyleng except they should not do so if the
553 action also includes use of yymore() (see below).
554 There are a number of special directives which can be included within
556 an action:
557 - ECHO copies yytext to the scanner's output.
559 - BEGIN followed by the name of a start condition places the scan‐
561 ner in the corresponding start condition (see below).
562 - REJECT directs the scanner to proceed on to the "second best"
564 rule which matched the input (or a prefix of the input). The
565 rule is chosen as described above in "How the Input is Matched",
566 and yytext and yyleng set up appropriately. It may either be
567 one which matched as much text as the originally chosen rule but
568 came later in the flex input file, or one which matched less
569 text. For example, the following will both count the words in
570 the input and call the routine special() whenever "frob" is
571 seen:
572 int word_count = 0;
574 %%
575 frob special(); REJECT;
577 [^ \t\n]+ ++word_count;
578 Without the REJECT, any "frob"'s in the input would not be
580 counted as words, since the scanner normally executes only one
581 action per token. Multiple REJECT's are allowed, each one find‐
582 ing the next best choice to the currently active rule. For
583 example, when the following scanner scans the token "abcd", it
584 will write "abcdabcaba" to the output:
585 %%
587 a |
588 ab |
589 abc |
590 abcd ECHO; REJECT;
591 .|\n /* eat up any unmatched character */
592 (The first three rules share the fourth's action since they use
594 the special '|' action.) REJECT is a particularly expensive
595 feature in terms of scanner performance; if it is used in any of
596 the scanner's actions it will slow down all of the scanner's
597 matching. Furthermore, REJECT cannot be used with the -Cf or
598 -CF options (see below).
599 Note also that unlike the other special actions, REJECT is a
601 branch; code immediately following it in the action will not be
602 executed.
603 - yymore() tells the scanner that the next time it matches a rule,
605 the corresponding token should be appended onto the current
606 value of yytext rather than replacing it. For example, given
607 the input "mega-kludge" the following will write "mega-mega-
608 kludge" to the output:
609 %%
611 mega- ECHO; yymore();
612 kludge ECHO;
613 First "mega-" is matched and echoed to the output. Then
615 "kludge" is matched, but the previous "mega-" is still hanging
616 around at the beginning of yytext so the ECHO for the "kludge"
617 rule will actually write "mega-kludge".
618 Two notes regarding use of yymore(). First, yymore() depends on the
620 value of yyleng correctly reflecting the size of the current token, so
621 you must not modify yyleng if you are using yymore(). Second, the
622 presence of yymore() in the scanner's action entails a minor perfor‐
623 mance penalty in the scanner's matching speed.
624 - yyless(n) returns all but the first n characters of the current
626 token back to the input stream, where they will be rescanned
627 when the scanner looks for the next match. yytext and yyleng
628 are adjusted appropriately (e.g., yyleng will now be equal to n
629 ). For example, on the input "foobar" the following will write
630 out "foobarbar":
631 %%
633 foobar ECHO; yyless(3);
634 [a-z]+ ECHO;
635 An argument of 0 to yyless will cause the entire current input
637 string to be scanned again. Unless you've changed how the scan‐
638 ner will subsequently process its input (using BEGIN, for exam‐
639 ple), this will result in an endless loop.
640 Note that yyless is a macro and can only be used in the flex input
642 file, not from other source files.
643 - unput(c) puts the character c back onto the input stream. It
645 will be the next character scanned. The following action will
646 take the current token and cause it to be rescanned enclosed in
647 parentheses.
648 {
650 int i;
651 /* Copy yytext because unput() trashes yytext */
652 char *yycopy = strdup( yytext );
653 unput( ')' );
654 for ( i = yyleng - 1; i >= 0; --i )
655 unput( yycopy[i] );
656 unput( '(' );
657 free( yycopy );
658 }
659 Note that since each unput() puts the given character back at
661 the beginning of the input stream, pushing back strings must be
662 done back-to-front.
663 An important potential problem when using unput() is that if you are
665 using %pointer (the default), a call to unput() destroys the contents
666 of yytext, starting with its rightmost character and devouring one
667 character to the left with each call. If you need the value of yytext
668 preserved after a call to unput() (as in the above example), you must
669 either first copy it elsewhere, or build your scanner using %array
670 instead (see How The Input Is Matched).
671 Finally, note that you cannot put back EOF to attempt to mark the input
673 stream with an end-of-file.
674 - input() reads the next character from the input stream. For
676 example, the following is one way to eat up C comments:
677 %%
679 "/*" {
680 register int c;
681 for ( ; ; )
683 {
684 while ( (c = input()) != '*' &&
685 c != EOF )
686 ; /* eat up text of comment */
687 if ( c == '*' )
689 {
690 while ( (c = input()) == '*' )
691 ;
692 if ( c == '/' )
693 break; /* found the end */
694 }
695 if ( c == EOF )
697 {
698 error( "EOF in comment" );
699 break;
700 }
701 }
702 }
703 (Note that if the scanner is compiled using C++, then input() is
705 instead referred to as yyinput(), in order to avoid a name clash
706 with the C++ stream by the name of input.)
707 - YY_FLUSH_BUFFER flushes the scanner's internal buffer so that
709 the next time the scanner attempts to match a token, it will
710 first refill the buffer using YY_INPUT (see The Generated Scan‐
711 ner, below). This action is a special case of the more general
712 yy_flush_buffer() function, described below in the section Mul‐
713 tiple Input Buffers.
714 - yyterminate() can be used in lieu of a return statement in an
716 action. It terminates the scanner and returns a 0 to the scan‐
717 ner's caller, indicating "all done". By default, yyterminate()
718 is also called when an end-of-file is encountered. It is a
719 macro and may be redefined.
720
722 The output of flex is the file lex.yy.c, which contains the scanning
723 routine yylex(), a number of tables used by it for matching tokens, and
724 a number of auxiliary routines and macros. By default, yylex() is
725 declared as follows:
726 int yylex()
728 {
729 ... various definitions and the actions in here ...
730 }
731 (If your environment supports function prototypes, then it will be "int
733 yylex( void )".) This definition may be changed by defining the
734 "YY_DECL" macro. For example, you could use:
735 #define YY_DECL float lexscan( a, b ) float a, b;
737 to give the scanning routine the name lexscan, returning a float, and
739 taking two floats as arguments. Note that if you give arguments to the
740 scanning routine using a K&R-style/non-prototyped function declaration,
741 you must terminate the definition with a semi-colon (;).
742 Whenever yylex() is called, it scans tokens from the global input file
744 yyin (which defaults to stdin). It continues until it either reaches
745 an end-of-file (at which point it returns the value 0) or one of its
746 actions executes a return statement.
747 If the scanner reaches an end-of-file, subsequent calls are undefined
749 unless either yyin is pointed at a new input file (in which case scan‐
750 ning continues from that file), or yyrestart() is called. yyrestart()
751 takes one argument, a FILE * pointer (which can be nil, if you've set
752 up YY_INPUT to scan from a source other than yyin), and initializes
753 yyin for scanning from that file. Essentially there is no difference
754 between just assigning yyin to a new input file or using yyrestart() to
755 do so; the latter is available for compatibility with previous versions
756 of flex, and because it can be used to switch input files in the middle
757 of scanning. It can also be used to throw away the current input buf‐
758 fer, by calling it with an argument of yyin; but better is to use
759 YY_FLUSH_BUFFER (see above). Note that yyrestart() does not reset the
760 start condition to INITIAL (see Start Conditions, below).
761 If yylex() stops scanning due to executing a return statement in one of
763 the actions, the scanner may then be called again and it will resume
764 scanning where it left off.
765 By default (and for purposes of efficiency), the scanner uses block-
767 reads rather than simple getc() calls to read characters from yyin.
768 The nature of how it gets its input can be controlled by defining the
769 YY_INPUT macro. YY_INPUT's calling sequence is
770 "YY_INPUT(buf,result,max_size)". Its action is to place up to max_size
771 characters in the character array buf and return in the integer vari‐
772 able result either the number of characters read or the constant
773 YY_NULL (0 on Unix systems) to indicate EOF. The default YY_INPUT
774 reads from the global file-pointer "yyin".
775 A sample definition of YY_INPUT (in the definitions section of the
777 input file):
778 %{
780 #define YY_INPUT(buf,result,max_size) \
781 { \
782 int c = getchar(); \
783 result = (c == EOF) ? YY_NULL : (buf[0] = c, 1); \
784 }
785 %}
786 This definition will change the input processing to occur one character
788 at a time.
789 When the scanner receives an end-of-file indication from YY_INPUT, it
791 then checks the yywrap() function. If yywrap() returns false (zero),
792 then it is assumed that the function has gone ahead and set up yyin to
793 point to another input file, and scanning continues. If it returns
794 true (non-zero), then the scanner terminates, returning 0 to its call‐
795 er. Note that in either case, the start condition remains unchanged;
796 it does not revert to INITIAL.
797 If you do not supply your own version of yywrap(), then you must either
799 use %option noyywrap (in which case the scanner behaves as though
800 yywrap() returned 1), or you must link with -lfl to obtain the default
801 version of the routine, which always returns 1.
802 Three routines are available for scanning from in-memory buffers rather
804 than files: yy_scan_string(), yy_scan_bytes(), and yy_scan_buffer().
805 See the discussion of them below in the section Multiple Input Buffers.
806 The scanner writes its ECHO output to the yyout global (default, std‐
808 out), which may be redefined by the user simply by assigning it to some
809 other FILE pointer.
810
812 flex provides a mechanism for conditionally activating rules. Any rule
813 whose pattern is prefixed with "<sc>" will only be active when the
814 scanner is in the start condition named "sc". For example,
815 <STRING>[^"]* { /* eat up the string body ... */
817 ...
818 }
819 will be active only when the scanner is in the "STRING" start condi‐
821 tion, and
822 <INITIAL,STRING,QUOTE>\. { /* handle an escape ... */
824 ...
825 }
826 will be active only when the current start condition is either "INI‐
828 TIAL", "STRING", or "QUOTE".
829 Start conditions are declared in the definitions (first) section of the
831 input using unindented lines beginning with either %s or %x followed by
832 a list of names. The former declares inclusive start conditions, the
833 latter exclusive start conditions. A start condition is activated
834 using the BEGIN action. Until the next BEGIN action is executed, rules
835 with the given start condition will be active and rules with other
836 start conditions will be inactive. If the start condition is inclu‐
837 sive, then rules with no start conditions at all will also be active.
838 If it is exclusive, then only rules qualified with the start condition
839 will be active. A set of rules contingent on the same exclusive start
840 condition describe a scanner which is independent of any of the other
841 rules in the flex input. Because of this, exclusive start conditions
842 make it easy to specify "mini-scanners" which scan portions of the
843 input that are syntactically different from the rest (e.g., comments).
844 If the distinction between inclusive and exclusive start conditions is
846 still a little vague, here's a simple example illustrating the connec‐
847 tion between the two. The set of rules:
848 %s example
850 %%
851 <example>foo do_something();
853 bar something_else();
855 is equivalent to
857 %x example
859 %%
860 <example>foo do_something();
862 <INITIAL,example>bar something_else();
864 Without the <INITIAL,example> qualifier, the bar pattern in the second
866 example wouldn't be active (i.e., couldn't match) when in start condi‐
867 tion example. If we just used <example> to qualify bar, though, then
868 it would only be active in example and not in INITIAL, while in the
869 first example it's active in both, because in the first example the
870 example startion condition is an inclusive (%s) start condition.
871 Also note that the special start-condition specifier <*> matches every
873 start condition. Thus, the above example could also have been written;
874 %x example
876 %%
877 <example>foo do_something();
879 <*>bar something_else();
881 The default rule (to ECHO any unmatched character) remains active in
884 start conditions. It is equivalent to:
885 <*>.|\n ECHO;
887 BEGIN(0) returns to the original state where only the rules with no
890 start conditions are active. This state can also be referred to as the
891 start-condition "INITIAL", so BEGIN(INITIAL) is equivalent to BEGIN(0).
892 (The parentheses around the start condition name are not required but
893 are considered good style.)
894 BEGIN actions can also be given as indented code at the beginning of
896 the rules section. For example, the following will cause the scanner
897 to enter the "SPECIAL" start condition whenever yylex() is called and
898 the global variable enter_special is true:
899 int enter_special;
901 %x SPECIAL
903 %%
904 if ( enter_special )
905 BEGIN(SPECIAL);
906 <SPECIAL>blahblahblah
908 ...more rules follow...
909 To illustrate the uses of start conditions, here is a scanner which
912 provides two different interpretations of a string like "123.456". By
913 default it will treat it as three tokens, the integer "123", a dot
914 ('.'), and the integer "456". But if the string is preceded earlier in
915 the line by the string "expect-floats" it will treat it as a single
916 token, the floating-point number 123.456:
917 %{
919 #include <math.h>
920 %}
921 %s expect
922 %%
924 expect-floats BEGIN(expect);
925 <expect>[0-9]+"."[0-9]+ {
927 printf( "found a float, = %f\n",
928 atof( yytext ) );
929 }
930 <expect>\n {
931 /* that's the end of the line, so
932 * we need another "expect-number"
933 * before we'll recognize any more
934 * numbers
935 */
936 BEGIN(INITIAL);
937 }
938 [0-9]+ {
940 printf( "found an integer, = %d\n",
941 atoi( yytext ) );
942 }
943 "." printf( "found a dot\n" );
945 Here is a scanner which recognizes (and discards) C comments while
947 maintaining a count of the current input line.
948 %x comment
950 %%
951 int line_num = 1;
952 "/*" BEGIN(comment);
954 <comment>[^*\n]* /* eat anything that's not a '*' */
956 <comment>"*"+[^*/\n]* /* eat up '*'s not followed by '/'s */
957 <comment>\n ++line_num;
958 <comment>"*"+"/" BEGIN(INITIAL);
959 This scanner goes to a bit of trouble to match as much text as possible
961 with each rule. In general, when attempting to write a high-speed
962 scanner try to match as much possible in each rule, as it's a big win.
963 Note that start-conditions names are really integer values and can be
965 stored as such. Thus, the above could be extended in the following
966 fashion:
967 %x comment foo
969 %%
970 int line_num = 1;
971 int comment_caller;
972 "/*" {
974 comment_caller = INITIAL;
975 BEGIN(comment);
976 }
977 ...
979 <foo>"/*" {
981 comment_caller = foo;
982 BEGIN(comment);
983 }
984 <comment>[^*\n]* /* eat anything that's not a '*' */
986 <comment>"*"+[^*/\n]* /* eat up '*'s not followed by '/'s */
987 <comment>\n ++line_num;
988 <comment>"*"+"/" BEGIN(comment_caller);
989 Furthermore, you can access the current start condition using the inte‐
991 ger-valued YY_START macro. For example, the above assignments to com‐
992 ment_caller could instead be written
993 comment_caller = YY_START;
995 Flex provides YYSTATE as an alias for YY_START (since that is what's
997 used by AT&T lex).
998 Note that start conditions do not have their own name-space; %s's and
1000 %x's declare names in the same fashion as #define's.
1001 Finally, here's an example of how to match C-style quoted strings using
1003 exclusive start conditions, including expanded escape sequences (but
1004 not including checking for a string that's too long):
1005 %x str
1007 %%
1009 char string_buf[MAX_STR_CONST];
1010 char *string_buf_ptr;
1011 \" string_buf_ptr = string_buf; BEGIN(str);
1014 <str>\" { /* saw closing quote - all done */
1016 BEGIN(INITIAL);
1017 *string_buf_ptr = '\0';
1018 /* return string constant token type and
1019 * value to parser
1020 */
1021 }
1022 <str>\n {
1024 /* error - unterminated string constant */
1025 /* generate error message */
1026 }
1027 <str>\\[0-7]{1,3} {
1029 /* octal escape sequence */
1030 int result;
1031 (void) sscanf( yytext + 1, "%o", &result );
1033 if ( result > 0xff )
1035 /* error, constant is out-of-bounds */
1036 *string_buf_ptr++ = result;
1038 }
1039 <str>\\[0-9]+ {
1041 /* generate error - bad escape sequence; something
1042 * like '\48' or '\0777777'
1043 */
1044 }
1045 <str>\\n *string_buf_ptr++ = '\n';
1047 <str>\\t *string_buf_ptr++ = '\t';
1048 <str>\\r *string_buf_ptr++ = '\r';
1049 <str>\\b *string_buf_ptr++ = '\b';
1050 <str>\\f *string_buf_ptr++ = '\f';
1051 <str>\\(.|\n) *string_buf_ptr++ = yytext[1];
1053 <str>[^\\\n\"]+ {
1055 char *yptr = yytext;
1056 while ( *yptr )
1058 *string_buf_ptr++ = *yptr++;
1059 }
1060 Often, such as in some of the examples above, you wind up writing a
1063 whole bunch of rules all preceded by the same start condition(s). Flex
1064 makes this a little easier and cleaner by introducing a notion of start
1065 condition scope. A start condition scope is begun with:
1066 <SCs>{
1068 where SCs is a list of one or more start conditions. Inside the start
1070 condition scope, every rule automatically has the prefix <SCs> applied
1071 to it, until a '}' which matches the initial '{'. So, for example,
1072 <ESC>{
1074 "\\n" return '\n';
1075 "\\r" return '\r';
1076 "\\f" return '\f';
1077 "\\0" return '\0';
1078 }
1079 is equivalent to:
1081 <ESC>"\\n" return '\n';
1083 <ESC>"\\r" return '\r';
1084 <ESC>"\\f" return '\f';
1085 <ESC>"\\0" return '\0';
1086 Start condition scopes may be nested.
1088 Three routines are available for manipulating stacks of start condi‐
1090 tions:
1091 void yy_push_state(int new_state)
1093 pushes the current start condition onto the top of the start
1094 condition stack and switches to new_state as though you had used
1095 BEGIN new_state (recall that start condition names are also
1096 integers).
1097 void yy_pop_state()
1099 pops the top of the stack and switches to it via BEGIN.
1100 int yy_top_state()
1102 returns the top of the stack without altering the stack's con‐
1103 tents.
1104 The start condition stack grows dynamically and so has no built-in size
1106 limitation. If memory is exhausted, program execution aborts.
1107 To use start condition stacks, your scanner must include a %option
1109 stack directive (see Options below).
1110
1112 Some scanners (such as those which support "include" files) require
1113 reading from several input streams. As flex scanners do a large amount
1114 of buffering, one cannot control where the next input will be read from
1115 by simply writing a YY_INPUT which is sensitive to the scanning con‐
1116 text. YY_INPUT is only called when the scanner reaches the end of its
1117 buffer, which may be a long time after scanning a statement such as an
1118 "include" which requires switching the input source.
1119 To negotiate these sorts of problems, flex provides a mechanism for
1121 creating and switching between multiple input buffers. An input buffer
1122 is created by using:
1123 YY_BUFFER_STATE yy_create_buffer( FILE *file, int size )
1125 which takes a FILE pointer and a size and creates a buffer associated
1127 with the given file and large enough to hold size characters (when in
1128 doubt, use YY_BUF_SIZE for the size). It returns a YY_BUFFER_STATE
1129 handle, which may then be passed to other routines (see below). The
1130 YY_BUFFER_STATE type is a pointer to an opaque struct yy_buffer_state
1131 structure, so you may safely initialize YY_BUFFER_STATE variables to
1132 ((YY_BUFFER_STATE) 0) if you wish, and also refer to the opaque struc‐
1133 ture in order to correctly declare input buffers in source files other
1134 than that of your scanner. Note that the FILE pointer in the call to
1135 yy_create_buffer is only used as the value of yyin seen by YY_INPUT; if
1136 you redefine YY_INPUT so it no longer uses yyin, then you can safely
1137 pass a nil FILE pointer to yy_create_buffer. You select a particular
1138 buffer to scan from using:
1139 void yy_switch_to_buffer( YY_BUFFER_STATE new_buffer )
1141 switches the scanner's input buffer so subsequent tokens will come from
1143 new_buffer. Note that yy_switch_to_buffer() may be used by yywrap() to
1144 set things up for continued scanning, instead of opening a new file and
1145 pointing yyin at it. Note also that switching input sources via either
1146 yy_switch_to_buffer() or yywrap() does not change the start condition.
1147 void yy_delete_buffer( YY_BUFFER_STATE buffer )
1149 is used to reclaim the storage associated with a buffer. ( buffer can
1151 be nil, in which case the routine does nothing.) You can also clear
1152 the current contents of a buffer using:
1153 void yy_flush_buffer( YY_BUFFER_STATE buffer )
1155 This function discards the buffer's contents, so the next time the
1157 scanner attempts to match a token from the buffer, it will first fill
1158 the buffer anew using YY_INPUT.
1159 yy_new_buffer() is an alias for yy_create_buffer(), provided for com‐
1161 patibility with the C++ use of new and delete for creating and destroy‐
1162 ing dynamic objects.
1163 Finally, the YY_CURRENT_BUFFER macro returns a YY_BUFFER_STATE handle
1165 to the current buffer.
1166 Here is an example of using these features for writing a scanner which
1168 expands include files (the <<EOF>> feature is discussed below):
1169 /* the "incl" state is used for picking up the name
1171 * of an include file
1172 */
1173 %x incl
1174 %{
1176 #define MAX_INCLUDE_DEPTH 10
1177 YY_BUFFER_STATE include_stack[MAX_INCLUDE_DEPTH];
1178 int include_stack_ptr = 0;
1179 %}
1180 %%
1182 include BEGIN(incl);
1183 [a-z]+ ECHO;
1185 [^a-z\n]*\n? ECHO;
1186 <incl>[ \t]* /* eat the whitespace */
1188 <incl>[^ \t\n]+ { /* got the include file name */
1189 if ( include_stack_ptr >= MAX_INCLUDE_DEPTH )
1190 {
1191 fprintf( stderr, "Includes nested too deeply" );
1192 exit( 1 );
1193 }
1194 include_stack[include_stack_ptr++] =
1196 YY_CURRENT_BUFFER;
1197 yyin = fopen( yytext, "r" );
1199 if ( ! yyin )
1201 error( ... );
1202 yy_switch_to_buffer(
1204 yy_create_buffer( yyin, YY_BUF_SIZE ) );
1205 BEGIN(INITIAL);
1207 }
1208 <<EOF>> {
1210 if ( --include_stack_ptr < 0 )
1211 {
1212 yyterminate();
1213 }
1214 else
1216 {
1217 yy_delete_buffer( YY_CURRENT_BUFFER );
1218 yy_switch_to_buffer(
1219 include_stack[include_stack_ptr] );
1220 }
1221 }
1222 Three routines are available for setting up input buffers for scanning
1224 in-memory strings instead of files. All of them create a new input
1225 buffer for scanning the string, and return a corresponding YY_BUF‐
1226 FER_STATE handle (which you should delete with yy_delete_buffer() when
1227 done with it). They also switch to the new buffer using
1228 yy_switch_to_buffer(), so the next call to yylex() will start scanning
1229 the string.
1230 yy_scan_string(const char *str)
1232 scans a NUL-terminated string.
1233 yy_scan_bytes(const char *bytes, int len)
1235 scans len bytes (including possibly NUL's) starting at location
1236 bytes.
1237 Note that both of these functions create and scan a copy of the string
1239 or bytes. (This may be desirable, since yylex() modifies the contents
1240 of the buffer it is scanning.) You can avoid the copy by using:
1241 yy_scan_buffer(char *base, yy_size_t size)
1243 which scans in place the buffer starting at base, consisting of
1244 size bytes, the last two bytes of which must be YY_END_OF_BUF‐
1245 FER_CHAR (ASCII NUL). These last two bytes are not scanned;
1246 thus, scanning consists of base[0] through base[size-2], inclu‐
1247 sive.
1248 If you fail to set up base in this manner (i.e., forget the
1250 final two YY_END_OF_BUFFER_CHAR bytes), then yy_scan_buffer()
1251 returns a nil pointer instead of creating a new input buffer.
1252 The type yy_size_t is an integral type to which you can cast an
1254 integer expression reflecting the size of the buffer.
1255
1257 The special rule "<<EOF>>" indicates actions which are to be taken when
1258 an end-of-file is encountered and yywrap() returns non-zero (i.e.,
1259 indicates no further files to process). The action must finish by
1260 doing one of four things:
1261 - assigning yyin to a new input file (in previous versions of
1263 flex, after doing the assignment you had to call the special
1264 action YY_NEW_FILE; this is no longer necessary);
1265 - executing a return statement;
1267 - executing the special yyterminate() action;
1269 - or, switching to a new buffer using yy_switch_to_buffer() as
1271 shown in the example above.
1272 <<EOF>> rules may not be used with other patterns; they may only be
1274 qualified with a list of start conditions. If an unqualified <<EOF>>
1275 rule is given, it applies to all start conditions which do not already
1276 have <<EOF>> actions. To specify an <<EOF>> rule for only the initial
1277 start condition, use
1278 <INITIAL><<EOF>>
1280 These rules are useful for catching things like unclosed comments. An
1283 example:
1284 %x quote
1286 %%
1287 ...other rules for dealing with quotes...
1289 <quote><<EOF>> {
1291 error( "unterminated quote" );
1292 yyterminate();
1293 }
1294 <<EOF>> {
1295 if ( *++filelist )
1296 yyin = fopen( *filelist, "r" );
1297 else
1298 yyterminate();
1299 }
1300
1303 The macro YY_USER_ACTION can be defined to provide an action which is
1304 always executed prior to the matched rule's action. For example, it
1305 could be #define'd to call a routine to convert yytext to lower-case.
1306 When YY_USER_ACTION is invoked, the variable yy_act gives the number of
1307 the matched rule (rules are numbered starting with 1). Suppose you
1308 want to profile how often each of your rules is matched. The following
1309 would do the trick:
1310 #define YY_USER_ACTION ++ctr[yy_act]
1312 where ctr is an array to hold the counts for the different rules. Note
1314 that the macro YY_NUM_RULES gives the total number of rules (including
1315 the default rule, even if you use -s), so a correct declaration for ctr
1316 is:
1317 int ctr[YY_NUM_RULES];
1319 The macro YY_USER_INIT may be defined to provide an action which is
1322 always executed before the first scan (and before the scanner's inter‐
1323 nal initializations are done). For example, it could be used to call a
1324 routine to read in a data table or open a logging file.
1325 The macro yy_set_interactive(is_interactive) can be used to control
1327 whether the current buffer is considered interactive. An interactive
1328 buffer is processed more slowly, but must be used when the scanner's
1329 input source is indeed interactive to avoid problems due to waiting to
1330 fill buffers (see the discussion of the -I flag below). A non-zero
1331 value in the macro invocation marks the buffer as interactive, a zero
1332 value as non-interactive. Note that use of this macro overrides
1333 %option always-interactive or %option never-interactive (see Options
1334 below). yy_set_interactive() must be invoked prior to beginning to
1335 scan the buffer that is (or is not) to be considered interactive.
1336 The macro yy_set_bol(at_bol) can be used to control whether the current
1338 buffer's scanning context for the next token match is done as though at
1339 the beginning of a line. A non-zero macro argument makes rules
1340 anchored with
1341 The macro YY_AT_BOL() returns true if the next token scanned from the
1343 current buffer will have '^' rules active, false otherwise.
1344 In the generated scanner, the actions are all gathered in one large
1346 switch statement and separated using YY_BREAK, which may be redefined.
1347 By default, it is simply a "break", to separate each rule's action from
1348 the following rule's. Redefining YY_BREAK allows, for example, C++
1349 users to #define YY_BREAK to do nothing (while being very careful that
1350 every rule ends with a "break" or a "return"!) to avoid suffering from
1351 unreachable statement warnings where because a rule's action ends with
1352 "return", the YY_BREAK is inaccessible.
1353
1355 This section summarizes the various values available to the user in the
1356 rule actions.
1357 - char *yytext holds the text of the current token. It may be
1359 modified but not lengthened (you cannot append characters to the
1360 end).
1361 If the special directive %array appears in the first section of
1363 the scanner description, then yytext is instead declared char
1364 yytext[YYLMAX], where YYLMAX is a macro definition that you can
1365 redefine in the first section if you don't like the default
1366 value (generally 8KB). Using %array results in somewhat slower
1367 scanners, but the value of yytext becomes immune to calls to
1368 input() and unput(), which potentially destroy its value when
1369 yytext is a character pointer. The opposite of %array is
1370 %pointer, which is the default.
1371 You cannot use %array when generating C++ scanner classes (the
1373 -+ flag).
1374 - int yyleng holds the length of the current token.
1376 - FILE *yyin is the file which by default flex reads from. It may
1378 be redefined but doing so only makes sense before scanning
1379 begins or after an EOF has been encountered. Changing it in the
1380 midst of scanning will have unexpected results since flex buf‐
1381 fers its input; use yyrestart() instead. Once scanning termi‐
1382 nates because an end-of-file has been seen, you can assign yyin
1383 at the new input file and then call the scanner again to con‐
1384 tinue scanning.
1385 - void yyrestart( FILE *new_file ) may be called to point yyin at
1387 the new input file. The switch-over to the new file is immedi‐
1388 ate (any previously buffered-up input is lost). Note that call‐
1389 ing yyrestart() with yyin as an argument thus throws away the
1390 current input buffer and continues scanning the same input file.
1391 - FILE *yyout is the file to which ECHO actions are done. It can
1393 be reassigned by the user.
1394 - YY_CURRENT_BUFFER returns a YY_BUFFER_STATE handle to the cur‐
1396 rent buffer.
1397 - YY_START returns an integer value corresponding to the current
1399 start condition. You can subsequently use this value with BEGIN
1400 to return to that start condition.
1401
1403 One of the main uses of flex is as a companion to the yacc parser-gen‐
1404 erator. yacc parsers expect to call a routine named yylex() to find
1405 the next input token. The routine is supposed to return the type of
1406 the next token as well as putting any associated value in the global
1407 yylval. To use flex with yacc, one specifies the -d option to yacc to
1408 instruct it to generate the file y.tab.h containing definitions of all
1409 the %tokens appearing in the yacc input. This file is then included in
1410 the flex scanner. For example, if one of the tokens is "TOK_NUMBER",
1411 part of the scanner might look like:
1412 %{
1414 #include "y.tab.h"
1415 %}
1416 %%
1418 [0-9]+ yylval = atoi( yytext ); return TOK_NUMBER;
1420
1423 flex has the following options:
1424 -b Generate backing-up information to lex.backup. This is a list
1426 of scanner states which require backing up and the input charac‐
1427 ters on which they do so. By adding rules one can remove back‐
1428 ing-up states. If all backing-up states are eliminated and -Cf
1429 or -CF is used, the generated scanner will run faster (see the
1430 -p flag). Only users who wish to squeeze every last cycle out
1431 of their scanners need worry about this option. (See the sec‐
1432 tion on Performance Considerations below.)
1433 -c is a do-nothing, deprecated option included for POSIX compli‐
1435 ance.
1436 -d makes the generated scanner run in debug mode. Whenever a pat‐
1438 tern is recognized and the global yy_flex_debug is non-zero
1439 (which is the default), the scanner will write to stderr a line
1440 of the form:
1441 --accepting rule at line 53 ("the matched text")
1443 The line number refers to the location of the rule in the file
1445 defining the scanner (i.e., the file that was fed to flex).
1446 Messages are also generated when the scanner backs up, accepts
1447 the default rule, reaches the end of its input buffer (or
1448 encounters a NUL; at this point, the two look the same as far as
1449 the scanner's concerned), or reaches an end-of-file.
1450 -f specifies fast scanner. No table compression is done and stdio
1452 is bypassed. The result is large but fast. This option is
1453 equivalent to -Cfr (see below).
1454 -h generates a "help" summary of flex's options to stdout and then
1456 exits. -? and --help are synonyms for -h.
1457 -i instructs flex to generate a case-insensitive scanner. The case
1459 of letters given in the flex input patterns will be ignored, and
1460 tokens in the input will be matched regardless of case. The
1461 matched text given in yytext will have the preserved case (i.e.,
1462 it will not be folded).
1463 -l turns on maximum compatibility with the original AT&T lex imple‐
1465 mentation. Note that this does not mean full compatibility.
1466 Use of this option costs a considerable amount of performance,
1467 and it cannot be used with the -+, -f, -F, -Cf, or -CF options.
1468 For details on the compatibilities it provides, see the section
1469 "Incompatibilities With Lex And POSIX" below. This option also
1470 results in the name YY_FLEX_LEX_COMPAT being #define'd in the
1471 generated scanner.
1472 -n is another do-nothing, deprecated option included only for POSIX
1474 compliance.
1475 -p generates a performance report to stderr. The report consists
1477 of comments regarding features of the flex input file which will
1478 cause a serious loss of performance in the resulting scanner.
1479 If you give the flag twice, you will also get comments regarding
1480 features that lead to minor performance losses.
1481 Note that the use of REJECT, %option yylineno, and variable
1483 trailing context (see the Deficiencies / Bugs section below)
1484 entails a substantial performance penalty; use of yymore(), the
1485 ^ operator, and the -I flag entail minor performance penalties.
1486 -s causes the default rule (that unmatched scanner input is echoed
1488 to stdout) to be suppressed. If the scanner encounters input
1489 that does not match any of its rules, it aborts with an error.
1490 This option is useful for finding holes in a scanner's rule set.
1491 -t instructs flex to write the scanner it generates to standard
1493 output instead of lex.yy.c.
1494 -v specifies that flex should write to stderr a summary of statis‐
1496 tics regarding the scanner it generates. Most of the statistics
1497 are meaningless to the casual flex user, but the first line
1498 identifies the version of flex (same as reported by -V), and the
1499 next line the flags used when generating the scanner, including
1500 those that are on by default.
1501 -w suppresses warning messages.
1503 -B instructs flex to generate a batch scanner, the opposite of
1505 interactive scanners generated by -I (see below). In general,
1506 you use -B when you are certain that your scanner will never be
1507 used interactively, and you want to squeeze a little more per‐
1508 formance out of it. If your goal is instead to squeeze out a
1509 lot more performance, you should be using the -Cf or -CF
1510 options (discussed below), which turn on -B automatically any‐
1511 way.
1512 -F specifies that the fast scanner table representation should be
1514 used (and stdio bypassed). This representation is about as fast
1515 as the full table representation (-f), and for some sets of pat‐
1516 terns will be considerably smaller (and for others, larger). In
1517 general, if the pattern set contains both "keywords" and a
1518 catch-all, "identifier" rule, such as in the set:
1519 "case" return TOK_CASE;
1521 "switch" return TOK_SWITCH;
1522 ...
1523 "default" return TOK_DEFAULT;
1524 [a-z]+ return TOK_ID;
1525 then you're better off using the full table representation. If
1527 only the "identifier" rule is present and you then use a hash
1528 table or some such to detect the keywords, you're better off
1529 using -F.
1530 This option is equivalent to -CFr (see below). It cannot be
1532 used with -+.
1533 -I instructs flex to generate an interactive scanner. An interac‐
1535 tive scanner is one that only looks ahead to decide what token
1536 has been matched if it absolutely must. It turns out that
1537 always looking one extra character ahead, even if the scanner
1538 has already seen enough text to disambiguate the current token,
1539 is a bit faster than only looking ahead when necessary. But
1540 scanners that always look ahead give dreadful interactive per‐
1541 formance; for example, when a user types a newline, it is not
1542 recognized as a newline token until they enter another token,
1543 which often means typing in another whole line.
1544 Flex scanners default to interactive unless you use the -Cf or
1546 -CF table-compression options (see below). That's because if
1547 you're looking for high-performance you should be using one of
1548 these options, so if you didn't, flex assumes you'd rather trade
1549 off a bit of run-time performance for intuitive interactive
1550 behavior. Note also that you cannot use -I in conjunction with
1551 -Cf or -CF. Thus, this option is not really needed; it is on by
1552 default for all those cases in which it is allowed.
1553 You can force a scanner to not be interactive by using -B (see
1555 above).
1556 -L instructs flex not to generate #line directives. Without this
1558 option, flex peppers the generated scanner with #line directives
1559 so error messages in the actions will be correctly located with
1560 respect to either the original flex input file (if the errors
1561 are due to code in the input file), or lex.yy.c (if the errors
1562 are flex's fault -- you should report these sorts of errors to
1563 the email address given below).
1564 -T makes flex run in trace mode. It will generate a lot of mes‐
1566 sages to stderr concerning the form of the input and the resul‐
1567 tant non-deterministic and deterministic finite automata. This
1568 option is mostly for use in maintaining flex.
1569 -V prints the version number to stdout and exits. --version is a
1571 synonym for -V.
1572 -7 instructs flex to generate a 7-bit scanner, i.e., one which can
1574 only recognized 7-bit characters in its input. The advantage of
1575 using -7 is that the scanner's tables can be up to half the size
1576 of those generated using the -8 option (see below). The disad‐
1577 vantage is that such scanners often hang or crash if their input
1578 contains an 8-bit character.
1579 Note, however, that unless you generate your scanner using the
1581 -Cf or -CF table compression options, use of -7 will save only a
1582 small amount of table space, and make your scanner considerably
1583 less portable. Flex's default behavior is to generate an 8-bit
1584 scanner unless you use the -Cf or -CF, in which case flex
1585 defaults to generating 7-bit scanners unless your site was
1586 always configured to generate 8-bit scanners (as will often be
1587 the case with non-USA sites). You can tell whether flex gener‐
1588 ated a 7-bit or an 8-bit scanner by inspecting the flag summary
1589 in the -v output as described above.
1590 Note that if you use -Cfe or -CFe (those table compression
1592 options, but also using equivalence classes as discussed see
1593 below), flex still defaults to generating an 8-bit scanner,
1594 since usually with these compression options full 8-bit tables
1595 are not much more expensive than 7-bit tables.
1596 -8 instructs flex to generate an 8-bit scanner, i.e., one which can
1598 recognize 8-bit characters. This flag is only needed for scan‐
1599 ners generated using -Cf or -CF, as otherwise flex defaults to
1600 generating an 8-bit scanner anyway.
1601 See the discussion of -7 above for flex's default behavior and
1603 the tradeoffs between 7-bit and 8-bit scanners.
1604 -+ specifies that you want flex to generate a C++ scanner class.
1606 See the section on Generating C++ Scanners below for details.
1607 -C[aefFmr]
1609 controls the degree of table compression and, more generally,
1610 trade-offs between small scanners and fast scanners.
1611 -Ca ("align") instructs flex to trade off larger tables in the
1613 generated scanner for faster performance because the elements of
1614 the tables are better aligned for memory access and computation.
1615 On some RISC architectures, fetching and manipulating longwords
1616 is more efficient than with smaller-sized units such as short‐
1617 words. This option can double the size of the tables used by
1618 your scanner.
1619 -Ce directs flex to construct equivalence classes, i.e., sets of
1621 characters which have identical lexical properties (for example,
1622 if the only appearance of digits in the flex input is in the
1623 character class "[0-9]" then the digits '0', '1', ..., '9' will
1624 all be put in the same equivalence class). Equivalence classes
1625 usually give dramatic reductions in the final table/object file
1626 sizes (typically a factor of 2-5) and are pretty cheap perfor‐
1627 mance-wise (one array look-up per character scanned).
1628 -Cf specifies that the full scanner tables should be generated -
1630 flex should not compress the tables by taking advantages of sim‐
1631 ilar transition functions for different states.
1632 -CF specifies that the alternate fast scanner representation
1634 (described above under the -F flag) should be used. This option
1635 cannot be used with -+.
1636 -Cm directs flex to construct meta-equivalence classes, which
1638 are sets of equivalence classes (or characters, if equivalence
1639 classes are not being used) that are commonly used together.
1640 Meta-equivalence classes are often a big win when using com‐
1641 pressed tables, but they have a moderate performance impact (one
1642 or two "if" tests and one array look-up per character scanned).
1643 -Cr causes the generated scanner to bypass use of the standard
1645 I/O library (stdio) for input. Instead of calling fread() or
1646 getc(), the scanner will use the read() system call, resulting
1647 in a performance gain which varies from system to system, but in
1648 general is probably negligible unless you are also using -Cf or
1649 -CF. Using -Cr can cause strange behavior if, for example, you
1650 read from yyin using stdio prior to calling the scanner (because
1651 the scanner will miss whatever text your previous reads left in
1652 the stdio input buffer).
1653 -Cr has no effect if you define YY_INPUT (see The Generated
1655 Scanner above).
1656 A lone -C specifies that the scanner tables should be compressed
1658 but neither equivalence classes nor meta-equivalence classes
1659 should be used.
1660 The options -Cf or -CF and -Cm do not make sense together -
1662 there is no opportunity for meta-equivalence classes if the ta‐
1663 ble is not being compressed. Otherwise the options may be
1664 freely mixed, and are cumulative.
1665 The default setting is -Cem, which specifies that flex should
1667 generate equivalence classes and meta-equivalence classes. This
1668 setting provides the highest degree of table compression. You
1669 can trade off faster-executing scanners at the cost of larger
1670 tables with the following generally being true:
1671 slowest & smallest
1673 -Cem
1674 -Cm
1675 -Ce
1676 -C
1677 -C{f,F}e
1678 -C{f,F}
1679 -C{f,F}a
1680 fastest & largest
1681 Note that scanners with the smallest tables are usually gener‐
1683 ated and compiled the quickest, so during development you will
1684 usually want to use the default, maximal compression.
1685 -Cfe is often a good compromise between speed and size for pro‐
1687 duction scanners.
1688 -ooutput
1690 directs flex to write the scanner to the file output instead of
1691 lex.yy.c. If you combine -o with the -t option, then the scan‐
1692 ner is written to stdout but its #line directives (see the -L
1693 option above) refer to the file output.
1694 -Pprefix
1696 changes the default yy prefix used by flex for all globally-vis‐
1697 ible variable and function names to instead be prefix. For
1698 example, -Pfoo changes the name of yytext to footext. It also
1699 changes the name of the default output file from lex.yy.c to
1700 lex.foo.c. Here are all of the names affected:
1701 yy_create_buffer
1703 yy_delete_buffer
1704 yy_flex_debug
1705 yy_init_buffer
1706 yy_flush_buffer
1707 yy_load_buffer_state
1708 yy_switch_to_buffer
1709 yyin
1710 yyleng
1711 yylex
1712 yylineno
1713 yyout
1714 yyrestart
1715 yytext
1716 yywrap
1717 (If you are using a C++ scanner, then only yywrap and
1719 yyFlexLexer are affected.) Within your scanner itself, you can
1720 still refer to the global variables and functions using either
1721 version of their name; but externally, they have the modified
1722 name.
1723 This option lets you easily link together multiple flex programs
1725 into the same executable. Note, though, that using this option
1726 also renames yywrap(), so you now must either provide your own
1727 (appropriately-named) version of the routine for your scanner,
1728 or use %option noyywrap, as linking with -lfl no longer provides
1729 one for you by default.
1730 -Sskeleton_file
1732 overrides the default skeleton file from which flex constructs
1733 its scanners. You'll never need this option unless you are
1734 doing flex maintenance or development.
1735 flex also provides a mechanism for controlling options within the scan‐
1737 ner specification itself, rather than from the flex command-line. This
1738 is done by including %option directives in the first section of the
1739 scanner specification. You can specify multiple options with a single
1740 %option directive, and multiple directives in the first section of your
1741 flex input file.
1742 Most options are given simply as names, optionally preceded by the word
1744 "no" (with no intervening whitespace) to negate their meaning. A num‐
1745 ber are equivalent to flex flags or their negation:
1746 7bit -7 option
1748 8bit -8 option
1749 align -Ca option
1750 backup -b option
1751 batch -B option
1752 c++ -+ option
1753 caseful or
1755 case-sensitive opposite of -i (default)
1756 case-insensitive or
1758 caseless -i option
1759 debug -d option
1761 default opposite of -s option
1762 ecs -Ce option
1763 fast -F option
1764 full -f option
1765 interactive -I option
1766 lex-compat -l option
1767 meta-ecs -Cm option
1768 perf-report -p option
1769 read -Cr option
1770 stdout -t option
1771 verbose -v option
1772 warn opposite of -w option
1773 (use "%option nowarn" for -w)
1774 array equivalent to "%array"
1776 pointer equivalent to "%pointer" (default)
1777 Some %option's provide features otherwise not available:
1779 always-interactive
1781 instructs flex to generate a scanner which always considers its
1782 input "interactive". Normally, on each new input file the scan‐
1783 ner calls isatty() in an attempt to determine whether the scan‐
1784 ner's input source is interactive and thus should be read a
1785 character at a time. When this option is used, however, then no
1786 such call is made.
1787 main directs flex to provide a default main() program for the scan‐
1789 ner, which simply calls yylex(). This option implies noyywrap
1790 (see below).
1791 never-interactive
1793 instructs flex to generate a scanner which never considers its
1794 input "interactive" (again, no call made to isatty()). This is
1795 the opposite of always-interactive.
1796 stack enables the use of start condition stacks (see Start Conditions
1798 above).
1799 stdinit
1801 if set (i.e., %option stdinit) initializes yyin and yyout to
1802 stdin and stdout, instead of the default of nil. Some existing
1803 lex programs depend on this behavior, even though it is not com‐
1804 pliant with ANSI C, which does not require stdin and stdout to
1805 be compile-time constant.
1806 yylineno
1808 directs flex to generate a scanner that maintains the number of
1809 the current line read from its input in the global variable
1810 yylineno. This option is implied by %option lex-compat.
1811 yywrap if unset (i.e., %option noyywrap), makes the scanner not call
1813 yywrap() upon an end-of-file, but simply assume that there are
1814 no more files to scan (until the user points yyin at a new file
1815 and calls yylex() again).
1816 flex scans your rule actions to determine whether you use the REJECT or
1818 yymore() features. The reject and yymore options are available to
1819 override its decision as to whether you use the options, either by set‐
1820 ting them (e.g., %option reject) to indicate the feature is indeed
1821 used, or unsetting them to indicate it actually is not used (e.g.,
1822 %option noyymore).
1823 Three options take string-delimited values, offset with '=':
1825 %option outfile="ABC"
1827 is equivalent to -oABC, and
1829 %option prefix="XYZ"
1831 is equivalent to -PXYZ. Finally,
1833 %option yyclass="foo"
1835 only applies when generating a C++ scanner ( -+ option). It informs
1837 flex that you have derived foo as a subclass of yyFlexLexer, so flex
1838 will place your actions in the member function foo::yylex() instead of
1839 yyFlexLexer::yylex(). It also generates a yyFlexLexer::yylex() member
1840 function that emits a run-time error (by invoking yyFlexLexer::Lexer‐
1841 Error()) if called. See Generating C++ Scanners, below, for additional
1842 information.
1843 A number of options are available for lint purists who want to suppress
1845 the appearance of unneeded routines in the generated scanner. Each of
1846 the following, if unset (e.g., %option nounput ), results in the corre‐
1847 sponding routine not appearing in the generated scanner:
1848 input, unput
1850 yy_push_state, yy_pop_state, yy_top_state
1851 yy_scan_buffer, yy_scan_bytes, yy_scan_string
1852 (though yy_push_state() and friends won't appear anyway unless you use
1854 %option stack).
1855
1857 The main design goal of flex is that it generate high-performance scan‐
1858 ners. It has been optimized for dealing well with large sets of rules.
1859 Aside from the effects on scanner speed of the table compression -C
1860 options outlined above, there are a number of options/actions which
1861 degrade performance. These are, from most expensive to least:
1862 REJECT
1864 %option yylineno
1865 arbitrary trailing context
1866 pattern sets that require backing up
1868 %array
1869 %option interactive
1870 %option always-interactive
1871 '^' beginning-of-line operator
1873 yymore()
1874 with the first three all being quite expensive and the last two being
1876 quite cheap. Note also that unput() is implemented as a routine call
1877 that potentially does quite a bit of work, while yyless() is a quite-
1878 cheap macro; so if just putting back some excess text you scanned, use
1879 yyless().
1880 REJECT should be avoided at all costs when performance is important.
1882 It is a particularly expensive option.
1883 Getting rid of backing up is messy and often may be an enormous amount
1885 of work for a complicated scanner. In principal, one begins by using
1886 the -b flag to generate a lex.backup file. For example, on the input
1887 %%
1889 foo return TOK_KEYWORD;
1890 foobar return TOK_KEYWORD;
1891 the file looks like:
1893 State #6 is non-accepting -
1895 associated rule line numbers:
1896 2 3
1897 out-transitions: [ o ]
1898 jam-transitions: EOF [ \001-n p-\177 ]
1899 State #8 is non-accepting -
1901 associated rule line numbers:
1902 3
1903 out-transitions: [ a ]
1904 jam-transitions: EOF [ \001-` b-\177 ]
1905 State #9 is non-accepting -
1907 associated rule line numbers:
1908 3
1909 out-transitions: [ r ]
1910 jam-transitions: EOF [ \001-q s-\177 ]
1911 Compressed tables always back up.
1913 The first few lines tell us that there's a scanner state in which it
1915 can make a transition on an 'o' but not on any other character, and
1916 that in that state the currently scanned text does not match any rule.
1917 The state occurs when trying to match the rules found at lines 2 and 3
1918 in the input file. If the scanner is in that state and then reads
1919 something other than an 'o', it will have to back up to find a rule
1920 which is matched. With a bit of headscratching one can see that this
1921 must be the state it's in when it has seen "fo". When this has hap‐
1922 pened, if anything other than another 'o' is seen, the scanner will
1923 have to back up to simply match the 'f' (by the default rule).
1924 The comment regarding State #8 indicates there's a problem when "foob"
1926 has been scanned. Indeed, on any character other than an 'a', the
1927 scanner will have to back up to accept "foo". Similarly, the comment
1928 for State #9 concerns when "fooba" has been scanned and an 'r' does not
1929 follow.
1930 The final comment reminds us that there's no point going to all the
1932 trouble of removing backing up from the rules unless we're using -Cf or
1933 -CF, since there's no performance gain doing so with compressed scan‐
1934 ners.
1935 The way to remove the backing up is to add "error" rules:
1937 %%
1939 foo return TOK_KEYWORD;
1940 foobar return TOK_KEYWORD;
1941 fooba |
1943 foob |
1944 fo {
1945 /* false alarm, not really a keyword */
1946 return TOK_ID;
1947 }
1948 Eliminating backing up among a list of keywords can also be done using
1951 a "catch-all" rule:
1952 %%
1954 foo return TOK_KEYWORD;
1955 foobar return TOK_KEYWORD;
1956 [a-z]+ return TOK_ID;
1958 This is usually the best solution when appropriate.
1960 Backing up messages tend to cascade. With a complicated set of rules
1962 it's not uncommon to get hundreds of messages. If one can decipher
1963 them, though, it often only takes a dozen or so rules to eliminate the
1964 backing up (though it's easy to make a mistake and have an error rule
1965 accidentally match a valid token. A possible future flex feature will
1966 be to automatically add rules to eliminate backing up).
1967 It's important to keep in mind that you gain the benefits of eliminat‐
1969 ing backing up only if you eliminate every instance of backing up.
1970 Leaving just one means you gain nothing.
1971 Variable trailing context (where both the leading and trailing parts do
1973 not have a fixed length) entails almost the same performance loss as
1974 REJECT (i.e., substantial). So when possible a rule like:
1975 %%
1977 mouse|rat/(cat|dog) run();
1978 is better written:
1980 %%
1982 mouse/cat|dog run();
1983 rat/cat|dog run();
1984 or as
1986 %%
1988 mouse|rat/cat run();
1989 mouse|rat/dog run();
1990 Note that here the special '|' action does not provide any savings, and
1992 can even make things worse (see Deficiencies / Bugs below).
1993 Another area where the user can increase a scanner's performance (and
1995 one that's easier to implement) arises from the fact that the longer
1996 the tokens matched, the faster the scanner will run. This is because
1997 with long tokens the processing of most input characters takes place in
1998 the (short) inner scanning loop, and does not often have to go through
1999 the additional work of setting up the scanning environment (e.g.,
2000 yytext) for the action. Recall the scanner for C comments:
2001 %x comment
2003 %%
2004 int line_num = 1;
2005 "/*" BEGIN(comment);
2007 <comment>[^*\n]*
2009 <comment>"*"+[^*/\n]*
2010 <comment>\n ++line_num;
2011 <comment>"*"+"/" BEGIN(INITIAL);
2012 This could be sped up by writing it as:
2014 %x comment
2016 %%
2017 int line_num = 1;
2018 "/*" BEGIN(comment);
2020 <comment>[^*\n]*
2022 <comment>[^*\n]*\n ++line_num;
2023 <comment>"*"+[^*/\n]*
2024 <comment>"*"+[^*/\n]*\n ++line_num;
2025 <comment>"*"+"/" BEGIN(INITIAL);
2026 Now instead of each newline requiring the processing of another action,
2028 recognizing the newlines is "distributed" over the other rules to keep
2029 the matched text as long as possible. Note that adding rules does not
2030 slow down the scanner! The speed of the scanner is independent of the
2031 number of rules or (modulo the considerations given at the beginning of
2032 this section) how complicated the rules are with regard to operators
2033 such as '*' and '|'.
2034 A final example in speeding up a scanner: suppose you want to scan
2036 through a file containing identifiers and keywords, one per line and
2037 with no other extraneous characters, and recognize all the keywords. A
2038 natural first approach is:
2039 %%
2041 asm |
2042 auto |
2043 break |
2044 ... etc ...
2045 volatile |
2046 while /* it's a keyword */
2047 .|\n /* it's not a keyword */
2049 To eliminate the back-tracking, introduce a catch-all rule:
2051 %%
2053 asm |
2054 auto |
2055 break |
2056 ... etc ...
2057 volatile |
2058 while /* it's a keyword */
2059 [a-z]+ |
2061 .|\n /* it's not a keyword */
2062 Now, if it's guaranteed that there's exactly one word per line, then we
2064 can reduce the total number of matches by a half by merging in the
2065 recognition of newlines with that of the other tokens:
2066 %%
2068 asm\n |
2069 auto\n |
2070 break\n |
2071 ... etc ...
2072 volatile\n |
2073 while\n /* it's a keyword */
2074 [a-z]+\n |
2076 .|\n /* it's not a keyword */
2077 One has to be careful here, as we have now reintroduced backing up into
2079 the scanner. In particular, while we know that there will never be any
2080 characters in the input stream other than letters or newlines, flex
2081 can't figure this out, and it will plan for possibly needing to back up
2082 when it has scanned a token like "auto" and then the next character is
2083 something other than a newline or a letter. Previously it would then
2084 just match the "auto" rule and be done, but now it has no "auto" rule,
2085 only a "auto\n" rule. To eliminate the possibility of backing up, we
2086 could either duplicate all rules but without final newlines, or, since
2087 we never expect to encounter such an input and therefore don't how it's
2088 classified, we can introduce one more catch-all rule, this one which
2089 doesn't include a newline:
2090 %%
2092 asm\n |
2093 auto\n |
2094 break\n |
2095 ... etc ...
2096 volatile\n |
2097 while\n /* it's a keyword */
2098 [a-z]+\n |
2100 [a-z]+ |
2101 .|\n /* it's not a keyword */
2102 Compiled with -Cf, this is about as fast as one can get a flex scanner
2104 to go for this particular problem.
2105 A final note: flex is slow when matching NUL's, particularly when a
2107 token contains multiple NUL's. It's best to write rules which match
2108 short amounts of text if it's anticipated that the text will often
2109 include NUL's.
2110 Another final note regarding performance: as mentioned above in the
2112 section How the Input is Matched, dynamically resizing yytext to accom‐
2113 modate huge tokens is a slow process because it presently requires that
2114 the (huge) token be rescanned from the beginning. Thus if performance
2115 is vital, you should attempt to match "large" quantities of text but
2116 not "huge" quantities, where the cutoff between the two is at about 8K
2117 characters/token.
2118
2120 flex provides two different ways to generate scanners for use with C++.
2121 The first way is to simply compile a scanner generated by flex using a
2122 C++ compiler instead of a C compiler. You should not encounter any
2123 compilations errors (please report any you find to the email address
2124 given in the Author section below). You can then use C++ code in your
2125 rule actions instead of C code. Note that the default input source for
2126 your scanner remains yyin, and default echoing is still done to yyout.
2127 Both of these remain FILE * variables and not C++ streams.
2128 You can also use flex to generate a C++ scanner class, using the -+
2130 option (or, equivalently, %option c++), which is automatically speci‐
2131 fied if the name of the flex executable ends in a '+', such as flex++.
2132 When using this option, flex defaults to generating the scanner to the
2133 file lex.yy.cc instead of lex.yy.c. The generated scanner includes the
2134 header file FlexLexer.h, which defines the interface to two C++
2135 classes.
2136 The first class, FlexLexer, provides an abstract base class defining
2138 the general scanner class interface. It provides the following member
2139 functions:
2140 const char* YYText()
2142 returns the text of the most recently matched token, the equiva‐
2143 lent of yytext.
2144 int YYLeng()
2146 returns the length of the most recently matched token, the
2147 equivalent of yyleng.
2148 int lineno() const
2150 returns the current input line number (see %option yylineno), or
2151 1 if %option yylineno was not used.
2152 void set_debug( int flag )
2154 sets the debugging flag for the scanner, equivalent to assigning
2155 to yy_flex_debug (see the Options section above). Note that you
2156 must build the scanner using %option debug to include debugging
2157 information in it.
2158 int debug() const
2160 returns the current setting of the debugging flag.
2161 Also provided are member functions equivalent to yy_switch_to_buffer(),
2163 yy_create_buffer() (though the first argument is an istream* object
2164 pointer and not a FILE*), yy_flush_buffer(), yy_delete_buffer(), and
2165 yyrestart() (again, the first argument is a istream* object pointer).
2166 The second class defined in FlexLexer.h is yyFlexLexer, which is
2168 derived from FlexLexer. It defines the following additional member
2169 functions:
2170 yyFlexLexer( istream* arg_yyin = 0, ostream* arg_yyout = 0 )
2172 constructs a yyFlexLexer object using the given streams for
2173 input and output. If not specified, the streams default to cin
2174 and cout, respectively.
2175 virtual int yylex()
2177 performs the same role is yylex() does for ordinary flex scan‐
2178 ners: it scans the input stream, consuming tokens, until a
2179 rule's action returns a value. If you derive a subclass S from
2180 yyFlexLexer and want to access the member functions and vari‐
2181 ables of S inside yylex(), then you need to use %option
2182 yyclass="S" to inform flex that you will be using that subclass
2183 instead of yyFlexLexer. In this case, rather than generating
2184 yyFlexLexer::yylex(), flex generates S::yylex() (and also gener‐
2185 ates a dummy yyFlexLexer::yylex() that calls yyFlexLexer::Lexer‐
2186 Error() if called).
2187 virtual void switch_streams(istream* new_in = 0,
2189 ostream* new_out = 0) reassigns yyin to new_in (if non-nil) and
2190 yyout to new_out (ditto), deleting the previous input buffer if
2191 yyin is reassigned.
2192 int yylex( istream* new_in, ostream* new_out = 0 )
2194 first switches the input streams via switch_streams( new_in,
2195 new_out ) and then returns the value of yylex().
2196 In addition, yyFlexLexer defines the following protected virtual func‐
2198 tions which you can redefine in derived classes to tailor the scanner:
2199 virtual int LexerInput( char* buf, int max_size )
2201 reads up to max_size characters into buf and returns the number
2202 of characters read. To indicate end-of-input, return 0 charac‐
2203 ters. Note that "interactive" scanners (see the -B and -I
2204 flags) define the macro YY_INTERACTIVE. If you redefine Lex‐
2205 erInput() and need to take different actions depending on
2206 whether or not the scanner might be scanning an interactive
2207 input source, you can test for the presence of this name via
2208 #ifdef.
2209 virtual void LexerOutput( const char* buf, int size )
2211 writes out size characters from the buffer buf, which, while
2212 NUL-terminated, may also contain "internal" NUL's if the scan‐
2213 ner's rules can match text with NUL's in them.
2214 virtual void LexerError( const char* msg )
2216 reports a fatal error message. The default version of this
2217 function writes the message to the stream cerr and exits.
2218 Note that a yyFlexLexer object contains its entire scanning state.
2220 Thus you can use such objects to create reentrant scanners. You can
2221 instantiate multiple instances of the same yyFlexLexer class, and you
2222 can also combine multiple C++ scanner classes together in the same pro‐
2223 gram using the -P option discussed above.
2224 Finally, note that the %array feature is not available to C++ scanner
2226 classes; you must use %pointer (the default).
2227 Here is an example of a simple C++ scanner:
2229 // An example of using the flex C++ scanner class.
2231 %{
2233 int mylineno = 0;
2234 %}
2235 string \"[^\n"]+\"
2237 ws [ \t]+
2239 alpha [A-Za-z]
2241 dig [0-9]
2242 name ({alpha}|{dig}|\$)({alpha}|{dig}|[_.\-/$])*
2243 num1 [-+]?{dig}+\.?([eE][-+]?{dig}+)?
2244 num2 [-+]?{dig}*\.{dig}+([eE][-+]?{dig}+)?
2245 number {num1}|{num2}
2246 %%
2248 {ws} /* skip blanks and tabs */
2250 "/*" {
2252 int c;
2253 while((c = yyinput()) != 0)
2255 {
2256 if(c == '\n')
2257 ++mylineno;
2258 else if(c == '*')
2260 {
2261 if((c = yyinput()) == '/')
2262 break;
2263 else
2264 unput(c);
2265 }
2266 }
2267 }
2268 {number} cout << "number " << YYText() << '\n';
2270 \n mylineno++;
2272 {name} cout << "name " << YYText() << '\n';
2274 {string} cout << "string " << YYText() << '\n';
2276 %%
2278 int main( int /* argc */, char** /* argv */ )
2280 {
2281 FlexLexer* lexer = new yyFlexLexer;
2282 while(lexer->yylex() != 0)
2283 ;
2284 return 0;
2285 }
2286 If you want to create multiple (different) lexer classes, you use the
2287 -P flag (or the prefix= option) to rename each yyFlexLexer to some
2288 other xxFlexLexer. You then can include <FlexLexer.h> in your other
2289 sources once per lexer class, first renaming yyFlexLexer as follows:
2290 #undef yyFlexLexer
2292 #define yyFlexLexer xxFlexLexer
2293 #include <FlexLexer.h>
2294 #undef yyFlexLexer
2296 #define yyFlexLexer zzFlexLexer
2297 #include <FlexLexer.h>
2298 if, for example, you used %option prefix="xx" for one of your scanners
2300 and %option prefix="zz" for the other.
2301 IMPORTANT: the present form of the scanning class is experimental and
2303 may change considerably between major releases.
2304
2306 flex is a rewrite of the AT&T Unix lex tool (the two implementations do
2307 not share any code, though), with some extensions and incompatibili‐
2308 ties, both of which are of concern to those who wish to write scanners
2309 acceptable to either implementation. Flex is fully compliant with the
2310 POSIX lex specification, except that when using %pointer (the default),
2311 a call to unput() destroys the contents of yytext, which is counter to
2312 the POSIX specification.
2313 In this section we discuss all of the known areas of incompatibility
2315 between flex, AT&T lex, and the POSIX specification.
2316 flex's -l option turns on maximum compatibility with the original AT&T
2318 lex implementation, at the cost of a major loss in the generated scan‐
2319 ner's performance. We note below which incompatibilities can be over‐
2320 come using the -l option.
2321 flex is fully compatible with lex with the following exceptions:
2323 - The undocumented lex scanner internal variable yylineno is not
2325 supported unless -l or %option yylineno is used.
2326 yylineno should be maintained on a per-buffer basis, rather than
2328 a per-scanner (single global variable) basis.
2329 yylineno is not part of the POSIX specification.
2331 - The input() routine is not redefinable, though it may be called
2333 to read characters following whatever has been matched by a
2334 rule. If input() encounters an end-of-file the normal yywrap()
2335 processing is done. A ``real'' end-of-file is returned by
2336 input() as EOF.
2337 Input is instead controlled by defining the YY_INPUT macro.
2339 The flex restriction that input() cannot be redefined is in
2341 accordance with the POSIX specification, which simply does not
2342 specify any way of controlling the scanner's input other than by
2343 making an initial assignment to yyin.
2344 - The unput() routine is not redefinable. This restriction is in
2346 accordance with POSIX.
2347 - flex scanners are not as reentrant as lex scanners. In particu‐
2349 lar, if you have an interactive scanner and an interrupt handler
2350 which long-jumps out of the scanner, and the scanner is subse‐
2351 quently called again, you may get the following message:
2352 fatal flex scanner internal error--end of buffer missed
2354 To reenter the scanner, first use
2356 yyrestart( yyin );
2358 Note that this call will throw away any buffered input; usually
2360 this isn't a problem with an interactive scanner.
2361 Also note that flex C++ scanner classes are reentrant, so if
2363 using C++ is an option for you, you should use them instead.
2364 See "Generating C++ Scanners" above for details.
2365 - output() is not supported. Output from the ECHO macro is done
2367 to the file-pointer yyout (default stdout).
2368 output() is not part of the POSIX specification.
2370 - lex does not support exclusive start conditions (%x), though
2372 they are in the POSIX specification.
2373 - When definitions are expanded, flex encloses them in parenthe‐
2375 ses. With lex, the following:
2376 NAME [A-Z][A-Z0-9]*
2378 %%
2379 foo{NAME}? printf( "Found it\n" );
2380 %%
2381 will not match the string "foo" because when the macro is
2383 expanded the rule is equivalent to "foo[A-Z][A-Z0-9]*?" and the
2384 precedence is such that the '?' is associated with "[A-Z0-9]*".
2385 With flex, the rule will be expanded to "foo([A-Z][A-Z0-9]*)?"
2386 and so the string "foo" will match.
2387 Note that if the definition begins with ^ or ends with $ then it
2389 is not expanded with parentheses, to allow these operators to
2390 appear in definitions without losing their special meanings.
2391 But the <s>, /, and <<EOF>> operators cannot be used in a flex
2392 definition.
2393 Using -l results in the lex behavior of no parentheses around
2395 the definition.
2396 The POSIX specification is that the definition be enclosed in
2398 parentheses.
2399 - Some implementations of lex allow a rule's action to begin on a
2401 separate line, if the rule's pattern has trailing whitespace:
2402 %%
2404 foo|bar<space here>
2405 { foobar_action(); }
2406 flex does not support this feature.
2408 - The lex %r (generate a Ratfor scanner) option is not supported.
2410 It is not part of the POSIX specification.
2411 - After a call to unput(), yytext is undefined until the next
2413 token is matched, unless the scanner was built using %array.
2414 This is not the case with lex or the POSIX specification. The
2415 -l option does away with this incompatibility.
2416 - The precedence of the {} (numeric range) operator is different.
2418 lex interprets "abc{1,3}" as "match one, two, or three occur‐
2419 rences of 'abc'", whereas flex interprets it as "match 'ab' fol‐
2420 lowed by one, two, or three occurrences of 'c'". The latter is
2421 in agreement with the POSIX specification.
2422 - The precedence of the ^ operator is different. lex interprets
2424 "^foo|bar" as "match either 'foo' at the beginning of a line, or
2425 'bar' anywhere", whereas flex interprets it as "match either
2426 'foo' or 'bar' if they come at the beginning of a line". The
2427 latter is in agreement with the POSIX specification.
2428 - The special table-size declarations such as %a supported by lex
2430 are not required by flex scanners; flex ignores them.
2431 - The name FLEX_SCANNER is #define'd so scanners may be written
2433 for use with either flex or lex. Scanners also include
2434 YY_FLEX_MAJOR_VERSION and YY_FLEX_MINOR_VERSION indicating which
2435 version of flex generated the scanner (for example, for the 2.5
2436 release, these defines would be 2 and 5 respectively).
2437 The following flex features are not included in lex or the POSIX speci‐
2439 fication:
2440 C++ scanners
2442 %option
2443 start condition scopes
2444 start condition stacks
2445 interactive/non-interactive scanners
2446 yy_scan_string() and friends
2447 yyterminate()
2448 yy_set_interactive()
2449 yy_set_bol()
2450 YY_AT_BOL()
2451 <<EOF>>
2452 <*>
2453 YY_DECL
2454 YY_START
2455 YY_USER_ACTION
2456 YY_USER_INIT
2457 #line directives
2458 %{}'s around actions
2459 multiple actions on a line
2460 plus almost all of the flex flags. The last feature in the list refers
2462 to the fact that with flex you can put multiple actions on the same
2463 line, separated with semi-colons, while with lex, the following
2464 foo handle_foo(); ++num_foos_seen;
2466 is (rather surprisingly) truncated to
2468 foo handle_foo();
2470 flex does not truncate the action. Actions that are not enclosed in
2472 braces are simply terminated at the end of the line.
2473
2475 warning, rule cannot be matched indicates that the given rule cannot be
2476 matched because it follows other rules that will always match the same
2477 text as it. For example, in the following "foo" cannot be matched
2478 because it comes after an identifier "catch-all" rule:
2479 [a-z]+ got_identifier();
2481 foo got_foo();
2482 Using REJECT in a scanner suppresses this warning.
2484 warning, -s option given but default rule can be matched means that it
2486 is possible (perhaps only in a particular start condition) that the
2487 default rule (match any single character) is the only one that will
2488 match a particular input. Since -s was given, presumably this is not
2489 intended.
2490 reject_used_but_not_detected undefined or yymore_used_but_not_detected
2492 undefined - These errors can occur at compile time. They indicate that
2493 the scanner uses REJECT or yymore() but that flex failed to notice the
2494 fact, meaning that flex scanned the first two sections looking for
2495 occurrences of these actions and failed to find any, but somehow you
2496 snuck some in (via a #include file, for example). Use %option reject
2497 or %option yymore to indicate to flex that you really do use these fea‐
2498 tures.
2499 flex scanner jammed - a scanner compiled with -s has encountered an
2501 input string which wasn't matched by any of its rules. This error can
2502 also occur due to internal problems.
2503 token too large, exceeds YYLMAX - your scanner uses %array and one of
2505 its rules matched a string longer than the YYLMAX constant (8K bytes by
2506 default). You can increase the value by #define'ing YYLMAX in the def‐
2507 initions section of your flex input.
2508 scanner requires -8 flag to use the character 'x' - Your scanner speci‐
2510 fication includes recognizing the 8-bit character 'x' and you did not
2511 specify the -8 flag, and your scanner defaulted to 7-bit because you
2512 used the -Cf or -CF table compression options. See the discussion of
2513 the -7 flag for details.
2514 flex scanner push-back overflow - you used unput() to push back so much
2516 text that the scanner's buffer could not hold both the pushed-back text
2517 and the current token in yytext. Ideally the scanner should dynami‐
2518 cally resize the buffer in this case, but at present it does not.
2519 input buffer overflow, can't enlarge buffer because scanner uses REJECT
2521 - the scanner was working on matching an extremely large token and
2522 needed to expand the input buffer. This doesn't work with scanners
2523 that use REJECT.
2524 fatal flex scanner internal error--end of buffer missed - This can
2526 occur in an scanner which is reentered after a long-jump has jumped out
2527 (or over) the scanner's activation frame. Before reentering the scan‐
2528 ner, use:
2529 yyrestart( yyin );
2531 or, as noted above, switch to using the C++ scanner class.
2533 too many start conditions in <> construct! - you listed more start con‐
2535 ditions in a <> construct than exist (so you must have listed at least
2536 one of them twice).
2537
2539 -lfl library with which scanners must be linked.
2540 lex.yy.c
2542 generated scanner (called lexyy.c on some systems).
2543 lex.yy.cc
2545 generated C++ scanner class, when using -+.
2546 <FlexLexer.h>
2548 header file defining the C++ scanner base class, FlexLexer, and
2549 its derived class, yyFlexLexer.
2550 flex.skl
2552 skeleton scanner. This file is only used when building flex,
2553 not when flex executes.
2554 lex.backup
2556 backing-up information for -b flag (called lex.bck on some sys‐
2557 tems).
2558
2560 Some trailing context patterns cannot be properly matched and generate
2561 warning messages ("dangerous trailing context"). These are patterns
2562 where the ending of the first part of the rule matches the beginning of
2563 the second part, such as "zx*/xy*", where the 'x*' matches the 'x' at
2564 the beginning of the trailing context. (Note that the POSIX draft
2565 states that the text matched by such patterns is undefined.)
2566 For some trailing context rules, parts which are actually fixed-length
2568 are not recognized as such, leading to the abovementioned performance
2569 loss. In particular, parts using '|' or {n} (such as "foo{3}") are
2570 always considered variable-length.
2571 Combining trailing context with the special '|' action can result in
2573 fixed trailing context being turned into the more expensive variable
2574 trailing context. For example, in the following:
2575 %%
2577 abc |
2578 xyz/def
2579 Use of unput() invalidates yytext and yyleng, unless the %array direc‐
2582 tive or the -l option has been used.
2583 Pattern-matching of NUL's is substantially slower than matching other
2585 characters.
2586 Dynamic resizing of the input buffer is slow, as it entails rescanning
2588 all the text matched so far by the current (generally huge) token.
2589 Due to both buffering of input and read-ahead, you cannot intermix
2591 calls to <stdio.h> routines, such as, for example, getchar(), with flex
2592 rules and expect it to work. Call input() instead.
2593 The total table entries listed by the -v flag excludes the number of
2595 table entries needed to determine what rule has been matched. The num‐
2596 ber of entries is equal to the number of DFA states if the scanner does
2597 not use REJECT, and somewhat greater than the number of states if it
2598 does.
2599 REJECT cannot be used with the -f or -F options.
2601 The flex internal algorithms need documentation.
2603
2605 lex(1), yacc(1), sed(1), awk(1).
2606 John Levine, Tony Mason, and Doug Brown, Lex & Yacc, O'Reilly and Asso‐
2608 ciates. Be sure to get the 2nd edition.
2609 M. E. Lesk and E. Schmidt, LEX - Lexical Analyzer Generator
2611 Alfred Aho, Ravi Sethi and Jeffrey Ullman, Compilers: Principles, Tech‐
2613 niques and Tools, Addison-Wesley (1986). Describes the pattern-match‐
2614 ing techniques used by flex (deterministic finite automata).
2615
2617 Vern Paxson, with the help of many ideas and much inspiration from Van
2618 Jacobson. Original version by Jef Poskanzer. The fast table represen‐
2619 tation is a partial implementation of a design done by Van Jacobson.
2620 The implementation was done by Kevin Gong and Vern Paxson.
2621 Thanks to the many flex beta-testers, feedbackers, and contributors,
2623 especially Francois Pinard, Casey Leedom, Robert Abramovitz, Stan Ader‐
2624 mann, Terry Allen, David Barker-Plummer, John Basrai, Neal Becker, Nel‐
2625 son H.F. Beebe, benson@odi.com, Karl Berry, Peter A. Bigot, Simon Blan‐
2626 chard, Keith Bostic, Frederic Brehm, Ian Brockbank, Kin Cho, Nick
2627 Christopher, Brian Clapper, J.T. Conklin, Jason Coughlin, Bill Cox,
2628 Nick Cropper, Dave Curtis, Scott David Daniels, Chris G. Demetriou,
2629 Theo Deraadt, Mike Donahue, Chuck Doucette, Tom Epperly, Leo Eskin,
2630 Chris Faylor, Chris Flatters, Jon Forrest, Jeffrey Friedl, Joe Gayda,
2631 Kaveh R. Ghazi, Wolfgang Glunz, Eric Goldman, Christopher M. Gould,
2632 Ulrich Grepel, Peer Griebel, Jan Hajic, Charles Hemphill, NORO Hideo,
2633 Jarkko Hietaniemi, Scott Hofmann, Jeff Honig, Dana Hudes, Eric Hughes,
2634 John Interrante, Ceriel Jacobs, Michal Jaegermann, Sakari Jalovaara,
2635 Jeffrey R. Jones, Henry Juengst, Klaus Kaempf, Jonathan I. Kamens, Ter‐
2636 rence O Kane, Amir Katz, ken@ken.hilco.com, Kevin B. Kenny, Steve
2637 Kirsch, Winfried Koenig, Marq Kole, Ronald Lamprecht, Greg Lee, Rohan
2638 Lenard, Craig Leres, John Levine, Steve Liddle, David Loffredo, Mike
2639 Long, Mohamed el Lozy, Brian Madsen, Malte, Joe Marshall, Bengt
2640 Martensson, Chris Metcalf, Luke Mewburn, Jim Meyering, R. Alexander
2641 Milowski, Erik Naggum, G.T. Nicol, Landon Noll, James Nordby, Marc
2642 Nozell, Richard Ohnemus, Karsten Pahnke, Sven Panne, Roland Pesch, Wal‐
2643 ter Pelissero, Gaumond Pierre, Esmond Pitt, Jef Poskanzer, Joe Rahmeh,
2644 Jarmo Raiha, Frederic Raimbault, Pat Rankin, Rick Richardson, Kevin
2645 Rodgers, Kai Uwe Rommel, Jim Roskind, Alberto Santini, Andreas Scherer,
2646 Darrell Schiebel, Raf Schietekat, Doug Schmidt, Philippe Schnoebelen,
2647 Andreas Schwab, Larry Schwimmer, Alex Siegel, Eckehard Stolz, Jan-Erik
2648 Strvmquist, Mike Stump, Paul Stuart, Dave Tallman, Ian Lance Taylor,
2649 Chris Thewalt, Richard M. Timoney, Jodi Tsai, Paul Tuinenga, Gary Weik,
2650 Frank Whaley, Gerhard Wilhelms, Kent Williams, Ken Yap, Ron Zellar,
2651 Nathan Zelle, David Zuhn, and those whose names have slipped my margin‐
2652 al mail-archiving skills but whose contributions are appreciated all
2653 the same.
2654 Thanks to Keith Bostic, Jon Forrest, Noah Friedman, John Gilmore, Craig
2656 Leres, John Levine, Bob Mulcahy, G.T. Nicol, Francois Pinard, Rich
2657 Salz, and Richard Stallman for help with various distribution
2658 headaches.
2659 Thanks to Esmond Pitt and Earle Horton for 8-bit character support; to
2661 Benson Margulies and Fred Burke for C++ support; to Kent Williams and
2662 Tom Epperly for C++ class support; to Ove Ewerlid for support of NUL's;
2663 and to Eric Hughes for support of multiple buffers.
2664 This work was primarily done when I was with the Real Time Systems
2666 Group at the Lawrence Berkeley Laboratory in Berkeley, CA. Many thanks
2667 to all there for the support I received.
2668 Send comments to vern@ee.lbl.gov.
2670Version 2.5 April 1995 FLEX(1)