1LEX(1P) POSIX Programmer's Manual LEX(1P)
2
6 This manual page is part of the POSIX Programmer's Manual. The Linux
7 implementation of this interface may differ (consult the corresponding
8 Linux manual page for details of Linux behavior), or the interface may
9 not be implemented on Linux.
10
13 lex — generate programs for lexical tasks (DEVELOPMENT)
14
16 lex [−t] [−n|−v] [file...]
17
19 The lex utility shall generate C programs to be used in lexical pro‐
20 cessing of character input, and that can be used as an interface to
21 yacc. The C programs shall be generated from lex source code and con‐
22 form to the ISO C standard, without depending on any undefined, unspec‐
23 ified, or implementation-defined behavior, except in cases where the
24 code is copied directly from the supplied source, or in cases that are
25 documented by the implementation. Usually, the lex utility shall write
26 the program it generates to the file lex.yy.c; the state of this file
27 is unspecified if lex exits with a non-zero exit status. See the
28 EXTENDED DESCRIPTION section for a complete description of the lex
29 input language.
30
32 The lex utility shall conform to the Base Definitions volume of
33 POSIX.1‐2008, Section 12.2, Utility Syntax Guidelines, except for
34 Guideline 9.
35 The following options shall be supported:
37 −n Suppress the summary of statistics usually written with the
39 −v option. If no table sizes are specified in the lex source
40 code and the −v option is not specified, then −n is implied.
41 −t Write the resulting program to standard output instead of
43 lex.yy.c.
44 −v Write a summary of lex statistics to the standard output.
46 (See the discussion of lex table sizes in Definitions in
47 lex.) If the −t option is specified and −n is not specified,
48 this report shall be written to standard error. If table
49 sizes are specified in the lex source code, and if the −n
50 option is not specified, the −v option may be enabled.
51
53 The following operand shall be supported:
54 file A pathname of an input file. If more than one such file is
56 specified, all files shall be concatenated to produce a sin‐
57 gle lex program. If no file operands are specified, or if a
58 file operand is '−', the standard input shall be used.
59
61 The standard input shall be used if no file operands are specified, or
62 if a file operand is '−'. See INPUT FILES.
63
65 The input files shall be text files containing lex source code, as
66 described in the EXTENDED DESCRIPTION section.
67
69 The following environment variables shall affect the execution of lex:
70 LANG Provide a default value for the internationalization vari‐
72 ables that are unset or null. (See the Base Definitions vol‐
73 ume of POSIX.1‐2008, Section 8.2, Internationalization Vari‐
74 ables for the precedence of internationalization variables
75 used to determine the values of locale categories.)
76 LC_ALL If set to a non-empty string value, override the values of
78 all the other internationalization variables.
79 LC_COLLATE
81 Determine the locale for the behavior of ranges, equivalence
82 classes, and multi-character collating elements within regu‐
83 lar expressions. If this variable is not set to the POSIX
84 locale, the results are unspecified.
85 LC_CTYPE Determine the locale for the interpretation of sequences of
87 bytes of text data as characters (for example, single-byte as
88 opposed to multi-byte characters in arguments and input
89 files), and the behavior of character classes within regular
90 expressions. If this variable is not set to the POSIX locale,
91 the results are unspecified.
92 LC_MESSAGES
94 Determine the locale that should be used to affect the format
95 and contents of diagnostic messages written to standard
96 error.
97 NLSPATH Determine the location of message catalogs for the processing
99 of LC_MESSAGES.
100
102 Default.
103
105 If the −t option is specified, the text file of C source code output of
106 lex shall be written to standard output.
107 If the −t option is not specified:
109 * Implementation-defined informational, error, and warning messages
111 concerning the contents of lex source code input shall be written
112 to either the standard output or standard error.
113 * If the −v option is specified and the −n option is not specified,
115 lex statistics shall also be written to either the standard output
116 or standard error, in an implementation-defined format. These sta‐
117 tistics may also be generated if table sizes are specified with a
118 '%' operator in the Definitions section, as long as the −n option
119 is not specified.
120
122 If the −t option is specified, implementation-defined informational,
123 error, and warning messages concerning the contents of lex source code
124 input shall be written to the standard error.
125 If the −t option is not specified:
127 1. Implementation-defined informational, error, and warning messages
129 concerning the contents of lex source code input shall be written
130 to either the standard output or standard error.
131 2. If the −v option is specified and the −n option is not specified,
133 lex statistics shall also be written to either the standard output
134 or standard error, in an implementation-defined format. These sta‐
135 tistics may also be generated if table sizes are specified with a
136 '%' operator in the Definitions section, as long as the −n option
137 is not specified.
138
140 A text file containing C source code shall be written to lex.yy.c, or
141 to the standard output if the −t option is present.
142
144 Each input file shall contain lex source code, which is a table of reg‐
145 ular expressions with corresponding actions in the form of C program
146 fragments.
147 When lex.yy.c is compiled and linked with the lex library (using the
149 −l l operand with c99), the resulting program shall read character
150 input from the standard input and shall partition it into strings that
151 match the given expressions.
152 When an expression is matched, these actions shall occur:
154 * The input string that was matched shall be left in yytext as a
156 null-terminated string; yytext shall either be an external charac‐
157 ter array or a pointer to a character string. As explained in Defi‐
158 nitions in lex, the type can be explicitly selected using the
159 %array or %pointer declarations, but the default is implementation-
160 defined.
161 * The external int yyleng shall be set to the length of the matching
163 string.
164 * The expression's corresponding program fragment, or action, shall
166 be executed.
167 During pattern matching, lex shall search the set of patterns for the
169 single longest possible match. Among rules that match the same number
170 of characters, the rule given first shall be chosen.
171 The general format of lex source shall be:
173 Definitions %% Rules %% UserSubroutines
175 The first "%%" is required to mark the beginning of the rules (regular
177 expressions and actions); the second "%%" is required only if user sub‐
178 routines follow.
179 Any line in the Definitions section beginning with a <blank> shall be
181 assumed to be a C program fragment and shall be copied to the external
182 definition area of the lex.yy.c file. Similarly, anything in the Defi‐
183 nitions section included between delimiter lines containing only "%{"
184 and "%}" shall also be copied unchanged to the external definition area
185 of the lex.yy.c file.
186 Any such input (beginning with a <blank> or within "%{" and "%}" delim‐
188 iter lines) appearing at the beginning of the Rules section before any
189 rules are specified shall be written to lex.yy.c after the declarations
190 of variables for the yylex() function and before the first line of code
191 in yylex(). Thus, user variables local to yylex() can be declared
192 here, as well as application code to execute upon entry to yylex().
193 The action taken by lex when encountering any input beginning with a
195 <blank> or within "%{" and "%}" delimiter lines appearing in the Rules
196 section but coming after one or more rules is undefined. The presence
197 of such input may result in an erroneous definition of the yylex()
198 function.
199 C-language code in the input shall not contain C-language trigraphs.
201 The C-language code within "%{" and "%}" delimiter lines shall not con‐
202 tain any lines consisting only of "%}", or only of "%%".
203 Definitions in lex
205 Definitions appear before the first "%%" delimiter. Any line in this
206 section not contained between "%{" and "%}" lines and not beginning
207 with a <blank> shall be assumed to define a lex substitution string.
208 The format of these lines shall be:
209 name substitute
211 If a name does not meet the requirements for identifiers in the ISO C
213 standard, the result is undefined. The string substitute shall replace
214 the string {name} when it is used in a rule. The name string shall be
215 recognized in this context only when the braces are provided and when
216 it does not appear within a bracket expression or within double-quotes.
217 In the Definitions section, any line beginning with a <percent-sign>
219 ('%') character and followed by an alphanumeric word beginning with
220 either 's' or 'S' shall define a set of start conditions. Any line
221 beginning with a '%' followed by a word beginning with either 'x' or
222 'X' shall define a set of exclusive start conditions. When the gener‐
223 ated scanner is in a %s state, patterns with no state specified shall
224 be also active; in a %x state, such patterns shall not be active. The
225 rest of the line, after the first word, shall be considered to be one
226 or more <blank>-separated names of start conditions. Start condition
227 names shall be constructed in the same way as definition names. Start
228 conditions can be used to restrict the matching of regular expressions
229 to one or more states as described in Regular Expressions in lex.
230 Implementations shall accept either of the following two mutually-
232 exclusive declarations in the Definitions section:
233 %array Declare the type of yytext to be a null-terminated character
235 array.
236 %pointer Declare the type of yytext to be a pointer to a null-termi‐
238 nated character string.
239 The default type of yytext is implementation-defined. If an application
241 refers to yytext outside of the scanner source file (that is, via an
242 extern), the application shall include the appropriate %array or
243 %pointer declaration in the scanner source file.
244 Implementations shall accept declarations in the Definitions section
246 for setting certain internal table sizes. The declarations are shown in
247 the following table.
248 Table: Table Size Declarations in lex
250 ┌────────────┬────────────────────────────────────┬───────────────┐
252 │Declaration │ Description │ Minimum Value │
253 ├────────────┼────────────────────────────────────┼───────────────┤
254 │%p n │ Number of positions │ 2500 │
255 │%n n │ Number of states │ 500 │
256 │%a n │ Number of transitions │ 2000 │
257 │%e n │ Number of parse tree nodes │ 1000 │
258 │%k n │ Number of packed character classes │ 1000 │
259 │%o n │ Size of the output array │ 3000 │
260 └────────────┴────────────────────────────────────┴───────────────┘
261 In the table, n represents a positive decimal integer, preceded by one
262 or more <blank> characters. The exact meaning of these table size num‐
263 bers is implementation-defined. The implementation shall document how
264 these numbers affect the lex utility and how they are related to any
265 output that may be generated by the implementation should limitations
266 be encountered during the execution of lex. It shall be possible to
267 determine from this output which of the table size values needs to be
268 modified to permit lex to successfully generate tables for the input
269 language. The values in the column Minimum Value represent the lowest
270 values conforming implementations shall provide.
271 Rules in lex
273 The rules in lex source files are a table in which the left column con‐
274 tains regular expressions and the right column contains actions (C pro‐
275 gram fragments) to be executed when the expressions are recognized.
276 ERE action
278 ERE action
279 ...
280 The extended regular expression (ERE) portion of a row shall be sepa‐
282 rated from action by one or more <blank> characters. A regular expres‐
283 sion containing <blank> characters shall be recognized under one of the
284 following conditions:
285 * The entire expression appears within double-quotes.
287 * The <blank> characters appear within double-quotes or square brack‐
289 ets.
290 * Each <blank> is preceded by a <backslash> character.
292 User Subroutines in lex
294 Anything in the user subroutines section shall be copied to lex.yy.c
295 following yylex().
296 Regular Expressions in lex
298 The lex utility shall support the set of extended regular expressions
299 (see the Base Definitions volume of POSIX.1‐2008, Section 9.4, Extended
300 Regular Expressions), with the following additions and exceptions to
301 the syntax:
302 "..." Any string enclosed in double-quotes shall represent the
304 characters within the double-quotes as themselves, except
305 that <backslash>-escapes (which appear in the following ta‐
306 ble) shall be recognized. Any <backslash>-escape sequence
307 shall be terminated by the closing quote. For example,
308 "\01""1" represents a single string: the octal value 1 fol‐
309 lowed by the character '1'.
310 <state>r, <state1,state2,...>r
312 The regular expression r shall be matched only when the pro‐
313 gram is in one of the start conditions indicated by state,
314 state1, and so on; see Actions in lex. (As an exception to
315 the typographical conventions of the rest of this volume of
316 POSIX.1‐2008, in this case <state> does not represent a
317 metavariable, but the literal angle-bracket characters sur‐
318 rounding a symbol.) The start condition shall be recognized
319 as such only at the beginning of a regular expression.
320 r/x The regular expression r shall be matched only if it is fol‐
322 lowed by an occurrence of regular expression x (x is the
323 instance of trailing context, further defined below). The
324 token returned in yytext shall only match r. If the trailing
325 portion of r matches the beginning of x, the result is
326 unspecified. The r expression cannot include further trailing
327 context or the '$' (match-end-of-line) operator; x cannot
328 include the '^' (match-beginning-of-line) operator, nor
329 trailing context, nor the '$' operator. That is, only one
330 occurrence of trailing context is allowed in a lex regular
331 expression, and the '^' operator only can be used at the
332 beginning of such an expression.
333 {name} When name is one of the substitution symbols from the Defini‐
335 tions section, the string, including the enclosing braces,
336 shall be replaced by the substitute value. The substitute
337 value shall be treated in the extended regular expression as
338 if it were enclosed in parentheses. No substitution shall
339 occur if {name} occurs within a bracket expression or within
340 double-quotes.
341 Within an ERE, a <backslash> character shall be considered to begin an
343 escape sequence as specified in the table in the Base Definitions vol‐
344 ume of POSIX.1‐2008, Chapter 5, File Format Notation ('\\', '\a', '\b',
345 '\f', '\n', '\r', '\t', '\v'). In addition, the escape sequences in
346 the following table shall be recognized.
347 A literal <newline> cannot occur within an ERE; the escape sequence
349 '\n' can be used to represent a <newline>. A <newline> shall not be
350 matched by a period operator.
351 Table: Escape Sequences in lex
353 ┌─────────┬──────────────────────────┬──────────────────────────┐
355 │ Escape │ │ │
356 │Sequence │ Description │ Meaning │
357 ├─────────┼──────────────────────────┼──────────────────────────┤
358 │\digits │ A <backslash> character │ The character whose │
359 │ │ followed by the longest │ encoding is represented │
360 │ │ sequence of one, two, or │ by the one, two, or │
361 │ │ three octal-digit char‐ │ three-digit octal inte‐ │
362 │ │ acters (01234567). If │ ger. Multi-byte charac‐ │
363 │ │ all of the digits are 0 │ ters require multiple, │
364 │ │ (that is, representation │ concatenated escape │
365 │ │ of the NUL character), │ sequences of this type, │
366 │ │ the behavior is unde‐ │ including the leading │
367 │ │ fined. │ <backslash> for each │
368 │ │ │ byte. │
369 ├─────────┼──────────────────────────┼──────────────────────────┤
370 │\xdigits │ A <backslash> character │ The character whose │
371 │ │ followed by the longest │ encoding is represented │
372 │ │ sequence of hexadecimal- │ by the hexadecimal inte‐ │
373 │ │ digit characters │ ger. │
374 │ │ (01234567abcdefABCDEF). │ │
375 │ │ If all of the digits are │ │
376 │ │ 0 (that is, representa‐ │ │
377 │ │ tion of the NUL charac‐ │ │
378 │ │ ter), the behavior is │ │
379 │ │ undefined. │ │
380 ├─────────┼──────────────────────────┼──────────────────────────┤
381 │\c │ A <backslash> character │ The character 'c', │
382 │ │ followed by any charac‐ │ unchanged. │
383 │ │ ter not described in │ │
384 │ │ this table or in the ta‐ │ │
385 │ │ ble in the Base Defini‐ │ │
386 │ │ tions volume of │ │
387 │ │ POSIX.1‐2008, Chapter 5, │ │
388 │ │ File Format Notation │ │
389 │ │ ('\\', '\a', '\b', '\f', │ │
390 │ │ '\n', '\r', '\t', '\v'). │ │
391 └─────────┴──────────────────────────┴──────────────────────────┘
392 Note: If a '\x' sequence needs to be immediately followed by a
393 hexadecimal digit character, a sequence such as "\x1""1" can
394 be used, which represents a character containing the value 1,
395 followed by the character '1'.
396 The order of precedence given to extended regular expressions for lex
398 differs from that specified in the Base Definitions volume of
399 POSIX.1‐2008, Section 9.4, Extended Regular Expressions. The order of
400 precedence for lex shall be as shown in the following table, from high
401 to low.
402 Note: The escaped characters entry is not meant to imply that these
404 are operators, but they are included in the table to show
405 their relationships to the true operators. The start condi‐
406 tion, trailing context, and anchoring notations have been
407 omitted from the table because of the placement restrictions
408 described in this section; they can only appear at the begin‐
409 ning or ending of an ERE.
410 Table: ERE Precedence in lex
412 ┌──────────────────────────────────┬──────────────────────┐
414 │ Extended Regular Expression │ Precedence │
415 ├──────────────────────────────────┼──────────────────────┤
416 │collation-related bracket symbols │ [= =] [: :] [. .] │
417 │escaped characters │ \<special character> │
418 │bracket expression │ [ ] │
419 │quoting │ "..." │
420 │grouping │ ( ) │
421 │definition │ {name} │
422 │single-character RE duplication │ * + ? │
423 │concatenation │ │
424 │interval expression │ {m,n} │
425 │alternation │ | │
426 └──────────────────────────────────┴──────────────────────┘
427 The ERE anchoring operators '^' and '$' do not appear in the table.
428 With lex regular expressions, these operators are restricted in their
429 use: the '^' operator can only be used at the beginning of an entire
430 regular expression, and the '$' operator only at the end. The operators
431 apply to the entire regular expression. Thus, for example, the pattern
432 "(^abc)|(def$)" is undefined; it can instead be written as two separate
433 rules, one with the regular expression "^abc" and one with "def$",
434 which share a common action via the special '|' action (see below). If
435 the pattern were written "^abc|def$", it would match either "abc" or
436 "def" on a line by itself.
437 Unlike the general ERE rules, embedded anchoring is not allowed by most
439 historical lex implementations. An example of embedded anchoring would
440 be for patterns such as "(^| )foo( |$)" to match "foo" when it exists
441 as a complete word. This functionality can be obtained using existing
442 lex features:
443 ^foo/[ \n] |
445 " foo"/[ \n] /* Found foo as a separate word. */
446 Note also that '$' is a form of trailing context (it is equivalent to
448 "/\n") and as such cannot be used with regular expressions containing
449 another instance of the operator (see the preceding discussion of
450 trailing context).
451 The additional regular expressions trailing-context operator '/' can be
453 used as an ordinary character if presented within double-quotes, "/";
454 preceded by a <backslash>, "\/"; or within a bracket expression, "[/]".
455 The start-condition '<' and '>' operators shall be special only in a
456 start condition at the beginning of a regular expression; elsewhere in
457 the regular expression they shall be treated as ordinary characters.
458 Actions in lex
460 The action to be taken when an ERE is matched can be a C program frag‐
461 ment or the special actions described below; the program fragment can
462 contain one or more C statements, and can also include special actions.
463 The empty C statement ';' shall be a valid action; any string in the
464 lex.yy.c input that matches the pattern portion of such a rule is
465 effectively ignored or skipped. However, the absence of an action shall
466 not be valid, and the action lex takes in such a condition is unde‐
467 fined.
468 The specification for an action, including C statements and special
470 actions, can extend across several lines if enclosed in braces:
471 ERE <one or more blanks> { program statement
473 program statement }
474 The program statements shall not contain unbalanced curly brace prepro‐
476 cessing tokens.
477 The default action when a string in the input to a lex.yy.c program is
479 not matched by any expression shall be to copy the string to the out‐
480 put. Because the default behavior of a program generated by lex is to
481 read the input and copy it to the output, a minimal lex source program
482 that has just "%%" shall generate a C program that simply copies the
483 input to the output unchanged.
484 Four special actions shall be available:
486 | ECHO; REJECT; BEGIN
488 | The action '|' means that the action for the next rule is the
490 action for this rule. Unlike the other three actions, '|'
491 cannot be enclosed in braces or be <semicolon>-terminated;
492 the application shall ensure that it is specified alone, with
493 no other actions.
494 ECHO; Write the contents of the string yytext on the output.
496 REJECT; Usually only a single expression is matched by a given string
498 in the input. REJECT means ``continue to the next expression
499 that matches the current input'', and shall cause whatever
500 rule was the second choice after the current rule to be exe‐
501 cuted for the same input. Thus, multiple rules can be matched
502 and executed for one input string or overlapping input
503 strings. For example, given the regular expressions "xyz" and
504 "xy" and the input "xyz", usually only the regular expression
505 "xyz" would match. The next attempted match would start after
506 z. If the last action in the "xyz" rule is REJECT, both this
507 rule and the "xy" rule would be executed. The REJECT action
508 may be implemented in such a fashion that flow of control
509 does not continue after it, as if it were equivalent to a
510 goto to another part of yylex(). The use of REJECT may
511 result in somewhat larger and slower scanners.
512 BEGIN The action:
514 BEGIN newstate;
516 switches the state (start condition) to newstate. If the
518 string newstate has not been declared previously as a start
519 condition in the Definitions section, the results are unspec‐
520 ified. The initial state is indicated by the digit '0' or the
521 token INITIAL.
522 The functions or macros described below are accessible to user code
524 included in the lex input. It is unspecified whether they appear in the
525 C code output of lex, or are accessible only through the −l l operand
526 to c99 (the lex library).
527 int yylex(void)
529 Performs lexical analysis on the input; this is the primary func‐
530 tion generated by the lex utility. The function shall return zero
531 when the end of input is reached; otherwise, it shall return non-
532 zero values (tokens) determined by the actions that are selected.
533 int yymore(void)
535 When called, indicates that when the next input string is recog‐
536 nized, it is to be appended to the current value of yytext rather
537 than replacing it; the value in yyleng shall be adjusted accord‐
538 ingly.
539 int yyless(int n)
541 Retains n initial characters in yytext, NUL-terminated, and
542 treats the remaining characters as if they had not been read; the
543 value in yyleng shall be adjusted accordingly.
544 int input(void)
546 Returns the next character from the input, or zero on end-of-
547 file. It shall obtain input from the stream pointer yyin,
548 although possibly via an intermediate buffer. Thus, once scanning
549 has begun, the effect of altering the value of yyin is undefined.
550 The character read shall be removed from the input stream of the
551 scanner without any processing by the scanner.
552 int unput(int c)
554 Returns the character 'c' to the input; yytext and yyleng are
555 undefined until the next expression is matched. The result of
556 using unput() for more characters than have been input is unspec‐
557 ified.
558 The following functions shall appear only in the lex library accessible
560 through the −l l operand; they can therefore be redefined by a conform‐
561 ing application:
562 int yywrap(void)
564 Called by yylex() at end-of-file; the default yywrap() shall
565 always return 1. If the application requires yylex() to continue
566 processing with another source of input, then the application can
567 include a function yywrap(), which associates another file with
568 the external variable FILE * yyin and shall return a value of
569 zero.
570 int main(int argc, char *argv[])
572 Calls yylex() to perform lexical analysis, then exits. The user
573 code can contain main() to perform application-specific opera‐
574 tions, calling yylex() as applicable.
575 Except for input(), unput(), and main(), all external and static names
577 generated by lex shall begin with the prefix yy or YY.
578
580 The following exit values shall be returned:
581 0 Successful completion.
583 >0 An error occurred.
585
587 Default.
588 The following sections are informative.
590
592 Conforming applications are warned that in the Rules section, an ERE
593 without an action is not acceptable, but need not be detected as erro‐
594 neous by lex. This may result in compilation or runtime errors.
595 The purpose of input() is to take characters off the input stream and
597 discard them as far as the lexical analysis is concerned. A common use
598 is to discard the body of a comment once the beginning of a comment is
599 recognized.
600 The lex utility is not fully internationalized in its treatment of reg‐
602 ular expressions in the lex source code or generated lexical analyzer.
603 It would seem desirable to have the lexical analyzer interpret the reg‐
604 ular expressions given in the lex source according to the environment
605 specified when the lexical analyzer is executed, but this is not possi‐
606 ble with the current lex technology. Furthermore, the very nature of
607 the lexical analyzers produced by lex must be closely tied to the lexi‐
608 cal requirements of the input language being described, which is fre‐
609 quently locale-specific anyway. (For example, writing an analyzer that
610 is used for French text is not automatically useful for processing
611 other languages.)
612
614 The following is an example of a lex program that implements a rudimen‐
615 tary scanner for a Pascal-like syntax:
616 %{
618 /* Need this for the call to atof() below. */
619 #include <math.h>
620 /* Need this for printf(), fopen(), and stdin below. */
621 #include <stdio.h>
622 %}
623 DIGIT [0−9]
625 ID [a−z][a−z0−9]*
626 %%
628 {DIGIT}+ {
630 printf("An integer: %s (%d)\n", yytext,
631 atoi(yytext));
632 }
633 {DIGIT}+"."{DIGIT}* {
635 printf("A float: %s (%g)\n", yytext,
636 atof(yytext));
637 }
638 if|then|begin|end|procedure|function {
640 printf("A keyword: %s\n", yytext);
641 }
642 {ID} printf("An identifier: %s\n", yytext);
644 "+"|"−"|"*"|"/" printf("An operator: %s\n", yytext);
646 "{"[^}\n]*"}" /* Eat up one-line comments. */
648 [ \t\n]+ /* Eat up white space. */
650 . printf("Unrecognized character: %s\n", yytext);
652 %%
654 int main(int argc, char *argv[])
656 {
657 ++argv, −−argc; /* Skip over program name. */
658 if (argc > 0)
659 yyin = fopen(argv[0], "r");
660 else
661 yyin = stdin;
662 yylex();
664 }
665
667 Even though the −c option and references to the C language are retained
668 in this description, lex may be generalized to other languages, as was
669 done at one time for EFL, the Extended FORTRAN Language. Since the lex
670 input specification is essentially language-independent, versions of
671 this utility could be written to produce Ada, Modula-2, or Pascal code,
672 and there are known historical implementations that do so.
673 The current description of lex bypasses the issue of dealing with
675 internationalized EREs in the lex source code or generated lexical ana‐
676 lyzer. If it follows the model used by awk (the source code is assumed
677 to be presented in the POSIX locale, but input and output are in the
678 locale specified by the environment variables), then the tables in the
679 lexical analyzer produced by lex would interpret EREs specified in the
680 lex source in terms of the environment variables specified when lex was
681 executed. The desired effect would be to have the lexical analyzer
682 interpret the EREs given in the lex source according to the environment
683 specified when the lexical analyzer is executed, but this is not possi‐
684 ble with the current lex technology.
685 The description of octal and hexadecimal-digit escape sequences agrees
687 with the ISO C standard use of escape sequences.
688 Earlier versions of this standard allowed for implementations with
690 bytes other than eight bits, but this has been modified in this ver‐
691 sion.
692 There is no detailed output format specification. The observed behavior
694 of lex under four different historical implementations was that none of
695 these implementations consistently reported the line numbers for error
696 and warning messages. Furthermore, there was a desire that lex be
697 allowed to output additional diagnostic messages. Leaving message for‐
698 mats unspecified avoids these formatting questions and problems with
699 internationalization.
700 Although the %x specifier for exclusive start conditions is not histor‐
702 ical practice, it is believed to be a minor change to historical imple‐
703 mentations and greatly enhances the usability of lex programs since it
704 permits an application to obtain the expected functionality with fewer
705 statements.
706 The %array and %pointer declarations were added as a compromise between
708 historical systems. The System V-based lex copies the matched text to
709 a yytext array. The flex program, supported in BSD and GNU systems,
710 uses a pointer. In the latter case, significant performance improve‐
711 ments are available for some scanners. Most historical programs should
712 require no change in porting from one system to another because the
713 string being referenced is null-terminated in both cases. (The method
714 used by flex in its case is to null-terminate the token in place by
715 remembering the character that used to come right after the token and
716 replacing it before continuing on to the next scan.) Multi-file pro‐
717 grams with external references to yytext outside the scanner source
718 file should continue to operate on their historical systems, but would
719 require one of the new declarations to be considered strictly portable.
720 The description of EREs avoids unnecessary duplication of ERE details
722 because their meanings within a lex ERE are the same as that for the
723 ERE in this volume of POSIX.1‐2008.
724 The reason for the undefined condition associated with text beginning
726 with a <blank> or within "%{" and "%}" delimiter lines appearing in the
727 Rules section is historical practice. Both the BSD and System V lex
728 copy the indented (or enclosed) input in the Rules section (except at
729 the beginning) to unreachable areas of the yylex() function (the code
730 is written directly after a break statement). In some cases, the System
731 V lex generates an error message or a syntax error, depending on the
732 form of indented input.
733 The intention in breaking the list of functions into those that may
735 appear in lex.yy.c versus those that only appear in libl.a is that only
736 those functions in libl.a can be reliably redefined by a conforming
737 application.
738 The descriptions of standard output and standard error are somewhat
740 complicated because historical lex implementations chose to issue diag‐
741 nostic messages to standard output (unless −t was given). POSIX.1‐2008
742 allows this behavior, but leaves an opening for the more expected
743 behavior of using standard error for diagnostics. Also, the System V
744 behavior of writing the statistics when any table sizes are given is
745 allowed, while BSD-derived systems can avoid it. The programmer can
746 always precisely obtain the desired results by using either the −t or
747 −n options.
748 The OPERANDS section does not mention the use of − as a synonym for
750 standard input; not all historical implementations support such usage
751 for any of the file operands.
752 A description of the translation table was deleted from early proposals
754 because of its relatively low usage in historical applications.
755 The change to the definition of the input() function that allows
757 buffering of input presents the opportunity for major performance gains
758 in some applications.
759 The following examples clarify the differences between lex regular
761 expressions and regular expressions appearing elsewhere in this volume
762 of POSIX.1‐2008. For regular expressions of the form "r/x", the string
763 matching r is always returned; confusion may arise when the beginning
764 of x matches the trailing portion of r. For example, given the regular
765 expression "a*b/cc" and the input "aaabcc", yytext would contain the
766 string "aaab" on this match. But given the regular expression "x*/xy"
767 and the input "xxxy", the token xxx, not xx, is returned by some imple‐
768 mentations because xxx matches "x*".
769 In the rule "ab*/bc", the "b*" at the end of r extends r's match into
771 the beginning of the trailing context, so the result is unspecified. If
772 this rule were "ab/bc", however, the rule matches the text "ab" when it
773 is followed by the text "bc". In this latter case, the matching of r
774 cannot extend into the beginning of x, so the result is specified.
775
777 None.
778
780 c99, ed, yacc
781 The Base Definitions volume of POSIX.1‐2008, Chapter 5, File Format
783 Notation, Chapter 8, Environment Variables, Chapter 9, Regular Expres‐
784 sions, Section 12.2, Utility Syntax Guidelines
785
787 Portions of this text are reprinted and reproduced in electronic form
788 from IEEE Std 1003.1, 2013 Edition, Standard for Information Technology
789 -- Portable Operating System Interface (POSIX), The Open Group Base
790 Specifications Issue 7, Copyright (C) 2013 by the Institute of Electri‐
791 cal and Electronics Engineers, Inc and The Open Group. (This is
792 POSIX.1-2008 with the 2013 Technical Corrigendum 1 applied.) In the
793 event of any discrepancy between this version and the original IEEE and
794 The Open Group Standard, the original IEEE and The Open Group Standard
795 is the referee document. The original Standard can be obtained online
796 at http://www.unix.org/online.html .
797 Any typographical or formatting errors that appear in this page are
799 most likely to have been introduced during the conversion of the source
800 files to man page format. To report such errors, see https://www.ker‐
801 nel.org/doc/man-pages/reporting_bugs.html .
802IEEE/The Open Group 2013 LEX(1P)