1LEX(1P)                    POSIX Programmer's Manual                   LEX(1P)
2
3
4

PROLOG

6       This  manual  page is part of the POSIX Programmer's Manual.  The Linux
7       implementation of this interface may differ (consult the  corresponding
8       Linux  manual page for details of Linux behavior), or the interface may
9       not be implemented on Linux.
10

NAME

12       lex — generate programs for lexical tasks (DEVELOPMENT)
13

SYNOPSIS

15       lex [-t] [-n|-v] [file...]
16

DESCRIPTION

18       The lex utility shall generate C programs to be used  in  lexical  pro‐
19       cessing  of  character  input,  and that can be used as an interface to
20       yacc.  The C programs shall be generated from lex source code and  con‐
21       form to the ISO C standard, without depending on any undefined, unspec‐
22       ified, or implementation-defined behavior, except in  cases  where  the
23       code  is copied directly from the supplied source, or in cases that are
24       documented by the implementation. Usually, the lex utility shall  write
25       the  program  it generates to the file lex.yy.c; the state of this file
26       is unspecified if lex exits  with  a  non-zero  exit  status.  See  the
27       EXTENDED  DESCRIPTION  section  for  a  complete description of the lex
28       input language.
29

OPTIONS

31       The lex utility  shall  conform  to  the  Base  Definitions  volume  of
32       POSIX.1‐2017,  Section  12.2,  Utility  Syntax  Guidelines,  except for
33       Guideline 9.
34
35       The following options shall be supported:
36
37       -n        Suppress the summary of statistics usually written  with  the
38                 -v  option. If no table sizes are specified in the lex source
39                 code and the -v option is not specified, then -n is implied.
40
41       -t        Write the resulting program to  standard  output  instead  of
42                 lex.yy.c.
43
44       -v        Write  a  summary  of  lex statistics to the standard output.
45                 (See the discussion of lex  table  sizes  in  Definitions  in
46                 lex.)  If the -t option is specified and -n is not specified,
47                 this report shall be written  to  standard  error.  If  table
48                 sizes  are  specified  in  the lex source code, and if the -n
49                 option is not specified, the -v option may be enabled.
50

OPERANDS

52       The following operand shall be supported:
53
54       file      A pathname of an input file. If more than one  such  file  is
55                 specified,  all files shall be concatenated to produce a sin‐
56                 gle lex program. If no file operands are specified, or  if  a
57                 file operand is '-', the standard input shall be used.
58

STDIN

60       The  standard input shall be used if no file operands are specified, or
61       if a file operand is '-'.  See INPUT FILES.
62

INPUT FILES

64       The input files shall be text files  containing  lex  source  code,  as
65       described in the EXTENDED DESCRIPTION section.
66

ENVIRONMENT VARIABLES

68       The following environment variables shall affect the execution of lex:
69
70       LANG      Provide  a  default  value for the internationalization vari‐
71                 ables that are unset or null. (See the Base Definitions  vol‐
72                 ume  of POSIX.1‐2017, Section 8.2, Internationalization Vari‐
73                 ables for the precedence  of  internationalization  variables
74                 used to determine the values of locale categories.)
75
76       LC_ALL    If  set  to  a non-empty string value, override the values of
77                 all the other internationalization variables.
78
79       LC_COLLATE
80                 Determine the locale for the behavior of ranges,  equivalence
81                 classes,  and multi-character collating elements within regu‐
82                 lar expressions. If this variable is not  set  to  the  POSIX
83                 locale, the results are unspecified.
84
85       LC_CTYPE  Determine  the  locale for the interpretation of sequences of
86                 bytes of text data as characters (for example, single-byte as
87                 opposed  to  multi-byte  characters  in  arguments  and input
88                 files), and the behavior of character classes within  regular
89                 expressions. If this variable is not set to the POSIX locale,
90                 the results are unspecified.
91
92       LC_MESSAGES
93                 Determine the locale that should be used to affect the format
94                 and  contents  of  diagnostic  messages  written  to standard
95                 error.
96
97       NLSPATH   Determine the location of message catalogs for the processing
98                 of LC_MESSAGES.
99

ASYNCHRONOUS EVENTS

101       Default.
102

STDOUT

104       If the -t option is specified, the text file of C source code output of
105       lex shall be written to standard output.
106
107       If the -t option is not specified:
108
109        *  Implementation-defined informational, error, and  warning  messages
110           concerning  the  contents of lex source code input shall be written
111           to either the standard output or standard error.
112
113        *  If the -v option is specified and the -n option is  not  specified,
114           lex  statistics shall also be written to either the standard output
115           or standard error, in an implementation-defined format. These  sta‐
116           tistics  may  also be generated if table sizes are specified with a
117           '%' operator in the Definitions section, as long as the  -n  option
118           is not specified.
119

STDERR

121       If  the  -t  option is specified, implementation-defined informational,
122       error, and warning messages concerning the contents of lex source  code
123       input shall be written to the standard error.
124
125       If the -t option is not specified:
126
127        1. Implementation-defined  informational,  error, and warning messages
128           concerning the contents of lex source code input shall  be  written
129           to either the standard output or standard error.
130
131        2. If  the  -v option is specified and the -n option is not specified,
132           lex statistics shall also be written to either the standard  output
133           or  standard error, in an implementation-defined format. These sta‐
134           tistics may also be generated if table sizes are specified  with  a
135           '%'  operator  in the Definitions section, as long as the -n option
136           is not specified.
137

OUTPUT FILES

139       A text file containing C source code shall be written to  lex.yy.c,  or
140       to the standard output if the -t option is present.
141

EXTENDED DESCRIPTION

143       Each input file shall contain lex source code, which is a table of reg‐
144       ular expressions with corresponding actions in the form  of  C  program
145       fragments.
146
147       When  lex.yy.c  is  compiled and linked with the lex library (using the
148       -l l operand with c99), the  resulting  program  shall  read  character
149       input  from the standard input and shall partition it into strings that
150       match the given expressions.
151
152       When an expression is matched, these actions shall occur:
153
154        *  The input string that was matched shall be  left  in  yytext  as  a
155           null-terminated  string; yytext shall either be an external charac‐
156           ter array or a pointer to a character string. As explained in Defi‐
157           nitions  in  lex,  the  type  can  be explicitly selected using the
158           %array or %pointer declarations, but the default is implementation-
159           defined.
160
161        *  The  external int yyleng shall be set to the length of the matching
162           string.
163
164        *  The expression's corresponding program fragment, or  action,  shall
165           be executed.
166
167       During  pattern  matching, lex shall search the set of patterns for the
168       single longest possible match. Among rules that match the  same  number
169       of characters, the rule given first shall be chosen.
170
171       The general format of lex source shall be:
172
173              Definitions %% Rules %% UserSubroutines
174
175       The  first "%%" is required to mark the beginning of the rules (regular
176       expressions and actions); the second "%%" is required only if user sub‐
177       routines follow.
178
179       Any  line  in the Definitions section beginning with a <blank> shall be
180       assumed to be a C program fragment and shall be copied to the  external
181       definition  area of the lex.yy.c file. Similarly, anything in the Defi‐
182       nitions section included between delimiter lines containing  only  "%{"
183       and "%}" shall also be copied unchanged to the external definition area
184       of the lex.yy.c file.
185
186       Any such input (beginning with a <blank> or within "%{" and "%}" delim‐
187       iter  lines) appearing at the beginning of the Rules section before any
188       rules are specified shall be written to lex.yy.c after the declarations
189       of variables for the yylex() function and before the first line of code
190       in yylex().  Thus, user variables local  to  yylex()  can  be  declared
191       here, as well as application code to execute upon entry to yylex().
192
193       The  action  taken  by lex when encountering any input beginning with a
194       <blank> or within "%{" and "%}" delimiter lines appearing in the  Rules
195       section  but  coming after one or more rules is undefined. The presence
196       of such input may result in an  erroneous  definition  of  the  yylex()
197       function.
198
199       C-language  code  in  the input shall not contain C-language trigraphs.
200       The C-language code within "%{" and "%}" delimiter lines shall not con‐
201       tain any lines consisting only of "%}", or only of "%%".
202
203   Definitions in lex
204       Definitions  appear  before  the first "%%" delimiter. Any line in this
205       section not contained between "%{" and "%}"  lines  and  not  beginning
206       with  a  <blank>  shall be assumed to define a lex substitution string.
207       The format of these lines shall be:
208
209
210           name substitute
211
212       If a name does not meet the requirements for identifiers in  the  ISO C
213       standard,  the result is undefined. The string substitute shall replace
214       the string {name} when it is used in a rule. The name string  shall  be
215       recognized  in  this context only when the braces are provided and when
216       it does not appear within a bracket expression or within double-quotes.
217
218       In the Definitions section, any line beginning  with  a  <percent-sign>
219       ('%')  character  and  followed  by an alphanumeric word beginning with
220       either 's' or 'S' shall define a set  of  start  conditions.  Any  line
221       beginning  with  a  '%' followed by a word beginning with either 'x' or
222       'X' shall define a set of exclusive start conditions. When  the  gener‐
223       ated  scanner  is in a %s state, patterns with no state specified shall
224       be also active; in a %x state, such patterns shall not be  active.  The
225       rest  of  the line, after the first word, shall be considered to be one
226       or more <blank>-separated names of start  conditions.  Start  condition
227       names  shall  be constructed in the same way as definition names. Start
228       conditions can be used to restrict the matching of regular  expressions
229       to one or more states as described in Regular Expressions in lex.
230
231       Implementations  shall  accept  either  of  the following two mutually-
232       exclusive declarations in the Definitions section:
233
234       %array    Declare the type of yytext to be a null-terminated  character
235                 array.
236
237       %pointer  Declare  the  type of yytext to be a pointer to a null-termi‐
238                 nated character string.
239
240       The default type of yytext is implementation-defined. If an application
241       refers  to  yytext  outside of the scanner source file (that is, via an
242       extern), the  application  shall  include  the  appropriate  %array  or
243       %pointer declaration in the scanner source file.
244
245       Implementations  shall  accept  declarations in the Definitions section
246       for setting certain internal table sizes. The declarations are shown in
247       the following table.
248
249                        Table: Table Size Declarations in lex
250
251         ┌────────────┬────────────────────────────────────┬───────────────┐
252Declaration Description             Minimum Value 
253         ├────────────┼────────────────────────────────────┼───────────────┤
254         │%p n        │ Number of positions                │     2500      │
255         │%n n        │ Number of states                   │      500      │
256         │%a n        │ Number of transitions              │     2000      │
257         │%e n        │ Number of parse tree nodes         │     1000      │
258         │%k n        │ Number of packed character classes │     1000      │
259         │%o n        │ Size of the output array           │     3000      │
260         └────────────┴────────────────────────────────────┴───────────────┘
261       In  the table, n represents a positive decimal integer, preceded by one
262       or more <blank> characters. The exact meaning of these table size  num‐
263       bers  is  implementation-defined. The implementation shall document how
264       these numbers affect the lex utility and how they are  related  to  any
265       output  that  may be generated by the implementation should limitations
266       be encountered during the execution of lex.  It shall  be  possible  to
267       determine  from  this output which of the table size values needs to be
268       modified to permit lex to successfully generate tables  for  the  input
269       language.  The  values in the column Minimum Value represent the lowest
270       values conforming implementations shall provide.
271
272   Rules in lex
273       The rules in lex source files are a table in which the left column con‐
274       tains regular expressions and the right column contains actions (C pro‐
275       gram fragments) to be executed when the expressions are recognized.
276
277
278           ERE action
279           ERE action
280           ...
281
282       The extended regular expression (ERE) portion of a row shall  be  sepa‐
283       rated  from action by one or more <blank> characters. A regular expres‐
284       sion containing <blank> characters shall be recognized under one of the
285       following conditions:
286
287        *  The entire expression appears within double-quotes.
288
289        *  The <blank> characters appear within double-quotes or square brack‐
290           ets.
291
292        *  Each <blank> is preceded by a <backslash> character.
293
294   User Subroutines in lex
295       Anything in the user subroutines section shall be  copied  to  lex.yy.c
296       following yylex().
297
298   Regular Expressions in lex
299       The  lex  utility shall support the set of extended regular expressions
300       (see the Base Definitions volume of POSIX.1‐2017, Section 9.4, Extended
301       Regular  Expressions),  with  the following additions and exceptions to
302       the syntax:
303
304       "..."     Any string enclosed  in  double-quotes  shall  represent  the
305                 characters  within  the  double-quotes  as themselves, except
306                 that <backslash>-escapes (which appear in the  following  ta‐
307                 ble)  shall  be  recognized.  Any <backslash>-escape sequence
308                 shall be  terminated  by  the  closing  quote.  For  example,
309                 "\01""1"  represents  a single string: the octal value 1 fol‐
310                 lowed by the character '1'.
311
312       <state>r, <state1,state2,...>r
313                 The regular expression r shall be matched only when the  pro‐
314                 gram  is  in  one of the start conditions indicated by state,
315                 state1, and so on; see Actions in lex.  (As an  exception  to
316                 the  typographical  conventions of the rest of this volume of
317                 POSIX.1‐2017, in this  case  <state>  does  not  represent  a
318                 metavariable,  but  the literal angle-bracket characters sur‐
319                 rounding a symbol.) The start condition shall  be  recognized
320                 as such only at the beginning of a regular expression.
321
322       r/x       The  regular expression r shall be matched only if it is fol‐
323                 lowed by an occurrence of regular  expression  x  (x  is  the
324                 instance  of  trailing  context,  further defined below). The
325                 token returned in yytext shall only match r.  If the trailing
326                 portion  of  r  matches  the  beginning  of  x, the result is
327                 unspecified. The r expression cannot include further trailing
328                 context  or  the  '$'  (match-end-of-line) operator; x cannot
329                 include  the  '^'  (match-beginning-of-line)  operator,   nor
330                 trailing  context,  nor  the  '$' operator. That is, only one
331                 occurrence of trailing context is allowed in  a  lex  regular
332                 expression,  and  the  '^'  operator  only can be used at the
333                 beginning of such an expression.
334
335       {name}    When name is one of the substitution symbols from the Defini‐
336                 tions  section,  the  string, including the enclosing braces,
337                 shall be replaced by the  substitute  value.  The  substitute
338                 value  shall be treated in the extended regular expression as
339                 if it were enclosed in  parentheses.  No  substitution  shall
340                 occur  if {name} occurs within a bracket expression or within
341                 double-quotes.
342
343       Within an ERE, a <backslash> character shall be considered to begin  an
344       escape  sequence as specified in the table in the Base Definitions vol‐
345       ume of POSIX.1‐2017, Chapter 5, File Format Notation ('\\', '\a', '\b',
346       '\f',  '\n',  '\r',  '\t', '\v').  In addition, the escape sequences in
347       the following table shall be recognized.
348
349       A literal <newline> cannot occur within an  ERE;  the  escape  sequence
350       '\n'  can  be  used to represent a <newline>.  A <newline> shall not be
351       matched by a period operator.
352
353                           Table: Escape Sequences in lex
354
355          ┌─────────┬──────────────────────────┬──────────────────────────┐
356Escape  │                          │                          │
357Sequence Description        Meaning          
358          ├─────────┼──────────────────────────┼──────────────────────────┤
359          │\digits  │ A <backslash> character  │ The character whose      │
360          │         │ followed by the longest  │ encoding is represented  │
361          │         │ sequence of one, two, or │ by the one, two, or      │
362          │         │ three octal-digit char‐  │ three-digit octal inte‐  │
363          │         │ acters (01234567). If    │ ger. Multi-byte charac‐  │
364          │         │ all of the digits are 0  │ ters require multiple,   │
365          │         │ (that is, representation │ concatenated escape      │
366          │         │ of the NUL character),   │ sequences of this type,  │
367          │         │ the behavior is unde‐    │ including the leading    │
368          │         │ fined.                   │ <backslash> for each     │
369          │         │                          │ byte.                    │
370          ├─────────┼──────────────────────────┼──────────────────────────┤
371          │\xdigits │ A <backslash> character  │ The character whose      │
372          │         │ followed by the longest  │ encoding is represented  │
373          │         │ sequence of hexadecimal- │ by the hexadecimal inte‐ │
374          │         │ digit characters         │ ger.                     │
375          │         │ (01234567abcdefABCDEF).  │                          │
376          │         │ If all of the digits are │                          │
377          │         │ 0 (that is, representa‐  │                          │
378          │         │ tion of the NUL charac‐  │                          │
379          │         │ ter), the behavior is    │                          │
380          │         │ undefined.               │                          │
381          ├─────────┼──────────────────────────┼──────────────────────────┤
382          │\c       │ A <backslash> character  │ The character 'c',       │
383          │         │ followed by any charac‐  │ unchanged.               │
384          │         │ ter not described in     │                          │
385          │         │ this table or in the ta‐ │                          │
386          │         │ ble in the Base Defini‐  │                          │
387          │         │ tions volume of          │                          │
388          │         │ POSIX.1‐2017, Chapter 5, │                          │
389          │         │ File Format Notation     │                          │
390          │         │ ('\\', '\a', '\b', '\f', │                          │
391          │         │ '\n', '\r', '\t', '\v'). │                          │
392          └─────────┴──────────────────────────┴──────────────────────────┘
393       Note:     If  a  '\x'  sequence  needs  to be immediately followed by a
394                 hexadecimal digit character, a sequence such as "\x1""1"  can
395                 be used, which represents a character containing the value 1,
396                 followed by the character '1'.
397
398       The order of precedence given to extended regular expressions  for  lex
399       differs   from  that  specified  in  the  Base  Definitions  volume  of
400       POSIX.1‐2017, Section 9.4, Extended Regular Expressions.  The order  of
401       precedence  for lex shall be as shown in the following table, from high
402       to low.
403
404       Note:     The escaped characters entry is not meant to imply that these
405                 are  operators,  but  they  are included in the table to show
406                 their relationships to the true operators. The  start  condi‐
407                 tion,  trailing  context,  and  anchoring notations have been
408                 omitted from the table because of the placement  restrictions
409                 described in this section; they can only appear at the begin‐
410                 ning or ending of an ERE.
411
412                            Table: ERE Precedence in lex
413
414             ┌──────────────────────────────────┬──────────────────────┐
415Extended Regular Expression    Precedence      
416             ├──────────────────────────────────┼──────────────────────┤
417collation-related bracket symbols │ [= =]  [: :]  [. .]  │
418escaped characters                │ \<special character> │
419bracket expression                │ [ ]                  │
420quoting                           │ "..."                │
421grouping                          │ ( )                  │
422definition                        │ {name}               │
423single-character RE duplication   │ * + ?                │
424concatenation                     │                      │
425interval expression               │ {m,n}                │
426alternation                       │ |                    │
427             └──────────────────────────────────┴──────────────────────┘
428       The ERE anchoring operators '^' and '$' do not  appear  in  the  table.
429       With  lex  regular expressions, these operators are restricted in their
430       use: the '^' operator can only be used at the beginning  of  an  entire
431       regular expression, and the '$' operator only at the end. The operators
432       apply to the entire regular expression. Thus, for example, the  pattern
433       "(^abc)|(def$)" is undefined; it can instead be written as two separate
434       rules, one with the regular expression  "^abc"  and  one  with  "def$",
435       which  share a common action via the special '|' action (see below). If
436       the pattern were written "^abc|def$", it would match  either  "abc"  or
437       "def" on a line by itself.
438
439       Unlike the general ERE rules, embedded anchoring is not allowed by most
440       historical lex implementations. An example of embedded anchoring  would
441       be  for  patterns such as "(^| )foo( |$)" to match "foo" when it exists
442       as a complete word. This functionality can be obtained  using  existing
443       lex features:
444
445
446           ^foo/[ \n]      |
447           " foo"/[ \n]    /* Found foo as a separate word. */
448
449       Note  also  that '$' is a form of trailing context (it is equivalent to
450       "/\n") and as such cannot be used with regular  expressions  containing
451       another  instance  of  the  operator  (see  the preceding discussion of
452       trailing context).
453
454       The additional regular expressions trailing-context operator '/' can be
455       used  as  an ordinary character if presented within double-quotes, "/";
456       preceded by a <backslash>, "\/"; or within a bracket expression, "[/]".
457       The  start-condition  '<'  and '>' operators shall be special only in a
458       start condition at the beginning of a regular expression; elsewhere  in
459       the regular expression they shall be treated as ordinary characters.
460
461   Actions in lex
462       The  action to be taken when an ERE is matched can be a C program frag‐
463       ment or the special actions described below; the program  fragment  can
464       contain one or more C statements, and can also include special actions.
465       The empty C statement ';' shall be a valid action; any  string  in  the
466       lex.yy.c  input  that  matches  the  pattern  portion of such a rule is
467       effectively ignored or skipped. However, the absence of an action shall
468       not  be  valid,  and  the action lex takes in such a condition is unde‐
469       fined.
470
471       The specification for an action, including  C  statements  and  special
472       actions, can extend across several lines if enclosed in braces:
473
474
475           ERE <one or more blanks> { program statement
476                                      program statement }
477
478       The program statements shall not contain unbalanced curly brace prepro‐
479       cessing tokens.
480
481       The default action when a string in the input to a lex.yy.c program  is
482       not  matched  by any expression shall be to copy the string to the out‐
483       put. Because the default behavior of a program generated by lex  is  to
484       read  the input and copy it to the output, a minimal lex source program
485       that has just "%%" shall generate a C program that  simply  copies  the
486       input to the output unchanged.
487
488       Four special actions shall be available:
489
490
491           |   ECHO;   REJECT;   BEGIN
492
493       |         The action '|' means that the action for the next rule is the
494                 action for this rule.  Unlike the other  three  actions,  '|'
495                 cannot  be  enclosed  in braces or be <semicolon>-terminated;
496                 the application shall ensure that it is specified alone, with
497                 no other actions.
498
499       ECHO;     Write the contents of the string yytext on the output.
500
501       REJECT;   Usually only a single expression is matched by a given string
502                 in the input.  REJECT means ``continue to the next expression
503                 that  matches  the  current input'', and shall cause whatever
504                 rule was the second choice after the current rule to be  exe‐
505                 cuted for the same input. Thus, multiple rules can be matched
506                 and executed  for  one  input  string  or  overlapping  input
507                 strings. For example, given the regular expressions "xyz" and
508                 "xy" and the input "xyz", usually only the regular expression
509                 "xyz" would match. The next attempted match would start after
510                 z.  If the last action in the "xyz" rule is REJECT, both this
511                 rule  and  the "xy" rule would be executed. The REJECT action
512                 may be implemented in such a fashion  that  flow  of  control
513                 does  not  continue  after  it, as if it were equivalent to a
514                 goto to another part of  yylex().   The  use  of  REJECT  may
515                 result in somewhat larger and slower scanners.
516
517       BEGIN     The action:
518
519
520                     BEGIN newstate;
521
522                 switches  the  state  (start  condition) to newstate.  If the
523                 string newstate has not been declared previously as  a  start
524                 condition in the Definitions section, the results are unspec‐
525                 ified. The initial state is indicated by the digit '0' or the
526                 token INITIAL.
527
528       The  functions  or  macros  described below are accessible to user code
529       included in the lex input. It is unspecified whether they appear in the
530       C  code  output of lex, or are accessible only through the -l l operand
531       to c99 (the lex library).
532
533       int yylex(void)
534             Performs lexical analysis on the input; this is the primary func‐
535             tion generated by the lex utility. The function shall return zero
536             when the end of input is reached; otherwise, it shall return non-
537             zero values (tokens) determined by the actions that are selected.
538
539       int yymore(void)
540             When  called, indicates that when the next input string is recog‐
541             nized, it is to be appended to the current value of yytext rather
542             than  replacing it; the value in yyleng shall be adjusted accord‐
543             ingly.
544
545       int yyless(int n)
546             Retains n  initial  characters  in  yytext,  NUL-terminated,  and
547             treats the remaining characters as if they had not been read; the
548             value in yyleng shall be adjusted accordingly.
549
550       int input(void)
551             Returns the next character from the input,  or  zero  on  end-of-
552             file.  It  shall  obtain  input  from  the  stream  pointer yyin,
553             although possibly via an intermediate buffer. Thus, once scanning
554             has begun, the effect of altering the value of yyin is undefined.
555             The character read shall be removed from the input stream of  the
556             scanner without any processing by the scanner.
557
558       int unput(int c)
559             Returns  the  character  'c'  to the input; yytext and yyleng are
560             undefined until the next expression is  matched.  The  result  of
561             using unput() for more characters than have been input is unspec‐
562             ified.
563
564       The following functions shall appear only in the lex library accessible
565       through the -l l operand; they can therefore be redefined by a conform‐
566       ing application:
567
568       int yywrap(void)
569             Called by yylex() at  end-of-file;  the  default  yywrap()  shall
570             always  return 1. If the application requires yylex() to continue
571             processing with another source of input, then the application can
572             include  a  function yywrap(), which associates another file with
573             the external variable FILE * yyin and shall  return  a  value  of
574             zero.
575
576       int main(int argc, char *argv[])
577             Calls  yylex()  to perform lexical analysis, then exits. The user
578             code can contain main() to  perform  application-specific  opera‐
579             tions, calling yylex() as applicable.
580
581       Except  for input(), unput(), and main(), all external and static names
582       generated by lex shall begin with the prefix yy or YY.
583

EXIT STATUS

585       The following exit values shall be returned:
586
587        0    Successful completion.
588
589       >0    An error occurred.
590

CONSEQUENCES OF ERRORS

592       Default.
593
594       The following sections are informative.
595

APPLICATION USAGE

597       Conforming applications are warned that in the Rules  section,  an  ERE
598       without  an action is not acceptable, but need not be detected as erro‐
599       neous by lex.  This may result in compilation or runtime errors.
600
601       The purpose of input() is to take characters off the input  stream  and
602       discard  them as far as the lexical analysis is concerned. A common use
603       is to discard the body of a comment once the beginning of a comment  is
604       recognized.
605
606       The lex utility is not fully internationalized in its treatment of reg‐
607       ular expressions in the lex source code or generated lexical  analyzer.
608       It would seem desirable to have the lexical analyzer interpret the reg‐
609       ular expressions given in the lex source according to  the  environment
610       specified when the lexical analyzer is executed, but this is not possi‐
611       ble with the current lex technology. Furthermore, the  very  nature  of
612       the lexical analyzers produced by lex must be closely tied to the lexi‐
613       cal requirements of the input language being described, which  is  fre‐
614       quently  locale-specific anyway. (For example, writing an analyzer that
615       is used for French text is  not  automatically  useful  for  processing
616       other languages.)
617

EXAMPLES

619       The following is an example of a lex program that implements a rudimen‐
620       tary scanner for a Pascal-like syntax:
621
622
623           %{
624           /* Need this for the call to atof() below. */
625           #include <math.h>
626           /* Need this for printf(), fopen(), and stdin below. */
627           #include <stdio.h>
628           %}
629
630           DIGIT    [0-9]
631           ID       [a-z][a-z0-9]*
632
633           %%
634
635           {DIGIT}+ {
636               printf("An integer: %s (%d)\n", yytext,
637                   atoi(yytext));
638               }
639
640           {DIGIT}+"."{DIGIT}*        {
641               printf("A float: %s (%g)\n", yytext,
642                   atof(yytext));
643               }
644
645           if|then|begin|end|procedure|function        {
646               printf("A keyword: %s\n", yytext);
647               }
648
649           {ID}    printf("An identifier: %s\n", yytext);
650
651           "+"|"-"|"*"|"/"        printf("An operator: %s\n", yytext);
652
653           "{"[^}\n]*"}"    /* Eat up one-line comments. */
654
655           [ \t\n]+        /* Eat up white space. */
656
657           .  printf("Unrecognized character: %s\n", yytext);
658
659           %%
660
661           int main(int argc, char *argv[])
662           {
663               ++argv, --argc;  /* Skip over program name. */
664               if (argc > 0)
665                   yyin = fopen(argv[0], "r");
666               else
667                   yyin = stdin;
668
669               yylex();
670           }
671

RATIONALE

673       Even though the -c option and references to the C language are retained
674       in  this description, lex may be generalized to other languages, as was
675       done at one time for EFL, the Extended FORTRAN Language. Since the  lex
676       input  specification  is  essentially language-independent, versions of
677       this utility could be written to produce Ada, Modula-2, or Pascal code,
678       and there are known historical implementations that do so.
679
680       The  current  description  of  lex  bypasses  the issue of dealing with
681       internationalized EREs in the lex source code or generated lexical ana‐
682       lyzer.  If it follows the model used by awk (the source code is assumed
683       to be presented in the POSIX locale, but input and output  are  in  the
684       locale  specified by the environment variables), then the tables in the
685       lexical analyzer produced by lex would interpret EREs specified in  the
686       lex source in terms of the environment variables specified when lex was
687       executed. The desired effect would be  to  have  the  lexical  analyzer
688       interpret the EREs given in the lex source according to the environment
689       specified when the lexical analyzer is executed, but this is not possi‐
690       ble with the current lex technology.
691
692       The  description of octal and hexadecimal-digit escape sequences agrees
693       with the ISO C standard use of escape sequences.
694
695       Earlier versions of this  standard  allowed  for  implementations  with
696       bytes  other  than  eight bits, but this has been modified in this ver‐
697       sion.
698
699       There is no detailed output format specification. The observed behavior
700       of lex under four different historical implementations was that none of
701       these implementations consistently reported the line numbers for  error
702       and  warning  messages.  Furthermore,  there  was  a desire that lex be
703       allowed to output additional diagnostic messages. Leaving message  for‐
704       mats  unspecified  avoids  these formatting questions and problems with
705       internationalization.
706
707       Although the %x specifier for exclusive start conditions is not histor‐
708       ical practice, it is believed to be a minor change to historical imple‐
709       mentations and greatly enhances the usability of lex programs since  it
710       permits  an application to obtain the expected functionality with fewer
711       statements.
712
713       The %array and %pointer declarations were added as a compromise between
714       historical  systems.  The System V-based lex copies the matched text to
715       a yytext array. The flex program, supported in  BSD  and  GNU  systems,
716       uses  a  pointer.  In the latter case, significant performance improve‐
717       ments are available for some scanners. Most historical programs  should
718       require  no  change  in  porting from one system to another because the
719       string being referenced is null-terminated in both cases.  (The  method
720       used  by  flex  in  its case is to null-terminate the token in place by
721       remembering the character that used to come right after the  token  and
722       replacing  it  before  continuing on to the next scan.) Multi-file pro‐
723       grams with external references to yytext  outside  the  scanner  source
724       file  should continue to operate on their historical systems, but would
725       require one of the new declarations to be considered strictly portable.
726
727       The description of EREs avoids unnecessary duplication of  ERE  details
728       because  their  meanings  within a lex ERE are the same as that for the
729       ERE in this volume of POSIX.1‐2017.
730
731       The reason for the undefined condition associated with  text  beginning
732       with a <blank> or within "%{" and "%}" delimiter lines appearing in the
733       Rules section is historical practice. Both the BSD  and  System  V  lex
734       copy  the  indented (or enclosed) input in the Rules section (except at
735       the beginning) to unreachable areas of the yylex() function  (the  code
736       is written directly after a break statement). In some cases, the System
737       V lex generates an error message or a syntax error,  depending  on  the
738       form of indented input.
739
740       The  intention  in  breaking  the list of functions into those that may
741       appear in lex.yy.c versus those that only appear in libl.a is that only
742       those  functions  in  libl.a  can be reliably redefined by a conforming
743       application.
744
745       The descriptions of standard output and  standard  error  are  somewhat
746       complicated because historical lex implementations chose to issue diag‐
747       nostic messages to standard output (unless -t was given).  POSIX.1‐2008
748       allows  this  behavior,  but  leaves  an  opening for the more expected
749       behavior of using standard error for diagnostics.  Also, the  System  V
750       behavior  of  writing  the statistics when any table sizes are given is
751       allowed, while BSD-derived systems can avoid  it.  The  programmer  can
752       always  precisely  obtain the desired results by using either the -t or
753       -n options.
754
755       The OPERANDS section does not mention the use of -  as  a  synonym  for
756       standard  input;  not all historical implementations support such usage
757       for any of the file operands.
758
759       A description of the translation table was deleted from early proposals
760       because of its relatively low usage in historical applications.
761
762       The  change  to  the  definition  of  the  input() function that allows
763       buffering of input presents the opportunity for major performance gains
764       in some applications.
765
766       The  following  examples  clarify  the  differences between lex regular
767       expressions and regular expressions appearing elsewhere in this  volume
768       of  POSIX.1‐2017. For regular expressions of the form "r/x", the string
769       matching r is always returned; confusion may arise when  the  beginning
770       of x matches the trailing portion of r.  For example, given the regular
771       expression "a*b/cc" and the input "aaabcc", yytext  would  contain  the
772       string  "aaab"  on this match. But given the regular expression "x*/xy"
773       and the input "xxxy", the token xxx, not xx, is returned by some imple‐
774       mentations because xxx matches "x*".
775
776       In  the  rule "ab*/bc", the "b*" at the end of r extends r's match into
777       the beginning of the trailing context, so the result is unspecified. If
778       this rule were "ab/bc", however, the rule matches the text "ab" when it
779       is followed by the text "bc".  In this latter case, the matching  of  r
780       cannot extend into the beginning of x, so the result is specified.
781

FUTURE DIRECTIONS

783       None.
784

SEE ALSO

786       c99, ed, yacc
787
788       The  Base  Definitions  volume  of POSIX.1‐2017, Chapter 5, File Format
789       Notation, Chapter 8, Environment Variables, Chapter 9, Regular  Expres‐
790       sions, Section 12.2, Utility Syntax Guidelines
791
793       Portions  of  this text are reprinted and reproduced in electronic form
794       from IEEE Std 1003.1-2017, Standard for Information Technology --  Por‐
795       table  Operating System Interface (POSIX), The Open Group Base Specifi‐
796       cations Issue 7, 2018 Edition, Copyright (C) 2018 by the  Institute  of
797       Electrical  and  Electronics Engineers, Inc and The Open Group.  In the
798       event of any discrepancy between this version and the original IEEE and
799       The  Open Group Standard, the original IEEE and The Open Group Standard
800       is the referee document. The original Standard can be  obtained  online
801       at http://www.opengroup.org/unix/online.html .
802
803       Any  typographical  or  formatting  errors that appear in this page are
804       most likely to have been introduced during the conversion of the source
805       files  to  man page format. To report such errors, see https://www.ker
806       nel.org/doc/man-pages/reporting_bugs.html .
807
808
809
810IEEE/The Open Group                  2017                              LEX(1P)
Impressum