1LEX(1P)                    POSIX Programmer's Manual                   LEX(1P)
2
3
4

PROLOG

6       This  manual  page is part of the POSIX Programmer's Manual.  The Linux
7       implementation of this interface may differ (consult the  corresponding
8       Linux  manual page for details of Linux behavior), or the interface may
9       not be implemented on Linux.
10
11

NAME

13       lex — generate programs for lexical tasks (DEVELOPMENT)
14

SYNOPSIS

16       lex [−t] [−n|−v] [file...]
17

DESCRIPTION

19       The lex utility shall generate C programs to be used  in  lexical  pro‐
20       cessing  of  character  input,  and that can be used as an interface to
21       yacc.  The C programs shall be generated from lex source code and  con‐
22       form to the ISO C standard, without depending on any undefined, unspec‐
23       ified, or implementation-defined behavior, except in  cases  where  the
24       code  is copied directly from the supplied source, or in cases that are
25       documented by the implementation. Usually, the lex utility shall  write
26       the  program  it generates to the file lex.yy.c; the state of this file
27       is unspecified if lex exits  with  a  non-zero  exit  status.  See  the
28       EXTENDED  DESCRIPTION  section  for  a  complete description of the lex
29       input language.
30

OPTIONS

32       The lex utility  shall  conform  to  the  Base  Definitions  volume  of
33       POSIX.1‐2008,  Section  12.2,  Utility  Syntax  Guidelines,  except for
34       Guideline 9.
35
36       The following options shall be supported:
37
38       −n        Suppress the summary of statistics usually written  with  the
39                 −v  option. If no table sizes are specified in the lex source
40                 code and the −v option is not specified, then −n is implied.
41
42       −t        Write the resulting program to  standard  output  instead  of
43                 lex.yy.c.
44
45       −v        Write  a  summary  of  lex statistics to the standard output.
46                 (See the discussion of lex  table  sizes  in  Definitions  in
47                 lex.)  If the −t option is specified and −n is not specified,
48                 this report shall be written  to  standard  error.  If  table
49                 sizes  are  specified  in  the lex source code, and if the −n
50                 option is not specified, the −v option may be enabled.
51

OPERANDS

53       The following operand shall be supported:
54
55       file      A pathname of an input file. If more than one  such  file  is
56                 specified,  all files shall be concatenated to produce a sin‐
57                 gle lex program. If no file operands are specified, or  if  a
58                 file operand is '−', the standard input shall be used.
59

STDIN

61       The  standard input shall be used if no file operands are specified, or
62       if a file operand is '−'.  See INPUT FILES.
63

INPUT FILES

65       The input files shall be text files  containing  lex  source  code,  as
66       described in the EXTENDED DESCRIPTION section.
67

ENVIRONMENT VARIABLES

69       The following environment variables shall affect the execution of lex:
70
71       LANG      Provide  a  default  value for the internationalization vari‐
72                 ables that are unset or null. (See the Base Definitions  vol‐
73                 ume  of POSIX.1‐2008, Section 8.2, Internationalization Vari‐
74                 ables for the precedence  of  internationalization  variables
75                 used to determine the values of locale categories.)
76
77       LC_ALL    If  set  to  a non-empty string value, override the values of
78                 all the other internationalization variables.
79
80       LC_COLLATE
81                 Determine the locale for the behavior of ranges,  equivalence
82                 classes,  and multi-character collating elements within regu‐
83                 lar expressions. If this variable is not  set  to  the  POSIX
84                 locale, the results are unspecified.
85
86       LC_CTYPE  Determine  the  locale for the interpretation of sequences of
87                 bytes of text data as characters (for example, single-byte as
88                 opposed  to  multi-byte  characters  in  arguments  and input
89                 files), and the behavior of character classes within  regular
90                 expressions. If this variable is not set to the POSIX locale,
91                 the results are unspecified.
92
93       LC_MESSAGES
94                 Determine the locale that should be used to affect the format
95                 and  contents  of  diagnostic  messages  written  to standard
96                 error.
97
98       NLSPATH   Determine the location of message catalogs for the processing
99                 of LC_MESSAGES.
100

ASYNCHRONOUS EVENTS

102       Default.
103

STDOUT

105       If the −t option is specified, the text file of C source code output of
106       lex shall be written to standard output.
107
108       If the −t option is not specified:
109
110        *  Implementation-defined informational, error, and  warning  messages
111           concerning  the  contents of lex source code input shall be written
112           to either the standard output or standard error.
113
114        *  If the −v option is specified and the −n option is  not  specified,
115           lex  statistics shall also be written to either the standard output
116           or standard error, in an implementation-defined format. These  sta‐
117           tistics  may  also be generated if table sizes are specified with a
118           '%' operator in the Definitions section, as long as the  −n  option
119           is not specified.
120

STDERR

122       If  the  −t  option is specified, implementation-defined informational,
123       error, and warning messages concerning the contents of lex source  code
124       input shall be written to the standard error.
125
126       If the −t option is not specified:
127
128        1. Implementation-defined  informational,  error, and warning messages
129           concerning the contents of lex source code input shall  be  written
130           to either the standard output or standard error.
131
132        2. If  the  −v option is specified and the −n option is not specified,
133           lex statistics shall also be written to either the standard  output
134           or  standard error, in an implementation-defined format. These sta‐
135           tistics may also be generated if table sizes are specified  with  a
136           '%'  operator  in the Definitions section, as long as the −n option
137           is not specified.
138

OUTPUT FILES

140       A text file containing C source code shall be written to  lex.yy.c,  or
141       to the standard output if the −t option is present.
142

EXTENDED DESCRIPTION

144       Each input file shall contain lex source code, which is a table of reg‐
145       ular expressions with corresponding actions in the form  of  C  program
146       fragments.
147
148       When  lex.yy.c  is  compiled and linked with the lex library (using the
149       −l l operand with c99), the  resulting  program  shall  read  character
150       input  from the standard input and shall partition it into strings that
151       match the given expressions.
152
153       When an expression is matched, these actions shall occur:
154
155        *  The input string that was matched shall be  left  in  yytext  as  a
156           null-terminated  string; yytext shall either be an external charac‐
157           ter array or a pointer to a character string. As explained in Defi‐
158           nitions  in  lex,  the  type  can  be explicitly selected using the
159           %array or %pointer declarations, but the default is implementation-
160           defined.
161
162        *  The  external int yyleng shall be set to the length of the matching
163           string.
164
165        *  The expression's corresponding program fragment, or  action,  shall
166           be executed.
167
168       During  pattern  matching, lex shall search the set of patterns for the
169       single longest possible match. Among rules that match the  same  number
170       of characters, the rule given first shall be chosen.
171
172       The general format of lex source shall be:
173
174              Definitions %% Rules %% UserSubroutines
175
176       The  first "%%" is required to mark the beginning of the rules (regular
177       expressions and actions); the second "%%" is required only if user sub‐
178       routines follow.
179
180       Any  line  in the Definitions section beginning with a <blank> shall be
181       assumed to be a C program fragment and shall be copied to the  external
182       definition  area of the lex.yy.c file. Similarly, anything in the Defi‐
183       nitions section included between delimiter lines containing  only  "%{"
184       and "%}" shall also be copied unchanged to the external definition area
185       of the lex.yy.c file.
186
187       Any such input (beginning with a <blank> or within "%{" and "%}" delim‐
188       iter  lines) appearing at the beginning of the Rules section before any
189       rules are specified shall be written to lex.yy.c after the declarations
190       of variables for the yylex() function and before the first line of code
191       in yylex().  Thus, user variables local  to  yylex()  can  be  declared
192       here, as well as application code to execute upon entry to yylex().
193
194       The  action  taken  by lex when encountering any input beginning with a
195       <blank> or within "%{" and "%}" delimiter lines appearing in the  Rules
196       section  but  coming after one or more rules is undefined. The presence
197       of such input may result in an  erroneous  definition  of  the  yylex()
198       function.
199
200       C-language  code  in  the input shall not contain C-language trigraphs.
201       The C-language code within "%{" and "%}" delimiter lines shall not con‐
202       tain any lines consisting only of "%}", or only of "%%".
203
204   Definitions in lex
205       Definitions  appear  before  the first "%%" delimiter. Any line in this
206       section not contained between "%{" and "%}"  lines  and  not  beginning
207       with  a  <blank>  shall be assumed to define a lex substitution string.
208       The format of these lines shall be:
209
210           name substitute
211
212       If a name does not meet the requirements for identifiers in  the  ISO C
213       standard,  the result is undefined. The string substitute shall replace
214       the string {name} when it is used in a rule. The name string  shall  be
215       recognized  in  this context only when the braces are provided and when
216       it does not appear within a bracket expression or within double-quotes.
217
218       In the Definitions section, any line beginning  with  a  <percent-sign>
219       ('%')  character  and  followed  by an alphanumeric word beginning with
220       either 's' or 'S' shall define a set  of  start  conditions.  Any  line
221       beginning  with  a  '%' followed by a word beginning with either 'x' or
222       'X' shall define a set of exclusive start conditions. When  the  gener‐
223       ated  scanner  is in a %s state, patterns with no state specified shall
224       be also active; in a %x state, such patterns shall not be  active.  The
225       rest  of  the line, after the first word, shall be considered to be one
226       or more <blank>-separated names of start  conditions.  Start  condition
227       names  shall  be constructed in the same way as definition names. Start
228       conditions can be used to restrict the matching of regular  expressions
229       to one or more states as described in Regular Expressions in lex.
230
231       Implementations  shall  accept  either  of  the following two mutually-
232       exclusive declarations in the Definitions section:
233
234       %array    Declare the type of yytext to be a null-terminated  character
235                 array.
236
237       %pointer  Declare  the  type of yytext to be a pointer to a null-termi‐
238                 nated character string.
239
240       The default type of yytext is implementation-defined. If an application
241       refers  to  yytext  outside of the scanner source file (that is, via an
242       extern), the  application  shall  include  the  appropriate  %array  or
243       %pointer declaration in the scanner source file.
244
245       Implementations  shall  accept  declarations in the Definitions section
246       for setting certain internal table sizes. The declarations are shown in
247       the following table.
248
249                        Table: Table Size Declarations in lex
250
251         ┌────────────┬────────────────────────────────────┬───────────────┐
252Declaration Description             Minimum Value 
253         ├────────────┼────────────────────────────────────┼───────────────┤
254         │%p n        │ Number of positions                │     2500      │
255         │%n n        │ Number of states                   │      500      │
256         │%a n        │ Number of transitions              │     2000      │
257         │%e n        │ Number of parse tree nodes         │     1000      │
258         │%k n        │ Number of packed character classes │     1000      │
259         │%o n        │ Size of the output array           │     3000      │
260         └────────────┴────────────────────────────────────┴───────────────┘
261       In  the table, n represents a positive decimal integer, preceded by one
262       or more <blank> characters. The exact meaning of these table size  num‐
263       bers  is  implementation-defined. The implementation shall document how
264       these numbers affect the lex utility and how they are  related  to  any
265       output  that  may be generated by the implementation should limitations
266       be encountered during the execution of lex.  It shall  be  possible  to
267       determine  from  this output which of the table size values needs to be
268       modified to permit lex to successfully generate tables  for  the  input
269       language.  The  values in the column Minimum Value represent the lowest
270       values conforming implementations shall provide.
271
272   Rules in lex
273       The rules in lex source files are a table in which the left column con‐
274       tains regular expressions and the right column contains actions (C pro‐
275       gram fragments) to be executed when the expressions are recognized.
276
277           ERE action
278           ERE action
279           ...
280
281       The extended regular expression (ERE) portion of a row shall  be  sepa‐
282       rated  from action by one or more <blank> characters. A regular expres‐
283       sion containing <blank> characters shall be recognized under one of the
284       following conditions:
285
286        *  The entire expression appears within double-quotes.
287
288        *  The <blank> characters appear within double-quotes or square brack‐
289           ets.
290
291        *  Each <blank> is preceded by a <backslash> character.
292
293   User Subroutines in lex
294       Anything in the user subroutines section shall be  copied  to  lex.yy.c
295       following yylex().
296
297   Regular Expressions in lex
298       The  lex  utility shall support the set of extended regular expressions
299       (see the Base Definitions volume of POSIX.1‐2008, Section 9.4, Extended
300       Regular  Expressions),  with  the following additions and exceptions to
301       the syntax:
302
303       "..."     Any string enclosed  in  double-quotes  shall  represent  the
304                 characters  within  the  double-quotes  as themselves, except
305                 that <backslash>-escapes (which appear in the  following  ta‐
306                 ble)  shall  be  recognized.  Any <backslash>-escape sequence
307                 shall be  terminated  by  the  closing  quote.  For  example,
308                 "\01""1"  represents  a single string: the octal value 1 fol‐
309                 lowed by the character '1'.
310
311       <state>r, <state1,state2,...>r
312                 The regular expression r shall be matched only when the  pro‐
313                 gram  is  in  one of the start conditions indicated by state,
314                 state1, and so on; see Actions in lex.  (As an  exception  to
315                 the  typographical  conventions of the rest of this volume of
316                 POSIX.1‐2008, in this  case  <state>  does  not  represent  a
317                 metavariable,  but  the literal angle-bracket characters sur‐
318                 rounding a symbol.) The start condition shall  be  recognized
319                 as such only at the beginning of a regular expression.
320
321       r/x       The  regular expression r shall be matched only if it is fol‐
322                 lowed by an occurrence of regular  expression  x  (x  is  the
323                 instance  of  trailing  context,  further defined below). The
324                 token returned in yytext shall only match r.  If the trailing
325                 portion  of  r  matches  the  beginning  of  x, the result is
326                 unspecified. The r expression cannot include further trailing
327                 context  or  the  '$'  (match-end-of-line) operator; x cannot
328                 include  the  '^'  (match-beginning-of-line)  operator,   nor
329                 trailing  context,  nor  the  '$' operator. That is, only one
330                 occurrence of trailing context is allowed in  a  lex  regular
331                 expression,  and  the  '^'  operator  only can be used at the
332                 beginning of such an expression.
333
334       {name}    When name is one of the substitution symbols from the Defini‐
335                 tions  section,  the  string, including the enclosing braces,
336                 shall be replaced by the  substitute  value.  The  substitute
337                 value  shall be treated in the extended regular expression as
338                 if it were enclosed in  parentheses.  No  substitution  shall
339                 occur  if {name} occurs within a bracket expression or within
340                 double-quotes.
341
342       Within an ERE, a <backslash> character shall be considered to begin  an
343       escape  sequence as specified in the table in the Base Definitions vol‐
344       ume of POSIX.1‐2008, Chapter 5, File Format Notation ('\\', '\a', '\b',
345       '\f',  '\n',  '\r',  '\t', '\v').  In addition, the escape sequences in
346       the following table shall be recognized.
347
348       A literal <newline> cannot occur within an  ERE;  the  escape  sequence
349       '\n'  can  be  used to represent a <newline>.  A <newline> shall not be
350       matched by a period operator.
351
352                           Table: Escape Sequences in lex
353
354          ┌─────────┬──────────────────────────┬──────────────────────────┐
355Escape  │                          │                          │
356Sequence Description        Meaning          
357          ├─────────┼──────────────────────────┼──────────────────────────┤
358          │\digits  │ A <backslash> character  │ The character whose      │
359          │         │ followed by the longest  │ encoding is represented  │
360          │         │ sequence of one, two, or │ by the one, two, or      │
361          │         │ three octal-digit char‐  │ three-digit octal inte‐  │
362          │         │ acters (01234567). If    │ ger. Multi-byte charac‐  │
363          │         │ all of the digits are 0  │ ters require multiple,   │
364          │         │ (that is, representation │ concatenated escape      │
365          │         │ of the NUL character),   │ sequences of this type,  │
366          │         │ the behavior is unde‐    │ including the leading    │
367          │         │ fined.                   │ <backslash> for each     │
368          │         │                          │ byte.                    │
369          ├─────────┼──────────────────────────┼──────────────────────────┤
370          │\xdigits │ A <backslash> character  │ The character whose      │
371          │         │ followed by the longest  │ encoding is represented  │
372          │         │ sequence of hexadecimal- │ by the hexadecimal inte‐ │
373          │         │ digit characters         │ ger.                     │
374          │         │ (01234567abcdefABCDEF).  │                          │
375          │         │ If all of the digits are │                          │
376          │         │ 0 (that is, representa‐  │                          │
377          │         │ tion of the NUL charac‐  │                          │
378          │         │ ter), the behavior is    │                          │
379          │         │ undefined.               │                          │
380          ├─────────┼──────────────────────────┼──────────────────────────┤
381          │\c       │ A <backslash> character  │ The character 'c',       │
382          │         │ followed by any charac‐  │ unchanged.               │
383          │         │ ter not described in     │                          │
384          │         │ this table or in the ta‐ │                          │
385          │         │ ble in the Base Defini‐  │                          │
386          │         │ tions volume of          │                          │
387          │         │ POSIX.1‐2008, Chapter 5, │                          │
388          │         │ File Format Notation     │                          │
389          │         │ ('\\', '\a', '\b', '\f', │                          │
390          │         │ '\n', '\r', '\t', '\v'). │                          │
391          └─────────┴──────────────────────────┴──────────────────────────┘
392       Note:     If  a  '\x'  sequence  needs  to be immediately followed by a
393                 hexadecimal digit character, a sequence such as "\x1""1"  can
394                 be used, which represents a character containing the value 1,
395                 followed by the character '1'.
396
397       The order of precedence given to extended regular expressions  for  lex
398       differs   from  that  specified  in  the  Base  Definitions  volume  of
399       POSIX.1‐2008, Section 9.4, Extended Regular Expressions.  The order  of
400       precedence  for lex shall be as shown in the following table, from high
401       to low.
402
403       Note:     The escaped characters entry is not meant to imply that these
404                 are  operators,  but  they  are included in the table to show
405                 their relationships to the true operators. The  start  condi‐
406                 tion,  trailing  context,  and  anchoring notations have been
407                 omitted from the table because of the placement  restrictions
408                 described in this section; they can only appear at the begin‐
409                 ning or ending of an ERE.
410
411                            Table: ERE Precedence in lex
412
413             ┌──────────────────────────────────┬──────────────────────┐
414Extended Regular Expression    Precedence      
415             ├──────────────────────────────────┼──────────────────────┤
416collation-related bracket symbols │ [= =]  [: :]  [. .]  │
417escaped characters                │ \<special character> │
418bracket expression                │ [ ]                  │
419quoting                           │ "..."                │
420grouping                          │ ( )                  │
421definition                        │ {name}               │
422single-character RE duplication   │ * + ?                │
423concatenation                     │                      │
424interval expression               │ {m,n}                │
425alternation                       │ |                    │
426             └──────────────────────────────────┴──────────────────────┘
427       The ERE anchoring operators '^' and '$' do not  appear  in  the  table.
428       With  lex  regular expressions, these operators are restricted in their
429       use: the '^' operator can only be used at the beginning  of  an  entire
430       regular expression, and the '$' operator only at the end. The operators
431       apply to the entire regular expression. Thus, for example, the  pattern
432       "(^abc)|(def$)" is undefined; it can instead be written as two separate
433       rules, one with the regular expression  "^abc"  and  one  with  "def$",
434       which  share a common action via the special '|' action (see below). If
435       the pattern were written "^abc|def$", it would match  either  "abc"  or
436       "def" on a line by itself.
437
438       Unlike the general ERE rules, embedded anchoring is not allowed by most
439       historical lex implementations. An example of embedded anchoring  would
440       be  for  patterns such as "(^| )foo( |$)" to match "foo" when it exists
441       as a complete word. This functionality can be obtained  using  existing
442       lex features:
443
444           ^foo/[ \n]      |
445           " foo"/[ \n]    /* Found foo as a separate word. */
446
447       Note  also  that '$' is a form of trailing context (it is equivalent to
448       "/\n") and as such cannot be used with regular  expressions  containing
449       another  instance  of  the  operator  (see  the preceding discussion of
450       trailing context).
451
452       The additional regular expressions trailing-context operator '/' can be
453       used  as  an ordinary character if presented within double-quotes, "/";
454       preceded by a <backslash>, "\/"; or within a bracket expression, "[/]".
455       The  start-condition  '<'  and '>' operators shall be special only in a
456       start condition at the beginning of a regular expression; elsewhere  in
457       the regular expression they shall be treated as ordinary characters.
458
459   Actions in lex
460       The  action to be taken when an ERE is matched can be a C program frag‐
461       ment or the special actions described below; the program  fragment  can
462       contain one or more C statements, and can also include special actions.
463       The empty C statement ';' shall be a valid action; any  string  in  the
464       lex.yy.c  input  that  matches  the  pattern  portion of such a rule is
465       effectively ignored or skipped. However, the absence of an action shall
466       not  be  valid,  and  the action lex takes in such a condition is unde‐
467       fined.
468
469       The specification for an action, including  C  statements  and  special
470       actions, can extend across several lines if enclosed in braces:
471
472           ERE <one or more blanks> { program statement
473                                      program statement }
474
475       The program statements shall not contain unbalanced curly brace prepro‐
476       cessing tokens.
477
478       The default action when a string in the input to a lex.yy.c program  is
479       not  matched  by any expression shall be to copy the string to the out‐
480       put. Because the default behavior of a program generated by lex  is  to
481       read  the input and copy it to the output, a minimal lex source program
482       that has just "%%" shall generate a C program that  simply  copies  the
483       input to the output unchanged.
484
485       Four special actions shall be available:
486
487           |   ECHO;   REJECT;   BEGIN
488
489       |         The action '|' means that the action for the next rule is the
490                 action for this rule.  Unlike the other  three  actions,  '|'
491                 cannot  be  enclosed  in braces or be <semicolon>-terminated;
492                 the application shall ensure that it is specified alone, with
493                 no other actions.
494
495       ECHO;     Write the contents of the string yytext on the output.
496
497       REJECT;   Usually only a single expression is matched by a given string
498                 in the input.  REJECT means ``continue to the next expression
499                 that  matches  the  current input'', and shall cause whatever
500                 rule was the second choice after the current rule to be  exe‐
501                 cuted for the same input. Thus, multiple rules can be matched
502                 and executed  for  one  input  string  or  overlapping  input
503                 strings. For example, given the regular expressions "xyz" and
504                 "xy" and the input "xyz", usually only the regular expression
505                 "xyz" would match. The next attempted match would start after
506                 z.  If the last action in the "xyz" rule is REJECT, both this
507                 rule  and  the "xy" rule would be executed. The REJECT action
508                 may be implemented in such a fashion  that  flow  of  control
509                 does  not  continue  after  it, as if it were equivalent to a
510                 goto to another part of  yylex().   The  use  of  REJECT  may
511                 result in somewhat larger and slower scanners.
512
513       BEGIN     The action:
514
515                     BEGIN newstate;
516
517                 switches  the  state  (start  condition) to newstate.  If the
518                 string newstate has not been declared previously as  a  start
519                 condition in the Definitions section, the results are unspec‐
520                 ified. The initial state is indicated by the digit '0' or the
521                 token INITIAL.
522
523       The  functions  or  macros  described below are accessible to user code
524       included in the lex input. It is unspecified whether they appear in the
525       C  code  output of lex, or are accessible only through the −l l operand
526       to c99 (the lex library).
527
528       int yylex(void)
529             Performs lexical analysis on the input; this is the primary func‐
530             tion generated by the lex utility. The function shall return zero
531             when the end of input is reached; otherwise, it shall return non-
532             zero values (tokens) determined by the actions that are selected.
533
534       int yymore(void)
535             When  called, indicates that when the next input string is recog‐
536             nized, it is to be appended to the current value of yytext rather
537             than  replacing it; the value in yyleng shall be adjusted accord‐
538             ingly.
539
540       int yyless(int n)
541             Retains n  initial  characters  in  yytext,  NUL-terminated,  and
542             treats the remaining characters as if they had not been read; the
543             value in yyleng shall be adjusted accordingly.
544
545       int input(void)
546             Returns the next character from the input,  or  zero  on  end-of-
547             file.  It  shall  obtain  input  from  the  stream  pointer yyin,
548             although possibly via an intermediate buffer. Thus, once scanning
549             has begun, the effect of altering the value of yyin is undefined.
550             The character read shall be removed from the input stream of  the
551             scanner without any processing by the scanner.
552
553       int unput(int c)
554             Returns  the  character  'c'  to the input; yytext and yyleng are
555             undefined until the next expression is  matched.  The  result  of
556             using unput() for more characters than have been input is unspec‐
557             ified.
558
559       The following functions shall appear only in the lex library accessible
560       through the −l l operand; they can therefore be redefined by a conform‐
561       ing application:
562
563       int yywrap(void)
564             Called by yylex() at  end-of-file;  the  default  yywrap()  shall
565             always  return 1. If the application requires yylex() to continue
566             processing with another source of input, then the application can
567             include  a  function yywrap(), which associates another file with
568             the external variable FILE * yyin and shall  return  a  value  of
569             zero.
570
571       int main(int argc, char *argv[])
572             Calls  yylex()  to perform lexical analysis, then exits. The user
573             code can contain main() to  perform  application-specific  opera‐
574             tions, calling yylex() as applicable.
575
576       Except  for input(), unput(), and main(), all external and static names
577       generated by lex shall begin with the prefix yy or YY.
578

EXIT STATUS

580       The following exit values shall be returned:
581
582        0    Successful completion.
583
584       >0    An error occurred.
585

CONSEQUENCES OF ERRORS

587       Default.
588
589       The following sections are informative.
590

APPLICATION USAGE

592       Conforming applications are warned that in the Rules  section,  an  ERE
593       without  an action is not acceptable, but need not be detected as erro‐
594       neous by lex.  This may result in compilation or runtime errors.
595
596       The purpose of input() is to take characters off the input  stream  and
597       discard  them as far as the lexical analysis is concerned. A common use
598       is to discard the body of a comment once the beginning of a comment  is
599       recognized.
600
601       The lex utility is not fully internationalized in its treatment of reg‐
602       ular expressions in the lex source code or generated lexical  analyzer.
603       It would seem desirable to have the lexical analyzer interpret the reg‐
604       ular expressions given in the lex source according to  the  environment
605       specified when the lexical analyzer is executed, but this is not possi‐
606       ble with the current lex technology. Furthermore, the  very  nature  of
607       the lexical analyzers produced by lex must be closely tied to the lexi‐
608       cal requirements of the input language being described, which  is  fre‐
609       quently  locale-specific anyway. (For example, writing an analyzer that
610       is used for French text is  not  automatically  useful  for  processing
611       other languages.)
612

EXAMPLES

614       The following is an example of a lex program that implements a rudimen‐
615       tary scanner for a Pascal-like syntax:
616
617           %{
618           /* Need this for the call to atof() below. */
619           #include <math.h>
620           /* Need this for printf(), fopen(), and stdin below. */
621           #include <stdio.h>
622           %}
623
624           DIGIT    [0−9]
625           ID       [a−z][a−z0−9]*
626
627           %%
628
629           {DIGIT}+ {
630               printf("An integer: %s (%d)\n", yytext,
631                   atoi(yytext));
632               }
633
634           {DIGIT}+"."{DIGIT}*        {
635               printf("A float: %s (%g)\n", yytext,
636                   atof(yytext));
637               }
638
639           if|then|begin|end|procedure|function        {
640               printf("A keyword: %s\n", yytext);
641               }
642
643           {ID}    printf("An identifier: %s\n", yytext);
644
645           "+"|"−"|"*"|"/"        printf("An operator: %s\n", yytext);
646
647           "{"[^}\n]*"}"    /* Eat up one-line comments. */
648
649           [ \t\n]+        /* Eat up white space. */
650
651           .  printf("Unrecognized character: %s\n", yytext);
652
653           %%
654
655           int main(int argc, char *argv[])
656           {
657               ++argv, −−argc;  /* Skip over program name. */
658               if (argc > 0)
659                   yyin = fopen(argv[0], "r");
660               else
661                   yyin = stdin;
662
663               yylex();
664           }
665

RATIONALE

667       Even though the −c option and references to the C language are retained
668       in  this description, lex may be generalized to other languages, as was
669       done at one time for EFL, the Extended FORTRAN Language. Since the  lex
670       input  specification  is  essentially language-independent, versions of
671       this utility could be written to produce Ada, Modula-2, or Pascal code,
672       and there are known historical implementations that do so.
673
674       The  current  description  of  lex  bypasses  the issue of dealing with
675       internationalized EREs in the lex source code or generated lexical ana‐
676       lyzer.  If it follows the model used by awk (the source code is assumed
677       to be presented in the POSIX locale, but input and output  are  in  the
678       locale  specified by the environment variables), then the tables in the
679       lexical analyzer produced by lex would interpret EREs specified in  the
680       lex source in terms of the environment variables specified when lex was
681       executed. The desired effect would be  to  have  the  lexical  analyzer
682       interpret the EREs given in the lex source according to the environment
683       specified when the lexical analyzer is executed, but this is not possi‐
684       ble with the current lex technology.
685
686       The  description of octal and hexadecimal-digit escape sequences agrees
687       with the ISO C standard use of escape sequences.
688
689       Earlier versions of this  standard  allowed  for  implementations  with
690       bytes  other  than  eight bits, but this has been modified in this ver‐
691       sion.
692
693       There is no detailed output format specification. The observed behavior
694       of lex under four different historical implementations was that none of
695       these implementations consistently reported the line numbers for  error
696       and  warning  messages.  Furthermore,  there  was  a desire that lex be
697       allowed to output additional diagnostic messages. Leaving message  for‐
698       mats  unspecified  avoids  these formatting questions and problems with
699       internationalization.
700
701       Although the %x specifier for exclusive start conditions is not histor‐
702       ical practice, it is believed to be a minor change to historical imple‐
703       mentations and greatly enhances the usability of lex programs since  it
704       permits  an application to obtain the expected functionality with fewer
705       statements.
706
707       The %array and %pointer declarations were added as a compromise between
708       historical  systems.  The System V-based lex copies the matched text to
709       a yytext array. The flex program, supported in  BSD  and  GNU  systems,
710       uses  a  pointer.  In the latter case, significant performance improve‐
711       ments are available for some scanners. Most historical programs  should
712       require  no  change  in  porting from one system to another because the
713       string being referenced is null-terminated in both cases.  (The  method
714       used  by  flex  in  its case is to null-terminate the token in place by
715       remembering the character that used to come right after the  token  and
716       replacing  it  before  continuing on to the next scan.) Multi-file pro‐
717       grams with external references to yytext  outside  the  scanner  source
718       file  should continue to operate on their historical systems, but would
719       require one of the new declarations to be considered strictly portable.
720
721       The description of EREs avoids unnecessary duplication of  ERE  details
722       because  their  meanings  within a lex ERE are the same as that for the
723       ERE in this volume of POSIX.1‐2008.
724
725       The reason for the undefined condition associated with  text  beginning
726       with a <blank> or within "%{" and "%}" delimiter lines appearing in the
727       Rules section is historical practice. Both the BSD  and  System  V  lex
728       copy  the  indented (or enclosed) input in the Rules section (except at
729       the beginning) to unreachable areas of the yylex() function  (the  code
730       is written directly after a break statement). In some cases, the System
731       V lex generates an error message or a syntax error,  depending  on  the
732       form of indented input.
733
734       The  intention  in  breaking  the list of functions into those that may
735       appear in lex.yy.c versus those that only appear in libl.a is that only
736       those  functions  in  libl.a  can be reliably redefined by a conforming
737       application.
738
739       The descriptions of standard output and  standard  error  are  somewhat
740       complicated because historical lex implementations chose to issue diag‐
741       nostic messages to standard output (unless −t was given).  POSIX.1‐2008
742       allows  this  behavior,  but  leaves  an  opening for the more expected
743       behavior of using standard error for diagnostics.  Also, the  System  V
744       behavior  of  writing  the statistics when any table sizes are given is
745       allowed, while BSD-derived systems can avoid  it.  The  programmer  can
746       always  precisely  obtain the desired results by using either the −t or
747       −n options.
748
749       The OPERANDS section does not mention the use of as  a  synonym  for
750       standard  input;  not all historical implementations support such usage
751       for any of the file operands.
752
753       A description of the translation table was deleted from early proposals
754       because of its relatively low usage in historical applications.
755
756       The  change  to  the  definition  of  the  input() function that allows
757       buffering of input presents the opportunity for major performance gains
758       in some applications.
759
760       The  following  examples  clarify  the  differences between lex regular
761       expressions and regular expressions appearing elsewhere in this  volume
762       of  POSIX.1‐2008. For regular expressions of the form "r/x", the string
763       matching r is always returned; confusion may arise when  the  beginning
764       of x matches the trailing portion of r.  For example, given the regular
765       expression "a*b/cc" and the input "aaabcc", yytext  would  contain  the
766       string  "aaab"  on this match. But given the regular expression "x*/xy"
767       and the input "xxxy", the token xxx, not xx, is returned by some imple‐
768       mentations because xxx matches "x*".
769
770       In  the  rule "ab*/bc", the "b*" at the end of r extends r's match into
771       the beginning of the trailing context, so the result is unspecified. If
772       this rule were "ab/bc", however, the rule matches the text "ab" when it
773       is followed by the text "bc".  In this latter case, the matching  of  r
774       cannot extend into the beginning of x, so the result is specified.
775

FUTURE DIRECTIONS

777       None.
778

SEE ALSO

780       c99, ed, yacc
781
782       The  Base  Definitions  volume  of POSIX.1‐2008, Chapter 5, File Format
783       Notation, Chapter 8, Environment Variables, Chapter 9, Regular  Expres‐
784       sions, Section 12.2, Utility Syntax Guidelines
785
787       Portions  of  this text are reprinted and reproduced in electronic form
788       from IEEE Std 1003.1, 2013 Edition, Standard for Information Technology
789       --  Portable  Operating  System  Interface (POSIX), The Open Group Base
790       Specifications Issue 7, Copyright (C) 2013 by the Institute of Electri‐
791       cal  and  Electronics  Engineers,  Inc  and  The  Open Group.  (This is
792       POSIX.1-2008 with the 2013 Technical Corrigendum  1  applied.)  In  the
793       event of any discrepancy between this version and the original IEEE and
794       The Open Group Standard, the original IEEE and The Open Group  Standard
795       is  the  referee document. The original Standard can be obtained online
796       at http://www.unix.org/online.html .
797
798       Any typographical or formatting errors that appear  in  this  page  are
799       most likely to have been introduced during the conversion of the source
800       files to man page format. To report such errors,  see  https://www.ker
801       nel.org/doc/man-pages/reporting_bugs.html .
802
803
804
805IEEE/The Open Group                  2013                              LEX(1P)
Impressum