1PCRETEST(1)                 General Commands Manual                PCRETEST(1)
2
3
4

NAME

6       pcretest - a program for testing Perl-compatible regular expressions.
7

SYNOPSIS

9
10       pcretest [options] [input file [output file]]
11
12       pcretest  was written as a test program for the PCRE regular expression
13       library itself, but it can also be used for experimenting with  regular
14       expressions.  This document describes the features of the test program;
15       for details of the regular expressions themselves, see the  pcrepattern
16       documentation. For details of the PCRE library function calls and their
17       options, see the pcreapi , pcre16 and pcre32 documentation.
18
19       The input for pcretest is a sequence of regular expression patterns and
20       strings  to be matched, as described below. The output shows the result
21       of each match. Options on the command line  and  the  patterns  control
22       PCRE options and exactly what is output.
23
24       As  PCRE has evolved, it has acquired many different features, and as a
25       result, pcretest now has rather a lot of obscure  options  for  testing
26       every possible feature. Some of these options are specifically designed
27       for use in conjunction with the test script and  data  files  that  are
28       distributed  as  part of PCRE, and are unlikely to be of use otherwise.
29       They are all documented here, but without much justification.
30

INPUT DATA FORMAT

32
33       Input to pcretest is processed line by line, either by  calling  the  C
34       library's fgets() function, or via the libreadline library (see below).
35       In Unix-like environments, fgets() treats any bytes other than  newline
36       as  data characters. However, in some Windows environments character 26
37       (hex 1A) causes an immediate end of file, and no further data is  read.
38       For  maximum  portability,  therefore,  it  is safest to use only ASCII
39       characters in pcretest input files.
40
41       The input is processed using using C's string functions,  so  must  not
42       contain  binary  zeroes, even though in Unix-like environments, fgets()
43       treats any bytes other than newline as data characters.
44

PCRE's 8-BIT, 16-BIT AND 32-BIT LIBRARIES

46
47       From release 8.30, two separate PCRE libraries can be built. The origi‐
48       nal  one  supports  8-bit  character  strings, whereas the newer 16-bit
49       library supports  character  strings  encoded  in  16-bit  units.  From
50       release  8.32,  a  third  library  can  be  built, supporting character
51       strings encoded in 32-bit units. The pcretest program can  be  used  to
52       test all three libraries. However, it is itself still an 8-bit program,
53       reading 8-bit input and writing 8-bit output.  When testing the  16-bit
54       or  32-bit  library, the patterns and data strings are converted to 16-
55       or 32-bit format before being passed to  the  PCRE  library  functions.
56       Results are converted to 8-bit for output.
57
58       References to functions and structures of the form pcre[16|32]_xx below
59       mean "pcre_xx when using the 8-bit library, pcre16_xx  when  using  the
60       16-bit library, or pcre32_xx when using the 32-bit library".
61

COMMAND LINE OPTIONS

63
64       -8        If the 8-bit library has been built, this option causes it to
65                 be used (this is the default). If the 8-bit library  has  not
66                 been built, this option causes an error.
67
68       -16       If  the  16-bit library has been built, this option causes it
69                 to be used. If only the 16-bit library has been  built,  this
70                 is  the  default.  If  the 16-bit library has not been built,
71                 this option causes an error.
72
73       -32       If the 32-bit library has been built, this option  causes  it
74                 to  be  used. If only the 32-bit library has been built, this
75                 is the default. If the 32-bit library  has  not  been  built,
76                 this option causes an error.
77
78       -b        Behave  as  if each pattern has the /B (show byte code) modi‐
79                 fier; the internal form is output after compilation.
80
81       -C        Output the version number of the PCRE library, and all avail‐
82                 able   information  about  the  optional  features  that  are
83                 included, and then  exit  with  zero  exit  code.  All  other
84                 options are ignored.
85
86       -C option Output  information  about a specific build-time option, then
87                 exit. This functionality is intended for use in scripts  such
88                 as  RunTest.  The  following options output the value and set
89                 the exit code as indicated:
90
91                   ebcdic-nl  the code for LF (= NL) in an EBCDIC environment:
92                                0x15 or 0x25
93                                0 if used in an ASCII environment
94                                exit code is always 0
95                   linksize   the configured internal link size (2, 3, or 4)
96                                exit code is set to the link size
97                   newline    the default newline setting:
98                                CR, LF, CRLF, ANYCRLF, or ANY
99                                exit code is always 0
100                   bsr        the default setting for what \R matches:
101                                ANYCRLF or ANY
102                                exit code is always 0
103
104                 The following options output 1 for true or 0 for  false,  and
105                 set the exit code to the same value:
106
107                   ebcdic     compiled for an EBCDIC environment
108                   jit        just-in-time support is available
109                   pcre16     the 16-bit library was built
110                   pcre32     the 32-bit library was built
111                   pcre8      the 8-bit library was built
112                   ucp        Unicode property support is available
113                   utf        UTF-8 and/or UTF-16 and/or UTF-32 support
114                                is available
115
116                 If  an  unknown  option is given, an error message is output;
117                 the exit code is 0.
118
119       -d        Behave as if each pattern has the /D  (debug)  modifier;  the
120                 internal  form  and information about the compiled pattern is
121                 output after compilation; -d is equivalent to -b -i.
122
123       -dfa      Behave as if each data line contains the \D escape  sequence;
124                 this    causes    the    alternative    matching    function,
125                 pcre[16|32]_dfa_exec(), to be used instead  of  the  standard
126                 pcre[16|32]_exec() function (more detail is given below).
127
128       -help     Output a brief summary these options and then exit.
129
130       -i        Behave  as  if  each pattern has the /I modifier; information
131                 about the compiled pattern is given after compilation.
132
133       -M        Behave as if each data line contains the \M escape  sequence;
134                 this  causes  PCRE  to  discover  the minimum MATCH_LIMIT and
135                 MATCH_LIMIT_RECURSION settings by calling  pcre[16|32]_exec()
136                 repeatedly with different limits.
137
138       -m        Output  the  size  of each compiled pattern after it has been
139                 compiled. This is equivalent to adding  /M  to  each  regular
140                 expression. The size is given in bytes for both libraries.
141
142       -O        Behave  as  if each pattern has the /O modifier, that is dis‐
143                 able auto-possessification for all patterns.
144
145       -o osize  Set the number of elements in the output vector that is  used
146                 when  calling pcre[16|32]_exec() or pcre[16|32]_dfa_exec() to
147                 be osize. The default value is 45, which  is  enough  for  14
148                 capturing subexpressions for pcre[16|32]_exec() or 22 differ‐
149                 ent matches for pcre[16|32]_dfa_exec().  The vector size  can
150                 be  changed  for individual matching calls by including \O in
151                 the data line (see below).
152
153       -p        Behave as if each pattern has  the  /P  modifier;  the  POSIX
154                 wrapper  API  is used to call PCRE. None of the other options
155                 has any effect when -p is set. This option can be  used  only
156                 with the 8-bit library.
157
158       -q        Do  not output the version number of pcretest at the start of
159                 execution.
160
161       -S size   On Unix-like systems, set the size of the run-time  stack  to
162                 size megabytes.
163
164       -s or -s+ Behave  as  if  each  pattern  has  the /S modifier; in other
165                 words, force each pattern to be studied. If -s+ is used,  all
166                 the  JIT  compile  options are passed to pcre[16|32]_study(),
167                 causing just-in-time optimization to  be  set  up  if  it  is
168                 available,  for  both full and partial matching. Specific JIT
169                 compile options can be selected by following -s+ with a digit
170                 in  the  range 1 to 7, which selects the JIT compile modes as
171                 follows:
172
173                   1  normal match only
174                   2  soft partial match only
175                   3  normal match and soft partial match
176                   4  hard partial match only
177                   6  soft and hard partial match
178                   7  all three modes (default)
179
180                 If -s++ is used instead of -s+ (with or without  a  following
181                 digit),  the  text  "(JIT)" is added to the first output line
182                 after a match or no match when JIT-compiled code was actually
183                 used.
184
185                 Note  that  there  are  pattern options that can override -s,
186                 either specifying no studying at all, or suppressing JIT com‐
187                 pilation.
188
189                 If  the  /I  or /D option is present on a pattern (requesting
190                 output about the compiled  pattern),  information  about  the
191                 result  of  studying  is not included when studying is caused
192                 only by -s and neither -i nor -d is present  on  the  command
193                 line.  This  behaviour  means that the output from tests that
194                 are run with and without -s should be identical, except  when
195                 options that output information about the actual running of a
196                 match are set.
197
198                 The -M, -t, and -tm options,  which  give  information  about
199                 resources  used,  are likely to produce different output with
200                 and without -s. Output may also differ if the  /C  option  is
201                 present on an individual pattern. This uses callouts to trace
202                 the the matching process, and this may be  different  between
203                 studied  and  non-studied  patterns.  If the pattern contains
204                 (*MARK) items there may also be  differences,  for  the  same
205                 reason. The -s command line option can be overridden for spe‐
206                 cific patterns that should never be studied (see the /S  pat‐
207                 tern modifier below).
208
209       -t        Run  each  compile, study, and match many times with a timer,
210                 and output the resulting times per compile, study,  or  match
211                 (in  milliseconds).  Do  not set -m with -t, because you will
212                 then get the size output a zillion times, and the timing will
213                 be  distorted.  You can control the number of iterations that
214                 are used for timing by following -t with a number (as a sepa‐
215                 rate  item on the command line). For example, "-t 1000" iter‐
216                 ates 1000 times.  The default is to iterate 500000 times.
217
218       -tm       This is like -t except that it times only the matching phase,
219                 not the compile or study phases.
220
221       -T -TM    These  behave like -t and -tm, but in addition, at the end of
222                 a run, the total times for all compiles, studies, and matches
223                 are output.
224

DESCRIPTION

226
227       If  pcretest  is  given two filename arguments, it reads from the first
228       and writes to the second. If it is given only one filename argument, it
229       reads  from  that  file  and writes to stdout. Otherwise, it reads from
230       stdin and writes to stdout, and prompts for each line of  input,  using
231       "re>" to prompt for regular expressions, and "data>" to prompt for data
232       lines.
233
234       When pcretest is built, a configuration  option  can  specify  that  it
235       should  be  linked  with the libreadline library. When this is done, if
236       the input is from a terminal, it is read using the readline() function.
237       This  provides line-editing and history facilities. The output from the
238       -help option states whether or not readline() will be used.
239
240       The program handles any number of sets of input on a single input file.
241       Each  set starts with a regular expression, and continues with any num‐
242       ber of data lines to be matched against that pattern.
243
244       Each data line is matched separately and independently. If you want  to
245       do multi-line matches, you have to use the \n escape sequence (or \r or
246       \r\n, etc., depending on the newline setting) in a single line of input
247       to  encode  the  newline  sequences. There is no limit on the length of
248       data lines; the input buffer is automatically extended  if  it  is  too
249       small.
250
251       An  empty  line signals the end of the data lines, at which point a new
252       regular expression is read. The regular expressions are given  enclosed
253       in any non-alphanumeric delimiters other than backslash, for example:
254
255         /(a|bc)x+yz/
256
257       White  space before the initial delimiter is ignored. A regular expres‐
258       sion may be continued over several input lines, in which case the  new‐
259       line  characters  are included within it. It is possible to include the
260       delimiter within the pattern by escaping it, for example
261
262         /abc\/def/
263
264       If you do so, the escape and the delimiter form part  of  the  pattern,
265       but  since delimiters are always non-alphanumeric, this does not affect
266       its interpretation.  If the terminating delimiter is  immediately  fol‐
267       lowed by a backslash, for example,
268
269         /abc/\
270
271       then  a  backslash  is added to the end of the pattern. This is done to
272       provide a way of testing the error condition that arises if  a  pattern
273       finishes with a backslash, because
274
275         /abc\/
276
277       is  interpreted as the first line of a pattern that starts with "abc/",
278       causing pcretest to read the next line as a continuation of the regular
279       expression.
280

PATTERN MODIFIERS

282
283       A  pattern may be followed by any number of modifiers, which are mostly
284       single characters, though some of these can  be  qualified  by  further
285       characters.   Following Perl usage, these are referred to below as, for
286       example, "the /i modifier", even though the delimiter  of  the  pattern
287       need  not  always  be  a slash, and no slash is used when writing modi‐
288       fiers. White space may appear between the final pattern  delimiter  and
289       the  first  modifier,  and between the modifiers themselves. For refer‐
290       ence, here is a complete list of  modifiers.  They  fall  into  several
291       groups that are described in detail in the following sections.
292
293         /8              set UTF mode
294         /9              set PCRE_NEVER_UTF (locks out UTF mode)
295         /?              disable UTF validity check
296         /+              show remainder of subject after match
297         /=              show all captures (not just those that are set)
298
299         /A              set PCRE_ANCHORED
300         /B              show compiled code
301         /C              set PCRE_AUTO_CALLOUT
302         /D              same as /B plus /I
303         /E              set PCRE_DOLLAR_ENDONLY
304         /F              flip byte order in compiled pattern
305         /f              set PCRE_FIRSTLINE
306         /G              find all matches (shorten string)
307         /g              find all matches (use startoffset)
308         /I              show information about pattern
309         /i              set PCRE_CASELESS
310         /J              set PCRE_DUPNAMES
311         /K              show backtracking control names
312         /L              set locale
313         /M              show compiled memory size
314         /m              set PCRE_MULTILINE
315         /N              set PCRE_NO_AUTO_CAPTURE
316         /O              set PCRE_NO_AUTO_POSSESS
317         /P              use the POSIX wrapper
318         /Q              test external stack check function
319         /S              study the pattern after compilation
320         /s              set PCRE_DOTALL
321         /T              select character tables
322         /U              set PCRE_UNGREEDY
323         /W              set PCRE_UCP
324         /X              set PCRE_EXTRA
325         /x              set PCRE_EXTENDED
326         /Y              set PCRE_NO_START_OPTIMIZE
327         /Z              don't show lengths in /B output
328
329         /<any>          set PCRE_NEWLINE_ANY
330         /<anycrlf>      set PCRE_NEWLINE_ANYCRLF
331         /<cr>           set PCRE_NEWLINE_CR
332         /<crlf>         set PCRE_NEWLINE_CRLF
333         /<lf>           set PCRE_NEWLINE_LF
334         /<bsr_anycrlf>  set PCRE_BSR_ANYCRLF
335         /<bsr_unicode>  set PCRE_BSR_UNICODE
336         /<JS>           set PCRE_JAVASCRIPT_COMPAT
337
338
339   Perl-compatible modifiers
340
341       The /i, /m, /s, and /x modifiers set the PCRE_CASELESS, PCRE_MULTILINE,
342       PCRE_DOTALL,   or    PCRE_EXTENDED    options,    respectively,    when
343       pcre[16|32]_compile()  is  called. These four modifier letters have the
344       same effect as they do in Perl. For example:
345
346         /caseless/i
347
348
349   Modifiers for other PCRE options
350
351       The following table shows additional modifiers for  setting  PCRE  com‐
352       pile-time options that do not correspond to anything in Perl:
353
354         /8              PCRE_UTF8           ) when using the 8-bit
355         /?              PCRE_NO_UTF8_CHECK  )   library
356
357         /8              PCRE_UTF16          ) when using the 16-bit
358         /?              PCRE_NO_UTF16_CHECK )   library
359
360         /8              PCRE_UTF32          ) when using the 32-bit
361         /?              PCRE_NO_UTF32_CHECK )   library
362
363         /9              PCRE_NEVER_UTF
364         /A              PCRE_ANCHORED
365         /C              PCRE_AUTO_CALLOUT
366         /E              PCRE_DOLLAR_ENDONLY
367         /f              PCRE_FIRSTLINE
368         /J              PCRE_DUPNAMES
369         /N              PCRE_NO_AUTO_CAPTURE
370         /O              PCRE_NO_AUTO_POSSESS
371         /U              PCRE_UNGREEDY
372         /W              PCRE_UCP
373         /X              PCRE_EXTRA
374         /Y              PCRE_NO_START_OPTIMIZE
375         /<any>          PCRE_NEWLINE_ANY
376         /<anycrlf>      PCRE_NEWLINE_ANYCRLF
377         /<cr>           PCRE_NEWLINE_CR
378         /<crlf>         PCRE_NEWLINE_CRLF
379         /<lf>           PCRE_NEWLINE_LF
380         /<bsr_anycrlf>  PCRE_BSR_ANYCRLF
381         /<bsr_unicode>  PCRE_BSR_UNICODE
382         /<JS>           PCRE_JAVASCRIPT_COMPAT
383
384       The  modifiers  that are enclosed in angle brackets are literal strings
385       as shown, including the angle brackets, but the letters within  can  be
386       in  either case.  This example sets multiline matching with CRLF as the
387       line ending sequence:
388
389         /^abc/m<CRLF>
390
391       As well as turning on  the  PCRE_UTF8/16/32  option,  the  /8  modifier
392       causes  all  non-printing  characters  in  output strings to be printed
393       using the \x{hh...} notation. Otherwise, those less than 0x100 are out‐
394       put in hex without the curly brackets.
395
396       Full  details  of  the PCRE options are given in the pcreapi documenta‐
397       tion.
398
399   Finding all matches in a string
400
401       Searching for all possible matches within each subject  string  can  be
402       requested  by  the  /g  or  /G modifier. After finding a match, PCRE is
403       called again to search the remainder of the subject string. The differ‐
404       ence between /g and /G is that the former uses the startoffset argument
405       to pcre[16|32]_exec() to start searching at  a  new  point  within  the
406       entire  string  (which is in effect what Perl does), whereas the latter
407       passes over a shortened substring.  This  makes  a  difference  to  the
408       matching  process  if  the  pattern  begins with a lookbehind assertion
409       (including \b or \B).
410
411       If any call to pcre[16|32]_exec() in a /g or  /G  sequence  matches  an
412       empty  string, the next call is done with the PCRE_NOTEMPTY_ATSTART and
413       PCRE_ANCHORED flags set in order  to  search  for  another,  non-empty,
414       match  at  the same point. If this second match fails, the start offset
415       is advanced, and the normal match is retried.  This  imitates  the  way
416       Perl handles such cases when using the /g modifier or the split() func‐
417       tion. Normally, the start offset is advanced by one character,  but  if
418       the  newline  convention  recognizes CRLF as a newline, and the current
419       character is CR followed by LF, an advance of two is used.
420
421   Other modifiers
422
423       There are yet more modifiers for controlling the way pcretest operates.
424
425       The /+ modifier requests that as well as outputting the substring  that
426       matched  the  entire  pattern,  pcretest  should in addition output the
427       remainder of the subject string. This is useful  for  tests  where  the
428       subject  contains multiple copies of the same substring. If the + modi‐
429       fier appears twice, the same action is taken for  captured  substrings.
430       In  each case the remainder is output on the following line with a plus
431       character following the capture number. Note that  this  modifier  must
432       not  immediately follow the /S modifier because /S+ and /S++ have other
433       meanings.
434
435       The /= modifier requests that the  values  of  all  potential  captured
436       parentheses  be  output after a match. By default, only those up to the
437       highest one actually used in the match are output (corresponding to the
438       return code from pcre[16|32]_exec()). Values in the offsets vector cor‐
439       responding to higher numbers should be set to -1, and these are  output
440       as  "<unset>".  This modifier gives a way of checking that this is hap‐
441       pening.
442
443       The /B modifier is a debugging feature. It requests that pcretest  out‐
444       put  a  representation of the compiled code after compilation. Normally
445       this information contains length and offset values; however, if  /Z  is
446       also  present,  this data is replaced by spaces. This is a special fea‐
447       ture for use in the automatic test scripts; it ensures  that  the  same
448       output is generated for different internal link sizes.
449
450       The  /D modifier is a PCRE debugging feature, and is equivalent to /BI,
451       that is, both the /B and the /I modifiers.
452
453       The /F modifier causes pcretest to flip the byte order  of  the  2-byte
454       and 4-byte fields in the compiled pattern. This facility is for testing
455       the feature in PCRE that allows it to execute patterns that  were  com‐
456       piled on a host with a different endianness. This feature is not avail‐
457       able when the POSIX interface to PCRE is being used, that is, when  the
458       /P pattern modifier is specified. See also the section about saving and
459       reloading compiled patterns below.
460
461       The /I modifier requests that pcretest  output  information  about  the
462       compiled  pattern (whether it is anchored, has a fixed first character,
463       and so on). It does this by calling pcre[16|32]_fullinfo()  after  com‐
464       piling  a  pattern.  If the pattern is studied, the results of that are
465       also output. In this output, the word "char" means a non-UTF character,
466       that  is,  the  value  of a single data item (8-bit, 16-bit, or 32-bit,
467       depending on the library that is being tested).
468
469       The /K modifier requests pcretest to show names from backtracking  con‐
470       trol  verbs  that  are  returned  from  calls to pcre[16|32]_exec(). It
471       causes pcretest to create a pcre[16|32]_extra  block  if  one  has  not
472       already  been  created by a call to pcre[16|32]_study(), and to set the
473       PCRE_EXTRA_MARK flag and the mark field  within  it,  every  time  that
474       pcre[16|32]_exec()  is  called.  If  the  variable  that the mark field
475       points to is  non-NULL  for  a  match,  non-match,  or  partial  match,
476       pcretest  prints  the  string  to which it points. For a match, this is
477       shown on a line by itself, tagged with "MK:". For  a  non-match  it  is
478       added to the message.
479
480       The  /L modifier must be followed directly by the name of a locale, for
481       example,
482
483         /pattern/Lfr_FR
484
485       For this reason, it must be the last modifier. The given locale is set,
486       pcre[16|32]_maketables()  is  called to build a set of character tables
487       for the locale, and this is then passed to  pcre[16|32]_compile()  when
488       compiling  the regular expression. Without an /L (or /T) modifier, NULL
489       is passed as the tables pointer;  that  is,  /L  applies  only  to  the
490       expression on which it appears.
491
492       The  /M  modifier  causes the size in bytes of the memory block used to
493       hold the compiled pattern to be output. This does not include the  size
494       of  the  pcre[16|32] block; it is just the actual compiled data. If the
495       pattern is successfully studied with the PCRE_STUDY_JIT_COMPILE option,
496       the size of the JIT compiled code is also output.
497
498       The /Q modifier is used to test the use of pcre_stack_guard. It must be
499       followed by '0' or '1', specifying the return code to be given from  an
500       external  function  that  is passed to PCRE and used for stack checking
501       during compilation (see the pcreapi documentation for details).
502
503       The /S modifier causes  pcre[16|32]_study()  to  be  called  after  the
504       expression  has been compiled, and the results used when the expression
505       is matched. There are a number of qualifying characters that may follow
506       /S.  They may appear in any order.
507
508       If /S is followed by an exclamation mark, pcre[16|32]_study() is called
509       with the PCRE_STUDY_EXTRA_NEEDED option, causing it always to return  a
510       pcre_extra block, even when studying discovers no useful information.
511
512       If /S is followed by a second S character, it suppresses studying, even
513       if it was requested externally by the  -s  command  line  option.  This
514       makes  it possible to specify that certain patterns are always studied,
515       and others are never studied, independently of -s. This feature is used
516       in the test files in a few cases where the output is different when the
517       pattern is studied.
518
519       If the  /S  modifier  is  followed  by  a  +  character,  the  call  to
520       pcre[16|32]_study()  is made with all the JIT study options, requesting
521       just-in-time optimization support if it is available, for  both  normal
522       and  partial matching. If you want to restrict the JIT compiling modes,
523       you can follow /S+ with a digit in the range 1 to 7:
524
525         1  normal match only
526         2  soft partial match only
527         3  normal match and soft partial match
528         4  hard partial match only
529         6  soft and hard partial match
530         7  all three modes (default)
531
532       If /S++ is used instead of /S+ (with or without a following digit), the
533       text  "(JIT)"  is  added  to  the first output line after a match or no
534       match when JIT-compiled code was actually used.
535
536       Note that there is also an independent /+  modifier;  it  must  not  be
537       given immediately after /S or /S+ because this will be misinterpreted.
538
539       If JIT studying is successful, the compiled JIT code will automatically
540       be used when pcre[16|32]_exec() is run, except when  incompatible  run-
541       time  options are specified. For more details, see the pcrejit documen‐
542       tation. See also the \J escape sequence below for a way of setting  the
543       size of the JIT stack.
544
545       Finally,  if  /S  is  followed by a minus character, JIT compilation is
546       suppressed, even if it was requested externally by the -s command  line
547       option.  This makes it possible to specify that JIT is never to be used
548       for certain patterns.
549
550       The /T modifier must be followed by a single digit. It  causes  a  spe‐
551       cific set of built-in character tables to be passed to pcre[16|32]_com‐
552       pile(). It is used in the standard PCRE tests to check  behaviour  with
553       different character tables. The digit specifies the tables as follows:
554
555         0   the default ASCII tables, as distributed in
556               pcre_chartables.c.dist
557         1   a set of tables defining ISO 8859 characters
558
559       In  table 1, some characters whose codes are greater than 128 are iden‐
560       tified as letters, digits, spaces, etc.
561
562   Using the POSIX wrapper API
563
564       The /P modifier causes pcretest to call PCRE via the POSIX wrapper  API
565       rather  than its native API. This supports only the 8-bit library. When
566       /P is set, the following modifiers set options for the regcomp()  func‐
567       tion:
568
569         /i    REG_ICASE
570         /m    REG_NEWLINE
571         /N    REG_NOSUB
572         /s    REG_DOTALL     )
573         /U    REG_UNGREEDY   ) These options are not part of
574         /W    REG_UCP        )   the POSIX standard
575         /8    REG_UTF8       )
576
577       The  /+  modifier  works  as  described  above. All other modifiers are
578       ignored.
579
580   Locking out certain modifiers
581
582       PCRE can be compiled with or without support for certain features  such
583       as  UTF-8/16/32  or Unicode properties. Accordingly, the standard tests
584       are split up into a number of different files  that  are  selected  for
585       running  depending  on  which features are available. When updating the
586       tests, it is all too easy to put a new test into the wrong file by mis‐
587       take;  for example, to put a test that requires UTF support into a file
588       that is used when it is not available. To help detect such mistakes  as
589       early  as  possible, there is a facility for locking out specific modi‐
590       fiers. If an input line for pcretest starts with the string "< forbid "
591       the  following  sequence  of characters is taken as a list of forbidden
592       modifiers. For example, in the test files that must not use UTF or Uni‐
593       code property support, this line appears:
594
595         < forbid 8W
596
597       This  locks out the /8 and /W modifiers. An immediate error is given if
598       they are subsequently encountered. If the character string  contains  <
599       but  not  >,  all  the  multi-character modifiers that begin with < are
600       locked out. Otherwise, such modifiers must be  explicitly  listed,  for
601       example:
602
603         < forbid <JS><cr>
604
605       There must be a single space between < and "forbid" for this feature to
606       be recognised. If there is not, the line is  interpreted  either  as  a
607       request  to  re-load  a pre-compiled pattern (see "SAVING AND RELOADING
608       COMPILED PATTERNS" below) or, if there is a another < character,  as  a
609       pattern that uses < as its delimiter.
610

DATA LINES

612
613       Before  each  data  line  is  passed to pcre[16|32]_exec(), leading and
614       trailing white space is removed, and it is then scanned for \  escapes.
615       Some  of  these are pretty esoteric features, intended for checking out
616       some of the more complicated features of PCRE. If you are just  testing
617       "ordinary"  regular  expressions, you probably don't need any of these.
618       The following escapes are recognized:
619
620         \a         alarm (BEL, \x07)
621         \b         backspace (\x08)
622         \e         escape (\x27)
623         \f         form feed (\x0c)
624         \n         newline (\x0a)
625         \qdd       set the PCRE_MATCH_LIMIT limit to dd
626                      (any number of digits)
627         \r         carriage return (\x0d)
628         \t         tab (\x09)
629         \v         vertical tab (\x0b)
630         \nnn       octal character (up to 3 octal digits); always
631                      a byte unless > 255 in UTF-8 or 16-bit or 32-bit mode
632         \o{dd...}  octal character (any number of octal digits}
633         \xhh       hexadecimal byte (up to 2 hex digits)
634         \x{hh...}  hexadecimal character (any number of hex digits)
635         \A         pass the PCRE_ANCHORED option to pcre[16|32]_exec()
636                      or pcre[16|32]_dfa_exec()
637         \B         pass the PCRE_NOTBOL option to pcre[16|32]_exec()
638                      or pcre[16|32]_dfa_exec()
639         \Cdd       call pcre[16|32]_copy_substring() for substring dd
640                      after a successful match (number less than 32)
641         \Cname     call pcre[16|32]_copy_named_substring() for substring
642                      "name" after a successful match (name terminated
643                      by next non alphanumeric character)
644         \C+        show the current captured substrings at callout
645                      time
646         \C-        do not supply a callout function
647         \C!n       return 1 instead of 0 when callout number n is
648                      reached
649         \C!n!m     return 1 instead of 0 when callout number n is
650                      reached for the nth time
651         \C*n       pass the number n (may be negative) as callout
652                      data; this is used as the callout return value
653         \D         use the pcre[16|32]_dfa_exec() match function
654         \F         only shortest match for pcre[16|32]_dfa_exec()
655         \Gdd       call pcre[16|32]_get_substring() for substring dd
656                      after a successful match (number less than 32)
657         \Gname     call pcre[16|32]_get_named_substring() for substring
658                      "name" after a successful match (name terminated
659                      by next non-alphanumeric character)
660         \Jdd       set up a JIT stack of dd kilobytes maximum (any
661                      number of digits)
662         \L         call pcre[16|32]_get_substringlist() after a
663                      successful match
664         \M         discover the minimum MATCH_LIMIT and
665                      MATCH_LIMIT_RECURSION settings
666         \N         pass the PCRE_NOTEMPTY option to pcre[16|32]_exec()
667                      or pcre[16|32]_dfa_exec(); if used twice, pass the
668                      PCRE_NOTEMPTY_ATSTART option
669         \Odd       set the size of the output vector passed to
670                      pcre[16|32]_exec() to dd (any number of digits)
671         \P         pass the PCRE_PARTIAL_SOFT option to pcre[16|32]_exec()
672                      or pcre[16|32]_dfa_exec(); if used twice, pass the
673                      PCRE_PARTIAL_HARD option
674         \Qdd       set the PCRE_MATCH_LIMIT_RECURSION limit to dd
675                      (any number of digits)
676         \R         pass the PCRE_DFA_RESTART option to pcre[16|32]_dfa_exec()
677         \S         output details of memory get/free calls during matching
678         \Y            pass    the    PCRE_NO_START_OPTIMIZE     option     to
679       pcre[16|32]_exec()
680                      or pcre[16|32]_dfa_exec()
681         \Z         pass the PCRE_NOTEOL option to pcre[16|32]_exec()
682                      or pcre[16|32]_dfa_exec()
683         \?         pass the PCRE_NO_UTF[8|16|32]_CHECK option to
684                      pcre[16|32]_exec() or pcre[16|32]_dfa_exec()
685         \>dd       start the match at offset dd (optional "-"; then
686                      any number of digits); this sets the startoffset
687                      argument         for        pcre[16|32]_exec()        or
688       pcre[16|32]_dfa_exec()
689         \<cr>      pass the PCRE_NEWLINE_CR option to pcre[16|32]_exec()
690                      or pcre[16|32]_dfa_exec()
691         \<lf>      pass the PCRE_NEWLINE_LF option to pcre[16|32]_exec()
692                      or pcre[16|32]_dfa_exec()
693         \<crlf>    pass the PCRE_NEWLINE_CRLF option to pcre[16|32]_exec()
694                      or pcre[16|32]_dfa_exec()
695         \<anycrlf> pass the PCRE_NEWLINE_ANYCRLF option to pcre[16|32]_exec()
696                      or pcre[16|32]_dfa_exec()
697         \<any>     pass the PCRE_NEWLINE_ANY option to pcre[16|32]_exec()
698                      or pcre[16|32]_dfa_exec()
699
700       The use of \x{hh...} is not dependent on the use of the /8 modifier  on
701       the  pattern. It is recognized always. There may be any number of hexa‐
702       decimal digits inside the braces; invalid  values  provoke  error  mes‐
703       sages.
704
705       Note  that  \xhh  specifies one byte rather than one character in UTF-8
706       mode; this makes it possible to construct invalid UTF-8  sequences  for
707       testing  purposes.  On the other hand, \x{hh} is interpreted as a UTF-8
708       character in UTF-8 mode, generating more than one byte if the value  is
709       greater  than  127.   When testing the 8-bit library not in UTF-8 mode,
710       \x{hh} generates one byte for values less than 256, and causes an error
711       for greater values.
712
713       In UTF-16 mode, all 4-digit \x{hhhh} values are accepted. This makes it
714       possible to construct invalid UTF-16 sequences for testing purposes.
715
716       In UTF-32 mode, all 4- to 8-digit \x{...}  values  are  accepted.  This
717       makes  it  possible  to  construct invalid UTF-32 sequences for testing
718       purposes.
719
720       The escapes that specify line ending  sequences  are  literal  strings,
721       exactly as shown. No more than one newline setting should be present in
722       any data line.
723
724       A backslash followed by anything else just escapes the  anything  else.
725       If  the very last character is a backslash, it is ignored. This gives a
726       way of passing an empty line as data, since a real  empty  line  termi‐
727       nates the data input.
728
729       The  \J escape provides a way of setting the maximum stack size that is
730       used by the just-in-time optimization code. It is ignored if JIT  opti‐
731       mization  is  not being used. Providing a stack that is larger than the
732       default 32K is necessary only for very complicated patterns.
733
734       If \M is present, pcretest calls pcre[16|32]_exec() several times, with
735       different values in the match_limit and match_limit_recursion fields of
736       the pcre[16|32]_extra data structure, until it finds the  minimum  num‐
737       bers for each parameter that allow pcre[16|32]_exec() to complete with‐
738       out error. Because this is testing a specific  feature  of  the  normal
739       interpretive pcre[16|32]_exec() execution, the use of any JIT optimiza‐
740       tion that might have been set up by the /S+ qualifier of -s+ option  is
741       disabled.
742
743       The  match_limit number is a measure of the amount of backtracking that
744       takes place, and checking it out can be instructive.  For  most  simple
745       matches,  the  number  is quite small, but for patterns with very large
746       numbers of matching possibilities, it can  become  large  very  quickly
747       with  increasing  length  of  subject string. The match_limit_recursion
748       number is a measure of how much stack (or, if  PCRE  is  compiled  with
749       NO_RECURSE,  how  much  heap)  memory  is  needed to complete the match
750       attempt.
751
752       When \O is used, the value specified may be higher or  lower  than  the
753       size set by the -O command line option (or defaulted to 45); \O applies
754       only to the call  of  pcre[16|32]_exec()  for  the  line  in  which  it
755       appears.
756
757       If  the /P modifier was present on the pattern, causing the POSIX wrap‐
758       per API to be used, the only option-setting  sequences  that  have  any
759       effect  are  \B,  \N,  and  \Z,  causing  REG_NOTBOL, REG_NOTEMPTY, and
760       REG_NOTEOL, respectively, to be passed to regexec().
761

THE ALTERNATIVE MATCHING FUNCTION

763
764       By  default,  pcretest  uses  the  standard  PCRE  matching   function,
765       pcre[16|32]_exec()  to  match  each  data  line.  PCRE also supports an
766       alternative matching function, pcre[16|32]_dfa_test(),  which  operates
767       in  a different way, and has some restrictions. The differences between
768       the two functions are described in the pcrematching documentation.
769
770       If a data line contains the \D escape sequence, or if the command  line
771       contains  the  -dfa  option, the alternative matching function is used.
772       This function finds all possible matches at a given point. If, however,
773       the  \F escape sequence is present in the data line, it stops after the
774       first match is found. This is always the shortest possible match.
775

DEFAULT OUTPUT FROM PCRETEST

777
778       This section describes the output when the  normal  matching  function,
779       pcre[16|32]_exec(), is being used.
780
781       When a match succeeds, pcretest outputs the list of captured substrings
782       that pcre[16|32]_exec() returns, starting with number 0 for the  string
783       that  matched  the whole pattern. Otherwise, it outputs "No match" when
784       the return is PCRE_ERROR_NOMATCH, and "Partial match:" followed by  the
785       partially    matching   substring   when   pcre[16|32]_exec()   returns
786       PCRE_ERROR_PARTIAL. (Note that this is the entire  substring  that  was
787       inspected  during  the  partial match; it may include characters before
788       the actual match start if a lookbehind assertion, \K,  \b,  or  \B  was
789       involved.)  For  any  other  return, pcretest outputs the PCRE negative
790       error number and a short descriptive phrase. If the error is  a  failed
791       UTF  string check, the offset of the start of the failing character and
792       the reason code are also output, provided that the size of  the  output
793       vector  is  at least two. Here is an example of an interactive pcretest
794       run.
795
796         $ pcretest
797         PCRE version 8.13 2011-04-30
798
799           re> /^abc(\d+)/
800         data> abc123
801          0: abc123
802          1: 123
803         data> xyz
804         No match
805
806       Unset capturing substrings that are not followed by one that is set are
807       not  returned  by pcre[16|32]_exec(), and are not shown by pcretest. In
808       the following example, there are two capturing substrings, but when the
809       first  data  line is matched, the second, unset substring is not shown.
810       An "internal" unset substring is shown as "<unset>", as for the  second
811       data line.
812
813           re> /(a)|(b)/
814         data> a
815          0: a
816          1: a
817         data> b
818          0: b
819          1: <unset>
820          2: b
821
822       If  the strings contain any non-printing characters, they are output as
823       \xhh escapes if the value is less than 256 and UTF  mode  is  not  set.
824       Otherwise they are output as \x{hh...} escapes. See below for the defi‐
825       nition of non-printing characters. If the pattern has the /+  modifier,
826       the  output  for substring 0 is followed by the the rest of the subject
827       string, identified by "0+" like this:
828
829           re> /cat/+
830         data> cataract
831          0: cat
832          0+ aract
833
834       If the pattern has the /g or /G modifier,  the  results  of  successive
835       matching attempts are output in sequence, like this:
836
837           re> /\Bi(\w\w)/g
838         data> Mississippi
839          0: iss
840          1: ss
841          0: iss
842          1: ss
843          0: ipp
844          1: pp
845
846       "No  match" is output only if the first match attempt fails. Here is an
847       example of a failure message (the offset 4 that is specified by \>4  is
848       past the end of the subject string):
849
850           re> /xyz/
851         data> xyz\>4
852         Error -24 (bad offset value)
853
854       If  any  of the sequences \C, \G, or \L are present in a data line that
855       is successfully matched, the substrings extracted  by  the  convenience
856       functions are output with C, G, or L after the string number instead of
857       a colon. This is in addition to the normal full list. The string length
858       (that  is,  the return from the extraction function) is given in paren‐
859       theses after each string for \C and \G.
860
861       Note that whereas patterns can be continued over several lines (a plain
862       ">" prompt is used for continuations), data lines may not. However new‐
863       lines can be included in data by means of the \n escape (or  \r,  \r\n,
864       etc., depending on the newline sequence setting).
865

OUTPUT FROM THE ALTERNATIVE MATCHING FUNCTION

867
868       When the alternative matching function, pcre[16|32]_dfa_exec(), is used
869       (by means of the \D escape sequence or the -dfa command  line  option),
870       the  output  consists  of  a  list of all the matches that start at the
871       first point in the subject where there is at least one match. For exam‐
872       ple:
873
874           re> /(tang|tangerine|tan)/
875         data> yellow tangerine\D
876          0: tangerine
877          1: tang
878          2: tan
879
880       (Using  the  normal  matching function on this data finds only "tang".)
881       The longest matching string is always given first (and numbered  zero).
882       After a PCRE_ERROR_PARTIAL return, the output is "Partial match:", fol‐
883       lowed by the partially matching  substring.  (Note  that  this  is  the
884       entire  substring  that  was inspected during the partial match; it may
885       include characters before the actual match start if a lookbehind asser‐
886       tion, \K, \b, or \B was involved.)
887
888       If /g is present on the pattern, the search for further matches resumes
889       at the end of the longest match. For example:
890
891           re> /(tang|tangerine|tan)/g
892         data> yellow tangerine and tangy sultana\D
893          0: tangerine
894          1: tang
895          2: tan
896          0: tang
897          1: tan
898          0: tan
899
900       Since the matching function does not  support  substring  capture,  the
901       escape  sequences  that  are concerned with captured substrings are not
902       relevant.
903

RESTARTING AFTER A PARTIAL MATCH

905
906       When the alternative matching function has given the PCRE_ERROR_PARTIAL
907       return,  indicating that the subject partially matched the pattern, you
908       can restart the match with additional subject data by means of  the  \R
909       escape sequence. For example:
910
911           re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/
912         data> 23ja\P\D
913         Partial match: 23ja
914         data> n05\R\D
915          0: n05
916
917       For  further  information  about  partial matching, see the pcrepartial
918       documentation.
919

CALLOUTS

921
922       If the pattern contains any callout requests, pcretest's callout  func‐
923       tion  is  called  during  matching. This works with both matching func‐
924       tions. By default, the called function displays the callout number, the
925       start  and  current  positions in the text at the callout time, and the
926       next pattern item to be tested. For example:
927
928         --->pqrabcdef
929           0    ^  ^     \d
930
931       This output indicates that  callout  number  0  occurred  for  a  match
932       attempt  starting  at  the fourth character of the subject string, when
933       the pointer was at the seventh character of the data, and when the next
934       pattern  item  was  \d.  Just one circumflex is output if the start and
935       current positions are the same.
936
937       Callouts numbered 255 are assumed to be automatic callouts, inserted as
938       a  result  of the /C pattern modifier. In this case, instead of showing
939       the callout number, the offset in the pattern, preceded by a  plus,  is
940       output. For example:
941
942           re> /\d?[A-E]\*/C
943         data> E*
944         --->E*
945          +0 ^      \d?
946          +3 ^      [A-E]
947          +8 ^^     \*
948         +10 ^ ^
949          0: E*
950
951       If a pattern contains (*MARK) items, an additional line is output when‐
952       ever a change of latest mark is passed to  the  callout  function.  For
953       example:
954
955           re> /a(*MARK:X)bc/C
956         data> abc
957         --->abc
958          +0 ^       a
959          +1 ^^      (*MARK:X)
960         +10 ^^      b
961         Latest Mark: X
962         +11 ^ ^     c
963         +12 ^  ^
964          0: abc
965
966       The  mark  changes between matching "a" and "b", but stays the same for
967       the rest of the match, so nothing more is output. If, as  a  result  of
968       backtracking,  the  mark  reverts to being unset, the text "<unset>" is
969       output.
970
971       The callout function in pcretest returns zero (carry  on  matching)  by
972       default,  but you can use a \C item in a data line (as described above)
973       to change this and other parameters of the callout.
974
975       Inserting callouts can be helpful when using pcretest to check  compli‐
976       cated  regular expressions. For further information about callouts, see
977       the pcrecallout documentation.
978

NON-PRINTING CHARACTERS

980
981       When pcretest is outputting text in the compiled version of a  pattern,
982       bytes  other  than 32-126 are always treated as non-printing characters
983       are are therefore shown as hex escapes.
984
985       When pcretest is outputting text that is a matched part  of  a  subject
986       string,  it behaves in the same way, unless a different locale has been
987       set for the  pattern  (using  the  /L  modifier).  In  this  case,  the
988       isprint() function to distinguish printing and non-printing characters.
989

SAVING AND RELOADING COMPILED PATTERNS

991
992       The  facilities  described  in  this section are not available when the
993       POSIX interface to PCRE is being used, that is,  when  the  /P  pattern
994       modifier is specified.
995
996       When the POSIX interface is not in use, you can cause pcretest to write
997       a compiled pattern to a file, by following the modifiers with >  and  a
998       file name.  For example:
999
1000         /pattern/im >/some/file
1001
1002       See  the pcreprecompile documentation for a discussion about saving and
1003       re-using compiled patterns.  Note that if the pattern was  successfully
1004       studied with JIT optimization, the JIT data cannot be saved.
1005
1006       The  data  that  is  written  is  binary. The first eight bytes are the
1007       length of the compiled pattern data  followed  by  the  length  of  the
1008       optional  study  data,  each  written as four bytes in big-endian order
1009       (most significant byte first). If there is no study  data  (either  the
1010       pattern was not studied, or studying did not return any data), the sec‐
1011       ond length is zero. The lengths are followed by an exact  copy  of  the
1012       compiled  pattern.  If  there is additional study data, this (excluding
1013       any JIT data) follows immediately after  the  compiled  pattern.  After
1014       writing the file, pcretest expects to read a new pattern.
1015
1016       A  saved  pattern  can  be reloaded into pcretest by specifying < and a
1017       file name instead of a pattern. There must be no space  between  <  and
1018       the  file  name,  which  must  not  contain a < character, as otherwise
1019       pcretest will interpret the line as a pattern delimited  by  <  charac‐
1020       ters. For example:
1021
1022          re> </some/file
1023         Compiled pattern loaded from /some/file
1024         No study data
1025
1026       If  the  pattern  was previously studied with the JIT optimization, the
1027       JIT information cannot be saved and restored, and so is lost. When  the
1028       pattern  has  been  loaded, pcretest proceeds to read data lines in the
1029       usual way.
1030
1031       You can copy a file written by pcretest to a different host and  reload
1032       it  there,  even  if the new host has opposite endianness to the one on
1033       which the pattern was compiled. For example, you can compile on an  i86
1034       machine  and  run  on  a SPARC machine. When a pattern is reloaded on a
1035       host with different endianness, the confirmation message is changed to:
1036
1037         Compiled pattern (byte-inverted) loaded from /some/file
1038
1039       The test suite contains some saved pre-compiled patterns with different
1040       endianness.  These  are  reloaded  using "<!" instead of just "<". This
1041       suppresses the "(byte-inverted)" text so that the output is the same on
1042       all  hosts.  It  also forces debugging output once the pattern has been
1043       reloaded.
1044
1045       File names for saving and reloading can be absolute  or  relative,  but
1046       note  that the shell facility of expanding a file name that starts with
1047       a tilde (~) is not available.
1048
1049       The ability to save and reload files in pcretest is intended for  test‐
1050       ing  and experimentation. It is not intended for production use because
1051       only a single pattern can be written to a file. Furthermore,  there  is
1052       no  facility  for  supplying  custom  character  tables  for use with a
1053       reloaded pattern. If the original  pattern  was  compiled  with  custom
1054       tables,  an  attempt to match a subject string using a reloaded pattern
1055       is likely to cause pcretest to crash.  Finally, if you attempt to  load
1056       a file that is not in the correct format, the result is undefined.
1057

SEE ALSO

1059
1060       pcre(3),  pcre16(3),  pcre32(3),  pcreapi(3),  pcrecallout(3), pcrejit,
1061       pcrematching(3), pcrepartial(d), pcrepattern(3), pcreprecompile(3).
1062

AUTHOR

1064
1065       Philip Hazel
1066       University Computing Service
1067       Cambridge CB2 3QH, England.
1068

REVISION

1070
1071       Last updated: 10 February 2020
1072       Copyright (c) 1997-2020 University of Cambridge.
1073
1074
1075
1076PCRE 8.44                      10 February 2020                    PCRETEST(1)
Impressum