1PERLRETUT(1)           Perl Programmers Reference Guide           PERLRETUT(1)
2
3
4

NAME

6       perlretut - Perl regular expressions tutorial
7

DESCRIPTION

9       This page provides a basic tutorial on understanding, creating and
10       using regular expressions in Perl.  It serves as a complement to the
11       reference page on regular expressions perlre.  Regular expressions are
12       an integral part of the "m//", "s///", "qr//" and "split" operators and
13       so this tutorial also overlaps with "Regexp Quote-Like Operators" in
14       perlop and "split" in perlfunc.
15
16       Perl is widely renowned for excellence in text processing, and regular
17       expressions are one of the big factors behind this fame.  Perl regular
18       expressions display an efficiency and flexibility unknown in most other
19       computer languages.  Mastering even the basics of regular expressions
20       will allow you to manipulate text with surprising ease.
21
22       What is a regular expression?  A regular expression is simply a string
23       that describes a pattern.  Patterns are in common use these days;
24       examples are the patterns typed into a search engine to find web pages
25       and the patterns used to list files in a directory, e.g., "ls *.txt" or
26       "dir *.*".  In Perl, the patterns described by regular expressions are
27       used to search strings, extract desired parts of strings, and to do
28       search and replace operations.
29
30       Regular expressions have the undeserved reputation of being abstract
31       and difficult to understand.  Regular expressions are constructed using
32       simple concepts like conditionals and loops and are no more difficult
33       to understand than the corresponding "if" conditionals and "while"
34       loops in the Perl language itself.  In fact, the main challenge in
35       learning regular expressions is just getting used to the terse notation
36       used to express these concepts.
37
38       This tutorial flattens the learning curve by discussing regular
39       expression concepts, along with their notation, one at a time and with
40       many examples.  The first part of the tutorial will progress from the
41       simplest word searches to the basic regular expression concepts.  If
42       you master the first part, you will have all the tools needed to solve
43       about 98% of your needs.  The second part of the tutorial is for those
44       comfortable with the basics and hungry for more power tools.  It
45       discusses the more advanced regular expression operators and introduces
46       the latest cutting-edge innovations.
47
48       A note: to save time, 'regular expression' is often abbreviated as
49       regexp or regex.  Regexp is a more natural abbreviation than regex, but
50       is harder to pronounce.  The Perl pod documentation is evenly split on
51       regexp vs regex; in Perl, there is more than one way to abbreviate it.
52       We'll use regexp in this tutorial.
53

Part 1: The basics

55   Simple word matching
56       The simplest regexp is simply a word, or more generally, a string of
57       characters.  A regexp consisting of a word matches any string that
58       contains that word:
59
60           "Hello World" =~ /World/;  # matches
61
62       What is this Perl statement all about? "Hello World" is a simple
63       double-quoted string.  "World" is the regular expression and the "//"
64       enclosing "/World/" tells Perl to search a string for a match.  The
65       operator "=~" associates the string with the regexp match and produces
66       a true value if the regexp matched, or false if the regexp did not
67       match.  In our case, "World" matches the second word in "Hello World",
68       so the expression is true.  Expressions like this are useful in
69       conditionals:
70
71           if ("Hello World" =~ /World/) {
72               print "It matches\n";
73           }
74           else {
75               print "It doesn't match\n";
76           }
77
78       There are useful variations on this theme.  The sense of the match can
79       be reversed by using the "!~" operator:
80
81           if ("Hello World" !~ /World/) {
82               print "It doesn't match\n";
83           }
84           else {
85               print "It matches\n";
86           }
87
88       The literal string in the regexp can be replaced by a variable:
89
90           $greeting = "World";
91           if ("Hello World" =~ /$greeting/) {
92               print "It matches\n";
93           }
94           else {
95               print "It doesn't match\n";
96           }
97
98       If you're matching against the special default variable $_, the "$_ =~"
99       part can be omitted:
100
101           $_ = "Hello World";
102           if (/World/) {
103               print "It matches\n";
104           }
105           else {
106               print "It doesn't match\n";
107           }
108
109       And finally, the "//" default delimiters for a match can be changed to
110       arbitrary delimiters by putting an 'm' out front:
111
112           "Hello World" =~ m!World!;   # matches, delimited by '!'
113           "Hello World" =~ m{World};   # matches, note the matching '{}'
114           "/usr/bin/perl" =~ m"/perl"; # matches after '/usr/bin',
115                                        # '/' becomes an ordinary char
116
117       "/World/", "m!World!", and "m{World}" all represent the same thing.
118       When, e.g., the quote (""") is used as a delimiter, the forward slash
119       '/' becomes an ordinary character and can be used in this regexp
120       without trouble.
121
122       Let's consider how different regexps would match "Hello World":
123
124           "Hello World" =~ /world/;  # doesn't match
125           "Hello World" =~ /o W/;    # matches
126           "Hello World" =~ /oW/;     # doesn't match
127           "Hello World" =~ /World /; # doesn't match
128
129       The first regexp "world" doesn't match because regexps are case-
130       sensitive.  The second regexp matches because the substring 'o W'
131       occurs in the string "Hello World".  The space character ' ' is treated
132       like any other character in a regexp and is needed to match in this
133       case.  The lack of a space character is the reason the third regexp
134       'oW' doesn't match.  The fourth regexp 'World ' doesn't match because
135       there is a space at the end of the regexp, but not at the end of the
136       string.  The lesson here is that regexps must match a part of the
137       string exactly in order for the statement to be true.
138
139       If a regexp matches in more than one place in the string, Perl will
140       always match at the earliest possible point in the string:
141
142           "Hello World" =~ /o/;       # matches 'o' in 'Hello'
143           "That hat is red" =~ /hat/; # matches 'hat' in 'That'
144
145       With respect to character matching, there are a few more points you
146       need to know about.   First of all, not all characters can be used 'as
147       is' in a match.  Some characters, called metacharacters, are reserved
148       for use in regexp notation.  The metacharacters are
149
150           {}[]()^$.|*+?\
151
152       The significance of each of these will be explained in the rest of the
153       tutorial, but for now, it is important only to know that a
154       metacharacter can be matched by putting a backslash before it:
155
156           "2+2=4" =~ /2+2/;    # doesn't match, + is a metacharacter
157           "2+2=4" =~ /2\+2/;   # matches, \+ is treated like an ordinary +
158           "The interval is [0,1)." =~ /[0,1)./     # is a syntax error!
159           "The interval is [0,1)." =~ /\[0,1\)\./  # matches
160           "#!/usr/bin/perl" =~ /#!\/usr\/bin\/perl/;  # matches
161
162       In the last regexp, the forward slash '/' is also backslashed, because
163       it is used to delimit the regexp.  This can lead to LTS (leaning
164       toothpick syndrome), however, and it is often more readable to change
165       delimiters.
166
167           "#!/usr/bin/perl" =~ m!#\!/usr/bin/perl!;  # easier to read
168
169       The backslash character '\' is a metacharacter itself and needs to be
170       backslashed:
171
172           'C:\WIN32' =~ /C:\\WIN/;   # matches
173
174       In addition to the metacharacters, there are some ASCII characters
175       which don't have printable character equivalents and are instead
176       represented by escape sequences.  Common examples are "\t" for a tab,
177       "\n" for a newline, "\r" for a carriage return and "\a" for a bell (or
178       alert).  If your string is better thought of as a sequence of arbitrary
179       bytes, the octal escape sequence, e.g., "\033", or hexadecimal escape
180       sequence, e.g., "\x1B" may be a more natural representation for your
181       bytes.  Here are some examples of escapes:
182
183           "1000\t2000" =~ m(0\t2)   # matches
184           "1000\n2000" =~ /0\n20/   # matches
185           "1000\t2000" =~ /\000\t2/ # doesn't match, "0" ne "\000"
186           "cat"   =~ /\o{143}\x61\x74/ # matches in ASCII, but a weird way
187                                        # to spell cat
188
189       If you've been around Perl a while, all this talk of escape sequences
190       may seem familiar.  Similar escape sequences are used in double-quoted
191       strings and in fact the regexps in Perl are mostly treated as double-
192       quoted strings.  This means that variables can be used in regexps as
193       well.  Just like double-quoted strings, the values of the variables in
194       the regexp will be substituted in before the regexp is evaluated for
195       matching purposes.  So we have:
196
197           $foo = 'house';
198           'housecat' =~ /$foo/;      # matches
199           'cathouse' =~ /cat$foo/;   # matches
200           'housecat' =~ /${foo}cat/; # matches
201
202       So far, so good.  With the knowledge above you can already perform
203       searches with just about any literal string regexp you can dream up.
204       Here is a very simple emulation of the Unix grep program:
205
206           % cat > simple_grep
207           #!/usr/bin/perl
208           $regexp = shift;
209           while (<>) {
210               print if /$regexp/;
211           }
212           ^D
213
214           % chmod +x simple_grep
215
216           % simple_grep abba /usr/dict/words
217           Babbage
218           cabbage
219           cabbages
220           sabbath
221           Sabbathize
222           Sabbathizes
223           sabbatical
224           scabbard
225           scabbards
226
227       This program is easy to understand.  "#!/usr/bin/perl" is the standard
228       way to invoke a perl program from the shell.  "$regexp = shift;" saves
229       the first command line argument as the regexp to be used, leaving the
230       rest of the command line arguments to be treated as files.
231       "while (<>)" loops over all the lines in all the files.  For each line,
232       "print if /$regexp/;" prints the line if the regexp matches the line.
233       In this line, both "print" and "/$regexp/" use the default variable $_
234       implicitly.
235
236       With all of the regexps above, if the regexp matched anywhere in the
237       string, it was considered a match.  Sometimes, however, we'd like to
238       specify where in the string the regexp should try to match.  To do
239       this, we would use the anchor metacharacters "^" and "$".  The anchor
240       "^" means match at the beginning of the string and the anchor "$" means
241       match at the end of the string, or before a newline at the end of the
242       string.  Here is how they are used:
243
244           "housekeeper" =~ /keeper/;    # matches
245           "housekeeper" =~ /^keeper/;   # doesn't match
246           "housekeeper" =~ /keeper$/;   # matches
247           "housekeeper\n" =~ /keeper$/; # matches
248
249       The second regexp doesn't match because "^" constrains "keeper" to
250       match only at the beginning of the string, but "housekeeper" has keeper
251       starting in the middle.  The third regexp does match, since the "$"
252       constrains "keeper" to match only at the end of the string.
253
254       When both "^" and "$" are used at the same time, the regexp has to
255       match both the beginning and the end of the string, i.e., the regexp
256       matches the whole string.  Consider
257
258           "keeper" =~ /^keep$/;      # doesn't match
259           "keeper" =~ /^keeper$/;    # matches
260           ""       =~ /^$/;          # ^$ matches an empty string
261
262       The first regexp doesn't match because the string has more to it than
263       "keep".  Since the second regexp is exactly the string, it matches.
264       Using both "^" and "$" in a regexp forces the complete string to match,
265       so it gives you complete control over which strings match and which
266       don't.  Suppose you are looking for a fellow named bert, off in a
267       string by himself:
268
269           "dogbert" =~ /bert/;   # matches, but not what you want
270
271           "dilbert" =~ /^bert/;  # doesn't match, but ..
272           "bertram" =~ /^bert/;  # matches, so still not good enough
273
274           "bertram" =~ /^bert$/; # doesn't match, good
275           "dilbert" =~ /^bert$/; # doesn't match, good
276           "bert"    =~ /^bert$/; # matches, perfect
277
278       Of course, in the case of a literal string, one could just as easily
279       use the string comparison "$string eq 'bert'" and it would be more
280       efficient.   The  "^...$" regexp really becomes useful when we add in
281       the more powerful regexp tools below.
282
283   Using character classes
284       Although one can already do quite a lot with the literal string regexps
285       above, we've only scratched the surface of regular expression
286       technology.  In this and subsequent sections we will introduce regexp
287       concepts (and associated metacharacter notations) that will allow a
288       regexp to represent not just a single character sequence, but a whole
289       class of them.
290
291       One such concept is that of a character class.  A character class
292       allows a set of possible characters, rather than just a single
293       character, to match at a particular point in a regexp.  Character
294       classes are denoted by brackets "[...]", with the set of characters to
295       be possibly matched inside.  Here are some examples:
296
297           /cat/;       # matches 'cat'
298           /[bcr]at/;   # matches 'bat, 'cat', or 'rat'
299           /item[0123456789]/;  # matches 'item0' or ... or 'item9'
300           "abc" =~ /[cab]/;    # matches 'a'
301
302       In the last statement, even though 'c' is the first character in the
303       class, 'a' matches because the first character position in the string
304       is the earliest point at which the regexp can match.
305
306           /[yY][eE][sS]/;      # match 'yes' in a case-insensitive way
307                                # 'yes', 'Yes', 'YES', etc.
308
309       This regexp displays a common task: perform a case-insensitive match.
310       Perl provides a way of avoiding all those brackets by simply appending
311       an 'i' to the end of the match.  Then "/[yY][eE][sS]/;" can be
312       rewritten as "/yes/i;".  The 'i' stands for case-insensitive and is an
313       example of a modifier of the matching operation.  We will meet other
314       modifiers later in the tutorial.
315
316       We saw in the section above that there were ordinary characters, which
317       represented themselves, and special characters, which needed a
318       backslash "\" to represent themselves.  The same is true in a character
319       class, but the sets of ordinary and special characters inside a
320       character class are different than those outside a character class.
321       The special characters for a character class are "-]\^$" (and the
322       pattern delimiter, whatever it is).  "]" is special because it denotes
323       the end of a character class.  "$" is special because it denotes a
324       scalar variable.  "\" is special because it is used in escape
325       sequences, just like above.  Here is how the special characters "]$\"
326       are handled:
327
328          /[\]c]def/; # matches ']def' or 'cdef'
329          $x = 'bcr';
330          /[$x]at/;   # matches 'bat', 'cat', or 'rat'
331          /[\$x]at/;  # matches '$at' or 'xat'
332          /[\\$x]at/; # matches '\at', 'bat, 'cat', or 'rat'
333
334       The last two are a little tricky.  In "[\$x]", the backslash protects
335       the dollar sign, so the character class has two members "$" and "x".
336       In "[\\$x]", the backslash is protected, so $x is treated as a variable
337       and substituted in double quote fashion.
338
339       The special character '-' acts as a range operator within character
340       classes, so that a contiguous set of characters can be written as a
341       range.  With ranges, the unwieldy "[0123456789]" and "[abc...xyz]"
342       become the svelte "[0-9]" and "[a-z]".  Some examples are
343
344           /item[0-9]/;  # matches 'item0' or ... or 'item9'
345           /[0-9bx-z]aa/;  # matches '0aa', ..., '9aa',
346                           # 'baa', 'xaa', 'yaa', or 'zaa'
347           /[0-9a-fA-F]/;  # matches a hexadecimal digit
348           /[0-9a-zA-Z_]/; # matches a "word" character,
349                           # like those in a Perl variable name
350
351       If '-' is the first or last character in a character class, it is
352       treated as an ordinary character; "[-ab]", "[ab-]" and "[a\-b]" are all
353       equivalent.
354
355       The special character "^" in the first position of a character class
356       denotes a negated character class, which matches any character but
357       those in the brackets.  Both "[...]" and "[^...]" must match a
358       character, or the match fails.  Then
359
360           /[^a]at/;  # doesn't match 'aat' or 'at', but matches
361                      # all other 'bat', 'cat, '0at', '%at', etc.
362           /[^0-9]/;  # matches a non-numeric character
363           /[a^]at/;  # matches 'aat' or '^at'; here '^' is ordinary
364
365       Now, even "[0-9]" can be a bother to write multiple times, so in the
366       interest of saving keystrokes and making regexps more readable, Perl
367       has several abbreviations for common character classes, as shown below.
368       Since the introduction of Unicode, unless the "//a" modifier is in
369       effect, these character classes match more than just a few characters
370       in the ASCII range.
371
372       ·   \d matches a digit, not just [0-9] but also digits from non-roman
373           scripts
374
375       ·   \s matches a whitespace character, the set [\ \t\r\n\f] and others
376
377       ·   \w matches a word character (alphanumeric or _), not just
378           [0-9a-zA-Z_] but also digits and characters from non-roman scripts
379
380       ·   \D is a negated \d; it represents any other character than a digit,
381           or [^\d]
382
383       ·   \S is a negated \s; it represents any non-whitespace character
384           [^\s]
385
386       ·   \W is a negated \w; it represents any non-word character [^\w]
387
388       ·   The period '.' matches any character but "\n" (unless the modifier
389           "//s" is in effect, as explained below).
390
391       ·   \N, like the period, matches any character but "\n", but it does so
392           regardless of whether the modifier "//s" is in effect.
393
394       The "//a" modifier, available starting in Perl 5.14,  is used to
395       restrict the matches of \d, \s, and \w to just those in the ASCII
396       range.  It is useful to keep your program from being needlessly exposed
397       to full Unicode (and its accompanying security considerations) when all
398       you want is to process English-like text.  (The "a" may be doubled,
399       "//aa", to provide even more restrictions, preventing case-insensitive
400       matching of ASCII with non-ASCII characters; otherwise a Unicode
401       "Kelvin Sign" would caselessly match a "k" or "K".)
402
403       The "\d\s\w\D\S\W" abbreviations can be used both inside and outside of
404       character classes.  Here are some in use:
405
406           /\d\d:\d\d:\d\d/; # matches a hh:mm:ss time format
407           /[\d\s]/;         # matches any digit or whitespace character
408           /\w\W\w/;         # matches a word char, followed by a
409                             # non-word char, followed by a word char
410           /..rt/;           # matches any two chars, followed by 'rt'
411           /end\./;          # matches 'end.'
412           /end[.]/;         # same thing, matches 'end.'
413
414       Because a period is a metacharacter, it needs to be escaped to match as
415       an ordinary period. Because, for example, "\d" and "\w" are sets of
416       characters, it is incorrect to think of "[^\d\w]" as "[\D\W]"; in fact
417       "[^\d\w]" is the same as "[^\w]", which is the same as "[\W]". Think
418       DeMorgan's laws.
419
420       An anchor useful in basic regexps is the word anchor "\b".  This
421       matches a boundary between a word character and a non-word character
422       "\w\W" or "\W\w":
423
424           $x = "Housecat catenates house and cat";
425           $x =~ /cat/;    # matches cat in 'housecat'
426           $x =~ /\bcat/;  # matches cat in 'catenates'
427           $x =~ /cat\b/;  # matches cat in 'housecat'
428           $x =~ /\bcat\b/;  # matches 'cat' at end of string
429
430       Note in the last example, the end of the string is considered a word
431       boundary.
432
433       You might wonder why '.' matches everything but "\n" - why not every
434       character? The reason is that often one is matching against lines and
435       would like to ignore the newline characters.  For instance, while the
436       string "\n" represents one line, we would like to think of it as empty.
437       Then
438
439           ""   =~ /^$/;    # matches
440           "\n" =~ /^$/;    # matches, $ anchors before "\n"
441
442           ""   =~ /./;      # doesn't match; it needs a char
443           ""   =~ /^.$/;    # doesn't match; it needs a char
444           "\n" =~ /^.$/;    # doesn't match; it needs a char other than "\n"
445           "a"  =~ /^.$/;    # matches
446           "a\n"  =~ /^.$/;  # matches, $ anchors before "\n"
447
448       This behavior is convenient, because we usually want to ignore newlines
449       when we count and match characters in a line.  Sometimes, however, we
450       want to keep track of newlines.  We might even want "^" and "$" to
451       anchor at the beginning and end of lines within the string, rather than
452       just the beginning and end of the string.  Perl allows us to choose
453       between ignoring and paying attention to newlines by using the "//s"
454       and "//m" modifiers.  "//s" and "//m" stand for single line and multi-
455       line and they determine whether a string is to be treated as one
456       continuous string, or as a set of lines.  The two modifiers affect two
457       aspects of how the regexp is interpreted: 1) how the '.' character
458       class is defined, and 2) where the anchors "^" and "$" are able to
459       match.  Here are the four possible combinations:
460
461       ·   no modifiers (//): Default behavior.  '.' matches any character
462           except "\n".  "^" matches only at the beginning of the string and
463           "$" matches only at the end or before a newline at the end.
464
465       ·   s modifier (//s): Treat string as a single long line.  '.' matches
466           any character, even "\n".  "^" matches only at the beginning of the
467           string and "$" matches only at the end or before a newline at the
468           end.
469
470       ·   m modifier (//m): Treat string as a set of multiple lines.  '.'
471           matches any character except "\n".  "^" and "$" are able to match
472           at the start or end of any line within the string.
473
474       ·   both s and m modifiers (//sm): Treat string as a single long line,
475           but detect multiple lines.  '.' matches any character, even "\n".
476           "^" and "$", however, are able to match at the start or end of any
477           line within the string.
478
479       Here are examples of "//s" and "//m" in action:
480
481           $x = "There once was a girl\nWho programmed in Perl\n";
482
483           $x =~ /^Who/;   # doesn't match, "Who" not at start of string
484           $x =~ /^Who/s;  # doesn't match, "Who" not at start of string
485           $x =~ /^Who/m;  # matches, "Who" at start of second line
486           $x =~ /^Who/sm; # matches, "Who" at start of second line
487
488           $x =~ /girl.Who/;   # doesn't match, "." doesn't match "\n"
489           $x =~ /girl.Who/s;  # matches, "." matches "\n"
490           $x =~ /girl.Who/m;  # doesn't match, "." doesn't match "\n"
491           $x =~ /girl.Who/sm; # matches, "." matches "\n"
492
493       Most of the time, the default behavior is what is wanted, but "//s" and
494       "//m" are occasionally very useful.  If "//m" is being used, the start
495       of the string can still be matched with "\A" and the end of the string
496       can still be matched with the anchors "\Z" (matches both the end and
497       the newline before, like "$"), and "\z" (matches only the end):
498
499           $x =~ /^Who/m;   # matches, "Who" at start of second line
500           $x =~ /\AWho/m;  # doesn't match, "Who" is not at start of string
501
502           $x =~ /girl$/m;  # matches, "girl" at end of first line
503           $x =~ /girl\Z/m; # doesn't match, "girl" is not at end of string
504
505           $x =~ /Perl\Z/m; # matches, "Perl" is at newline before end
506           $x =~ /Perl\z/m; # doesn't match, "Perl" is not at end of string
507
508       We now know how to create choices among classes of characters in a
509       regexp.  What about choices among words or character strings? Such
510       choices are described in the next section.
511
512   Matching this or that
513       Sometimes we would like our regexp to be able to match different
514       possible words or character strings.  This is accomplished by using the
515       alternation metacharacter "|".  To match "dog" or "cat", we form the
516       regexp "dog|cat".  As before, Perl will try to match the regexp at the
517       earliest possible point in the string.  At each character position,
518       Perl will first try to match the first alternative, "dog".  If "dog"
519       doesn't match, Perl will then try the next alternative, "cat".  If
520       "cat" doesn't match either, then the match fails and Perl moves to the
521       next position in the string.  Some examples:
522
523           "cats and dogs" =~ /cat|dog|bird/;  # matches "cat"
524           "cats and dogs" =~ /dog|cat|bird/;  # matches "cat"
525
526       Even though "dog" is the first alternative in the second regexp, "cat"
527       is able to match earlier in the string.
528
529           "cats"          =~ /c|ca|cat|cats/; # matches "c"
530           "cats"          =~ /cats|cat|ca|c/; # matches "cats"
531
532       Here, all the alternatives match at the first string position, so the
533       first alternative is the one that matches.  If some of the alternatives
534       are truncations of the others, put the longest ones first to give them
535       a chance to match.
536
537           "cab" =~ /a|b|c/ # matches "c"
538                            # /a|b|c/ == /[abc]/
539
540       The last example points out that character classes are like
541       alternations of characters.  At a given character position, the first
542       alternative that allows the regexp match to succeed will be the one
543       that matches.
544
545   Grouping things and hierarchical matching
546       Alternation allows a regexp to choose among alternatives, but by itself
547       it is unsatisfying.  The reason is that each alternative is a whole
548       regexp, but sometime we want alternatives for just part of a regexp.
549       For instance, suppose we want to search for housecats or housekeepers.
550       The regexp "housecat|housekeeper" fits the bill, but is inefficient
551       because we had to type "house" twice.  It would be nice to have parts
552       of the regexp be constant, like "house", and some parts have
553       alternatives, like "cat|keeper".
554
555       The grouping metacharacters "()" solve this problem.  Grouping allows
556       parts of a regexp to be treated as a single unit.  Parts of a regexp
557       are grouped by enclosing them in parentheses.  Thus we could solve the
558       "housecat|housekeeper" by forming the regexp as "house(cat|keeper)".
559       The regexp "house(cat|keeper)" means match "house" followed by either
560       "cat" or "keeper".  Some more examples are
561
562           /(a|b)b/;    # matches 'ab' or 'bb'
563           /(ac|b)b/;   # matches 'acb' or 'bb'
564           /(^a|b)c/;   # matches 'ac' at start of string or 'bc' anywhere
565           /(a|[bc])d/; # matches 'ad', 'bd', or 'cd'
566
567           /house(cat|)/;  # matches either 'housecat' or 'house'
568           /house(cat(s|)|)/;  # matches either 'housecats' or 'housecat' or
569                               # 'house'.  Note groups can be nested.
570
571           /(19|20|)\d\d/;  # match years 19xx, 20xx, or the Y2K problem, xx
572           "20" =~ /(19|20|)\d\d/;  # matches the null alternative '()\d\d',
573                                    # because '20\d\d' can't match
574
575       Alternations behave the same way in groups as out of them: at a given
576       string position, the leftmost alternative that allows the regexp to
577       match is taken.  So in the last example at the first string position,
578       "20" matches the second alternative, but there is nothing left over to
579       match the next two digits "\d\d".  So Perl moves on to the next
580       alternative, which is the null alternative and that works, since "20"
581       is two digits.
582
583       The process of trying one alternative, seeing if it matches, and moving
584       on to the next alternative, while going back in the string from where
585       the previous alternative was tried, if it doesn't, is called
586       backtracking.  The term 'backtracking' comes from the idea that
587       matching a regexp is like a walk in the woods.  Successfully matching a
588       regexp is like arriving at a destination.  There are many possible
589       trailheads, one for each string position, and each one is tried in
590       order, left to right.  From each trailhead there may be many paths,
591       some of which get you there, and some which are dead ends.  When you
592       walk along a trail and hit a dead end, you have to backtrack along the
593       trail to an earlier point to try another trail.  If you hit your
594       destination, you stop immediately and forget about trying all the other
595       trails.  You are persistent, and only if you have tried all the trails
596       from all the trailheads and not arrived at your destination, do you
597       declare failure.  To be concrete, here is a step-by-step analysis of
598       what Perl does when it tries to match the regexp
599
600           "abcde" =~ /(abd|abc)(df|d|de)/;
601
602       0   Start with the first letter in the string 'a'.
603
604       1   Try the first alternative in the first group 'abd'.
605
606       2   Match 'a' followed by 'b'. So far so good.
607
608       3   'd' in the regexp doesn't match 'c' in the string - a dead end.  So
609           backtrack two characters and pick the second alternative in the
610           first group 'abc'.
611
612       4   Match 'a' followed by 'b' followed by 'c'.  We are on a roll and
613           have satisfied the first group. Set $1 to 'abc'.
614
615       5   Move on to the second group and pick the first alternative 'df'.
616
617       6   Match the 'd'.
618
619       7   'f' in the regexp doesn't match 'e' in the string, so a dead end.
620           Backtrack one character and pick the second alternative in the
621           second group 'd'.
622
623       8   'd' matches. The second grouping is satisfied, so set $2 to 'd'.
624
625       9   We are at the end of the regexp, so we are done! We have matched
626           'abcd' out of the string "abcde".
627
628       There are a couple of things to note about this analysis.  First, the
629       third alternative in the second group 'de' also allows a match, but we
630       stopped before we got to it - at a given character position, leftmost
631       wins.  Second, we were able to get a match at the first character
632       position of the string 'a'.  If there were no matches at the first
633       position, Perl would move to the second character position 'b' and
634       attempt the match all over again.  Only when all possible paths at all
635       possible character positions have been exhausted does Perl give up and
636       declare "$string =~ /(abd|abc)(df|d|de)/;" to be false.
637
638       Even with all this work, regexp matching happens remarkably fast.  To
639       speed things up, Perl compiles the regexp into a compact sequence of
640       opcodes that can often fit inside a processor cache.  When the code is
641       executed, these opcodes can then run at full throttle and search very
642       quickly.
643
644   Extracting matches
645       The grouping metacharacters "()" also serve another completely
646       different function: they allow the extraction of the parts of a string
647       that matched.  This is very useful to find out what matched and for
648       text processing in general.  For each grouping, the part that matched
649       inside goes into the special variables $1, $2, etc.  They can be used
650       just as ordinary variables:
651
652           # extract hours, minutes, seconds
653           if ($time =~ /(\d\d):(\d\d):(\d\d)/) {    # match hh:mm:ss format
654               $hours = $1;
655               $minutes = $2;
656               $seconds = $3;
657           }
658
659       Now, we know that in scalar context, "$time =~ /(\d\d):(\d\d):(\d\d)/"
660       returns a true or false value.  In list context, however, it returns
661       the list of matched values "($1,$2,$3)".  So we could write the code
662       more compactly as
663
664           # extract hours, minutes, seconds
665           ($hours, $minutes, $second) = ($time =~ /(\d\d):(\d\d):(\d\d)/);
666
667       If the groupings in a regexp are nested, $1 gets the group with the
668       leftmost opening parenthesis, $2 the next opening parenthesis, etc.
669       Here is a regexp with nested groups:
670
671           /(ab(cd|ef)((gi)|j))/;
672            1  2      34
673
674       If this regexp matches, $1 contains a string starting with 'ab', $2 is
675       either set to 'cd' or 'ef', $3 equals either 'gi' or 'j', and $4 is
676       either set to 'gi', just like $3, or it remains undefined.
677
678       For convenience, Perl sets $+ to the string held by the highest
679       numbered $1, $2,... that got assigned (and, somewhat related, $^N to
680       the value of the $1, $2,... most-recently assigned; i.e. the $1, $2,...
681       associated with the rightmost closing parenthesis used in the match).
682
683   Backreferences
684       Closely associated with the matching variables $1, $2, ... are the
685       backreferences "\g1", "\g2",...  Backreferences are simply matching
686       variables that can be used inside a regexp.  This is a really nice
687       feature; what matches later in a regexp is made to depend on what
688       matched earlier in the regexp.  Suppose we wanted to look for doubled
689       words in a text, like 'the the'.  The following regexp finds all
690       3-letter doubles with a space in between:
691
692           /\b(\w\w\w)\s\g1\b/;
693
694       The grouping assigns a value to \g1, so that the same 3-letter sequence
695       is used for both parts.
696
697       A similar task is to find words consisting of two identical parts:
698
699           % simple_grep '^(\w\w\w\w|\w\w\w|\w\w|\w)\g1$' /usr/dict/words
700           beriberi
701           booboo
702           coco
703           mama
704           murmur
705           papa
706
707       The regexp has a single grouping which considers 4-letter combinations,
708       then 3-letter combinations, etc., and uses "\g1" to look for a repeat.
709       Although $1 and "\g1" represent the same thing, care should be taken to
710       use matched variables $1, $2,... only outside a regexp and
711       backreferences "\g1", "\g2",... only inside a regexp; not doing so may
712       lead to surprising and unsatisfactory results.
713
714   Relative backreferences
715       Counting the opening parentheses to get the correct number for a
716       backreference is error-prone as soon as there is more than one
717       capturing group.  A more convenient technique became available with
718       Perl 5.10: relative backreferences. To refer to the immediately
719       preceding capture group one now may write "\g{-1}", the next but last
720       is available via "\g{-2}", and so on.
721
722       Another good reason in addition to readability and maintainability for
723       using relative backreferences is illustrated by the following example,
724       where a simple pattern for matching peculiar strings is used:
725
726           $a99a = '([a-z])(\d)\g2\g1';   # matches a11a, g22g, x33x, etc.
727
728       Now that we have this pattern stored as a handy string, we might feel
729       tempted to use it as a part of some other pattern:
730
731           $line = "code=e99e";
732           if ($line =~ /^(\w+)=$a99a$/){   # unexpected behavior!
733               print "$1 is valid\n";
734           } else {
735               print "bad line: '$line'\n";
736           }
737
738       But this doesn't match, at least not the way one might expect. Only
739       after inserting the interpolated $a99a and looking at the resulting
740       full text of the regexp is it obvious that the backreferences have
741       backfired. The subexpression "(\w+)" has snatched number 1 and demoted
742       the groups in $a99a by one rank. This can be avoided by using relative
743       backreferences:
744
745           $a99a = '([a-z])(\d)\g{-1}\g{-2}';  # safe for being interpolated
746
747   Named backreferences
748       Perl 5.10 also introduced named capture groups and named
749       backreferences.  To attach a name to a capturing group, you write
750       either "(?<name>...)" or "(?'name'...)".  The backreference may then be
751       written as "\g{name}".  It is permissible to attach the same name to
752       more than one group, but then only the leftmost one of the eponymous
753       set can be referenced.  Outside of the pattern a named capture group is
754       accessible through the "%+" hash.
755
756       Assuming that we have to match calendar dates which may be given in one
757       of the three formats yyyy-mm-dd, mm/dd/yyyy or dd.mm.yyyy, we can write
758       three suitable patterns where we use 'd', 'm' and 'y' respectively as
759       the names of the groups capturing the pertaining components of a date.
760       The matching operation combines the three patterns as alternatives:
761
762           $fmt1 = '(?<y>\d\d\d\d)-(?<m>\d\d)-(?<d>\d\d)';
763           $fmt2 = '(?<m>\d\d)/(?<d>\d\d)/(?<y>\d\d\d\d)';
764           $fmt3 = '(?<d>\d\d)\.(?<m>\d\d)\.(?<y>\d\d\d\d)';
765           for my $d qw( 2006-10-21 15.01.2007 10/31/2005 ){
766               if ( $d =~ m{$fmt1|$fmt2|$fmt3} ){
767                   print "day=$+{d} month=$+{m} year=$+{y}\n";
768               }
769           }
770
771       If any of the alternatives matches, the hash "%+" is bound to contain
772       the three key-value pairs.
773
774   Alternative capture group numbering
775       Yet another capturing group numbering technique (also as from Perl
776       5.10) deals with the problem of referring to groups within a set of
777       alternatives.  Consider a pattern for matching a time of the day, civil
778       or military style:
779
780           if ( $time =~ /(\d\d|\d):(\d\d)|(\d\d)(\d\d)/ ){
781               # process hour and minute
782           }
783
784       Processing the results requires an additional if statement to determine
785       whether $1 and $2 or $3 and $4 contain the goodies. It would be easier
786       if we could use group numbers 1 and 2 in second alternative as well,
787       and this is exactly what the parenthesized construct "(?|...)", set
788       around an alternative achieves. Here is an extended version of the
789       previous pattern:
790
791           if ( $time =~ /(?|(\d\d|\d):(\d\d)|(\d\d)(\d\d))\s+([A-Z][A-Z][A-Z])/ ){
792               print "hour=$1 minute=$2 zone=$3\n";
793           }
794
795       Within the alternative numbering group, group numbers start at the same
796       position for each alternative. After the group, numbering continues
797       with one higher than the maximum reached across all the alternatives.
798
799   Position information
800       In addition to what was matched, Perl (since 5.6.0) also provides the
801       positions of what was matched as contents of the "@-" and "@+" arrays.
802       "$-[0]" is the position of the start of the entire match and $+[0] is
803       the position of the end. Similarly, "$-[n]" is the position of the
804       start of the $n match and $+[n] is the position of the end. If $n is
805       undefined, so are "$-[n]" and $+[n]. Then this code
806
807           $x = "Mmm...donut, thought Homer";
808           $x =~ /^(Mmm|Yech)\.\.\.(donut|peas)/; # matches
809           foreach $expr (1..$#-) {
810               print "Match $expr: '${$expr}' at position ($-[$expr],$+[$expr])\n";
811           }
812
813       prints
814
815           Match 1: 'Mmm' at position (0,3)
816           Match 2: 'donut' at position (6,11)
817
818       Even if there are no groupings in a regexp, it is still possible to
819       find out what exactly matched in a string.  If you use them, Perl will
820       set "$`" to the part of the string before the match, will set $& to the
821       part of the string that matched, and will set "$'" to the part of the
822       string after the match.  An example:
823
824           $x = "the cat caught the mouse";
825           $x =~ /cat/;  # $` = 'the ', $& = 'cat', $' = ' caught the mouse'
826           $x =~ /the/;  # $` = '', $& = 'the', $' = ' cat caught the mouse'
827
828       In the second match, "$`" equals '' because the regexp matched at the
829       first character position in the string and stopped; it never saw the
830       second 'the'.  It is important to note that using "$`" and "$'" slows
831       down regexp matching quite a bit, while $& slows it down to a lesser
832       extent, because if they are used in one regexp in a program, they are
833       generated for all regexps in the program.  So if raw performance is a
834       goal of your application, they should be avoided.  If you need to
835       extract the corresponding substrings, use "@-" and "@+" instead:
836
837           $` is the same as substr( $x, 0, $-[0] )
838           $& is the same as substr( $x, $-[0], $+[0]-$-[0] )
839           $' is the same as substr( $x, $+[0] )
840
841       As of Perl 5.10, the "${^PREMATCH}", "${^MATCH}" and "${^POSTMATCH}"
842       variables may be used. These are only set if the "/p" modifier is
843       present.  Consequently they do not penalize the rest of the program.
844
845   Non-capturing groupings
846       A group that is required to bundle a set of alternatives may or may not
847       be useful as a capturing group.  If it isn't, it just creates a
848       superfluous addition to the set of available capture group values,
849       inside as well as outside the regexp.  Non-capturing groupings, denoted
850       by "(?:regexp)", still allow the regexp to be treated as a single unit,
851       but don't establish a capturing group at the same time.  Both capturing
852       and non-capturing groupings are allowed to co-exist in the same regexp.
853       Because there is no extraction, non-capturing groupings are faster than
854       capturing groupings.  Non-capturing groupings are also handy for
855       choosing exactly which parts of a regexp are to be extracted to
856       matching variables:
857
858           # match a number, $1-$4 are set, but we only want $1
859           /([+-]?\ *(\d+(\.\d*)?|\.\d+)([eE][+-]?\d+)?)/;
860
861           # match a number faster , only $1 is set
862           /([+-]?\ *(?:\d+(?:\.\d*)?|\.\d+)(?:[eE][+-]?\d+)?)/;
863
864           # match a number, get $1 = whole number, $2 = exponent
865           /([+-]?\ *(?:\d+(?:\.\d*)?|\.\d+)(?:[eE]([+-]?\d+))?)/;
866
867       Non-capturing groupings are also useful for removing nuisance elements
868       gathered from a split operation where parentheses are required for some
869       reason:
870
871           $x = '12aba34ba5';
872           @num = split /(a|b)+/, $x;    # @num = ('12','a','34','a','5')
873           @num = split /(?:a|b)+/, $x;  # @num = ('12','34','5')
874
875   Matching repetitions
876       The examples in the previous section display an annoying weakness.  We
877       were only matching 3-letter words, or chunks of words of 4 letters or
878       less.  We'd like to be able to match words or, more generally, strings
879       of any length, without writing out tedious alternatives like
880       "\w\w\w\w|\w\w\w|\w\w|\w".
881
882       This is exactly the problem the quantifier metacharacters "?", "*",
883       "+", and "{}" were created for.  They allow us to delimit the number of
884       repeats for a portion of a regexp we consider to be a match.
885       Quantifiers are put immediately after the character, character class,
886       or grouping that we want to specify.  They have the following meanings:
887
888       ·   "a?" means: match 'a' 1 or 0 times
889
890       ·   "a*" means: match 'a' 0 or more times, i.e., any number of times
891
892       ·   "a+" means: match 'a' 1 or more times, i.e., at least once
893
894       ·   "a{n,m}" means: match at least "n" times, but not more than "m"
895           times.
896
897       ·   "a{n,}" means: match at least "n" or more times
898
899       ·   "a{n}" means: match exactly "n" times
900
901       Here are some examples:
902
903           /[a-z]+\s+\d*/;  # match a lowercase word, at least one space, and
904                            # any number of digits
905           /(\w+)\s+\g1/;    # match doubled words of arbitrary length
906           /y(es)?/i;       # matches 'y', 'Y', or a case-insensitive 'yes'
907           $year =~ /^\d{2,4}$/;  # make sure year is at least 2 but not more
908                                  # than 4 digits
909           $year =~ /^\d{4}$|^\d{2}$/;    # better match; throw out 3-digit dates
910           $year =~ /^\d{2}(\d{2})?$/;  # same thing written differently. However,
911                                        # this captures the last two digits in $1
912                                        # and the other does not.
913
914           % simple_grep '^(\w+)\g1$' /usr/dict/words   # isn't this easier?
915           beriberi
916           booboo
917           coco
918           mama
919           murmur
920           papa
921
922       For all of these quantifiers, Perl will try to match as much of the
923       string as possible, while still allowing the regexp to succeed.  Thus
924       with "/a?.../", Perl will first try to match the regexp with the "a"
925       present; if that fails, Perl will try to match the regexp without the
926       "a" present.  For the quantifier "*", we get the following:
927
928           $x = "the cat in the hat";
929           $x =~ /^(.*)(cat)(.*)$/; # matches,
930                                    # $1 = 'the '
931                                    # $2 = 'cat'
932                                    # $3 = ' in the hat'
933
934       Which is what we might expect, the match finds the only "cat" in the
935       string and locks onto it.  Consider, however, this regexp:
936
937           $x =~ /^(.*)(at)(.*)$/; # matches,
938                                   # $1 = 'the cat in the h'
939                                   # $2 = 'at'
940                                   # $3 = ''   (0 characters match)
941
942       One might initially guess that Perl would find the "at" in "cat" and
943       stop there, but that wouldn't give the longest possible string to the
944       first quantifier ".*".  Instead, the first quantifier ".*" grabs as
945       much of the string as possible while still having the regexp match.  In
946       this example, that means having the "at" sequence with the final "at"
947       in the string.  The other important principle illustrated here is that,
948       when there are two or more elements in a regexp, the leftmost
949       quantifier, if there is one, gets to grab as much of the string as
950       possible, leaving the rest of the regexp to fight over scraps.  Thus in
951       our example, the first quantifier ".*" grabs most of the string, while
952       the second quantifier ".*" gets the empty string.   Quantifiers that
953       grab as much of the string as possible are called maximal match or
954       greedy quantifiers.
955
956       When a regexp can match a string in several different ways, we can use
957       the principles above to predict which way the regexp will match:
958
959       ·   Principle 0: Taken as a whole, any regexp will be matched at the
960           earliest possible position in the string.
961
962       ·   Principle 1: In an alternation "a|b|c...", the leftmost alternative
963           that allows a match for the whole regexp will be the one used.
964
965       ·   Principle 2: The maximal matching quantifiers "?", "*", "+" and
966           "{n,m}" will in general match as much of the string as possible
967           while still allowing the whole regexp to match.
968
969       ·   Principle 3: If there are two or more elements in a regexp, the
970           leftmost greedy quantifier, if any, will match as much of the
971           string as possible while still allowing the whole regexp to match.
972           The next leftmost greedy quantifier, if any, will try to match as
973           much of the string remaining available to it as possible, while
974           still allowing the whole regexp to match.  And so on, until all the
975           regexp elements are satisfied.
976
977       As we have seen above, Principle 0 overrides the others. The regexp
978       will be matched as early as possible, with the other principles
979       determining how the regexp matches at that earliest character position.
980
981       Here is an example of these principles in action:
982
983           $x = "The programming republic of Perl";
984           $x =~ /^(.+)(e|r)(.*)$/;  # matches,
985                                     # $1 = 'The programming republic of Pe'
986                                     # $2 = 'r'
987                                     # $3 = 'l'
988
989       This regexp matches at the earliest string position, 'T'.  One might
990       think that "e", being leftmost in the alternation, would be matched,
991       but "r" produces the longest string in the first quantifier.
992
993           $x =~ /(m{1,2})(.*)$/;  # matches,
994                                   # $1 = 'mm'
995                                   # $2 = 'ing republic of Perl'
996
997       Here, The earliest possible match is at the first 'm' in "programming".
998       "m{1,2}" is the first quantifier, so it gets to match a maximal "mm".
999
1000           $x =~ /.*(m{1,2})(.*)$/;  # matches,
1001                                     # $1 = 'm'
1002                                     # $2 = 'ing republic of Perl'
1003
1004       Here, the regexp matches at the start of the string. The first
1005       quantifier ".*" grabs as much as possible, leaving just a single 'm'
1006       for the second quantifier "m{1,2}".
1007
1008           $x =~ /(.?)(m{1,2})(.*)$/;  # matches,
1009                                       # $1 = 'a'
1010                                       # $2 = 'mm'
1011                                       # $3 = 'ing republic of Perl'
1012
1013       Here, ".?" eats its maximal one character at the earliest possible
1014       position in the string, 'a' in "programming", leaving "m{1,2}" the
1015       opportunity to match both "m"'s. Finally,
1016
1017           "aXXXb" =~ /(X*)/; # matches with $1 = ''
1018
1019       because it can match zero copies of 'X' at the beginning of the string.
1020       If you definitely want to match at least one 'X', use "X+", not "X*".
1021
1022       Sometimes greed is not good.  At times, we would like quantifiers to
1023       match a minimal piece of string, rather than a maximal piece.  For this
1024       purpose, Larry Wall created the minimal match or non-greedy quantifiers
1025       "??", "*?", "+?", and "{}?".  These are the usual quantifiers with a
1026       "?" appended to them.  They have the following meanings:
1027
1028       ·   "a??" means: match 'a' 0 or 1 times. Try 0 first, then 1.
1029
1030       ·   "a*?" means: match 'a' 0 or more times, i.e., any number of times,
1031           but as few times as possible
1032
1033       ·   "a+?" means: match 'a' 1 or more times, i.e., at least once, but as
1034           few times as possible
1035
1036       ·   "a{n,m}?" means: match at least "n" times, not more than "m" times,
1037           as few times as possible
1038
1039       ·   "a{n,}?" means: match at least "n" times, but as few times as
1040           possible
1041
1042       ·   "a{n}?" means: match exactly "n" times.  Because we match exactly
1043           "n" times, "a{n}?" is equivalent to "a{n}" and is just there for
1044           notational consistency.
1045
1046       Let's look at the example above, but with minimal quantifiers:
1047
1048           $x = "The programming republic of Perl";
1049           $x =~ /^(.+?)(e|r)(.*)$/; # matches,
1050                                     # $1 = 'Th'
1051                                     # $2 = 'e'
1052                                     # $3 = ' programming republic of Perl'
1053
1054       The minimal string that will allow both the start of the string "^" and
1055       the alternation to match is "Th", with the alternation "e|r" matching
1056       "e".  The second quantifier ".*" is free to gobble up the rest of the
1057       string.
1058
1059           $x =~ /(m{1,2}?)(.*?)$/;  # matches,
1060                                     # $1 = 'm'
1061                                     # $2 = 'ming republic of Perl'
1062
1063       The first string position that this regexp can match is at the first
1064       'm' in "programming". At this position, the minimal "m{1,2}?"  matches
1065       just one 'm'.  Although the second quantifier ".*?" would prefer to
1066       match no characters, it is constrained by the end-of-string anchor "$"
1067       to match the rest of the string.
1068
1069           $x =~ /(.*?)(m{1,2}?)(.*)$/;  # matches,
1070                                         # $1 = 'The progra'
1071                                         # $2 = 'm'
1072                                         # $3 = 'ming republic of Perl'
1073
1074       In this regexp, you might expect the first minimal quantifier ".*?"  to
1075       match the empty string, because it is not constrained by a "^" anchor
1076       to match the beginning of the word.  Principle 0 applies here, however.
1077       Because it is possible for the whole regexp to match at the start of
1078       the string, it will match at the start of the string.  Thus the first
1079       quantifier has to match everything up to the first "m".  The second
1080       minimal quantifier matches just one "m" and the third quantifier
1081       matches the rest of the string.
1082
1083           $x =~ /(.??)(m{1,2})(.*)$/;  # matches,
1084                                        # $1 = 'a'
1085                                        # $2 = 'mm'
1086                                        # $3 = 'ing republic of Perl'
1087
1088       Just as in the previous regexp, the first quantifier ".??" can match
1089       earliest at position 'a', so it does.  The second quantifier is greedy,
1090       so it matches "mm", and the third matches the rest of the string.
1091
1092       We can modify principle 3 above to take into account non-greedy
1093       quantifiers:
1094
1095       ·   Principle 3: If there are two or more elements in a regexp, the
1096           leftmost greedy (non-greedy) quantifier, if any, will match as much
1097           (little) of the string as possible while still allowing the whole
1098           regexp to match.  The next leftmost greedy (non-greedy) quantifier,
1099           if any, will try to match as much (little) of the string remaining
1100           available to it as possible, while still allowing the whole regexp
1101           to match.  And so on, until all the regexp elements are satisfied.
1102
1103       Just like alternation, quantifiers are also susceptible to
1104       backtracking.  Here is a step-by-step analysis of the example
1105
1106           $x = "the cat in the hat";
1107           $x =~ /^(.*)(at)(.*)$/; # matches,
1108                                   # $1 = 'the cat in the h'
1109                                   # $2 = 'at'
1110                                   # $3 = ''   (0 matches)
1111
1112       0   Start with the first letter in the string 't'.
1113
1114       1   The first quantifier '.*' starts out by matching the whole string
1115           'the cat in the hat'.
1116
1117       2   'a' in the regexp element 'at' doesn't match the end of the string.
1118           Backtrack one character.
1119
1120       3   'a' in the regexp element 'at' still doesn't match the last letter
1121           of the string 't', so backtrack one more character.
1122
1123       4   Now we can match the 'a' and the 't'.
1124
1125       5   Move on to the third element '.*'.  Since we are at the end of the
1126           string and '.*' can match 0 times, assign it the empty string.
1127
1128       6   We are done!
1129
1130       Most of the time, all this moving forward and backtracking happens
1131       quickly and searching is fast. There are some pathological regexps,
1132       however, whose execution time exponentially grows with the size of the
1133       string.  A typical structure that blows up in your face is of the form
1134
1135           /(a|b+)*/;
1136
1137       The problem is the nested indeterminate quantifiers.  There are many
1138       different ways of partitioning a string of length n between the "+" and
1139       "*": one repetition with "b+" of length n, two repetitions with the
1140       first "b+" length k and the second with length n-k, m repetitions whose
1141       bits add up to length n, etc.  In fact there are an exponential number
1142       of ways to partition a string as a function of its length.  A regexp
1143       may get lucky and match early in the process, but if there is no match,
1144       Perl will try every possibility before giving up.  So be careful with
1145       nested "*"'s, "{n,m}"'s, and "+"'s.  The book Mastering Regular
1146       Expressions by Jeffrey Friedl gives a wonderful discussion of this and
1147       other efficiency issues.
1148
1149   Possessive quantifiers
1150       Backtracking during the relentless search for a match may be a waste of
1151       time, particularly when the match is bound to fail.  Consider the
1152       simple pattern
1153
1154           /^\w+\s+\w+$/; # a word, spaces, a word
1155
1156       Whenever this is applied to a string which doesn't quite meet the
1157       pattern's expectations such as "abc  " or "abc  def ", the regex engine
1158       will backtrack, approximately once for each character in the string.
1159       But we know that there is no way around taking all of the initial word
1160       characters to match the first repetition, that all spaces must be eaten
1161       by the middle part, and the same goes for the second word.
1162
1163       With the introduction of the possessive quantifiers in Perl 5.10, we
1164       have a way of instructing the regex engine not to backtrack, with the
1165       usual quantifiers with a "+" appended to them.  This makes them greedy
1166       as well as stingy; once they succeed they won't give anything back to
1167       permit another solution. They have the following meanings:
1168
1169       ·   "a{n,m}+" means: match at least "n" times, not more than "m" times,
1170           as many times as possible, and don't give anything up. "a?+" is
1171           short for "a{0,1}+"
1172
1173       ·   "a{n,}+" means: match at least "n" times, but as many times as
1174           possible, and don't give anything up. "a*+" is short for "a{0,}+"
1175           and "a++" is short for "a{1,}+".
1176
1177       ·   "a{n}+" means: match exactly "n" times.  It is just there for
1178           notational consistency.
1179
1180       These possessive quantifiers represent a special case of a more general
1181       concept, the independent subexpression, see below.
1182
1183       As an example where a possessive quantifier is suitable we consider
1184       matching a quoted string, as it appears in several programming
1185       languages.  The backslash is used as an escape character that indicates
1186       that the next character is to be taken literally, as another character
1187       for the string.  Therefore, after the opening quote, we expect a
1188       (possibly empty) sequence of alternatives: either some character except
1189       an unescaped quote or backslash or an escaped character.
1190
1191           /"(?:[^"\\]++|\\.)*+"/;
1192
1193   Building a regexp
1194       At this point, we have all the basic regexp concepts covered, so let's
1195       give a more involved example of a regular expression.  We will build a
1196       regexp that matches numbers.
1197
1198       The first task in building a regexp is to decide what we want to match
1199       and what we want to exclude.  In our case, we want to match both
1200       integers and floating point numbers and we want to reject any string
1201       that isn't a number.
1202
1203       The next task is to break the problem down into smaller problems that
1204       are easily converted into a regexp.
1205
1206       The simplest case is integers.  These consist of a sequence of digits,
1207       with an optional sign in front.  The digits we can represent with "\d+"
1208       and the sign can be matched with "[+-]".  Thus the integer regexp is
1209
1210           /[+-]?\d+/;  # matches integers
1211
1212       A floating point number potentially has a sign, an integral part, a
1213       decimal point, a fractional part, and an exponent.  One or more of
1214       these parts is optional, so we need to check out the different
1215       possibilities.  Floating point numbers which are in proper form include
1216       123., 0.345, .34, -1e6, and 25.4E-72.  As with integers, the sign out
1217       front is completely optional and can be matched by "[+-]?".  We can see
1218       that if there is no exponent, floating point numbers must have a
1219       decimal point, otherwise they are integers.  We might be tempted to
1220       model these with "\d*\.\d*", but this would also match just a single
1221       decimal point, which is not a number.  So the three cases of floating
1222       point number without exponent are
1223
1224          /[+-]?\d+\./;  # 1., 321., etc.
1225          /[+-]?\.\d+/;  # .1, .234, etc.
1226          /[+-]?\d+\.\d+/;  # 1.0, 30.56, etc.
1227
1228       These can be combined into a single regexp with a three-way
1229       alternation:
1230
1231          /[+-]?(\d+\.\d+|\d+\.|\.\d+)/;  # floating point, no exponent
1232
1233       In this alternation, it is important to put '\d+\.\d+' before '\d+\.'.
1234       If '\d+\.' were first, the regexp would happily match that and ignore
1235       the fractional part of the number.
1236
1237       Now consider floating point numbers with exponents.  The key
1238       observation here is that both integers and numbers with decimal points
1239       are allowed in front of an exponent.  Then exponents, like the overall
1240       sign, are independent of whether we are matching numbers with or
1241       without decimal points, and can be 'decoupled' from the mantissa.  The
1242       overall form of the regexp now becomes clear:
1243
1244           /^(optional sign)(integer | f.p. mantissa)(optional exponent)$/;
1245
1246       The exponent is an "e" or "E", followed by an integer.  So the exponent
1247       regexp is
1248
1249          /[eE][+-]?\d+/;  # exponent
1250
1251       Putting all the parts together, we get a regexp that matches numbers:
1252
1253          /^[+-]?(\d+\.\d+|\d+\.|\.\d+|\d+)([eE][+-]?\d+)?$/;  # Ta da!
1254
1255       Long regexps like this may impress your friends, but can be hard to
1256       decipher.  In complex situations like this, the "//x" modifier for a
1257       match is invaluable.  It allows one to put nearly arbitrary whitespace
1258       and comments into a regexp without affecting their meaning.  Using it,
1259       we can rewrite our 'extended' regexp in the more pleasing form
1260
1261          /^
1262             [+-]?         # first, match an optional sign
1263             (             # then match integers or f.p. mantissas:
1264                 \d+\.\d+  # mantissa of the form a.b
1265                |\d+\.     # mantissa of the form a.
1266                |\.\d+     # mantissa of the form .b
1267                |\d+       # integer of the form a
1268             )
1269             ([eE][+-]?\d+)?  # finally, optionally match an exponent
1270          $/x;
1271
1272       If whitespace is mostly irrelevant, how does one include space
1273       characters in an extended regexp? The answer is to backslash it '\ ' or
1274       put it in a character class "[ ]".  The same thing goes for pound
1275       signs: use "\#" or "[#]".  For instance, Perl allows a space between
1276       the sign and the mantissa or integer, and we could add this to our
1277       regexp as follows:
1278
1279          /^
1280             [+-]?\ *      # first, match an optional sign *and space*
1281             (             # then match integers or f.p. mantissas:
1282                 \d+\.\d+  # mantissa of the form a.b
1283                |\d+\.     # mantissa of the form a.
1284                |\.\d+     # mantissa of the form .b
1285                |\d+       # integer of the form a
1286             )
1287             ([eE][+-]?\d+)?  # finally, optionally match an exponent
1288          $/x;
1289
1290       In this form, it is easier to see a way to simplify the alternation.
1291       Alternatives 1, 2, and 4 all start with "\d+", so it could be factored
1292       out:
1293
1294          /^
1295             [+-]?\ *      # first, match an optional sign
1296             (             # then match integers or f.p. mantissas:
1297                 \d+       # start out with a ...
1298                 (
1299                     \.\d* # mantissa of the form a.b or a.
1300                 )?        # ? takes care of integers of the form a
1301                |\.\d+     # mantissa of the form .b
1302             )
1303             ([eE][+-]?\d+)?  # finally, optionally match an exponent
1304          $/x;
1305
1306       or written in the compact form,
1307
1308           /^[+-]?\ *(\d+(\.\d*)?|\.\d+)([eE][+-]?\d+)?$/;
1309
1310       This is our final regexp.  To recap, we built a regexp by
1311
1312       ·   specifying the task in detail,
1313
1314       ·   breaking down the problem into smaller parts,
1315
1316       ·   translating the small parts into regexps,
1317
1318       ·   combining the regexps,
1319
1320       ·   and optimizing the final combined regexp.
1321
1322       These are also the typical steps involved in writing a computer
1323       program.  This makes perfect sense, because regular expressions are
1324       essentially programs written in a little computer language that
1325       specifies patterns.
1326
1327   Using regular expressions in Perl
1328       The last topic of Part 1 briefly covers how regexps are used in Perl
1329       programs.  Where do they fit into Perl syntax?
1330
1331       We have already introduced the matching operator in its default
1332       "/regexp/" and arbitrary delimiter "m!regexp!" forms.  We have used the
1333       binding operator "=~" and its negation "!~" to test for string matches.
1334       Associated with the matching operator, we have discussed the single
1335       line "//s", multi-line "//m", case-insensitive "//i" and extended "//x"
1336       modifiers.  There are a few more things you might want to know about
1337       matching operators.
1338
1339       Prohibiting substitution
1340
1341       If you change $pattern after the first substitution happens, Perl will
1342       ignore it.  If you don't want any substitutions at all, use the special
1343       delimiter "m''":
1344
1345           @pattern = ('Seuss');
1346           while (<>) {
1347               print if m'@pattern';  # matches literal '@pattern', not 'Seuss'
1348           }
1349
1350       Similar to strings, "m''" acts like apostrophes on a regexp; all other
1351       "m" delimiters act like quotes.  If the regexp evaluates to the empty
1352       string, the regexp in the last successful match is used instead.  So we
1353       have
1354
1355           "dog" =~ /d/;  # 'd' matches
1356           "dogbert =~ //;  # this matches the 'd' regexp used before
1357
1358       Global matching
1359
1360       The final two modifiers we will discuss here, "//g" and "//c", concern
1361       multiple matches.  The modifier "//g" stands for global matching and
1362       allows the matching operator to match within a string as many times as
1363       possible.  In scalar context, successive invocations against a string
1364       will have "//g" jump from match to match, keeping track of position in
1365       the string as it goes along.  You can get or set the position with the
1366       "pos()" function.
1367
1368       The use of "//g" is shown in the following example.  Suppose we have a
1369       string that consists of words separated by spaces.  If we know how many
1370       words there are in advance, we could extract the words using groupings:
1371
1372           $x = "cat dog house"; # 3 words
1373           $x =~ /^\s*(\w+)\s+(\w+)\s+(\w+)\s*$/; # matches,
1374                                                  # $1 = 'cat'
1375                                                  # $2 = 'dog'
1376                                                  # $3 = 'house'
1377
1378       But what if we had an indeterminate number of words? This is the sort
1379       of task "//g" was made for.  To extract all words, form the simple
1380       regexp "(\w+)" and loop over all matches with "/(\w+)/g":
1381
1382           while ($x =~ /(\w+)/g) {
1383               print "Word is $1, ends at position ", pos $x, "\n";
1384           }
1385
1386       prints
1387
1388           Word is cat, ends at position 3
1389           Word is dog, ends at position 7
1390           Word is house, ends at position 13
1391
1392       A failed match or changing the target string resets the position.  If
1393       you don't want the position reset after failure to match, add the
1394       "//c", as in "/regexp/gc".  The current position in the string is
1395       associated with the string, not the regexp.  This means that different
1396       strings have different positions and their respective positions can be
1397       set or read independently.
1398
1399       In list context, "//g" returns a list of matched groupings, or if there
1400       are no groupings, a list of matches to the whole regexp.  So if we
1401       wanted just the words, we could use
1402
1403           @words = ($x =~ /(\w+)/g);  # matches,
1404                                       # $words[0] = 'cat'
1405                                       # $words[1] = 'dog'
1406                                       # $words[2] = 'house'
1407
1408       Closely associated with the "//g" modifier is the "\G" anchor.  The
1409       "\G" anchor matches at the point where the previous "//g" match left
1410       off.  "\G" allows us to easily do context-sensitive matching:
1411
1412           $metric = 1;  # use metric units
1413           ...
1414           $x = <FILE>;  # read in measurement
1415           $x =~ /^([+-]?\d+)\s*/g;  # get magnitude
1416           $weight = $1;
1417           if ($metric) { # error checking
1418               print "Units error!" unless $x =~ /\Gkg\./g;
1419           }
1420           else {
1421               print "Units error!" unless $x =~ /\Glbs\./g;
1422           }
1423           $x =~ /\G\s+(widget|sprocket)/g;  # continue processing
1424
1425       The combination of "//g" and "\G" allows us to process the string a bit
1426       at a time and use arbitrary Perl logic to decide what to do next.
1427       Currently, the "\G" anchor is only fully supported when used to anchor
1428       to the start of the pattern.
1429
1430       "\G" is also invaluable in processing fixed-length records with
1431       regexps.  Suppose we have a snippet of coding region DNA, encoded as
1432       base pair letters "ATCGTTGAAT..." and we want to find all the stop
1433       codons "TGA".  In a coding region, codons are 3-letter sequences, so we
1434       can think of the DNA snippet as a sequence of 3-letter records.  The
1435       naive regexp
1436
1437           # expanded, this is "ATC GTT GAA TGC AAA TGA CAT GAC"
1438           $dna = "ATCGTTGAATGCAAATGACATGAC";
1439           $dna =~ /TGA/;
1440
1441       doesn't work; it may match a "TGA", but there is no guarantee that the
1442       match is aligned with codon boundaries, e.g., the substring "GTT GAA"
1443       gives a match.  A better solution is
1444
1445           while ($dna =~ /(\w\w\w)*?TGA/g) {  # note the minimal *?
1446               print "Got a TGA stop codon at position ", pos $dna, "\n";
1447           }
1448
1449       which prints
1450
1451           Got a TGA stop codon at position 18
1452           Got a TGA stop codon at position 23
1453
1454       Position 18 is good, but position 23 is bogus.  What happened?
1455
1456       The answer is that our regexp works well until we get past the last
1457       real match.  Then the regexp will fail to match a synchronized "TGA"
1458       and start stepping ahead one character position at a time, not what we
1459       want.  The solution is to use "\G" to anchor the match to the codon
1460       alignment:
1461
1462           while ($dna =~ /\G(\w\w\w)*?TGA/g) {
1463               print "Got a TGA stop codon at position ", pos $dna, "\n";
1464           }
1465
1466       This prints
1467
1468           Got a TGA stop codon at position 18
1469
1470       which is the correct answer.  This example illustrates that it is
1471       important not only to match what is desired, but to reject what is not
1472       desired.
1473
1474       (There are other regexp modifiers that are available, such as "//o",
1475       but their specialized uses are beyond the scope of this introduction.
1476       )
1477
1478       Search and replace
1479
1480       Regular expressions also play a big role in search and replace
1481       operations in Perl.  Search and replace is accomplished with the "s///"
1482       operator.  The general form is "s/regexp/replacement/modifiers", with
1483       everything we know about regexps and modifiers applying in this case as
1484       well.  The "replacement" is a Perl double-quoted string that replaces
1485       in the string whatever is matched with the "regexp".  The operator "=~"
1486       is also used here to associate a string with "s///".  If matching
1487       against $_, the "$_ =~" can be dropped.  If there is a match, "s///"
1488       returns the number of substitutions made; otherwise it returns false.
1489       Here are a few examples:
1490
1491           $x = "Time to feed the cat!";
1492           $x =~ s/cat/hacker/;   # $x contains "Time to feed the hacker!"
1493           if ($x =~ s/^(Time.*hacker)!$/$1 now!/) {
1494               $more_insistent = 1;
1495           }
1496           $y = "'quoted words'";
1497           $y =~ s/^'(.*)'$/$1/;  # strip single quotes,
1498                                  # $y contains "quoted words"
1499
1500       In the last example, the whole string was matched, but only the part
1501       inside the single quotes was grouped.  With the "s///" operator, the
1502       matched variables $1, $2, etc. are immediately available for use in the
1503       replacement expression, so we use $1 to replace the quoted string with
1504       just what was quoted.  With the global modifier, "s///g" will search
1505       and replace all occurrences of the regexp in the string:
1506
1507           $x = "I batted 4 for 4";
1508           $x =~ s/4/four/;   # doesn't do it all:
1509                              # $x contains "I batted four for 4"
1510           $x = "I batted 4 for 4";
1511           $x =~ s/4/four/g;  # does it all:
1512                              # $x contains "I batted four for four"
1513
1514       If you prefer 'regex' over 'regexp' in this tutorial, you could use the
1515       following program to replace it:
1516
1517           % cat > simple_replace
1518           #!/usr/bin/perl
1519           $regexp = shift;
1520           $replacement = shift;
1521           while (<>) {
1522               s/$regexp/$replacement/g;
1523               print;
1524           }
1525           ^D
1526
1527           % simple_replace regexp regex perlretut.pod
1528
1529       In "simple_replace" we used the "s///g" modifier to replace all
1530       occurrences of the regexp on each line.  (Even though the regular
1531       expression appears in a loop, Perl is smart enough to compile it only
1532       once.)  As with "simple_grep", both the "print" and the
1533       "s/$regexp/$replacement/g" use $_ implicitly.
1534
1535       If you don't want "s///" to change your original variable you can use
1536       the non-destructive substitute modifier, "s///r".  This changes the
1537       behavior so that "s///r" returns the final substituted string (instead
1538       of the number of substitutions):
1539
1540           $x = "I like dogs.";
1541           $y = $x =~ s/dogs/cats/r;
1542           print "$x $y\n";
1543
1544       That example will print "I like dogs. I like cats". Notice the original
1545       $x variable has not been affected. The overall result of the
1546       substitution is instead stored in $y. If the substitution doesn't
1547       affect anything then the original string is returned:
1548
1549           $x = "I like dogs.";
1550           $y = $x =~ s/elephants/cougars/r;
1551           print "$x $y\n"; # prints "I like dogs. I like dogs."
1552
1553       One other interesting thing that the "s///r" flag allows is chaining
1554       substitutions:
1555
1556           $x = "Cats are great.";
1557           print $x =~ s/Cats/Dogs/r =~ s/Dogs/Frogs/r =~ s/Frogs/Hedgehogs/r, "\n";
1558           # prints "Hedgehogs are great."
1559
1560       A modifier available specifically to search and replace is the "s///e"
1561       evaluation modifier.  "s///e" treats the replacement text as Perl code,
1562       rather than a double-quoted string.  The value that the code returns is
1563       substituted for the matched substring.  "s///e" is useful if you need
1564       to do a bit of computation in the process of replacing text.  This
1565       example counts character frequencies in a line:
1566
1567           $x = "Bill the cat";
1568           $x =~ s/(.)/$chars{$1}++;$1/eg;  # final $1 replaces char with itself
1569           print "frequency of '$_' is $chars{$_}\n"
1570               foreach (sort {$chars{$b} <=> $chars{$a}} keys %chars);
1571
1572       This prints
1573
1574           frequency of ' ' is 2
1575           frequency of 't' is 2
1576           frequency of 'l' is 2
1577           frequency of 'B' is 1
1578           frequency of 'c' is 1
1579           frequency of 'e' is 1
1580           frequency of 'h' is 1
1581           frequency of 'i' is 1
1582           frequency of 'a' is 1
1583
1584       As with the match "m//" operator, "s///" can use other delimiters, such
1585       as "s!!!" and "s{}{}", and even "s{}//".  If single quotes are used
1586       "s'''", then the regexp and replacement are treated as single-quoted
1587       strings and there are no variable substitutions.  "s///" in list
1588       context returns the same thing as in scalar context, i.e., the number
1589       of matches.
1590
1591       The split function
1592
1593       The "split()" function is another place where a regexp is used.  "split
1594       /regexp/, string, limit" separates the "string" operand into a list of
1595       substrings and returns that list.  The regexp must be designed to match
1596       whatever constitutes the separators for the desired substrings.  The
1597       "limit", if present, constrains splitting into no more than "limit"
1598       number of strings.  For example, to split a string into words, use
1599
1600           $x = "Calvin and Hobbes";
1601           @words = split /\s+/, $x;  # $word[0] = 'Calvin'
1602                                      # $word[1] = 'and'
1603                                      # $word[2] = 'Hobbes'
1604
1605       If the empty regexp "//" is used, the regexp always matches and the
1606       string is split into individual characters.  If the regexp has
1607       groupings, then the resulting list contains the matched substrings from
1608       the groupings as well.  For instance,
1609
1610           $x = "/usr/bin/perl";
1611           @dirs = split m!/!, $x;  # $dirs[0] = ''
1612                                    # $dirs[1] = 'usr'
1613                                    # $dirs[2] = 'bin'
1614                                    # $dirs[3] = 'perl'
1615           @parts = split m!(/)!, $x;  # $parts[0] = ''
1616                                       # $parts[1] = '/'
1617                                       # $parts[2] = 'usr'
1618                                       # $parts[3] = '/'
1619                                       # $parts[4] = 'bin'
1620                                       # $parts[5] = '/'
1621                                       # $parts[6] = 'perl'
1622
1623       Since the first character of $x matched the regexp, "split" prepended
1624       an empty initial element to the list.
1625
1626       If you have read this far, congratulations! You now have all the basic
1627       tools needed to use regular expressions to solve a wide range of text
1628       processing problems.  If this is your first time through the tutorial,
1629       why not stop here and play around with regexps a while....  Part 2
1630       concerns the more esoteric aspects of regular expressions and those
1631       concepts certainly aren't needed right at the start.
1632

Part 2: Power tools

1634       OK, you know the basics of regexps and you want to know more.  If
1635       matching regular expressions is analogous to a walk in the woods, then
1636       the tools discussed in Part 1 are analogous to topo maps and a compass,
1637       basic tools we use all the time.  Most of the tools in part 2 are
1638       analogous to flare guns and satellite phones.  They aren't used too
1639       often on a hike, but when we are stuck, they can be invaluable.
1640
1641       What follows are the more advanced, less used, or sometimes esoteric
1642       capabilities of Perl regexps.  In Part 2, we will assume you are
1643       comfortable with the basics and concentrate on the advanced features.
1644
1645   More on characters, strings, and character classes
1646       There are a number of escape sequences and character classes that we
1647       haven't covered yet.
1648
1649       There are several escape sequences that convert characters or strings
1650       between upper and lower case, and they are also available within
1651       patterns.  "\l" and "\u" convert the next character to lower or upper
1652       case, respectively:
1653
1654           $x = "perl";
1655           $string =~ /\u$x/;  # matches 'Perl' in $string
1656           $x = "M(rs?|s)\\."; # note the double backslash
1657           $string =~ /\l$x/;  # matches 'mr.', 'mrs.', and 'ms.',
1658
1659       A "\L" or "\U" indicates a lasting conversion of case, until terminated
1660       by "\E" or thrown over by another "\U" or "\L":
1661
1662           $x = "This word is in lower case:\L SHOUT\E";
1663           $x =~ /shout/;       # matches
1664           $x = "I STILL KEYPUNCH CARDS FOR MY 360"
1665           $x =~ /\Ukeypunch/;  # matches punch card string
1666
1667       If there is no "\E", case is converted until the end of the string. The
1668       regexps "\L\u$word" or "\u\L$word" convert the first character of $word
1669       to uppercase and the rest of the characters to lowercase.
1670
1671       Control characters can be escaped with "\c", so that a control-Z
1672       character would be matched with "\cZ".  The escape sequence "\Q"..."\E"
1673       quotes, or protects most non-alphabetic characters.   For instance,
1674
1675           $x = "\QThat !^*&%~& cat!";
1676           $x =~ /\Q!^*&%~&\E/;  # check for rough language
1677
1678       It does not protect "$" or "@", so that variables can still be
1679       substituted.
1680
1681       "\Q", "\L", "\l", "\U", "\u" and "\E" are actually part of double-
1682       quotish syntax, and not part of regexp syntax proper.  They will work
1683       if they appear in a regular expression embedded directly in a program,
1684       but not when contained in a string that is interpolated in a pattern.
1685
1686       With the advent of 5.6.0, Perl regexps can handle more than just the
1687       standard ASCII character set.  Perl now supports Unicode, a standard
1688       for representing the alphabets from virtually all of the world's
1689       written languages, and a host of symbols.  Perl's text strings are
1690       Unicode strings, so they can contain characters with a value (codepoint
1691       or character number) higher than 255.
1692
1693       What does this mean for regexps? Well, regexp users don't need to know
1694       much about Perl's internal representation of strings.  But they do need
1695       to know 1) how to represent Unicode characters in a regexp and 2) that
1696       a matching operation will treat the string to be searched as a sequence
1697       of characters, not bytes.  The answer to 1) is that Unicode characters
1698       greater than "chr(255)" are represented using the "\x{hex}" notation,
1699       because \x hex (without curly braces) doesn't go further than 255.
1700       (Starting in Perl 5.14, if you're an octal fan, you can also use
1701       "\o{oct}".)
1702
1703           /\x{263a}/;  # match a Unicode smiley face :)
1704
1705       NOTE: In Perl 5.6.0 it used to be that one needed to say "use utf8" to
1706       use any Unicode features.  This is no more the case: for almost all
1707       Unicode processing, the explicit "utf8" pragma is not needed.  (The
1708       only case where it matters is if your Perl script is in Unicode and
1709       encoded in UTF-8, then an explicit "use utf8" is needed.)
1710
1711       Figuring out the hexadecimal sequence of a Unicode character you want
1712       or deciphering someone else's hexadecimal Unicode regexp is about as
1713       much fun as programming in machine code.  So another way to specify
1714       Unicode characters is to use the named character escape sequence
1715       "\N{name}".  name is a name for the Unicode character, as specified in
1716       the Unicode standard.  For instance, if we wanted to represent or match
1717       the astrological sign for the planet Mercury, we could use
1718
1719           $x = "abc\N{MERCURY}def";
1720           $x =~ /\N{MERCURY}/;   # matches
1721
1722       One can also use "short" names:
1723
1724           print "\N{GREEK SMALL LETTER SIGMA} is called sigma.\n";
1725           print "\N{greek:Sigma} is an upper-case sigma.\n";
1726
1727       You can also restrict names to a certain alphabet by specifying the
1728       charnames pragma:
1729
1730           use charnames qw(greek);
1731           print "\N{sigma} is Greek sigma\n";
1732
1733       An index of character names is available on-line from the Unicode
1734       Consortium, <http://www.unicode.org/charts/charindex.html>; explanatory
1735       material with links to other resources at
1736       <http://www.unicode.org/standard/where>.
1737
1738       The answer to requirement 2) is, as of 5.6.0, that a regexp (mostly)
1739       uses Unicode characters.  (The "mostly" is for messy backward
1740       compatibility reasons, but starting in Perl 5.14, any regex compiled in
1741       the scope of a "use feature 'unicode_strings'" (which is automatically
1742       turned on within the scope of a "use 5.012" or higher) will turn that
1743       "mostly" into "always".  If you want to handle Unicode properly, you
1744       should ensure that 'unicode_strings' is turned on.)  Internally, this
1745       is encoded to bytes using either UTF-8 or a native 8 bit encoding,
1746       depending on the history of the string, but conceptually it is a
1747       sequence of characters, not bytes. See perlunitut for a tutorial about
1748       that.
1749
1750       Let us now discuss Unicode character classes.  Just as with Unicode
1751       characters, there are named Unicode character classes represented by
1752       the "\p{name}" escape sequence.  Closely associated is the "\P{name}"
1753       character class, which is the negation of the "\p{name}" class.  For
1754       example, to match lower and uppercase characters,
1755
1756           $x = "BOB";
1757           $x =~ /^\p{IsUpper}/;   # matches, uppercase char class
1758           $x =~ /^\P{IsUpper}/;   # doesn't match, char class sans uppercase
1759           $x =~ /^\p{IsLower}/;   # doesn't match, lowercase char class
1760           $x =~ /^\P{IsLower}/;   # matches, char class sans lowercase
1761
1762       (The "Is" is optional.)
1763
1764       Here is the association between some Perl named classes and the
1765       traditional Unicode classes:
1766
1767           Perl class name  Unicode class name or regular expression
1768
1769           IsAlpha          /^[LM]/
1770           IsAlnum          /^[LMN]/
1771           IsASCII          $code <= 127
1772           IsCntrl          /^C/
1773           IsBlank          $code =~ /^(0020|0009)$/ || /^Z[^lp]/
1774           IsDigit          Nd
1775           IsGraph          /^([LMNPS]|Co)/
1776           IsLower          Ll
1777           IsPrint          /^([LMNPS]|Co|Zs)/
1778           IsPunct          /^P/
1779           IsSpace          /^Z/ || ($code =~ /^(0009|000A|000B|000C|000D)$/
1780           IsSpacePerl      /^Z/ || ($code =~ /^(0009|000A|000C|000D|0085|2028|2029)$/
1781           IsUpper          /^L[ut]/
1782           IsWord           /^[LMN]/ || $code eq "005F"
1783           IsXDigit         $code =~ /^00(3[0-9]|[46][1-6])$/
1784
1785       You can also use the official Unicode class names with "\p" and "\P",
1786       like "\p{L}" for Unicode 'letters', "\p{Lu}" for uppercase letters, or
1787       "\P{Nd}" for non-digits.  If a "name" is just one letter, the braces
1788       can be dropped.  For instance, "\pM" is the character class of Unicode
1789       'marks', for example accent marks.  For the full list see perlunicode.
1790
1791       Unicode has also been separated into various sets of characters which
1792       you can test with "\p{...}" (in) and "\P{...}" (not in).  To test
1793       whether a character is (or is not) an element of a script you would use
1794       the script name, for example "\p{Latin}", "\p{Greek}", or
1795       "\P{Katakana}".
1796
1797       What we have described so far is the single form of the "\p{...}"
1798       character classes.  There is also a compound form which you may run
1799       into.  These look like "\p{name=value}" or "\p{name:value}" (the equals
1800       sign and colon can be used interchangeably).  These are more general
1801       than the single form, and in fact most of the single forms are just
1802       Perl-defined shortcuts for common compound forms.  For example, the
1803       script examples in the previous paragraph could be written equivalently
1804       as "\p{Script=Latin}", "\p{Script:Greek}", and "\P{script=katakana}"
1805       (case is irrelevant between the "{}" braces).  You may never have to
1806       use the compound forms, but sometimes it is necessary, and their use
1807       can make your code easier to understand.
1808
1809       "\X" is an abbreviation for a character class that comprises a Unicode
1810       extended grapheme cluster.  This represents a "logical character": what
1811       appears to be a single character, but may be represented internally by
1812       more than one.  As an example, using the Unicode full names, e.g.,
1813       "A + COMBINING RING" is a grapheme cluster with base character "A" and
1814       combining character "COMBINING RING", which translates in Danish to A
1815       with the circle atop it, as in the word Angstrom.
1816
1817       For the full and latest information about Unicode see the latest
1818       Unicode standard, or the Unicode Consortium's website
1819       <http://www.unicode.org>
1820
1821       As if all those classes weren't enough, Perl also defines POSIX-style
1822       character classes.  These have the form "[:name:]", with "name" the
1823       name of the POSIX class.  The POSIX classes are "alpha", "alnum",
1824       "ascii", "cntrl", "digit", "graph", "lower", "print", "punct", "space",
1825       "upper", and "xdigit", and two extensions, "word" (a Perl extension to
1826       match "\w"), and "blank" (a GNU extension).  The "//a" modifier
1827       restricts these to matching just in the ASCII range; otherwise they can
1828       match the same as their corresponding Perl Unicode classes: "[:upper:]"
1829       is the same as "\p{IsUpper}", etc.  (There are some exceptions and
1830       gotchas with this; see perlrecharclass for a full discussion.) The
1831       "[:digit:]", "[:word:]", and "[:space:]" correspond to the familiar
1832       "\d", "\w", and "\s" character classes.  To negate a POSIX class, put a
1833       "^" in front of the name, so that, e.g., "[:^digit:]" corresponds to
1834       "\D" and, under Unicode, "\P{IsDigit}".  The Unicode and POSIX
1835       character classes can be used just like "\d", with the exception that
1836       POSIX character classes can only be used inside of a character class:
1837
1838           /\s+[abc[:digit:]xyz]\s*/;  # match a,b,c,x,y,z, or a digit
1839           /^=item\s[[:digit:]]/;      # match '=item',
1840                                       # followed by a space and a digit
1841           /\s+[abc\p{IsDigit}xyz]\s+/;  # match a,b,c,x,y,z, or a digit
1842           /^=item\s\p{IsDigit}/;        # match '=item',
1843                                         # followed by a space and a digit
1844
1845       Whew! That is all the rest of the characters and character classes.
1846
1847   Compiling and saving regular expressions
1848       In Part 1 we mentioned that Perl compiles a regexp into a compact
1849       sequence of opcodes.  Thus, a compiled regexp is a data structure that
1850       can be stored once and used again and again.  The regexp quote "qr//"
1851       does exactly that: "qr/string/" compiles the "string" as a regexp and
1852       transforms the result into a form that can be assigned to a variable:
1853
1854           $reg = qr/foo+bar?/;  # reg contains a compiled regexp
1855
1856       Then $reg can be used as a regexp:
1857
1858           $x = "fooooba";
1859           $x =~ $reg;     # matches, just like /foo+bar?/
1860           $x =~ /$reg/;   # same thing, alternate form
1861
1862       $reg can also be interpolated into a larger regexp:
1863
1864           $x =~ /(abc)?$reg/;  # still matches
1865
1866       As with the matching operator, the regexp quote can use different
1867       delimiters, e.g., "qr!!", "qr{}" or "qr~~".  Apostrophes as delimiters
1868       ("qr''") inhibit any interpolation.
1869
1870       Pre-compiled regexps are useful for creating dynamic matches that don't
1871       need to be recompiled each time they are encountered.  Using pre-
1872       compiled regexps, we write a "grep_step" program which greps for a
1873       sequence of patterns, advancing to the next pattern as soon as one has
1874       been satisfied.
1875
1876           % cat > grep_step
1877           #!/usr/bin/perl
1878           # grep_step - match <number> regexps, one after the other
1879           # usage: multi_grep <number> regexp1 regexp2 ... file1 file2 ...
1880
1881           $number = shift;
1882           $regexp[$_] = shift foreach (0..$number-1);
1883           @compiled = map qr/$_/, @regexp;
1884           while ($line = <>) {
1885               if ($line =~ /$compiled[0]/) {
1886                   print $line;
1887                   shift @compiled;
1888                   last unless @compiled;
1889               }
1890           }
1891           ^D
1892
1893           % grep_step 3 shift print last grep_step
1894           $number = shift;
1895                   print $line;
1896                   last unless @compiled;
1897
1898       Storing pre-compiled regexps in an array @compiled allows us to simply
1899       loop through the regexps without any recompilation, thus gaining
1900       flexibility without sacrificing speed.
1901
1902   Composing regular expressions at runtime
1903       Backtracking is more efficient than repeated tries with different
1904       regular expressions.  If there are several regular expressions and a
1905       match with any of them is acceptable, then it is possible to combine
1906       them into a set of alternatives.  If the individual expressions are
1907       input data, this can be done by programming a join operation.  We'll
1908       exploit this idea in an improved version of the "simple_grep" program:
1909       a program that matches multiple patterns:
1910
1911           % cat > multi_grep
1912           #!/usr/bin/perl
1913           # multi_grep - match any of <number> regexps
1914           # usage: multi_grep <number> regexp1 regexp2 ... file1 file2 ...
1915
1916           $number = shift;
1917           $regexp[$_] = shift foreach (0..$number-1);
1918           $pattern = join '|', @regexp;
1919
1920           while ($line = <>) {
1921               print $line if $line =~ /$pattern/;
1922           }
1923           ^D
1924
1925           % multi_grep 2 shift for multi_grep
1926           $number = shift;
1927           $regexp[$_] = shift foreach (0..$number-1);
1928
1929       Sometimes it is advantageous to construct a pattern from the input that
1930       is to be analyzed and use the permissible values on the left hand side
1931       of the matching operations.  As an example for this somewhat
1932       paradoxical situation, let's assume that our input contains a command
1933       verb which should match one out of a set of available command verbs,
1934       with the additional twist that commands may be abbreviated as long as
1935       the given string is unique. The program below demonstrates the basic
1936       algorithm.
1937
1938           % cat > keymatch
1939           #!/usr/bin/perl
1940           $kwds = 'copy compare list print';
1941           while( $command = <> ){
1942               $command =~ s/^\s+|\s+$//g;  # trim leading and trailing spaces
1943               if( ( @matches = $kwds =~ /\b$command\w*/g ) == 1 ){
1944                   print "command: '@matches'\n";
1945               } elsif( @matches == 0 ){
1946                   print "no such command: '$command'\n";
1947               } else {
1948                   print "not unique: '$command' (could be one of: @matches)\n";
1949               }
1950           }
1951           ^D
1952
1953           % keymatch
1954           li
1955           command: 'list'
1956           co
1957           not unique: 'co' (could be one of: copy compare)
1958           printer
1959           no such command: 'printer'
1960
1961       Rather than trying to match the input against the keywords, we match
1962       the combined set of keywords against the input.  The pattern matching
1963       operation "$kwds =~ /\b($command\w*)/g" does several things at the same
1964       time. It makes sure that the given command begins where a keyword
1965       begins ("\b"). It tolerates abbreviations due to the added "\w*". It
1966       tells us the number of matches ("scalar @matches") and all the keywords
1967       that were actually matched.  You could hardly ask for more.
1968
1969   Embedding comments and modifiers in a regular expression
1970       Starting with this section, we will be discussing Perl's set of
1971       extended patterns.  These are extensions to the traditional regular
1972       expression syntax that provide powerful new tools for pattern matching.
1973       We have already seen extensions in the form of the minimal matching
1974       constructs "??", "*?", "+?", "{n,m}?", and "{n,}?".  Most of the
1975       extensions below have the form "(?char...)", where the "char" is a
1976       character that determines the type of extension.
1977
1978       The first extension is an embedded comment "(?#text)".  This embeds a
1979       comment into the regular expression without affecting its meaning.  The
1980       comment should not have any closing parentheses in the text.  An
1981       example is
1982
1983           /(?# Match an integer:)[+-]?\d+/;
1984
1985       This style of commenting has been largely superseded by the raw,
1986       freeform commenting that is allowed with the "//x" modifier.
1987
1988       Most modifiers, such as "//i", "//m", "//s" and "//x" (or any
1989       combination thereof) can also be embedded in a regexp using "(?i)",
1990       "(?m)", "(?s)", and "(?x)".  For instance,
1991
1992           /(?i)yes/;  # match 'yes' case insensitively
1993           /yes/i;     # same thing
1994           /(?x)(          # freeform version of an integer regexp
1995                    [+-]?  # match an optional sign
1996                    \d+    # match a sequence of digits
1997                )
1998           /x;
1999
2000       Embedded modifiers can have two important advantages over the usual
2001       modifiers.  Embedded modifiers allow a custom set of modifiers to each
2002       regexp pattern.  This is great for matching an array of regexps that
2003       must have different modifiers:
2004
2005           $pattern[0] = '(?i)doctor';
2006           $pattern[1] = 'Johnson';
2007           ...
2008           while (<>) {
2009               foreach $patt (@pattern) {
2010                   print if /$patt/;
2011               }
2012           }
2013
2014       The second advantage is that embedded modifiers (except "//p", which
2015       modifies the entire regexp) only affect the regexp inside the group the
2016       embedded modifier is contained in.  So grouping can be used to localize
2017       the modifier's effects:
2018
2019           /Answer: ((?i)yes)/;  # matches 'Answer: yes', 'Answer: YES', etc.
2020
2021       Embedded modifiers can also turn off any modifiers already present by
2022       using, e.g., "(?-i)".  Modifiers can also be combined into a single
2023       expression, e.g., "(?s-i)" turns on single line mode and turns off case
2024       insensitivity.
2025
2026       Embedded modifiers may also be added to a non-capturing grouping.
2027       "(?i-m:regexp)" is a non-capturing grouping that matches "regexp" case
2028       insensitively and turns off multi-line mode.
2029
2030   Looking ahead and looking behind
2031       This section concerns the lookahead and lookbehind assertions.  First,
2032       a little background.
2033
2034       In Perl regular expressions, most regexp elements 'eat up' a certain
2035       amount of string when they match.  For instance, the regexp element
2036       "[abc}]" eats up one character of the string when it matches, in the
2037       sense that Perl moves to the next character position in the string
2038       after the match.  There are some elements, however, that don't eat up
2039       characters (advance the character position) if they match.  The
2040       examples we have seen so far are the anchors.  The anchor "^" matches
2041       the beginning of the line, but doesn't eat any characters.  Similarly,
2042       the word boundary anchor "\b" matches wherever a character matching
2043       "\w" is next to a character that doesn't, but it doesn't eat up any
2044       characters itself.  Anchors are examples of zero-width assertions:
2045       zero-width, because they consume no characters, and assertions, because
2046       they test some property of the string.  In the context of our walk in
2047       the woods analogy to regexp matching, most regexp elements move us
2048       along a trail, but anchors have us stop a moment and check our
2049       surroundings.  If the local environment checks out, we can proceed
2050       forward.  But if the local environment doesn't satisfy us, we must
2051       backtrack.
2052
2053       Checking the environment entails either looking ahead on the trail,
2054       looking behind, or both.  "^" looks behind, to see that there are no
2055       characters before.  "$" looks ahead, to see that there are no
2056       characters after.  "\b" looks both ahead and behind, to see if the
2057       characters on either side differ in their "word-ness".
2058
2059       The lookahead and lookbehind assertions are generalizations of the
2060       anchor concept.  Lookahead and lookbehind are zero-width assertions
2061       that let us specify which characters we want to test for.  The
2062       lookahead assertion is denoted by "(?=regexp)" and the lookbehind
2063       assertion is denoted by "(?<=fixed-regexp)".  Some examples are
2064
2065           $x = "I catch the housecat 'Tom-cat' with catnip";
2066           $x =~ /cat(?=\s)/;   # matches 'cat' in 'housecat'
2067           @catwords = ($x =~ /(?<=\s)cat\w+/g);  # matches,
2068                                                  # $catwords[0] = 'catch'
2069                                                  # $catwords[1] = 'catnip'
2070           $x =~ /\bcat\b/;  # matches 'cat' in 'Tom-cat'
2071           $x =~ /(?<=\s)cat(?=\s)/; # doesn't match; no isolated 'cat' in
2072                                     # middle of $x
2073
2074       Note that the parentheses in "(?=regexp)" and "(?<=regexp)" are non-
2075       capturing, since these are zero-width assertions.  Thus in the second
2076       regexp, the substrings captured are those of the whole regexp itself.
2077       Lookahead "(?=regexp)" can match arbitrary regexps, but lookbehind
2078       "(?<=fixed-regexp)" only works for regexps of fixed width, i.e., a
2079       fixed number of characters long.  Thus "(?<=(ab|bc))" is fine, but
2080       "(?<=(ab)*)" is not.  The negated versions of the lookahead and
2081       lookbehind assertions are denoted by "(?!regexp)" and
2082       "(?<!fixed-regexp)" respectively.  They evaluate true if the regexps do
2083       not match:
2084
2085           $x = "foobar";
2086           $x =~ /foo(?!bar)/;  # doesn't match, 'bar' follows 'foo'
2087           $x =~ /foo(?!baz)/;  # matches, 'baz' doesn't follow 'foo'
2088           $x =~ /(?<!\s)foo/;  # matches, there is no \s before 'foo'
2089
2090       The "\C" is unsupported in lookbehind, because the already treacherous
2091       definition of "\C" would become even more so when going backwards.
2092
2093       Here is an example where a string containing blank-separated words,
2094       numbers and single dashes is to be split into its components.  Using
2095       "/\s+/" alone won't work, because spaces are not required between
2096       dashes, or a word or a dash. Additional places for a split are
2097       established by looking ahead and behind:
2098
2099           $str = "one two - --6-8";
2100           @toks = split / \s+              # a run of spaces
2101                         | (?<=\S) (?=-)    # any non-space followed by '-'
2102                         | (?<=-)  (?=\S)   # a '-' followed by any non-space
2103                         /x, $str;          # @toks = qw(one two - - - 6 - 8)
2104
2105   Using independent subexpressions to prevent backtracking
2106       Independent subexpressions are regular expressions, in the context of a
2107       larger regular expression, that function independently of the larger
2108       regular expression.  That is, they consume as much or as little of the
2109       string as they wish without regard for the ability of the larger regexp
2110       to match.  Independent subexpressions are represented by "(?>regexp)".
2111       We can illustrate their behavior by first considering an ordinary
2112       regexp:
2113
2114           $x = "ab";
2115           $x =~ /a*ab/;  # matches
2116
2117       This obviously matches, but in the process of matching, the
2118       subexpression "a*" first grabbed the "a".  Doing so, however, wouldn't
2119       allow the whole regexp to match, so after backtracking, "a*" eventually
2120       gave back the "a" and matched the empty string.  Here, what "a*"
2121       matched was dependent on what the rest of the regexp matched.
2122
2123       Contrast that with an independent subexpression:
2124
2125           $x =~ /(?>a*)ab/;  # doesn't match!
2126
2127       The independent subexpression "(?>a*)" doesn't care about the rest of
2128       the regexp, so it sees an "a" and grabs it.  Then the rest of the
2129       regexp "ab" cannot match.  Because "(?>a*)" is independent, there is no
2130       backtracking and the independent subexpression does not give up its
2131       "a".  Thus the match of the regexp as a whole fails.  A similar
2132       behavior occurs with completely independent regexps:
2133
2134           $x = "ab";
2135           $x =~ /a*/g;   # matches, eats an 'a'
2136           $x =~ /\Gab/g; # doesn't match, no 'a' available
2137
2138       Here "//g" and "\G" create a 'tag team' handoff of the string from one
2139       regexp to the other.  Regexps with an independent subexpression are
2140       much like this, with a handoff of the string to the independent
2141       subexpression, and a handoff of the string back to the enclosing
2142       regexp.
2143
2144       The ability of an independent subexpression to prevent backtracking can
2145       be quite useful.  Suppose we want to match a non-empty string enclosed
2146       in parentheses up to two levels deep.  Then the following regexp
2147       matches:
2148
2149           $x = "abc(de(fg)h";  # unbalanced parentheses
2150           $x =~ /\( ( [^()]+ | \([^()]*\) )+ \)/x;
2151
2152       The regexp matches an open parenthesis, one or more copies of an
2153       alternation, and a close parenthesis.  The alternation is two-way, with
2154       the first alternative "[^()]+" matching a substring with no parentheses
2155       and the second alternative "\([^()]*\)"  matching a substring delimited
2156       by parentheses.  The problem with this regexp is that it is
2157       pathological: it has nested indeterminate quantifiers of the form
2158       "(a+|b)+".  We discussed in Part 1 how nested quantifiers like this
2159       could take an exponentially long time to execute if there was no match
2160       possible.  To prevent the exponential blowup, we need to prevent
2161       useless backtracking at some point.  This can be done by enclosing the
2162       inner quantifier as an independent subexpression:
2163
2164           $x =~ /\( ( (?>[^()]+) | \([^()]*\) )+ \)/x;
2165
2166       Here, "(?>[^()]+)" breaks the degeneracy of string partitioning by
2167       gobbling up as much of the string as possible and keeping it.   Then
2168       match failures fail much more quickly.
2169
2170   Conditional expressions
2171       A conditional expression is a form of if-then-else statement that
2172       allows one to choose which patterns are to be matched, based on some
2173       condition.  There are two types of conditional expression:
2174       "(?(condition)yes-regexp)" and "(?(condition)yes-regexp|no-regexp)".
2175       "(?(condition)yes-regexp)" is like an 'if () {}' statement in Perl.  If
2176       the "condition" is true, the "yes-regexp" will be matched.  If the
2177       "condition" is false, the "yes-regexp" will be skipped and Perl will
2178       move onto the next regexp element.  The second form is like an
2179       'if () {} else {}' statement in Perl.  If the "condition" is true, the
2180       "yes-regexp" will be matched, otherwise the "no-regexp" will be
2181       matched.
2182
2183       The "condition" can have several forms.  The first form is simply an
2184       integer in parentheses "(integer)".  It is true if the corresponding
2185       backreference "\integer" matched earlier in the regexp.  The same thing
2186       can be done with a name associated with a capture group, written as
2187       "(<name>)" or "('name')".  The second form is a bare zero-width
2188       assertion "(?...)", either a lookahead, a lookbehind, or a code
2189       assertion (discussed in the next section).  The third set of forms
2190       provides tests that return true if the expression is executed within a
2191       recursion ("(R)") or is being called from some capturing group,
2192       referenced either by number ("(R1)", "(R2)",...) or by name
2193       ("(R&name)").
2194
2195       The integer or name form of the "condition" allows us to choose, with
2196       more flexibility, what to match based on what matched earlier in the
2197       regexp. This searches for words of the form "$x$x" or "$x$y$y$x":
2198
2199           % simple_grep '^(\w+)(\w+)?(?(2)\g2\g1|\g1)$' /usr/dict/words
2200           beriberi
2201           coco
2202           couscous
2203           deed
2204           ...
2205           toot
2206           toto
2207           tutu
2208
2209       The lookbehind "condition" allows, along with backreferences, an
2210       earlier part of the match to influence a later part of the match.  For
2211       instance,
2212
2213           /[ATGC]+(?(?<=AA)G|C)$/;
2214
2215       matches a DNA sequence such that it either ends in "AAG", or some other
2216       base pair combination and "C".  Note that the form is "(?(?<=AA)G|C)"
2217       and not "(?((?<=AA))G|C)"; for the lookahead, lookbehind or code
2218       assertions, the parentheses around the conditional are not needed.
2219
2220   Defining named patterns
2221       Some regular expressions use identical subpatterns in several places.
2222       Starting with Perl 5.10, it is possible to define named subpatterns in
2223       a section of the pattern so that they can be called up by name anywhere
2224       in the pattern.  This syntactic pattern for this definition group is
2225       "(?(DEFINE)(?<name>pattern)...)".  An insertion of a named pattern is
2226       written as "(?&name)".
2227
2228       The example below illustrates this feature using the pattern for
2229       floating point numbers that was presented earlier on.  The three
2230       subpatterns that are used more than once are the optional sign, the
2231       digit sequence for an integer and the decimal fraction.  The DEFINE
2232       group at the end of the pattern contains their definition.  Notice that
2233       the decimal fraction pattern is the first place where we can reuse the
2234       integer pattern.
2235
2236          /^ (?&osg)\ * ( (?&int)(?&dec)? | (?&dec) )
2237             (?: [eE](?&osg)(?&int) )?
2238           $
2239           (?(DEFINE)
2240             (?<osg>[-+]?)         # optional sign
2241             (?<int>\d++)          # integer
2242             (?<dec>\.(?&int))     # decimal fraction
2243           )/x
2244
2245   Recursive patterns
2246       This feature (introduced in Perl 5.10) significantly extends the power
2247       of Perl's pattern matching.  By referring to some other capture group
2248       anywhere in the pattern with the construct "(?group-ref)", the pattern
2249       within the referenced group is used as an independent subpattern in
2250       place of the group reference itself.  Because the group reference may
2251       be contained within the group it refers to, it is now possible to apply
2252       pattern matching to tasks that hitherto required a recursive parser.
2253
2254       To illustrate this feature, we'll design a pattern that matches if a
2255       string contains a palindrome. (This is a word or a sentence that, while
2256       ignoring spaces, interpunctuation and case, reads the same backwards as
2257       forwards. We begin by observing that the empty string or a string
2258       containing just one word character is a palindrome. Otherwise it must
2259       have a word character up front and the same at its end, with another
2260       palindrome in between.
2261
2262           /(?: (\w) (?...Here be a palindrome...) \g{-1} | \w? )/x
2263
2264       Adding "\W*" at either end to eliminate what is to be ignored, we
2265       already have the full pattern:
2266
2267           my $pp = qr/^(\W* (?: (\w) (?1) \g{-1} | \w? ) \W*)$/ix;
2268           for $s ( "saippuakauppias", "A man, a plan, a canal: Panama!" ){
2269               print "'$s' is a palindrome\n" if $s =~ /$pp/;
2270           }
2271
2272       In "(?...)" both absolute and relative backreferences may be used.  The
2273       entire pattern can be reinserted with "(?R)" or "(?0)".  If you prefer
2274       to name your groups, you can use "(?&name)" to recurse into that group.
2275
2276   A bit of magic: executing Perl code in a regular expression
2277       Normally, regexps are a part of Perl expressions.  Code evaluation
2278       expressions turn that around by allowing arbitrary Perl code to be a
2279       part of a regexp.  A code evaluation expression is denoted "(?{code})",
2280       with code a string of Perl statements.
2281
2282       Be warned that this feature is considered experimental, and may be
2283       changed without notice.
2284
2285       Code expressions are zero-width assertions, and the value they return
2286       depends on their environment.  There are two possibilities: either the
2287       code expression is used as a conditional in a conditional expression
2288       "(?(condition)...)", or it is not.  If the code expression is a
2289       conditional, the code is evaluated and the result (i.e., the result of
2290       the last statement) is used to determine truth or falsehood.  If the
2291       code expression is not used as a conditional, the assertion always
2292       evaluates true and the result is put into the special variable $^R.
2293       The variable $^R can then be used in code expressions later in the
2294       regexp.  Here are some silly examples:
2295
2296           $x = "abcdef";
2297           $x =~ /abc(?{print "Hi Mom!";})def/; # matches,
2298                                                # prints 'Hi Mom!'
2299           $x =~ /aaa(?{print "Hi Mom!";})def/; # doesn't match,
2300                                                # no 'Hi Mom!'
2301
2302       Pay careful attention to the next example:
2303
2304           $x =~ /abc(?{print "Hi Mom!";})ddd/; # doesn't match,
2305                                                # no 'Hi Mom!'
2306                                                # but why not?
2307
2308       At first glance, you'd think that it shouldn't print, because obviously
2309       the "ddd" isn't going to match the target string. But look at this
2310       example:
2311
2312           $x =~ /abc(?{print "Hi Mom!";})[dD]dd/; # doesn't match,
2313                                                   # but _does_ print
2314
2315       Hmm. What happened here? If you've been following along, you know that
2316       the above pattern should be effectively (almost) the same as the last
2317       one; enclosing the "d" in a character class isn't going to change what
2318       it matches. So why does the first not print while the second one does?
2319
2320       The answer lies in the optimizations the regex engine makes. In the
2321       first case, all the engine sees are plain old characters (aside from
2322       the "?{}" construct). It's smart enough to realize that the string
2323       'ddd' doesn't occur in our target string before actually running the
2324       pattern through. But in the second case, we've tricked it into thinking
2325       that our pattern is more complicated. It takes a look, sees our
2326       character class, and decides that it will have to actually run the
2327       pattern to determine whether or not it matches, and in the process of
2328       running it hits the print statement before it discovers that we don't
2329       have a match.
2330
2331       To take a closer look at how the engine does optimizations, see the
2332       section "Pragmas and debugging" below.
2333
2334       More fun with "?{}":
2335
2336           $x =~ /(?{print "Hi Mom!";})/;       # matches,
2337                                                # prints 'Hi Mom!'
2338           $x =~ /(?{$c = 1;})(?{print "$c";})/;  # matches,
2339                                                  # prints '1'
2340           $x =~ /(?{$c = 1;})(?{print "$^R";})/; # matches,
2341                                                  # prints '1'
2342
2343       The bit of magic mentioned in the section title occurs when the regexp
2344       backtracks in the process of searching for a match.  If the regexp
2345       backtracks over a code expression and if the variables used within are
2346       localized using "local", the changes in the variables produced by the
2347       code expression are undone! Thus, if we wanted to count how many times
2348       a character got matched inside a group, we could use, e.g.,
2349
2350           $x = "aaaa";
2351           $count = 0;  # initialize 'a' count
2352           $c = "bob";  # test if $c gets clobbered
2353           $x =~ /(?{local $c = 0;})         # initialize count
2354                  ( a                        # match 'a'
2355                    (?{local $c = $c + 1;})  # increment count
2356                  )*                         # do this any number of times,
2357                  aa                         # but match 'aa' at the end
2358                  (?{$count = $c;})          # copy local $c var into $count
2359                 /x;
2360           print "'a' count is $count, \$c variable is '$c'\n";
2361
2362       This prints
2363
2364           'a' count is 2, $c variable is 'bob'
2365
2366       If we replace the " (?{local $c = $c + 1;})" with " (?{$c = $c + 1;})",
2367       the variable changes are not undone during backtracking, and we get
2368
2369           'a' count is 4, $c variable is 'bob'
2370
2371       Note that only localized variable changes are undone.  Other side
2372       effects of code expression execution are permanent.  Thus
2373
2374           $x = "aaaa";
2375           $x =~ /(a(?{print "Yow\n";}))*aa/;
2376
2377       produces
2378
2379          Yow
2380          Yow
2381          Yow
2382          Yow
2383
2384       The result $^R is automatically localized, so that it will behave
2385       properly in the presence of backtracking.
2386
2387       This example uses a code expression in a conditional to match a
2388       definite article, either 'the' in English or 'der|die|das' in German:
2389
2390           $lang = 'DE';  # use German
2391           ...
2392           $text = "das";
2393           print "matched\n"
2394               if $text =~ /(?(?{
2395                                 $lang eq 'EN'; # is the language English?
2396                                })
2397                              the |             # if so, then match 'the'
2398                              (der|die|das)     # else, match 'der|die|das'
2399                            )
2400                           /xi;
2401
2402       Note that the syntax here is "(?(?{...})yes-regexp|no-regexp)", not
2403       "(?((?{...}))yes-regexp|no-regexp)".  In other words, in the case of a
2404       code expression, we don't need the extra parentheses around the
2405       conditional.
2406
2407       If you try to use code expressions with interpolating variables, Perl
2408       may surprise you:
2409
2410           $bar = 5;
2411           $pat = '(?{ 1 })';
2412           /foo(?{ $bar })bar/; # compiles ok, $bar not interpolated
2413           /foo(?{ 1 })$bar/;   # compile error!
2414           /foo${pat}bar/;      # compile error!
2415
2416           $pat = qr/(?{ $foo = 1 })/;  # precompile code regexp
2417           /foo${pat}bar/;      # compiles ok
2418
2419       If a regexp has (1) code expressions and interpolating variables, or
2420       (2) a variable that interpolates a code expression, Perl treats the
2421       regexp as an error. If the code expression is precompiled into a
2422       variable, however, interpolating is ok. The question is, why is this an
2423       error?
2424
2425       The reason is that variable interpolation and code expressions together
2426       pose a security risk.  The combination is dangerous because many
2427       programmers who write search engines often take user input and plug it
2428       directly into a regexp:
2429
2430           $regexp = <>;       # read user-supplied regexp
2431           $chomp $regexp;     # get rid of possible newline
2432           $text =~ /$regexp/; # search $text for the $regexp
2433
2434       If the $regexp variable contains a code expression, the user could then
2435       execute arbitrary Perl code.  For instance, some joker could search for
2436       "system('rm -rf *');" to erase your files.  In this sense, the
2437       combination of interpolation and code expressions taints your regexp.
2438       So by default, using both interpolation and code expressions in the
2439       same regexp is not allowed.  If you're not concerned about malicious
2440       users, it is possible to bypass this security check by invoking
2441       "use re 'eval'":
2442
2443           use re 'eval';       # throw caution out the door
2444           $bar = 5;
2445           $pat = '(?{ 1 })';
2446           /foo(?{ 1 })$bar/;   # compiles ok
2447           /foo${pat}bar/;      # compiles ok
2448
2449       Another form of code expression is the pattern code expression.  The
2450       pattern code expression is like a regular code expression, except that
2451       the result of the code evaluation is treated as a regular expression
2452       and matched immediately.  A simple example is
2453
2454           $length = 5;
2455           $char = 'a';
2456           $x = 'aaaaabb';
2457           $x =~ /(??{$char x $length})/x; # matches, there are 5 of 'a'
2458
2459       This final example contains both ordinary and pattern code expressions.
2460       It detects whether a binary string 1101010010001... has a Fibonacci
2461       spacing 0,1,1,2,3,5,...  of the 1's:
2462
2463           $x = "1101010010001000001";
2464           $z0 = ''; $z1 = '0';   # initial conditions
2465           print "It is a Fibonacci sequence\n"
2466               if $x =~ /^1         # match an initial '1'
2467                           (?:
2468                              ((??{ $z0 })) # match some '0'
2469                              1             # and then a '1'
2470                              (?{ $z0 = $z1; $z1 .= $^N; })
2471                           )+   # repeat as needed
2472                         $      # that is all there is
2473                        /x;
2474           printf "Largest sequence matched was %d\n", length($z1)-length($z0);
2475
2476       Remember that $^N is set to whatever was matched by the last completed
2477       capture group. This prints
2478
2479           It is a Fibonacci sequence
2480           Largest sequence matched was 5
2481
2482       Ha! Try that with your garden variety regexp package...
2483
2484       Note that the variables $z0 and $z1 are not substituted when the regexp
2485       is compiled, as happens for ordinary variables outside a code
2486       expression.  Rather, the code expressions are evaluated when Perl
2487       encounters them during the search for a match.
2488
2489       The regexp without the "//x" modifier is
2490
2491           /^1(?:((??{ $z0 }))1(?{ $z0 = $z1; $z1 .= $^N; }))+$/
2492
2493       which shows that spaces are still possible in the code parts.
2494       Nevertheless, when working with code and conditional expressions, the
2495       extended form of regexps is almost necessary in creating and debugging
2496       regexps.
2497
2498   Backtracking control verbs
2499       Perl 5.10 introduced a number of control verbs intended to provide
2500       detailed control over the backtracking process, by directly influencing
2501       the regexp engine and by providing monitoring techniques.  As all the
2502       features in this group are experimental and subject to change or
2503       removal in a future version of Perl, the interested reader is referred
2504       to "Special Backtracking Control Verbs" in perlre for a detailed
2505       description.
2506
2507       Below is just one example, illustrating the control verb "(*FAIL)",
2508       which may be abbreviated as "(*F)". If this is inserted in a regexp it
2509       will cause it to fail, just as it would at some mismatch between the
2510       pattern and the string. Processing of the regexp continues as it would
2511       after any "normal" failure, so that, for instance, the next position in
2512       the string or another alternative will be tried. As failing to match
2513       doesn't preserve capture groups or produce results, it may be necessary
2514       to use this in combination with embedded code.
2515
2516          %count = ();
2517          "supercalifragilisticexpialidocious" =~
2518              /([aeiou])(?{ $count{$1}++; })(*FAIL)/i;
2519          printf "%3d '%s'\n", $count{$_}, $_ for (sort keys %count);
2520
2521       The pattern begins with a class matching a subset of letters.  Whenever
2522       this matches, a statement like "$count{'a'}++;" is executed,
2523       incrementing the letter's counter. Then "(*FAIL)" does what it says,
2524       and the regexp engine proceeds according to the book: as long as the
2525       end of the string hasn't been reached, the position is advanced before
2526       looking for another vowel. Thus, match or no match makes no difference,
2527       and the regexp engine proceeds until the entire string has been
2528       inspected.  (It's remarkable that an alternative solution using
2529       something like
2530
2531          $count{lc($_)}++ for split('', "supercalifragilisticexpialidocious");
2532          printf "%3d '%s'\n", $count2{$_}, $_ for ( qw{ a e i o u } );
2533
2534       is considerably slower.)
2535
2536   Pragmas and debugging
2537       Speaking of debugging, there are several pragmas available to control
2538       and debug regexps in Perl.  We have already encountered one pragma in
2539       the previous section, "use re 'eval';", that allows variable
2540       interpolation and code expressions to coexist in a regexp.  The other
2541       pragmas are
2542
2543           use re 'taint';
2544           $tainted = <>;
2545           @parts = ($tainted =~ /(\w+)\s+(\w+)/; # @parts is now tainted
2546
2547       The "taint" pragma causes any substrings from a match with a tainted
2548       variable to be tainted as well.  This is not normally the case, as
2549       regexps are often used to extract the safe bits from a tainted
2550       variable.  Use "taint" when you are not extracting safe bits, but are
2551       performing some other processing.  Both "taint" and "eval" pragmas are
2552       lexically scoped, which means they are in effect only until the end of
2553       the block enclosing the pragmas.
2554
2555           use re '/m';  # or any other flags
2556           $multiline_string =~ /^foo/; # /m is implied
2557
2558       The "re '/flags'" pragma (introduced in Perl 5.14) turns on the given
2559       regular expression flags until the end of the lexical scope.  See
2560       "'/flags' mode" in re for more detail.
2561
2562           use re 'debug';
2563           /^(.*)$/s;       # output debugging info
2564
2565           use re 'debugcolor';
2566           /^(.*)$/s;       # output debugging info in living color
2567
2568       The global "debug" and "debugcolor" pragmas allow one to get detailed
2569       debugging info about regexp compilation and execution.  "debugcolor" is
2570       the same as debug, except the debugging information is displayed in
2571       color on terminals that can display termcap color sequences.  Here is
2572       example output:
2573
2574           % perl -e 'use re "debug"; "abc" =~ /a*b+c/;'
2575           Compiling REx 'a*b+c'
2576           size 9 first at 1
2577              1: STAR(4)
2578              2:   EXACT <a>(0)
2579              4: PLUS(7)
2580              5:   EXACT <b>(0)
2581              7: EXACT <c>(9)
2582              9: END(0)
2583           floating 'bc' at 0..2147483647 (checking floating) minlen 2
2584           Guessing start of match, REx 'a*b+c' against 'abc'...
2585           Found floating substr 'bc' at offset 1...
2586           Guessed: match at offset 0
2587           Matching REx 'a*b+c' against 'abc'
2588             Setting an EVAL scope, savestack=3
2589              0 <> <abc>             |  1:  STAR
2590                                      EXACT <a> can match 1 times out of 32767...
2591             Setting an EVAL scope, savestack=3
2592              1 <a> <bc>             |  4:    PLUS
2593                                      EXACT <b> can match 1 times out of 32767...
2594             Setting an EVAL scope, savestack=3
2595              2 <ab> <c>             |  7:      EXACT <c>
2596              3 <abc> <>             |  9:      END
2597           Match successful!
2598           Freeing REx: 'a*b+c'
2599
2600       If you have gotten this far into the tutorial, you can probably guess
2601       what the different parts of the debugging output tell you.  The first
2602       part
2603
2604           Compiling REx 'a*b+c'
2605           size 9 first at 1
2606              1: STAR(4)
2607              2:   EXACT <a>(0)
2608              4: PLUS(7)
2609              5:   EXACT <b>(0)
2610              7: EXACT <c>(9)
2611              9: END(0)
2612
2613       describes the compilation stage.  STAR(4) means that there is a starred
2614       object, in this case 'a', and if it matches, goto line 4, i.e.,
2615       PLUS(7).  The middle lines describe some heuristics and optimizations
2616       performed before a match:
2617
2618           floating 'bc' at 0..2147483647 (checking floating) minlen 2
2619           Guessing start of match, REx 'a*b+c' against 'abc'...
2620           Found floating substr 'bc' at offset 1...
2621           Guessed: match at offset 0
2622
2623       Then the match is executed and the remaining lines describe the
2624       process:
2625
2626           Matching REx 'a*b+c' against 'abc'
2627             Setting an EVAL scope, savestack=3
2628              0 <> <abc>             |  1:  STAR
2629                                      EXACT <a> can match 1 times out of 32767...
2630             Setting an EVAL scope, savestack=3
2631              1 <a> <bc>             |  4:    PLUS
2632                                      EXACT <b> can match 1 times out of 32767...
2633             Setting an EVAL scope, savestack=3
2634              2 <ab> <c>             |  7:      EXACT <c>
2635              3 <abc> <>             |  9:      END
2636           Match successful!
2637           Freeing REx: 'a*b+c'
2638
2639       Each step is of the form "n <x> <y>", with "<x>" the part of the string
2640       matched and "<y>" the part not yet matched.  The "|  1:  STAR" says
2641       that Perl is at line number 1 in the compilation list above.  See
2642       "Debugging Regular Expressions" in perldebguts for much more detail.
2643
2644       An alternative method of debugging regexps is to embed "print"
2645       statements within the regexp.  This provides a blow-by-blow account of
2646       the backtracking in an alternation:
2647
2648           "that this" =~ m@(?{print "Start at position ", pos, "\n";})
2649                            t(?{print "t1\n";})
2650                            h(?{print "h1\n";})
2651                            i(?{print "i1\n";})
2652                            s(?{print "s1\n";})
2653                                |
2654                            t(?{print "t2\n";})
2655                            h(?{print "h2\n";})
2656                            a(?{print "a2\n";})
2657                            t(?{print "t2\n";})
2658                            (?{print "Done at position ", pos, "\n";})
2659                           @x;
2660
2661       prints
2662
2663           Start at position 0
2664           t1
2665           h1
2666           t2
2667           h2
2668           a2
2669           t2
2670           Done at position 4
2671

BUGS

2673       Code expressions, conditional expressions, and independent expressions
2674       are experimental.  Don't use them in production code.  Yet.
2675

SEE ALSO

2677       This is just a tutorial.  For the full story on Perl regular
2678       expressions, see the perlre regular expressions reference page.
2679
2680       For more information on the matching "m//" and substitution "s///"
2681       operators, see "Regexp Quote-Like Operators" in perlop.  For
2682       information on the "split" operation, see "split" in perlfunc.
2683
2684       For an excellent all-around resource on the care and feeding of regular
2685       expressions, see the book Mastering Regular Expressions by Jeffrey
2686       Friedl (published by O'Reilly, ISBN 1556592-257-3).
2687
2689       Copyright (c) 2000 Mark Kvale All rights reserved.
2690
2691       This document may be distributed under the same terms as Perl itself.
2692
2693   Acknowledgments
2694       The inspiration for the stop codon DNA example came from the ZIP code
2695       example in chapter 7 of Mastering Regular Expressions.
2696
2697       The author would like to thank Jeff Pinyan, Andrew Johnson, Peter
2698       Haworth, Ronald J Kimball, and Joe Smith for all their helpful
2699       comments.
2700
2701
2702
2703perl v5.16.3                      2013-03-04                      PERLRETUT(1)
Impressum