1PERLRETUT(1) Perl Programmers Reference Guide PERLRETUT(1)
2
6 perlretut - Perl regular expressions tutorial
7
9 This page provides a basic tutorial on understanding, creating and
10 using regular expressions in Perl. It serves as a complement to the
11 reference page on regular expressions perlre. Regular expressions are
12 an integral part of the "m//", "s///", "qr//" and "split" operators and
13 so this tutorial also overlaps with "Regexp Quote-Like Operators" in
14 perlop and "split" in perlfunc.
15 Perl is widely renowned for excellence in text processing, and regular
17 expressions are one of the big factors behind this fame. Perl regular
18 expressions display an efficiency and flexibility unknown in most other
19 computer languages. Mastering even the basics of regular expressions
20 will allow you to manipulate text with surprising ease.
21 What is a regular expression? A regular expression is simply a string
23 that describes a pattern. Patterns are in common use these days;
24 examples are the patterns typed into a search engine to find web pages
25 and the patterns used to list files in a directory, e.g., "ls *.txt" or
26 "dir *.*". In Perl, the patterns described by regular expressions are
27 used to search strings, extract desired parts of strings, and to do
28 search and replace operations.
29 Regular expressions have the undeserved reputation of being abstract
31 and difficult to understand. Regular expressions are constructed using
32 simple concepts like conditionals and loops and are no more difficult
33 to understand than the corresponding "if" conditionals and "while"
34 loops in the Perl language itself. In fact, the main challenge in
35 learning regular expressions is just getting used to the terse notation
36 used to express these concepts.
37 This tutorial flattens the learning curve by discussing regular
39 expression concepts, along with their notation, one at a time and with
40 many examples. The first part of the tutorial will progress from the
41 simplest word searches to the basic regular expression concepts. If
42 you master the first part, you will have all the tools needed to solve
43 about 98% of your needs. The second part of the tutorial is for those
44 comfortable with the basics and hungry for more power tools. It
45 discusses the more advanced regular expression operators and introduces
46 the latest cutting-edge innovations.
47 A note: to save time, 'regular expression' is often abbreviated as
49 regexp or regex. Regexp is a more natural abbreviation than regex, but
50 is harder to pronounce. The Perl pod documentation is evenly split on
51 regexp vs regex; in Perl, there is more than one way to abbreviate it.
52 We'll use regexp in this tutorial.
53
55 Simple word matching
56 The simplest regexp is simply a word, or more generally, a string of
57 characters. A regexp consisting of a word matches any string that
58 contains that word:
59 "Hello World" =~ /World/; # matches
61 What is this Perl statement all about? "Hello World" is a simple
63 double-quoted string. "World" is the regular expression and the "//"
64 enclosing "/World/" tells Perl to search a string for a match. The
65 operator "=~" associates the string with the regexp match and produces
66 a true value if the regexp matched, or false if the regexp did not
67 match. In our case, "World" matches the second word in "Hello World",
68 so the expression is true. Expressions like this are useful in
69 conditionals:
70 if ("Hello World" =~ /World/) {
72 print "It matches\n";
73 }
74 else {
75 print "It doesn't match\n";
76 }
77 There are useful variations on this theme. The sense of the match can
79 be reversed by using the "!~" operator:
80 if ("Hello World" !~ /World/) {
82 print "It doesn't match\n";
83 }
84 else {
85 print "It matches\n";
86 }
87 The literal string in the regexp can be replaced by a variable:
89 $greeting = "World";
91 if ("Hello World" =~ /$greeting/) {
92 print "It matches\n";
93 }
94 else {
95 print "It doesn't match\n";
96 }
97 If you're matching against the special default variable $_, the "$_ =~"
99 part can be omitted:
100 $_ = "Hello World";
102 if (/World/) {
103 print "It matches\n";
104 }
105 else {
106 print "It doesn't match\n";
107 }
108 And finally, the "//" default delimiters for a match can be changed to
110 arbitrary delimiters by putting an 'm' out front:
111 "Hello World" =~ m!World!; # matches, delimited by '!'
113 "Hello World" =~ m{World}; # matches, note the matching '{}'
114 "/usr/bin/perl" =~ m"/perl"; # matches after '/usr/bin',
115 # '/' becomes an ordinary char
116 "/World/", "m!World!", and "m{World}" all represent the same thing.
118 When, e.g., the quote (""") is used as a delimiter, the forward slash
119 '/' becomes an ordinary character and can be used in this regexp
120 without trouble.
121 Let's consider how different regexps would match "Hello World":
123 "Hello World" =~ /world/; # doesn't match
125 "Hello World" =~ /o W/; # matches
126 "Hello World" =~ /oW/; # doesn't match
127 "Hello World" =~ /World /; # doesn't match
128 The first regexp "world" doesn't match because regexps are case-
130 sensitive. The second regexp matches because the substring 'o W'
131 occurs in the string "Hello World". The space character ' ' is treated
132 like any other character in a regexp and is needed to match in this
133 case. The lack of a space character is the reason the third regexp
134 'oW' doesn't match. The fourth regexp 'World ' doesn't match because
135 there is a space at the end of the regexp, but not at the end of the
136 string. The lesson here is that regexps must match a part of the
137 string exactly in order for the statement to be true.
138 If a regexp matches in more than one place in the string, Perl will
140 always match at the earliest possible point in the string:
141 "Hello World" =~ /o/; # matches 'o' in 'Hello'
143 "That hat is red" =~ /hat/; # matches 'hat' in 'That'
144 With respect to character matching, there are a few more points you
146 need to know about. First of all, not all characters can be used 'as
147 is' in a match. Some characters, called metacharacters, are reserved
148 for use in regexp notation. The metacharacters are
149 {}[]()^$.|*+?\
151 The significance of each of these will be explained in the rest of the
153 tutorial, but for now, it is important only to know that a
154 metacharacter can be matched by putting a backslash before it:
155 "2+2=4" =~ /2+2/; # doesn't match, + is a metacharacter
157 "2+2=4" =~ /2\+2/; # matches, \+ is treated like an ordinary +
158 "The interval is [0,1)." =~ /[0,1)./ # is a syntax error!
159 "The interval is [0,1)." =~ /\[0,1\)\./ # matches
160 "#!/usr/bin/perl" =~ /#!\/usr\/bin\/perl/; # matches
161 In the last regexp, the forward slash '/' is also backslashed, because
163 it is used to delimit the regexp. This can lead to LTS (leaning
164 toothpick syndrome), however, and it is often more readable to change
165 delimiters.
166 "#!/usr/bin/perl" =~ m!#\!/usr/bin/perl!; # easier to read
168 The backslash character '\' is a metacharacter itself and needs to be
170 backslashed:
171 'C:\WIN32' =~ /C:\\WIN/; # matches
173 In addition to the metacharacters, there are some ASCII characters
175 which don't have printable character equivalents and are instead
176 represented by escape sequences. Common examples are "\t" for a tab,
177 "\n" for a newline, "\r" for a carriage return and "\a" for a bell (or
178 alert). If your string is better thought of as a sequence of arbitrary
179 bytes, the octal escape sequence, e.g., "\033", or hexadecimal escape
180 sequence, e.g., "\x1B" may be a more natural representation for your
181 bytes. Here are some examples of escapes:
182 "1000\t2000" =~ m(0\t2) # matches
184 "1000\n2000" =~ /0\n20/ # matches
185 "1000\t2000" =~ /\000\t2/ # doesn't match, "0" ne "\000"
186 "cat" =~ /\o{143}\x61\x74/ # matches in ASCII, but a weird way
187 # to spell cat
188 If you've been around Perl a while, all this talk of escape sequences
190 may seem familiar. Similar escape sequences are used in double-quoted
191 strings and in fact the regexps in Perl are mostly treated as double-
192 quoted strings. This means that variables can be used in regexps as
193 well. Just like double-quoted strings, the values of the variables in
194 the regexp will be substituted in before the regexp is evaluated for
195 matching purposes. So we have:
196 $foo = 'house';
198 'housecat' =~ /$foo/; # matches
199 'cathouse' =~ /cat$foo/; # matches
200 'housecat' =~ /${foo}cat/; # matches
201 So far, so good. With the knowledge above you can already perform
203 searches with just about any literal string regexp you can dream up.
204 Here is a very simple emulation of the Unix grep program:
205 % cat > simple_grep
207 #!/usr/bin/perl
208 $regexp = shift;
209 while (<>) {
210 print if /$regexp/;
211 }
212 ^D
213 % chmod +x simple_grep
215 % simple_grep abba /usr/dict/words
217 Babbage
218 cabbage
219 cabbages
220 sabbath
221 Sabbathize
222 Sabbathizes
223 sabbatical
224 scabbard
225 scabbards
226 This program is easy to understand. "#!/usr/bin/perl" is the standard
228 way to invoke a perl program from the shell. "$regexp = shift;" saves
229 the first command line argument as the regexp to be used, leaving the
230 rest of the command line arguments to be treated as files.
231 "while (<>)" loops over all the lines in all the files. For each line,
232 "print if /$regexp/;" prints the line if the regexp matches the line.
233 In this line, both "print" and "/$regexp/" use the default variable $_
234 implicitly.
235 With all of the regexps above, if the regexp matched anywhere in the
237 string, it was considered a match. Sometimes, however, we'd like to
238 specify where in the string the regexp should try to match. To do
239 this, we would use the anchor metacharacters "^" and "$". The anchor
240 "^" means match at the beginning of the string and the anchor "$" means
241 match at the end of the string, or before a newline at the end of the
242 string. Here is how they are used:
243 "housekeeper" =~ /keeper/; # matches
245 "housekeeper" =~ /^keeper/; # doesn't match
246 "housekeeper" =~ /keeper$/; # matches
247 "housekeeper\n" =~ /keeper$/; # matches
248 The second regexp doesn't match because "^" constrains "keeper" to
250 match only at the beginning of the string, but "housekeeper" has keeper
251 starting in the middle. The third regexp does match, since the "$"
252 constrains "keeper" to match only at the end of the string.
253 When both "^" and "$" are used at the same time, the regexp has to
255 match both the beginning and the end of the string, i.e., the regexp
256 matches the whole string. Consider
257 "keeper" =~ /^keep$/; # doesn't match
259 "keeper" =~ /^keeper$/; # matches
260 "" =~ /^$/; # ^$ matches an empty string
261 The first regexp doesn't match because the string has more to it than
263 "keep". Since the second regexp is exactly the string, it matches.
264 Using both "^" and "$" in a regexp forces the complete string to match,
265 so it gives you complete control over which strings match and which
266 don't. Suppose you are looking for a fellow named bert, off in a
267 string by himself:
268 "dogbert" =~ /bert/; # matches, but not what you want
270 "dilbert" =~ /^bert/; # doesn't match, but ..
272 "bertram" =~ /^bert/; # matches, so still not good enough
273 "bertram" =~ /^bert$/; # doesn't match, good
275 "dilbert" =~ /^bert$/; # doesn't match, good
276 "bert" =~ /^bert$/; # matches, perfect
277 Of course, in the case of a literal string, one could just as easily
279 use the string comparison "$string eq 'bert'" and it would be more
280 efficient. The "^...$" regexp really becomes useful when we add in
281 the more powerful regexp tools below.
282 Using character classes
284 Although one can already do quite a lot with the literal string regexps
285 above, we've only scratched the surface of regular expression
286 technology. In this and subsequent sections we will introduce regexp
287 concepts (and associated metacharacter notations) that will allow a
288 regexp to represent not just a single character sequence, but a whole
289 class of them.
290 One such concept is that of a character class. A character class
292 allows a set of possible characters, rather than just a single
293 character, to match at a particular point in a regexp. Character
294 classes are denoted by brackets "[...]", with the set of characters to
295 be possibly matched inside. Here are some examples:
296 /cat/; # matches 'cat'
298 /[bcr]at/; # matches 'bat, 'cat', or 'rat'
299 /item[0123456789]/; # matches 'item0' or ... or 'item9'
300 "abc" =~ /[cab]/; # matches 'a'
301 In the last statement, even though 'c' is the first character in the
303 class, 'a' matches because the first character position in the string
304 is the earliest point at which the regexp can match.
305 /[yY][eE][sS]/; # match 'yes' in a case-insensitive way
307 # 'yes', 'Yes', 'YES', etc.
308 This regexp displays a common task: perform a case-insensitive match.
310 Perl provides a way of avoiding all those brackets by simply appending
311 an 'i' to the end of the match. Then "/[yY][eE][sS]/;" can be
312 rewritten as "/yes/i;". The 'i' stands for case-insensitive and is an
313 example of a modifier of the matching operation. We will meet other
314 modifiers later in the tutorial.
315 We saw in the section above that there were ordinary characters, which
317 represented themselves, and special characters, which needed a
318 backslash "\" to represent themselves. The same is true in a character
319 class, but the sets of ordinary and special characters inside a
320 character class are different than those outside a character class.
321 The special characters for a character class are "-]\^$" (and the
322 pattern delimiter, whatever it is). "]" is special because it denotes
323 the end of a character class. "$" is special because it denotes a
324 scalar variable. "\" is special because it is used in escape
325 sequences, just like above. Here is how the special characters "]$\"
326 are handled:
327 /[\]c]def/; # matches ']def' or 'cdef'
329 $x = 'bcr';
330 /[$x]at/; # matches 'bat', 'cat', or 'rat'
331 /[\$x]at/; # matches '$at' or 'xat'
332 /[\\$x]at/; # matches '\at', 'bat, 'cat', or 'rat'
333 The last two are a little tricky. In "[\$x]", the backslash protects
335 the dollar sign, so the character class has two members "$" and "x".
336 In "[\\$x]", the backslash is protected, so $x is treated as a variable
337 and substituted in double quote fashion.
338 The special character '-' acts as a range operator within character
340 classes, so that a contiguous set of characters can be written as a
341 range. With ranges, the unwieldy "[0123456789]" and "[abc...xyz]"
342 become the svelte "[0-9]" and "[a-z]". Some examples are
343 /item[0-9]/; # matches 'item0' or ... or 'item9'
345 /[0-9bx-z]aa/; # matches '0aa', ..., '9aa',
346 # 'baa', 'xaa', 'yaa', or 'zaa'
347 /[0-9a-fA-F]/; # matches a hexadecimal digit
348 /[0-9a-zA-Z_]/; # matches a "word" character,
349 # like those in a Perl variable name
350 If '-' is the first or last character in a character class, it is
352 treated as an ordinary character; "[-ab]", "[ab-]" and "[a\-b]" are all
353 equivalent.
354 The special character "^" in the first position of a character class
356 denotes a negated character class, which matches any character but
357 those in the brackets. Both "[...]" and "[^...]" must match a
358 character, or the match fails. Then
359 /[^a]at/; # doesn't match 'aat' or 'at', but matches
361 # all other 'bat', 'cat, '0at', '%at', etc.
362 /[^0-9]/; # matches a non-numeric character
363 /[a^]at/; # matches 'aat' or '^at'; here '^' is ordinary
364 Now, even "[0-9]" can be a bother to write multiple times, so in the
366 interest of saving keystrokes and making regexps more readable, Perl
367 has several abbreviations for common character classes, as shown below.
368 Since the introduction of Unicode, unless the "//a" modifier is in
369 effect, these character classes match more than just a few characters
370 in the ASCII range.
371 · \d matches a digit, not just [0-9] but also digits from non-roman
373 scripts
374 · \s matches a whitespace character, the set [\ \t\r\n\f] and others
376 · \w matches a word character (alphanumeric or _), not just
378 [0-9a-zA-Z_] but also digits and characters from non-roman scripts
379 · \D is a negated \d; it represents any other character than a digit,
381 or [^\d]
382 · \S is a negated \s; it represents any non-whitespace character
384 [^\s]
385 · \W is a negated \w; it represents any non-word character [^\w]
387 · The period '.' matches any character but "\n" (unless the modifier
389 "//s" is in effect, as explained below).
390 · \N, like the period, matches any character but "\n", but it does so
392 regardless of whether the modifier "//s" is in effect.
393 The "//a" modifier, available starting in Perl 5.14, is used to
395 restrict the matches of \d, \s, and \w to just those in the ASCII
396 range. It is useful to keep your program from being needlessly exposed
397 to full Unicode (and its accompanying security considerations) when all
398 you want is to process English-like text. (The "a" may be doubled,
399 "//aa", to provide even more restrictions, preventing case-insensitive
400 matching of ASCII with non-ASCII characters; otherwise a Unicode
401 "Kelvin Sign" would caselessly match a "k" or "K".)
402 The "\d\s\w\D\S\W" abbreviations can be used both inside and outside of
404 character classes. Here are some in use:
405 /\d\d:\d\d:\d\d/; # matches a hh:mm:ss time format
407 /[\d\s]/; # matches any digit or whitespace character
408 /\w\W\w/; # matches a word char, followed by a
409 # non-word char, followed by a word char
410 /..rt/; # matches any two chars, followed by 'rt'
411 /end\./; # matches 'end.'
412 /end[.]/; # same thing, matches 'end.'
413 Because a period is a metacharacter, it needs to be escaped to match as
415 an ordinary period. Because, for example, "\d" and "\w" are sets of
416 characters, it is incorrect to think of "[^\d\w]" as "[\D\W]"; in fact
417 "[^\d\w]" is the same as "[^\w]", which is the same as "[\W]". Think
418 DeMorgan's laws.
419 An anchor useful in basic regexps is the word anchor "\b". This
421 matches a boundary between a word character and a non-word character
422 "\w\W" or "\W\w":
423 $x = "Housecat catenates house and cat";
425 $x =~ /cat/; # matches cat in 'housecat'
426 $x =~ /\bcat/; # matches cat in 'catenates'
427 $x =~ /cat\b/; # matches cat in 'housecat'
428 $x =~ /\bcat\b/; # matches 'cat' at end of string
429 Note in the last example, the end of the string is considered a word
431 boundary.
432 You might wonder why '.' matches everything but "\n" - why not every
434 character? The reason is that often one is matching against lines and
435 would like to ignore the newline characters. For instance, while the
436 string "\n" represents one line, we would like to think of it as empty.
437 Then
438 "" =~ /^$/; # matches
440 "\n" =~ /^$/; # matches, $ anchors before "\n"
441 "" =~ /./; # doesn't match; it needs a char
443 "" =~ /^.$/; # doesn't match; it needs a char
444 "\n" =~ /^.$/; # doesn't match; it needs a char other than "\n"
445 "a" =~ /^.$/; # matches
446 "a\n" =~ /^.$/; # matches, $ anchors before "\n"
447 This behavior is convenient, because we usually want to ignore newlines
449 when we count and match characters in a line. Sometimes, however, we
450 want to keep track of newlines. We might even want "^" and "$" to
451 anchor at the beginning and end of lines within the string, rather than
452 just the beginning and end of the string. Perl allows us to choose
453 between ignoring and paying attention to newlines by using the "//s"
454 and "//m" modifiers. "//s" and "//m" stand for single line and multi-
455 line and they determine whether a string is to be treated as one
456 continuous string, or as a set of lines. The two modifiers affect two
457 aspects of how the regexp is interpreted: 1) how the '.' character
458 class is defined, and 2) where the anchors "^" and "$" are able to
459 match. Here are the four possible combinations:
460 · no modifiers (//): Default behavior. '.' matches any character
462 except "\n". "^" matches only at the beginning of the string and
463 "$" matches only at the end or before a newline at the end.
464 · s modifier (//s): Treat string as a single long line. '.' matches
466 any character, even "\n". "^" matches only at the beginning of the
467 string and "$" matches only at the end or before a newline at the
468 end.
469 · m modifier (//m): Treat string as a set of multiple lines. '.'
471 matches any character except "\n". "^" and "$" are able to match
472 at the start or end of any line within the string.
473 · both s and m modifiers (//sm): Treat string as a single long line,
475 but detect multiple lines. '.' matches any character, even "\n".
476 "^" and "$", however, are able to match at the start or end of any
477 line within the string.
478 Here are examples of "//s" and "//m" in action:
480 $x = "There once was a girl\nWho programmed in Perl\n";
482 $x =~ /^Who/; # doesn't match, "Who" not at start of string
484 $x =~ /^Who/s; # doesn't match, "Who" not at start of string
485 $x =~ /^Who/m; # matches, "Who" at start of second line
486 $x =~ /^Who/sm; # matches, "Who" at start of second line
487 $x =~ /girl.Who/; # doesn't match, "." doesn't match "\n"
489 $x =~ /girl.Who/s; # matches, "." matches "\n"
490 $x =~ /girl.Who/m; # doesn't match, "." doesn't match "\n"
491 $x =~ /girl.Who/sm; # matches, "." matches "\n"
492 Most of the time, the default behavior is what is wanted, but "//s" and
494 "//m" are occasionally very useful. If "//m" is being used, the start
495 of the string can still be matched with "\A" and the end of the string
496 can still be matched with the anchors "\Z" (matches both the end and
497 the newline before, like "$"), and "\z" (matches only the end):
498 $x =~ /^Who/m; # matches, "Who" at start of second line
500 $x =~ /\AWho/m; # doesn't match, "Who" is not at start of string
501 $x =~ /girl$/m; # matches, "girl" at end of first line
503 $x =~ /girl\Z/m; # doesn't match, "girl" is not at end of string
504 $x =~ /Perl\Z/m; # matches, "Perl" is at newline before end
506 $x =~ /Perl\z/m; # doesn't match, "Perl" is not at end of string
507 We now know how to create choices among classes of characters in a
509 regexp. What about choices among words or character strings? Such
510 choices are described in the next section.
511 Matching this or that
513 Sometimes we would like our regexp to be able to match different
514 possible words or character strings. This is accomplished by using the
515 alternation metacharacter "|". To match "dog" or "cat", we form the
516 regexp "dog|cat". As before, Perl will try to match the regexp at the
517 earliest possible point in the string. At each character position,
518 Perl will first try to match the first alternative, "dog". If "dog"
519 doesn't match, Perl will then try the next alternative, "cat". If
520 "cat" doesn't match either, then the match fails and Perl moves to the
521 next position in the string. Some examples:
522 "cats and dogs" =~ /cat|dog|bird/; # matches "cat"
524 "cats and dogs" =~ /dog|cat|bird/; # matches "cat"
525 Even though "dog" is the first alternative in the second regexp, "cat"
527 is able to match earlier in the string.
528 "cats" =~ /c|ca|cat|cats/; # matches "c"
530 "cats" =~ /cats|cat|ca|c/; # matches "cats"
531 Here, all the alternatives match at the first string position, so the
533 first alternative is the one that matches. If some of the alternatives
534 are truncations of the others, put the longest ones first to give them
535 a chance to match.
536 "cab" =~ /a|b|c/ # matches "c"
538 # /a|b|c/ == /[abc]/
539 The last example points out that character classes are like
541 alternations of characters. At a given character position, the first
542 alternative that allows the regexp match to succeed will be the one
543 that matches.
544 Grouping things and hierarchical matching
546 Alternation allows a regexp to choose among alternatives, but by itself
547 it is unsatisfying. The reason is that each alternative is a whole
548 regexp, but sometime we want alternatives for just part of a regexp.
549 For instance, suppose we want to search for housecats or housekeepers.
550 The regexp "housecat|housekeeper" fits the bill, but is inefficient
551 because we had to type "house" twice. It would be nice to have parts
552 of the regexp be constant, like "house", and some parts have
553 alternatives, like "cat|keeper".
554 The grouping metacharacters "()" solve this problem. Grouping allows
556 parts of a regexp to be treated as a single unit. Parts of a regexp
557 are grouped by enclosing them in parentheses. Thus we could solve the
558 "housecat|housekeeper" by forming the regexp as "house(cat|keeper)".
559 The regexp "house(cat|keeper)" means match "house" followed by either
560 "cat" or "keeper". Some more examples are
561 /(a|b)b/; # matches 'ab' or 'bb'
563 /(ac|b)b/; # matches 'acb' or 'bb'
564 /(^a|b)c/; # matches 'ac' at start of string or 'bc' anywhere
565 /(a|[bc])d/; # matches 'ad', 'bd', or 'cd'
566 /house(cat|)/; # matches either 'housecat' or 'house'
568 /house(cat(s|)|)/; # matches either 'housecats' or 'housecat' or
569 # 'house'. Note groups can be nested.
570 /(19|20|)\d\d/; # match years 19xx, 20xx, or the Y2K problem, xx
572 "20" =~ /(19|20|)\d\d/; # matches the null alternative '()\d\d',
573 # because '20\d\d' can't match
574 Alternations behave the same way in groups as out of them: at a given
576 string position, the leftmost alternative that allows the regexp to
577 match is taken. So in the last example at the first string position,
578 "20" matches the second alternative, but there is nothing left over to
579 match the next two digits "\d\d". So Perl moves on to the next
580 alternative, which is the null alternative and that works, since "20"
581 is two digits.
582 The process of trying one alternative, seeing if it matches, and moving
584 on to the next alternative, while going back in the string from where
585 the previous alternative was tried, if it doesn't, is called
586 backtracking. The term 'backtracking' comes from the idea that
587 matching a regexp is like a walk in the woods. Successfully matching a
588 regexp is like arriving at a destination. There are many possible
589 trailheads, one for each string position, and each one is tried in
590 order, left to right. From each trailhead there may be many paths,
591 some of which get you there, and some which are dead ends. When you
592 walk along a trail and hit a dead end, you have to backtrack along the
593 trail to an earlier point to try another trail. If you hit your
594 destination, you stop immediately and forget about trying all the other
595 trails. You are persistent, and only if you have tried all the trails
596 from all the trailheads and not arrived at your destination, do you
597 declare failure. To be concrete, here is a step-by-step analysis of
598 what Perl does when it tries to match the regexp
599 "abcde" =~ /(abd|abc)(df|d|de)/;
601 0 Start with the first letter in the string 'a'.
603 1 Try the first alternative in the first group 'abd'.
605 2 Match 'a' followed by 'b'. So far so good.
607 3 'd' in the regexp doesn't match 'c' in the string - a dead end. So
609 backtrack two characters and pick the second alternative in the
610 first group 'abc'.
611 4 Match 'a' followed by 'b' followed by 'c'. We are on a roll and
613 have satisfied the first group. Set $1 to 'abc'.
614 5 Move on to the second group and pick the first alternative 'df'.
616 6 Match the 'd'.
618 7 'f' in the regexp doesn't match 'e' in the string, so a dead end.
620 Backtrack one character and pick the second alternative in the
621 second group 'd'.
622 8 'd' matches. The second grouping is satisfied, so set $2 to 'd'.
624 9 We are at the end of the regexp, so we are done! We have matched
626 'abcd' out of the string "abcde".
627 There are a couple of things to note about this analysis. First, the
629 third alternative in the second group 'de' also allows a match, but we
630 stopped before we got to it - at a given character position, leftmost
631 wins. Second, we were able to get a match at the first character
632 position of the string 'a'. If there were no matches at the first
633 position, Perl would move to the second character position 'b' and
634 attempt the match all over again. Only when all possible paths at all
635 possible character positions have been exhausted does Perl give up and
636 declare "$string =~ /(abd|abc)(df|d|de)/;" to be false.
637 Even with all this work, regexp matching happens remarkably fast. To
639 speed things up, Perl compiles the regexp into a compact sequence of
640 opcodes that can often fit inside a processor cache. When the code is
641 executed, these opcodes can then run at full throttle and search very
642 quickly.
643 Extracting matches
645 The grouping metacharacters "()" also serve another completely
646 different function: they allow the extraction of the parts of a string
647 that matched. This is very useful to find out what matched and for
648 text processing in general. For each grouping, the part that matched
649 inside goes into the special variables $1, $2, etc. They can be used
650 just as ordinary variables:
651 # extract hours, minutes, seconds
653 if ($time =~ /(\d\d):(\d\d):(\d\d)/) { # match hh:mm:ss format
654 $hours = $1;
655 $minutes = $2;
656 $seconds = $3;
657 }
658 Now, we know that in scalar context, "$time =~ /(\d\d):(\d\d):(\d\d)/"
660 returns a true or false value. In list context, however, it returns
661 the list of matched values "($1,$2,$3)". So we could write the code
662 more compactly as
663 # extract hours, minutes, seconds
665 ($hours, $minutes, $second) = ($time =~ /(\d\d):(\d\d):(\d\d)/);
666 If the groupings in a regexp are nested, $1 gets the group with the
668 leftmost opening parenthesis, $2 the next opening parenthesis, etc.
669 Here is a regexp with nested groups:
670 /(ab(cd|ef)((gi)|j))/;
672 1 2 34
673 If this regexp matches, $1 contains a string starting with 'ab', $2 is
675 either set to 'cd' or 'ef', $3 equals either 'gi' or 'j', and $4 is
676 either set to 'gi', just like $3, or it remains undefined.
677 For convenience, Perl sets $+ to the string held by the highest
679 numbered $1, $2,... that got assigned (and, somewhat related, $^N to
680 the value of the $1, $2,... most-recently assigned; i.e. the $1, $2,...
681 associated with the rightmost closing parenthesis used in the match).
682 Backreferences
684 Closely associated with the matching variables $1, $2, ... are the
685 backreferences "\g1", "\g2",... Backreferences are simply matching
686 variables that can be used inside a regexp. This is a really nice
687 feature; what matches later in a regexp is made to depend on what
688 matched earlier in the regexp. Suppose we wanted to look for doubled
689 words in a text, like 'the the'. The following regexp finds all
690 3-letter doubles with a space in between:
691 /\b(\w\w\w)\s\g1\b/;
693 The grouping assigns a value to \g1, so that the same 3-letter sequence
695 is used for both parts.
696 A similar task is to find words consisting of two identical parts:
698 % simple_grep '^(\w\w\w\w|\w\w\w|\w\w|\w)\g1$' /usr/dict/words
700 beriberi
701 booboo
702 coco
703 mama
704 murmur
705 papa
706 The regexp has a single grouping which considers 4-letter combinations,
708 then 3-letter combinations, etc., and uses "\g1" to look for a repeat.
709 Although $1 and "\g1" represent the same thing, care should be taken to
710 use matched variables $1, $2,... only outside a regexp and
711 backreferences "\g1", "\g2",... only inside a regexp; not doing so may
712 lead to surprising and unsatisfactory results.
713 Relative backreferences
715 Counting the opening parentheses to get the correct number for a
716 backreference is error-prone as soon as there is more than one
717 capturing group. A more convenient technique became available with
718 Perl 5.10: relative backreferences. To refer to the immediately
719 preceding capture group one now may write "\g{-1}", the next but last
720 is available via "\g{-2}", and so on.
721 Another good reason in addition to readability and maintainability for
723 using relative backreferences is illustrated by the following example,
724 where a simple pattern for matching peculiar strings is used:
725 $a99a = '([a-z])(\d)\g2\g1'; # matches a11a, g22g, x33x, etc.
727 Now that we have this pattern stored as a handy string, we might feel
729 tempted to use it as a part of some other pattern:
730 $line = "code=e99e";
732 if ($line =~ /^(\w+)=$a99a$/){ # unexpected behavior!
733 print "$1 is valid\n";
734 } else {
735 print "bad line: '$line'\n";
736 }
737 But this doesn't match, at least not the way one might expect. Only
739 after inserting the interpolated $a99a and looking at the resulting
740 full text of the regexp is it obvious that the backreferences have
741 backfired. The subexpression "(\w+)" has snatched number 1 and demoted
742 the groups in $a99a by one rank. This can be avoided by using relative
743 backreferences:
744 $a99a = '([a-z])(\d)\g{-1}\g{-2}'; # safe for being interpolated
746 Named backreferences
748 Perl 5.10 also introduced named capture groups and named
749 backreferences. To attach a name to a capturing group, you write
750 either "(?<name>...)" or "(?'name'...)". The backreference may then be
751 written as "\g{name}". It is permissible to attach the same name to
752 more than one group, but then only the leftmost one of the eponymous
753 set can be referenced. Outside of the pattern a named capture group is
754 accessible through the "%+" hash.
755 Assuming that we have to match calendar dates which may be given in one
757 of the three formats yyyy-mm-dd, mm/dd/yyyy or dd.mm.yyyy, we can write
758 three suitable patterns where we use 'd', 'm' and 'y' respectively as
759 the names of the groups capturing the pertaining components of a date.
760 The matching operation combines the three patterns as alternatives:
761 $fmt1 = '(?<y>\d\d\d\d)-(?<m>\d\d)-(?<d>\d\d)';
763 $fmt2 = '(?<m>\d\d)/(?<d>\d\d)/(?<y>\d\d\d\d)';
764 $fmt3 = '(?<d>\d\d)\.(?<m>\d\d)\.(?<y>\d\d\d\d)';
765 for my $d qw( 2006-10-21 15.01.2007 10/31/2005 ){
766 if ( $d =~ m{$fmt1|$fmt2|$fmt3} ){
767 print "day=$+{d} month=$+{m} year=$+{y}\n";
768 }
769 }
770 If any of the alternatives matches, the hash "%+" is bound to contain
772 the three key-value pairs.
773 Alternative capture group numbering
775 Yet another capturing group numbering technique (also as from Perl
776 5.10) deals with the problem of referring to groups within a set of
777 alternatives. Consider a pattern for matching a time of the day, civil
778 or military style:
779 if ( $time =~ /(\d\d|\d):(\d\d)|(\d\d)(\d\d)/ ){
781 # process hour and minute
782 }
783 Processing the results requires an additional if statement to determine
785 whether $1 and $2 or $3 and $4 contain the goodies. It would be easier
786 if we could use group numbers 1 and 2 in second alternative as well,
787 and this is exactly what the parenthesized construct "(?|...)", set
788 around an alternative achieves. Here is an extended version of the
789 previous pattern:
790 if ( $time =~ /(?|(\d\d|\d):(\d\d)|(\d\d)(\d\d))\s+([A-Z][A-Z][A-Z])/ ){
792 print "hour=$1 minute=$2 zone=$3\n";
793 }
794 Within the alternative numbering group, group numbers start at the same
796 position for each alternative. After the group, numbering continues
797 with one higher than the maximum reached across all the alternatives.
798 Position information
800 In addition to what was matched, Perl (since 5.6.0) also provides the
801 positions of what was matched as contents of the "@-" and "@+" arrays.
802 "$-[0]" is the position of the start of the entire match and $+[0] is
803 the position of the end. Similarly, "$-[n]" is the position of the
804 start of the $n match and $+[n] is the position of the end. If $n is
805 undefined, so are "$-[n]" and $+[n]. Then this code
806 $x = "Mmm...donut, thought Homer";
808 $x =~ /^(Mmm|Yech)\.\.\.(donut|peas)/; # matches
809 foreach $expr (1..$#-) {
810 print "Match $expr: '${$expr}' at position ($-[$expr],$+[$expr])\n";
811 }
812 prints
814 Match 1: 'Mmm' at position (0,3)
816 Match 2: 'donut' at position (6,11)
817 Even if there are no groupings in a regexp, it is still possible to
819 find out what exactly matched in a string. If you use them, Perl will
820 set "$`" to the part of the string before the match, will set $& to the
821 part of the string that matched, and will set "$'" to the part of the
822 string after the match. An example:
823 $x = "the cat caught the mouse";
825 $x =~ /cat/; # $` = 'the ', $& = 'cat', $' = ' caught the mouse'
826 $x =~ /the/; # $` = '', $& = 'the', $' = ' cat caught the mouse'
827 In the second match, "$`" equals '' because the regexp matched at the
829 first character position in the string and stopped; it never saw the
830 second 'the'. It is important to note that using "$`" and "$'" slows
831 down regexp matching quite a bit, while $& slows it down to a lesser
832 extent, because if they are used in one regexp in a program, they are
833 generated for all regexps in the program. So if raw performance is a
834 goal of your application, they should be avoided. If you need to
835 extract the corresponding substrings, use "@-" and "@+" instead:
836 $` is the same as substr( $x, 0, $-[0] )
838 $& is the same as substr( $x, $-[0], $+[0]-$-[0] )
839 $' is the same as substr( $x, $+[0] )
840 As of Perl 5.10, the "${^PREMATCH}", "${^MATCH}" and "${^POSTMATCH}"
842 variables may be used. These are only set if the "/p" modifier is
843 present. Consequently they do not penalize the rest of the program.
844 Non-capturing groupings
846 A group that is required to bundle a set of alternatives may or may not
847 be useful as a capturing group. If it isn't, it just creates a
848 superfluous addition to the set of available capture group values,
849 inside as well as outside the regexp. Non-capturing groupings, denoted
850 by "(?:regexp)", still allow the regexp to be treated as a single unit,
851 but don't establish a capturing group at the same time. Both capturing
852 and non-capturing groupings are allowed to co-exist in the same regexp.
853 Because there is no extraction, non-capturing groupings are faster than
854 capturing groupings. Non-capturing groupings are also handy for
855 choosing exactly which parts of a regexp are to be extracted to
856 matching variables:
857 # match a number, $1-$4 are set, but we only want $1
859 /([+-]?\ *(\d+(\.\d*)?|\.\d+)([eE][+-]?\d+)?)/;
860 # match a number faster , only $1 is set
862 /([+-]?\ *(?:\d+(?:\.\d*)?|\.\d+)(?:[eE][+-]?\d+)?)/;
863 # match a number, get $1 = whole number, $2 = exponent
865 /([+-]?\ *(?:\d+(?:\.\d*)?|\.\d+)(?:[eE]([+-]?\d+))?)/;
866 Non-capturing groupings are also useful for removing nuisance elements
868 gathered from a split operation where parentheses are required for some
869 reason:
870 $x = '12aba34ba5';
872 @num = split /(a|b)+/, $x; # @num = ('12','a','34','a','5')
873 @num = split /(?:a|b)+/, $x; # @num = ('12','34','5')
874 Matching repetitions
876 The examples in the previous section display an annoying weakness. We
877 were only matching 3-letter words, or chunks of words of 4 letters or
878 less. We'd like to be able to match words or, more generally, strings
879 of any length, without writing out tedious alternatives like
880 "\w\w\w\w|\w\w\w|\w\w|\w".
881 This is exactly the problem the quantifier metacharacters "?", "*",
883 "+", and "{}" were created for. They allow us to delimit the number of
884 repeats for a portion of a regexp we consider to be a match.
885 Quantifiers are put immediately after the character, character class,
886 or grouping that we want to specify. They have the following meanings:
887 · "a?" means: match 'a' 1 or 0 times
889 · "a*" means: match 'a' 0 or more times, i.e., any number of times
891 · "a+" means: match 'a' 1 or more times, i.e., at least once
893 · "a{n,m}" means: match at least "n" times, but not more than "m"
895 times.
896 · "a{n,}" means: match at least "n" or more times
898 · "a{n}" means: match exactly "n" times
900 Here are some examples:
902 /[a-z]+\s+\d*/; # match a lowercase word, at least one space, and
904 # any number of digits
905 /(\w+)\s+\g1/; # match doubled words of arbitrary length
906 /y(es)?/i; # matches 'y', 'Y', or a case-insensitive 'yes'
907 $year =~ /^\d{2,4}$/; # make sure year is at least 2 but not more
908 # than 4 digits
909 $year =~ /^\d{4}$|^\d{2}$/; # better match; throw out 3-digit dates
910 $year =~ /^\d{2}(\d{2})?$/; # same thing written differently. However,
911 # this captures the last two digits in $1
912 # and the other does not.
913 % simple_grep '^(\w+)\g1$' /usr/dict/words # isn't this easier?
915 beriberi
916 booboo
917 coco
918 mama
919 murmur
920 papa
921 For all of these quantifiers, Perl will try to match as much of the
923 string as possible, while still allowing the regexp to succeed. Thus
924 with "/a?.../", Perl will first try to match the regexp with the "a"
925 present; if that fails, Perl will try to match the regexp without the
926 "a" present. For the quantifier "*", we get the following:
927 $x = "the cat in the hat";
929 $x =~ /^(.*)(cat)(.*)$/; # matches,
930 # $1 = 'the '
931 # $2 = 'cat'
932 # $3 = ' in the hat'
933 Which is what we might expect, the match finds the only "cat" in the
935 string and locks onto it. Consider, however, this regexp:
936 $x =~ /^(.*)(at)(.*)$/; # matches,
938 # $1 = 'the cat in the h'
939 # $2 = 'at'
940 # $3 = '' (0 characters match)
941 One might initially guess that Perl would find the "at" in "cat" and
943 stop there, but that wouldn't give the longest possible string to the
944 first quantifier ".*". Instead, the first quantifier ".*" grabs as
945 much of the string as possible while still having the regexp match. In
946 this example, that means having the "at" sequence with the final "at"
947 in the string. The other important principle illustrated here is that,
948 when there are two or more elements in a regexp, the leftmost
949 quantifier, if there is one, gets to grab as much of the string as
950 possible, leaving the rest of the regexp to fight over scraps. Thus in
951 our example, the first quantifier ".*" grabs most of the string, while
952 the second quantifier ".*" gets the empty string. Quantifiers that
953 grab as much of the string as possible are called maximal match or
954 greedy quantifiers.
955 When a regexp can match a string in several different ways, we can use
957 the principles above to predict which way the regexp will match:
958 · Principle 0: Taken as a whole, any regexp will be matched at the
960 earliest possible position in the string.
961 · Principle 1: In an alternation "a|b|c...", the leftmost alternative
963 that allows a match for the whole regexp will be the one used.
964 · Principle 2: The maximal matching quantifiers "?", "*", "+" and
966 "{n,m}" will in general match as much of the string as possible
967 while still allowing the whole regexp to match.
968 · Principle 3: If there are two or more elements in a regexp, the
970 leftmost greedy quantifier, if any, will match as much of the
971 string as possible while still allowing the whole regexp to match.
972 The next leftmost greedy quantifier, if any, will try to match as
973 much of the string remaining available to it as possible, while
974 still allowing the whole regexp to match. And so on, until all the
975 regexp elements are satisfied.
976 As we have seen above, Principle 0 overrides the others. The regexp
978 will be matched as early as possible, with the other principles
979 determining how the regexp matches at that earliest character position.
980 Here is an example of these principles in action:
982 $x = "The programming republic of Perl";
984 $x =~ /^(.+)(e|r)(.*)$/; # matches,
985 # $1 = 'The programming republic of Pe'
986 # $2 = 'r'
987 # $3 = 'l'
988 This regexp matches at the earliest string position, 'T'. One might
990 think that "e", being leftmost in the alternation, would be matched,
991 but "r" produces the longest string in the first quantifier.
992 $x =~ /(m{1,2})(.*)$/; # matches,
994 # $1 = 'mm'
995 # $2 = 'ing republic of Perl'
996 Here, The earliest possible match is at the first 'm' in "programming".
998 "m{1,2}" is the first quantifier, so it gets to match a maximal "mm".
999 $x =~ /.*(m{1,2})(.*)$/; # matches,
1001 # $1 = 'm'
1002 # $2 = 'ing republic of Perl'
1003 Here, the regexp matches at the start of the string. The first
1005 quantifier ".*" grabs as much as possible, leaving just a single 'm'
1006 for the second quantifier "m{1,2}".
1007 $x =~ /(.?)(m{1,2})(.*)$/; # matches,
1009 # $1 = 'a'
1010 # $2 = 'mm'
1011 # $3 = 'ing republic of Perl'
1012 Here, ".?" eats its maximal one character at the earliest possible
1014 position in the string, 'a' in "programming", leaving "m{1,2}" the
1015 opportunity to match both "m"'s. Finally,
1016 "aXXXb" =~ /(X*)/; # matches with $1 = ''
1018 because it can match zero copies of 'X' at the beginning of the string.
1020 If you definitely want to match at least one 'X', use "X+", not "X*".
1021 Sometimes greed is not good. At times, we would like quantifiers to
1023 match a minimal piece of string, rather than a maximal piece. For this
1024 purpose, Larry Wall created the minimal match or non-greedy quantifiers
1025 "??", "*?", "+?", and "{}?". These are the usual quantifiers with a
1026 "?" appended to them. They have the following meanings:
1027 · "a??" means: match 'a' 0 or 1 times. Try 0 first, then 1.
1029 · "a*?" means: match 'a' 0 or more times, i.e., any number of times,
1031 but as few times as possible
1032 · "a+?" means: match 'a' 1 or more times, i.e., at least once, but as
1034 few times as possible
1035 · "a{n,m}?" means: match at least "n" times, not more than "m" times,
1037 as few times as possible
1038 · "a{n,}?" means: match at least "n" times, but as few times as
1040 possible
1041 · "a{n}?" means: match exactly "n" times. Because we match exactly
1043 "n" times, "a{n}?" is equivalent to "a{n}" and is just there for
1044 notational consistency.
1045 Let's look at the example above, but with minimal quantifiers:
1047 $x = "The programming republic of Perl";
1049 $x =~ /^(.+?)(e|r)(.*)$/; # matches,
1050 # $1 = 'Th'
1051 # $2 = 'e'
1052 # $3 = ' programming republic of Perl'
1053 The minimal string that will allow both the start of the string "^" and
1055 the alternation to match is "Th", with the alternation "e|r" matching
1056 "e". The second quantifier ".*" is free to gobble up the rest of the
1057 string.
1058 $x =~ /(m{1,2}?)(.*?)$/; # matches,
1060 # $1 = 'm'
1061 # $2 = 'ming republic of Perl'
1062 The first string position that this regexp can match is at the first
1064 'm' in "programming". At this position, the minimal "m{1,2}?" matches
1065 just one 'm'. Although the second quantifier ".*?" would prefer to
1066 match no characters, it is constrained by the end-of-string anchor "$"
1067 to match the rest of the string.
1068 $x =~ /(.*?)(m{1,2}?)(.*)$/; # matches,
1070 # $1 = 'The progra'
1071 # $2 = 'm'
1072 # $3 = 'ming republic of Perl'
1073 In this regexp, you might expect the first minimal quantifier ".*?" to
1075 match the empty string, because it is not constrained by a "^" anchor
1076 to match the beginning of the word. Principle 0 applies here, however.
1077 Because it is possible for the whole regexp to match at the start of
1078 the string, it will match at the start of the string. Thus the first
1079 quantifier has to match everything up to the first "m". The second
1080 minimal quantifier matches just one "m" and the third quantifier
1081 matches the rest of the string.
1082 $x =~ /(.??)(m{1,2})(.*)$/; # matches,
1084 # $1 = 'a'
1085 # $2 = 'mm'
1086 # $3 = 'ing republic of Perl'
1087 Just as in the previous regexp, the first quantifier ".??" can match
1089 earliest at position 'a', so it does. The second quantifier is greedy,
1090 so it matches "mm", and the third matches the rest of the string.
1091 We can modify principle 3 above to take into account non-greedy
1093 quantifiers:
1094 · Principle 3: If there are two or more elements in a regexp, the
1096 leftmost greedy (non-greedy) quantifier, if any, will match as much
1097 (little) of the string as possible while still allowing the whole
1098 regexp to match. The next leftmost greedy (non-greedy) quantifier,
1099 if any, will try to match as much (little) of the string remaining
1100 available to it as possible, while still allowing the whole regexp
1101 to match. And so on, until all the regexp elements are satisfied.
1102 Just like alternation, quantifiers are also susceptible to
1104 backtracking. Here is a step-by-step analysis of the example
1105 $x = "the cat in the hat";
1107 $x =~ /^(.*)(at)(.*)$/; # matches,
1108 # $1 = 'the cat in the h'
1109 # $2 = 'at'
1110 # $3 = '' (0 matches)
1111 0 Start with the first letter in the string 't'.
1113 1 The first quantifier '.*' starts out by matching the whole string
1115 'the cat in the hat'.
1116 2 'a' in the regexp element 'at' doesn't match the end of the string.
1118 Backtrack one character.
1119 3 'a' in the regexp element 'at' still doesn't match the last letter
1121 of the string 't', so backtrack one more character.
1122 4 Now we can match the 'a' and the 't'.
1124 5 Move on to the third element '.*'. Since we are at the end of the
1126 string and '.*' can match 0 times, assign it the empty string.
1127 6 We are done!
1129 Most of the time, all this moving forward and backtracking happens
1131 quickly and searching is fast. There are some pathological regexps,
1132 however, whose execution time exponentially grows with the size of the
1133 string. A typical structure that blows up in your face is of the form
1134 /(a|b+)*/;
1136 The problem is the nested indeterminate quantifiers. There are many
1138 different ways of partitioning a string of length n between the "+" and
1139 "*": one repetition with "b+" of length n, two repetitions with the
1140 first "b+" length k and the second with length n-k, m repetitions whose
1141 bits add up to length n, etc. In fact there are an exponential number
1142 of ways to partition a string as a function of its length. A regexp
1143 may get lucky and match early in the process, but if there is no match,
1144 Perl will try every possibility before giving up. So be careful with
1145 nested "*"'s, "{n,m}"'s, and "+"'s. The book Mastering Regular
1146 Expressions by Jeffrey Friedl gives a wonderful discussion of this and
1147 other efficiency issues.
1148 Possessive quantifiers
1150 Backtracking during the relentless search for a match may be a waste of
1151 time, particularly when the match is bound to fail. Consider the
1152 simple pattern
1153 /^\w+\s+\w+$/; # a word, spaces, a word
1155 Whenever this is applied to a string which doesn't quite meet the
1157 pattern's expectations such as "abc " or "abc def ", the regex engine
1158 will backtrack, approximately once for each character in the string.
1159 But we know that there is no way around taking all of the initial word
1160 characters to match the first repetition, that all spaces must be eaten
1161 by the middle part, and the same goes for the second word.
1162 With the introduction of the possessive quantifiers in Perl 5.10, we
1164 have a way of instructing the regex engine not to backtrack, with the
1165 usual quantifiers with a "+" appended to them. This makes them greedy
1166 as well as stingy; once they succeed they won't give anything back to
1167 permit another solution. They have the following meanings:
1168 · "a{n,m}+" means: match at least "n" times, not more than "m" times,
1170 as many times as possible, and don't give anything up. "a?+" is
1171 short for "a{0,1}+"
1172 · "a{n,}+" means: match at least "n" times, but as many times as
1174 possible, and don't give anything up. "a*+" is short for "a{0,}+"
1175 and "a++" is short for "a{1,}+".
1176 · "a{n}+" means: match exactly "n" times. It is just there for
1178 notational consistency.
1179 These possessive quantifiers represent a special case of a more general
1181 concept, the independent subexpression, see below.
1182 As an example where a possessive quantifier is suitable we consider
1184 matching a quoted string, as it appears in several programming
1185 languages. The backslash is used as an escape character that indicates
1186 that the next character is to be taken literally, as another character
1187 for the string. Therefore, after the opening quote, we expect a
1188 (possibly empty) sequence of alternatives: either some character except
1189 an unescaped quote or backslash or an escaped character.
1190 /"(?:[^"\\]++|\\.)*+"/;
1192 Building a regexp
1194 At this point, we have all the basic regexp concepts covered, so let's
1195 give a more involved example of a regular expression. We will build a
1196 regexp that matches numbers.
1197 The first task in building a regexp is to decide what we want to match
1199 and what we want to exclude. In our case, we want to match both
1200 integers and floating point numbers and we want to reject any string
1201 that isn't a number.
1202 The next task is to break the problem down into smaller problems that
1204 are easily converted into a regexp.
1205 The simplest case is integers. These consist of a sequence of digits,
1207 with an optional sign in front. The digits we can represent with "\d+"
1208 and the sign can be matched with "[+-]". Thus the integer regexp is
1209 /[+-]?\d+/; # matches integers
1211 A floating point number potentially has a sign, an integral part, a
1213 decimal point, a fractional part, and an exponent. One or more of
1214 these parts is optional, so we need to check out the different
1215 possibilities. Floating point numbers which are in proper form include
1216 123., 0.345, .34, -1e6, and 25.4E-72. As with integers, the sign out
1217 front is completely optional and can be matched by "[+-]?". We can see
1218 that if there is no exponent, floating point numbers must have a
1219 decimal point, otherwise they are integers. We might be tempted to
1220 model these with "\d*\.\d*", but this would also match just a single
1221 decimal point, which is not a number. So the three cases of floating
1222 point number without exponent are
1223 /[+-]?\d+\./; # 1., 321., etc.
1225 /[+-]?\.\d+/; # .1, .234, etc.
1226 /[+-]?\d+\.\d+/; # 1.0, 30.56, etc.
1227 These can be combined into a single regexp with a three-way
1229 alternation:
1230 /[+-]?(\d+\.\d+|\d+\.|\.\d+)/; # floating point, no exponent
1232 In this alternation, it is important to put '\d+\.\d+' before '\d+\.'.
1234 If '\d+\.' were first, the regexp would happily match that and ignore
1235 the fractional part of the number.
1236 Now consider floating point numbers with exponents. The key
1238 observation here is that both integers and numbers with decimal points
1239 are allowed in front of an exponent. Then exponents, like the overall
1240 sign, are independent of whether we are matching numbers with or
1241 without decimal points, and can be 'decoupled' from the mantissa. The
1242 overall form of the regexp now becomes clear:
1243 /^(optional sign)(integer | f.p. mantissa)(optional exponent)$/;
1245 The exponent is an "e" or "E", followed by an integer. So the exponent
1247 regexp is
1248 /[eE][+-]?\d+/; # exponent
1250 Putting all the parts together, we get a regexp that matches numbers:
1252 /^[+-]?(\d+\.\d+|\d+\.|\.\d+|\d+)([eE][+-]?\d+)?$/; # Ta da!
1254 Long regexps like this may impress your friends, but can be hard to
1256 decipher. In complex situations like this, the "//x" modifier for a
1257 match is invaluable. It allows one to put nearly arbitrary whitespace
1258 and comments into a regexp without affecting their meaning. Using it,
1259 we can rewrite our 'extended' regexp in the more pleasing form
1260 /^
1262 [+-]? # first, match an optional sign
1263 ( # then match integers or f.p. mantissas:
1264 \d+\.\d+ # mantissa of the form a.b
1265 |\d+\. # mantissa of the form a.
1266 |\.\d+ # mantissa of the form .b
1267 |\d+ # integer of the form a
1268 )
1269 ([eE][+-]?\d+)? # finally, optionally match an exponent
1270 $/x;
1271 If whitespace is mostly irrelevant, how does one include space
1273 characters in an extended regexp? The answer is to backslash it '\ ' or
1274 put it in a character class "[ ]". The same thing goes for pound
1275 signs: use "\#" or "[#]". For instance, Perl allows a space between
1276 the sign and the mantissa or integer, and we could add this to our
1277 regexp as follows:
1278 /^
1280 [+-]?\ * # first, match an optional sign *and space*
1281 ( # then match integers or f.p. mantissas:
1282 \d+\.\d+ # mantissa of the form a.b
1283 |\d+\. # mantissa of the form a.
1284 |\.\d+ # mantissa of the form .b
1285 |\d+ # integer of the form a
1286 )
1287 ([eE][+-]?\d+)? # finally, optionally match an exponent
1288 $/x;
1289 In this form, it is easier to see a way to simplify the alternation.
1291 Alternatives 1, 2, and 4 all start with "\d+", so it could be factored
1292 out:
1293 /^
1295 [+-]?\ * # first, match an optional sign
1296 ( # then match integers or f.p. mantissas:
1297 \d+ # start out with a ...
1298 (
1299 \.\d* # mantissa of the form a.b or a.
1300 )? # ? takes care of integers of the form a
1301 |\.\d+ # mantissa of the form .b
1302 )
1303 ([eE][+-]?\d+)? # finally, optionally match an exponent
1304 $/x;
1305 or written in the compact form,
1307 /^[+-]?\ *(\d+(\.\d*)?|\.\d+)([eE][+-]?\d+)?$/;
1309 This is our final regexp. To recap, we built a regexp by
1311 · specifying the task in detail,
1313 · breaking down the problem into smaller parts,
1315 · translating the small parts into regexps,
1317 · combining the regexps,
1319 · and optimizing the final combined regexp.
1321 These are also the typical steps involved in writing a computer
1323 program. This makes perfect sense, because regular expressions are
1324 essentially programs written in a little computer language that
1325 specifies patterns.
1326 Using regular expressions in Perl
1328 The last topic of Part 1 briefly covers how regexps are used in Perl
1329 programs. Where do they fit into Perl syntax?
1330 We have already introduced the matching operator in its default
1332 "/regexp/" and arbitrary delimiter "m!regexp!" forms. We have used the
1333 binding operator "=~" and its negation "!~" to test for string matches.
1334 Associated with the matching operator, we have discussed the single
1335 line "//s", multi-line "//m", case-insensitive "//i" and extended "//x"
1336 modifiers. There are a few more things you might want to know about
1337 matching operators.
1338 Prohibiting substitution
1340 If you change $pattern after the first substitution happens, Perl will
1342 ignore it. If you don't want any substitutions at all, use the special
1343 delimiter "m''":
1344 @pattern = ('Seuss');
1346 while (<>) {
1347 print if m'@pattern'; # matches literal '@pattern', not 'Seuss'
1348 }
1349 Similar to strings, "m''" acts like apostrophes on a regexp; all other
1351 "m" delimiters act like quotes. If the regexp evaluates to the empty
1352 string, the regexp in the last successful match is used instead. So we
1353 have
1354 "dog" =~ /d/; # 'd' matches
1356 "dogbert =~ //; # this matches the 'd' regexp used before
1357 Global matching
1359 The final two modifiers we will discuss here, "//g" and "//c", concern
1361 multiple matches. The modifier "//g" stands for global matching and
1362 allows the matching operator to match within a string as many times as
1363 possible. In scalar context, successive invocations against a string
1364 will have "//g" jump from match to match, keeping track of position in
1365 the string as it goes along. You can get or set the position with the
1366 "pos()" function.
1367 The use of "//g" is shown in the following example. Suppose we have a
1369 string that consists of words separated by spaces. If we know how many
1370 words there are in advance, we could extract the words using groupings:
1371 $x = "cat dog house"; # 3 words
1373 $x =~ /^\s*(\w+)\s+(\w+)\s+(\w+)\s*$/; # matches,
1374 # $1 = 'cat'
1375 # $2 = 'dog'
1376 # $3 = 'house'
1377 But what if we had an indeterminate number of words? This is the sort
1379 of task "//g" was made for. To extract all words, form the simple
1380 regexp "(\w+)" and loop over all matches with "/(\w+)/g":
1381 while ($x =~ /(\w+)/g) {
1383 print "Word is $1, ends at position ", pos $x, "\n";
1384 }
1385 prints
1387 Word is cat, ends at position 3
1389 Word is dog, ends at position 7
1390 Word is house, ends at position 13
1391 A failed match or changing the target string resets the position. If
1393 you don't want the position reset after failure to match, add the
1394 "//c", as in "/regexp/gc". The current position in the string is
1395 associated with the string, not the regexp. This means that different
1396 strings have different positions and their respective positions can be
1397 set or read independently.
1398 In list context, "//g" returns a list of matched groupings, or if there
1400 are no groupings, a list of matches to the whole regexp. So if we
1401 wanted just the words, we could use
1402 @words = ($x =~ /(\w+)/g); # matches,
1404 # $words[0] = 'cat'
1405 # $words[1] = 'dog'
1406 # $words[2] = 'house'
1407 Closely associated with the "//g" modifier is the "\G" anchor. The
1409 "\G" anchor matches at the point where the previous "//g" match left
1410 off. "\G" allows us to easily do context-sensitive matching:
1411 $metric = 1; # use metric units
1413 ...
1414 $x = <FILE>; # read in measurement
1415 $x =~ /^([+-]?\d+)\s*/g; # get magnitude
1416 $weight = $1;
1417 if ($metric) { # error checking
1418 print "Units error!" unless $x =~ /\Gkg\./g;
1419 }
1420 else {
1421 print "Units error!" unless $x =~ /\Glbs\./g;
1422 }
1423 $x =~ /\G\s+(widget|sprocket)/g; # continue processing
1424 The combination of "//g" and "\G" allows us to process the string a bit
1426 at a time and use arbitrary Perl logic to decide what to do next.
1427 Currently, the "\G" anchor is only fully supported when used to anchor
1428 to the start of the pattern.
1429 "\G" is also invaluable in processing fixed-length records with
1431 regexps. Suppose we have a snippet of coding region DNA, encoded as
1432 base pair letters "ATCGTTGAAT..." and we want to find all the stop
1433 codons "TGA". In a coding region, codons are 3-letter sequences, so we
1434 can think of the DNA snippet as a sequence of 3-letter records. The
1435 naive regexp
1436 # expanded, this is "ATC GTT GAA TGC AAA TGA CAT GAC"
1438 $dna = "ATCGTTGAATGCAAATGACATGAC";
1439 $dna =~ /TGA/;
1440 doesn't work; it may match a "TGA", but there is no guarantee that the
1442 match is aligned with codon boundaries, e.g., the substring "GTT GAA"
1443 gives a match. A better solution is
1444 while ($dna =~ /(\w\w\w)*?TGA/g) { # note the minimal *?
1446 print "Got a TGA stop codon at position ", pos $dna, "\n";
1447 }
1448 which prints
1450 Got a TGA stop codon at position 18
1452 Got a TGA stop codon at position 23
1453 Position 18 is good, but position 23 is bogus. What happened?
1455 The answer is that our regexp works well until we get past the last
1457 real match. Then the regexp will fail to match a synchronized "TGA"
1458 and start stepping ahead one character position at a time, not what we
1459 want. The solution is to use "\G" to anchor the match to the codon
1460 alignment:
1461 while ($dna =~ /\G(\w\w\w)*?TGA/g) {
1463 print "Got a TGA stop codon at position ", pos $dna, "\n";
1464 }
1465 This prints
1467 Got a TGA stop codon at position 18
1469 which is the correct answer. This example illustrates that it is
1471 important not only to match what is desired, but to reject what is not
1472 desired.
1473 (There are other regexp modifiers that are available, such as "//o",
1475 but their specialized uses are beyond the scope of this introduction.
1476 )
1477 Search and replace
1479 Regular expressions also play a big role in search and replace
1481 operations in Perl. Search and replace is accomplished with the "s///"
1482 operator. The general form is "s/regexp/replacement/modifiers", with
1483 everything we know about regexps and modifiers applying in this case as
1484 well. The "replacement" is a Perl double-quoted string that replaces
1485 in the string whatever is matched with the "regexp". The operator "=~"
1486 is also used here to associate a string with "s///". If matching
1487 against $_, the "$_ =~" can be dropped. If there is a match, "s///"
1488 returns the number of substitutions made; otherwise it returns false.
1489 Here are a few examples:
1490 $x = "Time to feed the cat!";
1492 $x =~ s/cat/hacker/; # $x contains "Time to feed the hacker!"
1493 if ($x =~ s/^(Time.*hacker)!$/$1 now!/) {
1494 $more_insistent = 1;
1495 }
1496 $y = "'quoted words'";
1497 $y =~ s/^'(.*)'$/$1/; # strip single quotes,
1498 # $y contains "quoted words"
1499 In the last example, the whole string was matched, but only the part
1501 inside the single quotes was grouped. With the "s///" operator, the
1502 matched variables $1, $2, etc. are immediately available for use in the
1503 replacement expression, so we use $1 to replace the quoted string with
1504 just what was quoted. With the global modifier, "s///g" will search
1505 and replace all occurrences of the regexp in the string:
1506 $x = "I batted 4 for 4";
1508 $x =~ s/4/four/; # doesn't do it all:
1509 # $x contains "I batted four for 4"
1510 $x = "I batted 4 for 4";
1511 $x =~ s/4/four/g; # does it all:
1512 # $x contains "I batted four for four"
1513 If you prefer 'regex' over 'regexp' in this tutorial, you could use the
1515 following program to replace it:
1516 % cat > simple_replace
1518 #!/usr/bin/perl
1519 $regexp = shift;
1520 $replacement = shift;
1521 while (<>) {
1522 s/$regexp/$replacement/g;
1523 print;
1524 }
1525 ^D
1526 % simple_replace regexp regex perlretut.pod
1528 In "simple_replace" we used the "s///g" modifier to replace all
1530 occurrences of the regexp on each line. (Even though the regular
1531 expression appears in a loop, Perl is smart enough to compile it only
1532 once.) As with "simple_grep", both the "print" and the
1533 "s/$regexp/$replacement/g" use $_ implicitly.
1534 If you don't want "s///" to change your original variable you can use
1536 the non-destructive substitute modifier, "s///r". This changes the
1537 behavior so that "s///r" returns the final substituted string (instead
1538 of the number of substitutions):
1539 $x = "I like dogs.";
1541 $y = $x =~ s/dogs/cats/r;
1542 print "$x $y\n";
1543 That example will print "I like dogs. I like cats". Notice the original
1545 $x variable has not been affected. The overall result of the
1546 substitution is instead stored in $y. If the substitution doesn't
1547 affect anything then the original string is returned:
1548 $x = "I like dogs.";
1550 $y = $x =~ s/elephants/cougars/r;
1551 print "$x $y\n"; # prints "I like dogs. I like dogs."
1552 One other interesting thing that the "s///r" flag allows is chaining
1554 substitutions:
1555 $x = "Cats are great.";
1557 print $x =~ s/Cats/Dogs/r =~ s/Dogs/Frogs/r =~ s/Frogs/Hedgehogs/r, "\n";
1558 # prints "Hedgehogs are great."
1559 A modifier available specifically to search and replace is the "s///e"
1561 evaluation modifier. "s///e" treats the replacement text as Perl code,
1562 rather than a double-quoted string. The value that the code returns is
1563 substituted for the matched substring. "s///e" is useful if you need
1564 to do a bit of computation in the process of replacing text. This
1565 example counts character frequencies in a line:
1566 $x = "Bill the cat";
1568 $x =~ s/(.)/$chars{$1}++;$1/eg; # final $1 replaces char with itself
1569 print "frequency of '$_' is $chars{$_}\n"
1570 foreach (sort {$chars{$b} <=> $chars{$a}} keys %chars);
1571 This prints
1573 frequency of ' ' is 2
1575 frequency of 't' is 2
1576 frequency of 'l' is 2
1577 frequency of 'B' is 1
1578 frequency of 'c' is 1
1579 frequency of 'e' is 1
1580 frequency of 'h' is 1
1581 frequency of 'i' is 1
1582 frequency of 'a' is 1
1583 As with the match "m//" operator, "s///" can use other delimiters, such
1585 as "s!!!" and "s{}{}", and even "s{}//". If single quotes are used
1586 "s'''", then the regexp and replacement are treated as single-quoted
1587 strings and there are no variable substitutions. "s///" in list
1588 context returns the same thing as in scalar context, i.e., the number
1589 of matches.
1590 The split function
1592 The "split()" function is another place where a regexp is used. "split
1594 /regexp/, string, limit" separates the "string" operand into a list of
1595 substrings and returns that list. The regexp must be designed to match
1596 whatever constitutes the separators for the desired substrings. The
1597 "limit", if present, constrains splitting into no more than "limit"
1598 number of strings. For example, to split a string into words, use
1599 $x = "Calvin and Hobbes";
1601 @words = split /\s+/, $x; # $word[0] = 'Calvin'
1602 # $word[1] = 'and'
1603 # $word[2] = 'Hobbes'
1604 If the empty regexp "//" is used, the regexp always matches and the
1606 string is split into individual characters. If the regexp has
1607 groupings, then the resulting list contains the matched substrings from
1608 the groupings as well. For instance,
1609 $x = "/usr/bin/perl";
1611 @dirs = split m!/!, $x; # $dirs[0] = ''
1612 # $dirs[1] = 'usr'
1613 # $dirs[2] = 'bin'
1614 # $dirs[3] = 'perl'
1615 @parts = split m!(/)!, $x; # $parts[0] = ''
1616 # $parts[1] = '/'
1617 # $parts[2] = 'usr'
1618 # $parts[3] = '/'
1619 # $parts[4] = 'bin'
1620 # $parts[5] = '/'
1621 # $parts[6] = 'perl'
1622 Since the first character of $x matched the regexp, "split" prepended
1624 an empty initial element to the list.
1625 If you have read this far, congratulations! You now have all the basic
1627 tools needed to use regular expressions to solve a wide range of text
1628 processing problems. If this is your first time through the tutorial,
1629 why not stop here and play around with regexps a while.... Part 2
1630 concerns the more esoteric aspects of regular expressions and those
1631 concepts certainly aren't needed right at the start.
1632
1634 OK, you know the basics of regexps and you want to know more. If
1635 matching regular expressions is analogous to a walk in the woods, then
1636 the tools discussed in Part 1 are analogous to topo maps and a compass,
1637 basic tools we use all the time. Most of the tools in part 2 are
1638 analogous to flare guns and satellite phones. They aren't used too
1639 often on a hike, but when we are stuck, they can be invaluable.
1640 What follows are the more advanced, less used, or sometimes esoteric
1642 capabilities of Perl regexps. In Part 2, we will assume you are
1643 comfortable with the basics and concentrate on the advanced features.
1644 More on characters, strings, and character classes
1646 There are a number of escape sequences and character classes that we
1647 haven't covered yet.
1648 There are several escape sequences that convert characters or strings
1650 between upper and lower case, and they are also available within
1651 patterns. "\l" and "\u" convert the next character to lower or upper
1652 case, respectively:
1653 $x = "perl";
1655 $string =~ /\u$x/; # matches 'Perl' in $string
1656 $x = "M(rs?|s)\\."; # note the double backslash
1657 $string =~ /\l$x/; # matches 'mr.', 'mrs.', and 'ms.',
1658 A "\L" or "\U" indicates a lasting conversion of case, until terminated
1660 by "\E" or thrown over by another "\U" or "\L":
1661 $x = "This word is in lower case:\L SHOUT\E";
1663 $x =~ /shout/; # matches
1664 $x = "I STILL KEYPUNCH CARDS FOR MY 360"
1665 $x =~ /\Ukeypunch/; # matches punch card string
1666 If there is no "\E", case is converted until the end of the string. The
1668 regexps "\L\u$word" or "\u\L$word" convert the first character of $word
1669 to uppercase and the rest of the characters to lowercase.
1670 Control characters can be escaped with "\c", so that a control-Z
1672 character would be matched with "\cZ". The escape sequence "\Q"..."\E"
1673 quotes, or protects most non-alphabetic characters. For instance,
1674 $x = "\QThat !^*&%~& cat!";
1676 $x =~ /\Q!^*&%~&\E/; # check for rough language
1677 It does not protect "$" or "@", so that variables can still be
1679 substituted.
1680 "\Q", "\L", "\l", "\U", "\u" and "\E" are actually part of double-
1682 quotish syntax, and not part of regexp syntax proper. They will work
1683 if they appear in a regular expression embedded directly in a program,
1684 but not when contained in a string that is interpolated in a pattern.
1685 With the advent of 5.6.0, Perl regexps can handle more than just the
1687 standard ASCII character set. Perl now supports Unicode, a standard
1688 for representing the alphabets from virtually all of the world's
1689 written languages, and a host of symbols. Perl's text strings are
1690 Unicode strings, so they can contain characters with a value (codepoint
1691 or character number) higher than 255.
1692 What does this mean for regexps? Well, regexp users don't need to know
1694 much about Perl's internal representation of strings. But they do need
1695 to know 1) how to represent Unicode characters in a regexp and 2) that
1696 a matching operation will treat the string to be searched as a sequence
1697 of characters, not bytes. The answer to 1) is that Unicode characters
1698 greater than "chr(255)" are represented using the "\x{hex}" notation,
1699 because \x hex (without curly braces) doesn't go further than 255.
1700 (Starting in Perl 5.14, if you're an octal fan, you can also use
1701 "\o{oct}".)
1702 /\x{263a}/; # match a Unicode smiley face :)
1704 NOTE: In Perl 5.6.0 it used to be that one needed to say "use utf8" to
1706 use any Unicode features. This is no more the case: for almost all
1707 Unicode processing, the explicit "utf8" pragma is not needed. (The
1708 only case where it matters is if your Perl script is in Unicode and
1709 encoded in UTF-8, then an explicit "use utf8" is needed.)
1710 Figuring out the hexadecimal sequence of a Unicode character you want
1712 or deciphering someone else's hexadecimal Unicode regexp is about as
1713 much fun as programming in machine code. So another way to specify
1714 Unicode characters is to use the named character escape sequence
1715 "\N{name}". name is a name for the Unicode character, as specified in
1716 the Unicode standard. For instance, if we wanted to represent or match
1717 the astrological sign for the planet Mercury, we could use
1718 $x = "abc\N{MERCURY}def";
1720 $x =~ /\N{MERCURY}/; # matches
1721 One can also use "short" names:
1723 print "\N{GREEK SMALL LETTER SIGMA} is called sigma.\n";
1725 print "\N{greek:Sigma} is an upper-case sigma.\n";
1726 You can also restrict names to a certain alphabet by specifying the
1728 charnames pragma:
1729 use charnames qw(greek);
1731 print "\N{sigma} is Greek sigma\n";
1732 An index of character names is available on-line from the Unicode
1734 Consortium, <http://www.unicode.org/charts/charindex.html>; explanatory
1735 material with links to other resources at
1736 <http://www.unicode.org/standard/where>.
1737 The answer to requirement 2) is, as of 5.6.0, that a regexp (mostly)
1739 uses Unicode characters. (The "mostly" is for messy backward
1740 compatibility reasons, but starting in Perl 5.14, any regex compiled in
1741 the scope of a "use feature 'unicode_strings'" (which is automatically
1742 turned on within the scope of a "use 5.012" or higher) will turn that
1743 "mostly" into "always". If you want to handle Unicode properly, you
1744 should ensure that 'unicode_strings' is turned on.) Internally, this
1745 is encoded to bytes using either UTF-8 or a native 8 bit encoding,
1746 depending on the history of the string, but conceptually it is a
1747 sequence of characters, not bytes. See perlunitut for a tutorial about
1748 that.
1749 Let us now discuss Unicode character classes. Just as with Unicode
1751 characters, there are named Unicode character classes represented by
1752 the "\p{name}" escape sequence. Closely associated is the "\P{name}"
1753 character class, which is the negation of the "\p{name}" class. For
1754 example, to match lower and uppercase characters,
1755 $x = "BOB";
1757 $x =~ /^\p{IsUpper}/; # matches, uppercase char class
1758 $x =~ /^\P{IsUpper}/; # doesn't match, char class sans uppercase
1759 $x =~ /^\p{IsLower}/; # doesn't match, lowercase char class
1760 $x =~ /^\P{IsLower}/; # matches, char class sans lowercase
1761 (The "Is" is optional.)
1763 Here is the association between some Perl named classes and the
1765 traditional Unicode classes:
1766 Perl class name Unicode class name or regular expression
1768 IsAlpha /^[LM]/
1770 IsAlnum /^[LMN]/
1771 IsASCII $code <= 127
1772 IsCntrl /^C/
1773 IsBlank $code =~ /^(0020|0009)$/ || /^Z[^lp]/
1774 IsDigit Nd
1775 IsGraph /^([LMNPS]|Co)/
1776 IsLower Ll
1777 IsPrint /^([LMNPS]|Co|Zs)/
1778 IsPunct /^P/
1779 IsSpace /^Z/ || ($code =~ /^(0009|000A|000B|000C|000D)$/
1780 IsSpacePerl /^Z/ || ($code =~ /^(0009|000A|000C|000D|0085|2028|2029)$/
1781 IsUpper /^L[ut]/
1782 IsWord /^[LMN]/ || $code eq "005F"
1783 IsXDigit $code =~ /^00(3[0-9]|[46][1-6])$/
1784 You can also use the official Unicode class names with "\p" and "\P",
1786 like "\p{L}" for Unicode 'letters', "\p{Lu}" for uppercase letters, or
1787 "\P{Nd}" for non-digits. If a "name" is just one letter, the braces
1788 can be dropped. For instance, "\pM" is the character class of Unicode
1789 'marks', for example accent marks. For the full list see perlunicode.
1790 Unicode has also been separated into various sets of characters which
1792 you can test with "\p{...}" (in) and "\P{...}" (not in). To test
1793 whether a character is (or is not) an element of a script you would use
1794 the script name, for example "\p{Latin}", "\p{Greek}", or
1795 "\P{Katakana}".
1796 What we have described so far is the single form of the "\p{...}"
1798 character classes. There is also a compound form which you may run
1799 into. These look like "\p{name=value}" or "\p{name:value}" (the equals
1800 sign and colon can be used interchangeably). These are more general
1801 than the single form, and in fact most of the single forms are just
1802 Perl-defined shortcuts for common compound forms. For example, the
1803 script examples in the previous paragraph could be written equivalently
1804 as "\p{Script=Latin}", "\p{Script:Greek}", and "\P{script=katakana}"
1805 (case is irrelevant between the "{}" braces). You may never have to
1806 use the compound forms, but sometimes it is necessary, and their use
1807 can make your code easier to understand.
1808 "\X" is an abbreviation for a character class that comprises a Unicode
1810 extended grapheme cluster. This represents a "logical character": what
1811 appears to be a single character, but may be represented internally by
1812 more than one. As an example, using the Unicode full names, e.g.,
1813 "A + COMBINING RING" is a grapheme cluster with base character "A" and
1814 combining character "COMBINING RING", which translates in Danish to A
1815 with the circle atop it, as in the word Angstrom.
1816 For the full and latest information about Unicode see the latest
1818 Unicode standard, or the Unicode Consortium's website
1819 <http://www.unicode.org>
1820 As if all those classes weren't enough, Perl also defines POSIX-style
1822 character classes. These have the form "[:name:]", with "name" the
1823 name of the POSIX class. The POSIX classes are "alpha", "alnum",
1824 "ascii", "cntrl", "digit", "graph", "lower", "print", "punct", "space",
1825 "upper", and "xdigit", and two extensions, "word" (a Perl extension to
1826 match "\w"), and "blank" (a GNU extension). The "//a" modifier
1827 restricts these to matching just in the ASCII range; otherwise they can
1828 match the same as their corresponding Perl Unicode classes: "[:upper:]"
1829 is the same as "\p{IsUpper}", etc. (There are some exceptions and
1830 gotchas with this; see perlrecharclass for a full discussion.) The
1831 "[:digit:]", "[:word:]", and "[:space:]" correspond to the familiar
1832 "\d", "\w", and "\s" character classes. To negate a POSIX class, put a
1833 "^" in front of the name, so that, e.g., "[:^digit:]" corresponds to
1834 "\D" and, under Unicode, "\P{IsDigit}". The Unicode and POSIX
1835 character classes can be used just like "\d", with the exception that
1836 POSIX character classes can only be used inside of a character class:
1837 /\s+[abc[:digit:]xyz]\s*/; # match a,b,c,x,y,z, or a digit
1839 /^=item\s[[:digit:]]/; # match '=item',
1840 # followed by a space and a digit
1841 /\s+[abc\p{IsDigit}xyz]\s+/; # match a,b,c,x,y,z, or a digit
1842 /^=item\s\p{IsDigit}/; # match '=item',
1843 # followed by a space and a digit
1844 Whew! That is all the rest of the characters and character classes.
1846 Compiling and saving regular expressions
1848 In Part 1 we mentioned that Perl compiles a regexp into a compact
1849 sequence of opcodes. Thus, a compiled regexp is a data structure that
1850 can be stored once and used again and again. The regexp quote "qr//"
1851 does exactly that: "qr/string/" compiles the "string" as a regexp and
1852 transforms the result into a form that can be assigned to a variable:
1853 $reg = qr/foo+bar?/; # reg contains a compiled regexp
1855 Then $reg can be used as a regexp:
1857 $x = "fooooba";
1859 $x =~ $reg; # matches, just like /foo+bar?/
1860 $x =~ /$reg/; # same thing, alternate form
1861 $reg can also be interpolated into a larger regexp:
1863 $x =~ /(abc)?$reg/; # still matches
1865 As with the matching operator, the regexp quote can use different
1867 delimiters, e.g., "qr!!", "qr{}" or "qr~~". Apostrophes as delimiters
1868 ("qr''") inhibit any interpolation.
1869 Pre-compiled regexps are useful for creating dynamic matches that don't
1871 need to be recompiled each time they are encountered. Using pre-
1872 compiled regexps, we write a "grep_step" program which greps for a
1873 sequence of patterns, advancing to the next pattern as soon as one has
1874 been satisfied.
1875 % cat > grep_step
1877 #!/usr/bin/perl
1878 # grep_step - match <number> regexps, one after the other
1879 # usage: multi_grep <number> regexp1 regexp2 ... file1 file2 ...
1880 $number = shift;
1882 $regexp[$_] = shift foreach (0..$number-1);
1883 @compiled = map qr/$_/, @regexp;
1884 while ($line = <>) {
1885 if ($line =~ /$compiled[0]/) {
1886 print $line;
1887 shift @compiled;
1888 last unless @compiled;
1889 }
1890 }
1891 ^D
1892 % grep_step 3 shift print last grep_step
1894 $number = shift;
1895 print $line;
1896 last unless @compiled;
1897 Storing pre-compiled regexps in an array @compiled allows us to simply
1899 loop through the regexps without any recompilation, thus gaining
1900 flexibility without sacrificing speed.
1901 Composing regular expressions at runtime
1903 Backtracking is more efficient than repeated tries with different
1904 regular expressions. If there are several regular expressions and a
1905 match with any of them is acceptable, then it is possible to combine
1906 them into a set of alternatives. If the individual expressions are
1907 input data, this can be done by programming a join operation. We'll
1908 exploit this idea in an improved version of the "simple_grep" program:
1909 a program that matches multiple patterns:
1910 % cat > multi_grep
1912 #!/usr/bin/perl
1913 # multi_grep - match any of <number> regexps
1914 # usage: multi_grep <number> regexp1 regexp2 ... file1 file2 ...
1915 $number = shift;
1917 $regexp[$_] = shift foreach (0..$number-1);
1918 $pattern = join '|', @regexp;
1919 while ($line = <>) {
1921 print $line if $line =~ /$pattern/;
1922 }
1923 ^D
1924 % multi_grep 2 shift for multi_grep
1926 $number = shift;
1927 $regexp[$_] = shift foreach (0..$number-1);
1928 Sometimes it is advantageous to construct a pattern from the input that
1930 is to be analyzed and use the permissible values on the left hand side
1931 of the matching operations. As an example for this somewhat
1932 paradoxical situation, let's assume that our input contains a command
1933 verb which should match one out of a set of available command verbs,
1934 with the additional twist that commands may be abbreviated as long as
1935 the given string is unique. The program below demonstrates the basic
1936 algorithm.
1937 % cat > keymatch
1939 #!/usr/bin/perl
1940 $kwds = 'copy compare list print';
1941 while( $command = <> ){
1942 $command =~ s/^\s+|\s+$//g; # trim leading and trailing spaces
1943 if( ( @matches = $kwds =~ /\b$command\w*/g ) == 1 ){
1944 print "command: '@matches'\n";
1945 } elsif( @matches == 0 ){
1946 print "no such command: '$command'\n";
1947 } else {
1948 print "not unique: '$command' (could be one of: @matches)\n";
1949 }
1950 }
1951 ^D
1952 % keymatch
1954 li
1955 command: 'list'
1956 co
1957 not unique: 'co' (could be one of: copy compare)
1958 printer
1959 no such command: 'printer'
1960 Rather than trying to match the input against the keywords, we match
1962 the combined set of keywords against the input. The pattern matching
1963 operation "$kwds =~ /\b($command\w*)/g" does several things at the same
1964 time. It makes sure that the given command begins where a keyword
1965 begins ("\b"). It tolerates abbreviations due to the added "\w*". It
1966 tells us the number of matches ("scalar @matches") and all the keywords
1967 that were actually matched. You could hardly ask for more.
1968 Embedding comments and modifiers in a regular expression
1970 Starting with this section, we will be discussing Perl's set of
1971 extended patterns. These are extensions to the traditional regular
1972 expression syntax that provide powerful new tools for pattern matching.
1973 We have already seen extensions in the form of the minimal matching
1974 constructs "??", "*?", "+?", "{n,m}?", and "{n,}?". Most of the
1975 extensions below have the form "(?char...)", where the "char" is a
1976 character that determines the type of extension.
1977 The first extension is an embedded comment "(?#text)". This embeds a
1979 comment into the regular expression without affecting its meaning. The
1980 comment should not have any closing parentheses in the text. An
1981 example is
1982 /(?# Match an integer:)[+-]?\d+/;
1984 This style of commenting has been largely superseded by the raw,
1986 freeform commenting that is allowed with the "//x" modifier.
1987 Most modifiers, such as "//i", "//m", "//s" and "//x" (or any
1989 combination thereof) can also be embedded in a regexp using "(?i)",
1990 "(?m)", "(?s)", and "(?x)". For instance,
1991 /(?i)yes/; # match 'yes' case insensitively
1993 /yes/i; # same thing
1994 /(?x)( # freeform version of an integer regexp
1995 [+-]? # match an optional sign
1996 \d+ # match a sequence of digits
1997 )
1998 /x;
1999 Embedded modifiers can have two important advantages over the usual
2001 modifiers. Embedded modifiers allow a custom set of modifiers to each
2002 regexp pattern. This is great for matching an array of regexps that
2003 must have different modifiers:
2004 $pattern[0] = '(?i)doctor';
2006 $pattern[1] = 'Johnson';
2007 ...
2008 while (<>) {
2009 foreach $patt (@pattern) {
2010 print if /$patt/;
2011 }
2012 }
2013 The second advantage is that embedded modifiers (except "//p", which
2015 modifies the entire regexp) only affect the regexp inside the group the
2016 embedded modifier is contained in. So grouping can be used to localize
2017 the modifier's effects:
2018 /Answer: ((?i)yes)/; # matches 'Answer: yes', 'Answer: YES', etc.
2020 Embedded modifiers can also turn off any modifiers already present by
2022 using, e.g., "(?-i)". Modifiers can also be combined into a single
2023 expression, e.g., "(?s-i)" turns on single line mode and turns off case
2024 insensitivity.
2025 Embedded modifiers may also be added to a non-capturing grouping.
2027 "(?i-m:regexp)" is a non-capturing grouping that matches "regexp" case
2028 insensitively and turns off multi-line mode.
2029 Looking ahead and looking behind
2031 This section concerns the lookahead and lookbehind assertions. First,
2032 a little background.
2033 In Perl regular expressions, most regexp elements 'eat up' a certain
2035 amount of string when they match. For instance, the regexp element
2036 "[abc}]" eats up one character of the string when it matches, in the
2037 sense that Perl moves to the next character position in the string
2038 after the match. There are some elements, however, that don't eat up
2039 characters (advance the character position) if they match. The
2040 examples we have seen so far are the anchors. The anchor "^" matches
2041 the beginning of the line, but doesn't eat any characters. Similarly,
2042 the word boundary anchor "\b" matches wherever a character matching
2043 "\w" is next to a character that doesn't, but it doesn't eat up any
2044 characters itself. Anchors are examples of zero-width assertions:
2045 zero-width, because they consume no characters, and assertions, because
2046 they test some property of the string. In the context of our walk in
2047 the woods analogy to regexp matching, most regexp elements move us
2048 along a trail, but anchors have us stop a moment and check our
2049 surroundings. If the local environment checks out, we can proceed
2050 forward. But if the local environment doesn't satisfy us, we must
2051 backtrack.
2052 Checking the environment entails either looking ahead on the trail,
2054 looking behind, or both. "^" looks behind, to see that there are no
2055 characters before. "$" looks ahead, to see that there are no
2056 characters after. "\b" looks both ahead and behind, to see if the
2057 characters on either side differ in their "word-ness".
2058 The lookahead and lookbehind assertions are generalizations of the
2060 anchor concept. Lookahead and lookbehind are zero-width assertions
2061 that let us specify which characters we want to test for. The
2062 lookahead assertion is denoted by "(?=regexp)" and the lookbehind
2063 assertion is denoted by "(?<=fixed-regexp)". Some examples are
2064 $x = "I catch the housecat 'Tom-cat' with catnip";
2066 $x =~ /cat(?=\s)/; # matches 'cat' in 'housecat'
2067 @catwords = ($x =~ /(?<=\s)cat\w+/g); # matches,
2068 # $catwords[0] = 'catch'
2069 # $catwords[1] = 'catnip'
2070 $x =~ /\bcat\b/; # matches 'cat' in 'Tom-cat'
2071 $x =~ /(?<=\s)cat(?=\s)/; # doesn't match; no isolated 'cat' in
2072 # middle of $x
2073 Note that the parentheses in "(?=regexp)" and "(?<=regexp)" are non-
2075 capturing, since these are zero-width assertions. Thus in the second
2076 regexp, the substrings captured are those of the whole regexp itself.
2077 Lookahead "(?=regexp)" can match arbitrary regexps, but lookbehind
2078 "(?<=fixed-regexp)" only works for regexps of fixed width, i.e., a
2079 fixed number of characters long. Thus "(?<=(ab|bc))" is fine, but
2080 "(?<=(ab)*)" is not. The negated versions of the lookahead and
2081 lookbehind assertions are denoted by "(?!regexp)" and
2082 "(?<!fixed-regexp)" respectively. They evaluate true if the regexps do
2083 not match:
2084 $x = "foobar";
2086 $x =~ /foo(?!bar)/; # doesn't match, 'bar' follows 'foo'
2087 $x =~ /foo(?!baz)/; # matches, 'baz' doesn't follow 'foo'
2088 $x =~ /(?<!\s)foo/; # matches, there is no \s before 'foo'
2089 The "\C" is unsupported in lookbehind, because the already treacherous
2091 definition of "\C" would become even more so when going backwards.
2092 Here is an example where a string containing blank-separated words,
2094 numbers and single dashes is to be split into its components. Using
2095 "/\s+/" alone won't work, because spaces are not required between
2096 dashes, or a word or a dash. Additional places for a split are
2097 established by looking ahead and behind:
2098 $str = "one two - --6-8";
2100 @toks = split / \s+ # a run of spaces
2101 | (?<=\S) (?=-) # any non-space followed by '-'
2102 | (?<=-) (?=\S) # a '-' followed by any non-space
2103 /x, $str; # @toks = qw(one two - - - 6 - 8)
2104 Using independent subexpressions to prevent backtracking
2106 Independent subexpressions are regular expressions, in the context of a
2107 larger regular expression, that function independently of the larger
2108 regular expression. That is, they consume as much or as little of the
2109 string as they wish without regard for the ability of the larger regexp
2110 to match. Independent subexpressions are represented by "(?>regexp)".
2111 We can illustrate their behavior by first considering an ordinary
2112 regexp:
2113 $x = "ab";
2115 $x =~ /a*ab/; # matches
2116 This obviously matches, but in the process of matching, the
2118 subexpression "a*" first grabbed the "a". Doing so, however, wouldn't
2119 allow the whole regexp to match, so after backtracking, "a*" eventually
2120 gave back the "a" and matched the empty string. Here, what "a*"
2121 matched was dependent on what the rest of the regexp matched.
2122 Contrast that with an independent subexpression:
2124 $x =~ /(?>a*)ab/; # doesn't match!
2126 The independent subexpression "(?>a*)" doesn't care about the rest of
2128 the regexp, so it sees an "a" and grabs it. Then the rest of the
2129 regexp "ab" cannot match. Because "(?>a*)" is independent, there is no
2130 backtracking and the independent subexpression does not give up its
2131 "a". Thus the match of the regexp as a whole fails. A similar
2132 behavior occurs with completely independent regexps:
2133 $x = "ab";
2135 $x =~ /a*/g; # matches, eats an 'a'
2136 $x =~ /\Gab/g; # doesn't match, no 'a' available
2137 Here "//g" and "\G" create a 'tag team' handoff of the string from one
2139 regexp to the other. Regexps with an independent subexpression are
2140 much like this, with a handoff of the string to the independent
2141 subexpression, and a handoff of the string back to the enclosing
2142 regexp.
2143 The ability of an independent subexpression to prevent backtracking can
2145 be quite useful. Suppose we want to match a non-empty string enclosed
2146 in parentheses up to two levels deep. Then the following regexp
2147 matches:
2148 $x = "abc(de(fg)h"; # unbalanced parentheses
2150 $x =~ /\( ( [^()]+ | \([^()]*\) )+ \)/x;
2151 The regexp matches an open parenthesis, one or more copies of an
2153 alternation, and a close parenthesis. The alternation is two-way, with
2154 the first alternative "[^()]+" matching a substring with no parentheses
2155 and the second alternative "\([^()]*\)" matching a substring delimited
2156 by parentheses. The problem with this regexp is that it is
2157 pathological: it has nested indeterminate quantifiers of the form
2158 "(a+|b)+". We discussed in Part 1 how nested quantifiers like this
2159 could take an exponentially long time to execute if there was no match
2160 possible. To prevent the exponential blowup, we need to prevent
2161 useless backtracking at some point. This can be done by enclosing the
2162 inner quantifier as an independent subexpression:
2163 $x =~ /\( ( (?>[^()]+) | \([^()]*\) )+ \)/x;
2165 Here, "(?>[^()]+)" breaks the degeneracy of string partitioning by
2167 gobbling up as much of the string as possible and keeping it. Then
2168 match failures fail much more quickly.
2169 Conditional expressions
2171 A conditional expression is a form of if-then-else statement that
2172 allows one to choose which patterns are to be matched, based on some
2173 condition. There are two types of conditional expression:
2174 "(?(condition)yes-regexp)" and "(?(condition)yes-regexp|no-regexp)".
2175 "(?(condition)yes-regexp)" is like an 'if () {}' statement in Perl. If
2176 the "condition" is true, the "yes-regexp" will be matched. If the
2177 "condition" is false, the "yes-regexp" will be skipped and Perl will
2178 move onto the next regexp element. The second form is like an
2179 'if () {} else {}' statement in Perl. If the "condition" is true, the
2180 "yes-regexp" will be matched, otherwise the "no-regexp" will be
2181 matched.
2182 The "condition" can have several forms. The first form is simply an
2184 integer in parentheses "(integer)". It is true if the corresponding
2185 backreference "\integer" matched earlier in the regexp. The same thing
2186 can be done with a name associated with a capture group, written as
2187 "(<name>)" or "('name')". The second form is a bare zero-width
2188 assertion "(?...)", either a lookahead, a lookbehind, or a code
2189 assertion (discussed in the next section). The third set of forms
2190 provides tests that return true if the expression is executed within a
2191 recursion ("(R)") or is being called from some capturing group,
2192 referenced either by number ("(R1)", "(R2)",...) or by name
2193 ("(R&name)").
2194 The integer or name form of the "condition" allows us to choose, with
2196 more flexibility, what to match based on what matched earlier in the
2197 regexp. This searches for words of the form "$x$x" or "$x$y$y$x":
2198 % simple_grep '^(\w+)(\w+)?(?(2)\g2\g1|\g1)$' /usr/dict/words
2200 beriberi
2201 coco
2202 couscous
2203 deed
2204 ...
2205 toot
2206 toto
2207 tutu
2208 The lookbehind "condition" allows, along with backreferences, an
2210 earlier part of the match to influence a later part of the match. For
2211 instance,
2212 /[ATGC]+(?(?<=AA)G|C)$/;
2214 matches a DNA sequence such that it either ends in "AAG", or some other
2216 base pair combination and "C". Note that the form is "(?(?<=AA)G|C)"
2217 and not "(?((?<=AA))G|C)"; for the lookahead, lookbehind or code
2218 assertions, the parentheses around the conditional are not needed.
2219 Defining named patterns
2221 Some regular expressions use identical subpatterns in several places.
2222 Starting with Perl 5.10, it is possible to define named subpatterns in
2223 a section of the pattern so that they can be called up by name anywhere
2224 in the pattern. This syntactic pattern for this definition group is
2225 "(?(DEFINE)(?<name>pattern)...)". An insertion of a named pattern is
2226 written as "(?&name)".
2227 The example below illustrates this feature using the pattern for
2229 floating point numbers that was presented earlier on. The three
2230 subpatterns that are used more than once are the optional sign, the
2231 digit sequence for an integer and the decimal fraction. The DEFINE
2232 group at the end of the pattern contains their definition. Notice that
2233 the decimal fraction pattern is the first place where we can reuse the
2234 integer pattern.
2235 /^ (?&osg)\ * ( (?&int)(?&dec)? | (?&dec) )
2237 (?: [eE](?&osg)(?&int) )?
2238 $
2239 (?(DEFINE)
2240 (?<osg>[-+]?) # optional sign
2241 (?<int>\d++) # integer
2242 (?<dec>\.(?&int)) # decimal fraction
2243 )/x
2244 Recursive patterns
2246 This feature (introduced in Perl 5.10) significantly extends the power
2247 of Perl's pattern matching. By referring to some other capture group
2248 anywhere in the pattern with the construct "(?group-ref)", the pattern
2249 within the referenced group is used as an independent subpattern in
2250 place of the group reference itself. Because the group reference may
2251 be contained within the group it refers to, it is now possible to apply
2252 pattern matching to tasks that hitherto required a recursive parser.
2253 To illustrate this feature, we'll design a pattern that matches if a
2255 string contains a palindrome. (This is a word or a sentence that, while
2256 ignoring spaces, interpunctuation and case, reads the same backwards as
2257 forwards. We begin by observing that the empty string or a string
2258 containing just one word character is a palindrome. Otherwise it must
2259 have a word character up front and the same at its end, with another
2260 palindrome in between.
2261 /(?: (\w) (?...Here be a palindrome...) \g{-1} | \w? )/x
2263 Adding "\W*" at either end to eliminate what is to be ignored, we
2265 already have the full pattern:
2266 my $pp = qr/^(\W* (?: (\w) (?1) \g{-1} | \w? ) \W*)$/ix;
2268 for $s ( "saippuakauppias", "A man, a plan, a canal: Panama!" ){
2269 print "'$s' is a palindrome\n" if $s =~ /$pp/;
2270 }
2271 In "(?...)" both absolute and relative backreferences may be used. The
2273 entire pattern can be reinserted with "(?R)" or "(?0)". If you prefer
2274 to name your groups, you can use "(?&name)" to recurse into that group.
2275 A bit of magic: executing Perl code in a regular expression
2277 Normally, regexps are a part of Perl expressions. Code evaluation
2278 expressions turn that around by allowing arbitrary Perl code to be a
2279 part of a regexp. A code evaluation expression is denoted "(?{code})",
2280 with code a string of Perl statements.
2281 Be warned that this feature is considered experimental, and may be
2283 changed without notice.
2284 Code expressions are zero-width assertions, and the value they return
2286 depends on their environment. There are two possibilities: either the
2287 code expression is used as a conditional in a conditional expression
2288 "(?(condition)...)", or it is not. If the code expression is a
2289 conditional, the code is evaluated and the result (i.e., the result of
2290 the last statement) is used to determine truth or falsehood. If the
2291 code expression is not used as a conditional, the assertion always
2292 evaluates true and the result is put into the special variable $^R.
2293 The variable $^R can then be used in code expressions later in the
2294 regexp. Here are some silly examples:
2295 $x = "abcdef";
2297 $x =~ /abc(?{print "Hi Mom!";})def/; # matches,
2298 # prints 'Hi Mom!'
2299 $x =~ /aaa(?{print "Hi Mom!";})def/; # doesn't match,
2300 # no 'Hi Mom!'
2301 Pay careful attention to the next example:
2303 $x =~ /abc(?{print "Hi Mom!";})ddd/; # doesn't match,
2305 # no 'Hi Mom!'
2306 # but why not?
2307 At first glance, you'd think that it shouldn't print, because obviously
2309 the "ddd" isn't going to match the target string. But look at this
2310 example:
2311 $x =~ /abc(?{print "Hi Mom!";})[dD]dd/; # doesn't match,
2313 # but _does_ print
2314 Hmm. What happened here? If you've been following along, you know that
2316 the above pattern should be effectively (almost) the same as the last
2317 one; enclosing the "d" in a character class isn't going to change what
2318 it matches. So why does the first not print while the second one does?
2319 The answer lies in the optimizations the regex engine makes. In the
2321 first case, all the engine sees are plain old characters (aside from
2322 the "?{}" construct). It's smart enough to realize that the string
2323 'ddd' doesn't occur in our target string before actually running the
2324 pattern through. But in the second case, we've tricked it into thinking
2325 that our pattern is more complicated. It takes a look, sees our
2326 character class, and decides that it will have to actually run the
2327 pattern to determine whether or not it matches, and in the process of
2328 running it hits the print statement before it discovers that we don't
2329 have a match.
2330 To take a closer look at how the engine does optimizations, see the
2332 section "Pragmas and debugging" below.
2333 More fun with "?{}":
2335 $x =~ /(?{print "Hi Mom!";})/; # matches,
2337 # prints 'Hi Mom!'
2338 $x =~ /(?{$c = 1;})(?{print "$c";})/; # matches,
2339 # prints '1'
2340 $x =~ /(?{$c = 1;})(?{print "$^R";})/; # matches,
2341 # prints '1'
2342 The bit of magic mentioned in the section title occurs when the regexp
2344 backtracks in the process of searching for a match. If the regexp
2345 backtracks over a code expression and if the variables used within are
2346 localized using "local", the changes in the variables produced by the
2347 code expression are undone! Thus, if we wanted to count how many times
2348 a character got matched inside a group, we could use, e.g.,
2349 $x = "aaaa";
2351 $count = 0; # initialize 'a' count
2352 $c = "bob"; # test if $c gets clobbered
2353 $x =~ /(?{local $c = 0;}) # initialize count
2354 ( a # match 'a'
2355 (?{local $c = $c + 1;}) # increment count
2356 )* # do this any number of times,
2357 aa # but match 'aa' at the end
2358 (?{$count = $c;}) # copy local $c var into $count
2359 /x;
2360 print "'a' count is $count, \$c variable is '$c'\n";
2361 This prints
2363 'a' count is 2, $c variable is 'bob'
2365 If we replace the " (?{local $c = $c + 1;})" with " (?{$c = $c + 1;})",
2367 the variable changes are not undone during backtracking, and we get
2368 'a' count is 4, $c variable is 'bob'
2370 Note that only localized variable changes are undone. Other side
2372 effects of code expression execution are permanent. Thus
2373 $x = "aaaa";
2375 $x =~ /(a(?{print "Yow\n";}))*aa/;
2376 produces
2378 Yow
2380 Yow
2381 Yow
2382 Yow
2383 The result $^R is automatically localized, so that it will behave
2385 properly in the presence of backtracking.
2386 This example uses a code expression in a conditional to match a
2388 definite article, either 'the' in English or 'der|die|das' in German:
2389 $lang = 'DE'; # use German
2391 ...
2392 $text = "das";
2393 print "matched\n"
2394 if $text =~ /(?(?{
2395 $lang eq 'EN'; # is the language English?
2396 })
2397 the | # if so, then match 'the'
2398 (der|die|das) # else, match 'der|die|das'
2399 )
2400 /xi;
2401 Note that the syntax here is "(?(?{...})yes-regexp|no-regexp)", not
2403 "(?((?{...}))yes-regexp|no-regexp)". In other words, in the case of a
2404 code expression, we don't need the extra parentheses around the
2405 conditional.
2406 If you try to use code expressions with interpolating variables, Perl
2408 may surprise you:
2409 $bar = 5;
2411 $pat = '(?{ 1 })';
2412 /foo(?{ $bar })bar/; # compiles ok, $bar not interpolated
2413 /foo(?{ 1 })$bar/; # compile error!
2414 /foo${pat}bar/; # compile error!
2415 $pat = qr/(?{ $foo = 1 })/; # precompile code regexp
2417 /foo${pat}bar/; # compiles ok
2418 If a regexp has (1) code expressions and interpolating variables, or
2420 (2) a variable that interpolates a code expression, Perl treats the
2421 regexp as an error. If the code expression is precompiled into a
2422 variable, however, interpolating is ok. The question is, why is this an
2423 error?
2424 The reason is that variable interpolation and code expressions together
2426 pose a security risk. The combination is dangerous because many
2427 programmers who write search engines often take user input and plug it
2428 directly into a regexp:
2429 $regexp = <>; # read user-supplied regexp
2431 $chomp $regexp; # get rid of possible newline
2432 $text =~ /$regexp/; # search $text for the $regexp
2433 If the $regexp variable contains a code expression, the user could then
2435 execute arbitrary Perl code. For instance, some joker could search for
2436 "system('rm -rf *');" to erase your files. In this sense, the
2437 combination of interpolation and code expressions taints your regexp.
2438 So by default, using both interpolation and code expressions in the
2439 same regexp is not allowed. If you're not concerned about malicious
2440 users, it is possible to bypass this security check by invoking
2441 "use re 'eval'":
2442 use re 'eval'; # throw caution out the door
2444 $bar = 5;
2445 $pat = '(?{ 1 })';
2446 /foo(?{ 1 })$bar/; # compiles ok
2447 /foo${pat}bar/; # compiles ok
2448 Another form of code expression is the pattern code expression. The
2450 pattern code expression is like a regular code expression, except that
2451 the result of the code evaluation is treated as a regular expression
2452 and matched immediately. A simple example is
2453 $length = 5;
2455 $char = 'a';
2456 $x = 'aaaaabb';
2457 $x =~ /(??{$char x $length})/x; # matches, there are 5 of 'a'
2458 This final example contains both ordinary and pattern code expressions.
2460 It detects whether a binary string 1101010010001... has a Fibonacci
2461 spacing 0,1,1,2,3,5,... of the 1's:
2462 $x = "1101010010001000001";
2464 $z0 = ''; $z1 = '0'; # initial conditions
2465 print "It is a Fibonacci sequence\n"
2466 if $x =~ /^1 # match an initial '1'
2467 (?:
2468 ((??{ $z0 })) # match some '0'
2469 1 # and then a '1'
2470 (?{ $z0 = $z1; $z1 .= $^N; })
2471 )+ # repeat as needed
2472 $ # that is all there is
2473 /x;
2474 printf "Largest sequence matched was %d\n", length($z1)-length($z0);
2475 Remember that $^N is set to whatever was matched by the last completed
2477 capture group. This prints
2478 It is a Fibonacci sequence
2480 Largest sequence matched was 5
2481 Ha! Try that with your garden variety regexp package...
2483 Note that the variables $z0 and $z1 are not substituted when the regexp
2485 is compiled, as happens for ordinary variables outside a code
2486 expression. Rather, the code expressions are evaluated when Perl
2487 encounters them during the search for a match.
2488 The regexp without the "//x" modifier is
2490 /^1(?:((??{ $z0 }))1(?{ $z0 = $z1; $z1 .= $^N; }))+$/
2492 which shows that spaces are still possible in the code parts.
2494 Nevertheless, when working with code and conditional expressions, the
2495 extended form of regexps is almost necessary in creating and debugging
2496 regexps.
2497 Backtracking control verbs
2499 Perl 5.10 introduced a number of control verbs intended to provide
2500 detailed control over the backtracking process, by directly influencing
2501 the regexp engine and by providing monitoring techniques. As all the
2502 features in this group are experimental and subject to change or
2503 removal in a future version of Perl, the interested reader is referred
2504 to "Special Backtracking Control Verbs" in perlre for a detailed
2505 description.
2506 Below is just one example, illustrating the control verb "(*FAIL)",
2508 which may be abbreviated as "(*F)". If this is inserted in a regexp it
2509 will cause it to fail, just as it would at some mismatch between the
2510 pattern and the string. Processing of the regexp continues as it would
2511 after any "normal" failure, so that, for instance, the next position in
2512 the string or another alternative will be tried. As failing to match
2513 doesn't preserve capture groups or produce results, it may be necessary
2514 to use this in combination with embedded code.
2515 %count = ();
2517 "supercalifragilisticexpialidocious" =~
2518 /([aeiou])(?{ $count{$1}++; })(*FAIL)/i;
2519 printf "%3d '%s'\n", $count{$_}, $_ for (sort keys %count);
2520 The pattern begins with a class matching a subset of letters. Whenever
2522 this matches, a statement like "$count{'a'}++;" is executed,
2523 incrementing the letter's counter. Then "(*FAIL)" does what it says,
2524 and the regexp engine proceeds according to the book: as long as the
2525 end of the string hasn't been reached, the position is advanced before
2526 looking for another vowel. Thus, match or no match makes no difference,
2527 and the regexp engine proceeds until the entire string has been
2528 inspected. (It's remarkable that an alternative solution using
2529 something like
2530 $count{lc($_)}++ for split('', "supercalifragilisticexpialidocious");
2532 printf "%3d '%s'\n", $count2{$_}, $_ for ( qw{ a e i o u } );
2533 is considerably slower.)
2535 Pragmas and debugging
2537 Speaking of debugging, there are several pragmas available to control
2538 and debug regexps in Perl. We have already encountered one pragma in
2539 the previous section, "use re 'eval';", that allows variable
2540 interpolation and code expressions to coexist in a regexp. The other
2541 pragmas are
2542 use re 'taint';
2544 $tainted = <>;
2545 @parts = ($tainted =~ /(\w+)\s+(\w+)/; # @parts is now tainted
2546 The "taint" pragma causes any substrings from a match with a tainted
2548 variable to be tainted as well. This is not normally the case, as
2549 regexps are often used to extract the safe bits from a tainted
2550 variable. Use "taint" when you are not extracting safe bits, but are
2551 performing some other processing. Both "taint" and "eval" pragmas are
2552 lexically scoped, which means they are in effect only until the end of
2553 the block enclosing the pragmas.
2554 use re '/m'; # or any other flags
2556 $multiline_string =~ /^foo/; # /m is implied
2557 The "re '/flags'" pragma (introduced in Perl 5.14) turns on the given
2559 regular expression flags until the end of the lexical scope. See
2560 "'/flags' mode" in re for more detail.
2561 use re 'debug';
2563 /^(.*)$/s; # output debugging info
2564 use re 'debugcolor';
2566 /^(.*)$/s; # output debugging info in living color
2567 The global "debug" and "debugcolor" pragmas allow one to get detailed
2569 debugging info about regexp compilation and execution. "debugcolor" is
2570 the same as debug, except the debugging information is displayed in
2571 color on terminals that can display termcap color sequences. Here is
2572 example output:
2573 % perl -e 'use re "debug"; "abc" =~ /a*b+c/;'
2575 Compiling REx 'a*b+c'
2576 size 9 first at 1
2577 1: STAR(4)
2578 2: EXACT <a>(0)
2579 4: PLUS(7)
2580 5: EXACT <b>(0)
2581 7: EXACT <c>(9)
2582 9: END(0)
2583 floating 'bc' at 0..2147483647 (checking floating) minlen 2
2584 Guessing start of match, REx 'a*b+c' against 'abc'...
2585 Found floating substr 'bc' at offset 1...
2586 Guessed: match at offset 0
2587 Matching REx 'a*b+c' against 'abc'
2588 Setting an EVAL scope, savestack=3
2589 0 <> <abc> | 1: STAR
2590 EXACT <a> can match 1 times out of 32767...
2591 Setting an EVAL scope, savestack=3
2592 1 <a> <bc> | 4: PLUS
2593 EXACT <b> can match 1 times out of 32767...
2594 Setting an EVAL scope, savestack=3
2595 2 <ab> <c> | 7: EXACT <c>
2596 3 <abc> <> | 9: END
2597 Match successful!
2598 Freeing REx: 'a*b+c'
2599 If you have gotten this far into the tutorial, you can probably guess
2601 what the different parts of the debugging output tell you. The first
2602 part
2603 Compiling REx 'a*b+c'
2605 size 9 first at 1
2606 1: STAR(4)
2607 2: EXACT <a>(0)
2608 4: PLUS(7)
2609 5: EXACT <b>(0)
2610 7: EXACT <c>(9)
2611 9: END(0)
2612 describes the compilation stage. STAR(4) means that there is a starred
2614 object, in this case 'a', and if it matches, goto line 4, i.e.,
2615 PLUS(7). The middle lines describe some heuristics and optimizations
2616 performed before a match:
2617 floating 'bc' at 0..2147483647 (checking floating) minlen 2
2619 Guessing start of match, REx 'a*b+c' against 'abc'...
2620 Found floating substr 'bc' at offset 1...
2621 Guessed: match at offset 0
2622 Then the match is executed and the remaining lines describe the
2624 process:
2625 Matching REx 'a*b+c' against 'abc'
2627 Setting an EVAL scope, savestack=3
2628 0 <> <abc> | 1: STAR
2629 EXACT <a> can match 1 times out of 32767...
2630 Setting an EVAL scope, savestack=3
2631 1 <a> <bc> | 4: PLUS
2632 EXACT <b> can match 1 times out of 32767...
2633 Setting an EVAL scope, savestack=3
2634 2 <ab> <c> | 7: EXACT <c>
2635 3 <abc> <> | 9: END
2636 Match successful!
2637 Freeing REx: 'a*b+c'
2638 Each step is of the form "n <x> <y>", with "<x>" the part of the string
2640 matched and "<y>" the part not yet matched. The "| 1: STAR" says
2641 that Perl is at line number 1 in the compilation list above. See
2642 "Debugging Regular Expressions" in perldebguts for much more detail.
2643 An alternative method of debugging regexps is to embed "print"
2645 statements within the regexp. This provides a blow-by-blow account of
2646 the backtracking in an alternation:
2647 "that this" =~ m@(?{print "Start at position ", pos, "\n";})
2649 t(?{print "t1\n";})
2650 h(?{print "h1\n";})
2651 i(?{print "i1\n";})
2652 s(?{print "s1\n";})
2653 |
2654 t(?{print "t2\n";})
2655 h(?{print "h2\n";})
2656 a(?{print "a2\n";})
2657 t(?{print "t2\n";})
2658 (?{print "Done at position ", pos, "\n";})
2659 @x;
2660 prints
2662 Start at position 0
2664 t1
2665 h1
2666 t2
2667 h2
2668 a2
2669 t2
2670 Done at position 4
2671
2673 Code expressions, conditional expressions, and independent expressions
2674 are experimental. Don't use them in production code. Yet.
2675
2677 This is just a tutorial. For the full story on Perl regular
2678 expressions, see the perlre regular expressions reference page.
2679 For more information on the matching "m//" and substitution "s///"
2681 operators, see "Regexp Quote-Like Operators" in perlop. For
2682 information on the "split" operation, see "split" in perlfunc.
2683 For an excellent all-around resource on the care and feeding of regular
2685 expressions, see the book Mastering Regular Expressions by Jeffrey
2686 Friedl (published by O'Reilly, ISBN 1556592-257-3).
2687
2689 Copyright (c) 2000 Mark Kvale All rights reserved.
2690 This document may be distributed under the same terms as Perl itself.
2692 Acknowledgments
2694 The inspiration for the stop codon DNA example came from the ZIP code
2695 example in chapter 7 of Mastering Regular Expressions.
2696 The author would like to thank Jeff Pinyan, Andrew Johnson, Peter
2698 Haworth, Ronald J Kimball, and Joe Smith for all their helpful
2699 comments.
2700perl v5.16.3 2013-03-04 PERLRETUT(1)