1Text::Balanced(3pm)    Perl Programmers Reference Guide    Text::Balanced(3pm)
2
3
4

NAME

6       Text::Balanced - Extract delimited text sequences from strings.
7

SYNOPSIS

9        use Text::Balanced qw (
10                               extract_delimited
11                               extract_bracketed
12                               extract_quotelike
13                               extract_codeblock
14                               extract_variable
15                               extract_tagged
16                               extract_multiple
17                               gen_delimited_pat
18                               gen_extract_tagged
19                              );
20
21        # Extract the initial substring of $text that is delimited by
22        # two (unescaped) instances of the first character in $delim.
23
24               ($extracted, $remainder) = extract_delimited($text,$delim);
25
26
27        # Extract the initial substring of $text that is bracketed
28        # with a delimiter(s) specified by $delim (where the string
29        # in $delim contains one or more of '(){}[]<>').
30
31               ($extracted, $remainder) = extract_bracketed($text,$delim);
32
33
34        # Extract the initial substring of $text that is bounded by
35        # an XML tag.
36
37               ($extracted, $remainder) = extract_tagged($text);
38
39
40        # Extract the initial substring of $text that is bounded by
41        # a C<BEGIN>...C<END> pair. Don't allow nested C<BEGIN> tags
42
43               ($extracted, $remainder) =
44                       extract_tagged($text,"BEGIN","END",undef,{bad=>["BEGIN"]});
45
46
47        # Extract the initial substring of $text that represents a
48        # Perl "quote or quote-like operation"
49
50               ($extracted, $remainder) = extract_quotelike($text);
51
52
53        # Extract the initial substring of $text that represents a block
54        # of Perl code, bracketed by any of character(s) specified by $delim
55        # (where the string $delim contains one or more of '(){}[]<>').
56
57               ($extracted, $remainder) = extract_codeblock($text,$delim);
58
59
60        # Extract the initial substrings of $text that would be extracted by
61        # one or more sequential applications of the specified functions
62        # or regular expressions
63
64               @extracted = extract_multiple($text,
65                                             [ \&extract_bracketed,
66                                               \&extract_quotelike,
67                                               \&some_other_extractor_sub,
68                                               qr/[xyz]*/,
69                                               'literal',
70                                             ]);
71
72       # Create a string representing an optimized pattern (a la Friedl) #
73       that matches a substring delimited by any of the specified characters #
74       (in this case: any type of quote or a slash)
75
76               $patstring = gen_delimited_pat(q{'"`/});
77
78       # Generate a reference to an anonymous sub that is just like
79       extract_tagged # but pre-compiled and optimized for a specific pair of
80       tags, and consequently # much faster (i.e. 3 times faster). It uses
81       qr// for better performance on # repeated calls, so it only works under
82       Perl 5.005 or later.
83
84               $extract_head = gen_extract_tagged('<HEAD>','</HEAD>');
85
86               ($extracted, $remainder) = $extract_head->($text);
87

DESCRIPTION

89       The various "extract_..." subroutines may be used to extract a
90       delimited substring, possibly after skipping a specified prefix string.
91       By default, that prefix is optional whitespace ("/\s*/"), but you can
92       change it to whatever you wish (see below).
93
94       The substring to be extracted must appear at the current "pos" location
95       of the string's variable (or at index zero, if no "pos" position is
96       defined).  In other words, the "extract_..." subroutines don't extract
97       the first occurrence of a substring anywhere in a string (like an
98       unanchored regex would). Rather, they extract an occurrence of the
99       substring appearing immediately at the current matching position in the
100       string (like a "\G"-anchored regex would).
101
102   General behaviour in list contexts
103       In a list context, all the subroutines return a list, the first three
104       elements of which are always:
105
106       [0] The extracted string, including the specified delimiters.  If the
107           extraction fails "undef" is returned.
108
109       [1] The remainder of the input string (i.e. the characters after the
110           extracted string). On failure, the entire string is returned.
111
112       [2] The skipped prefix (i.e. the characters before the extracted
113           string).  On failure, "undef" is returned.
114
115       Note that in a list context, the contents of the original input text
116       (the first argument) are not modified in any way.
117
118       However, if the input text was passed in a variable, that variable's
119       "pos" value is updated to point at the first character after the
120       extracted text. That means that in a list context the various
121       subroutines can be used much like regular expressions. For example:
122
123               while ( $next = (extract_quotelike($text))[0] )
124               {
125                       # process next quote-like (in $next)
126               }
127
128   General behaviour in scalar and void contexts
129       In a scalar context, the extracted string is returned, having first
130       been removed from the input text. Thus, the following code also
131       processes each quote-like operation, but actually removes them from
132       $text:
133
134               while ( $next = extract_quotelike($text) )
135               {
136                       # process next quote-like (in $next)
137               }
138
139       Note that if the input text is a read-only string (i.e. a literal), no
140       attempt is made to remove the extracted text.
141
142       In a void context the behaviour of the extraction subroutines is
143       exactly the same as in a scalar context, except (of course) that the
144       extracted substring is not returned.
145
146   A note about prefixes
147       Prefix patterns are matched without any trailing modifiers ("/gimsox"
148       etc.)  This can bite you if you're expecting a prefix specification
149       like '.*?(?=<H1>)' to skip everything up to the first <H1> tag. Such a
150       prefix pattern will only succeed if the <H1> tag is on the current
151       line, since . normally doesn't match newlines.
152
153       To overcome this limitation, you need to turn on /s matching within the
154       prefix pattern, using the "(?s)" directive: '(?s).*?(?=<H1>)'
155
156   "extract_delimited"
157       The "extract_delimited" function formalizes the common idiom of
158       extracting a single-character-delimited substring from the start of a
159       string. For example, to extract a single-quote delimited string, the
160       following code is typically used:
161
162               ($remainder = $text) =~ s/\A('(\\.|[^'])*')//s;
163               $extracted = $1;
164
165       but with "extract_delimited" it can be simplified to:
166
167               ($extracted,$remainder) = extract_delimited($text, "'");
168
169       "extract_delimited" takes up to four scalars (the input text, the
170       delimiters, a prefix pattern to be skipped, and any escape characters)
171       and extracts the initial substring of the text that is appropriately
172       delimited. If the delimiter string has multiple characters, the first
173       one encountered in the text is taken to delimit the substring.  The
174       third argument specifies a prefix pattern that is to be skipped (but
175       must be present!) before the substring is extracted.  The final
176       argument specifies the escape character to be used for each delimiter.
177
178       All arguments are optional. If the escape characters are not specified,
179       every delimiter is escaped with a backslash ("\").  If the prefix is
180       not specified, the pattern '\s*' - optional whitespace - is used. If
181       the delimiter set is also not specified, the set "/["'`]/" is used. If
182       the text to be processed is not specified either, $_ is used.
183
184       In list context, "extract_delimited" returns a array of three elements,
185       the extracted substring (including the surrounding delimiters), the
186       remainder of the text, and the skipped prefix (if any). If a suitable
187       delimited substring is not found, the first element of the array is the
188       empty string, the second is the complete original text, and the prefix
189       returned in the third element is an empty string.
190
191       In a scalar context, just the extracted substring is returned. In a
192       void context, the extracted substring (and any prefix) are simply
193       removed from the beginning of the first argument.
194
195       Examples:
196
197               # Remove a single-quoted substring from the very beginning of $text:
198
199                       $substring = extract_delimited($text, "'", '');
200
201               # Remove a single-quoted Pascalish substring (i.e. one in which
202               # doubling the quote character escapes it) from the very
203               # beginning of $text:
204
205                       $substring = extract_delimited($text, "'", '', "'");
206
207               # Extract a single- or double- quoted substring from the
208               # beginning of $text, optionally after some whitespace
209               # (note the list context to protect $text from modification):
210
211                       ($substring) = extract_delimited $text, q{"'};
212
213               # Delete the substring delimited by the first '/' in $text:
214
215                       $text = join '', (extract_delimited($text,'/','[^/]*')[2,1];
216
217       Note that this last example is not the same as deleting the first
218       quote-like pattern. For instance, if $text contained the string:
219
220               "if ('./cmd' =~ m/$UNIXCMD/s) { $cmd = $1; }"
221
222       then after the deletion it would contain:
223
224               "if ('.$UNIXCMD/s) { $cmd = $1; }"
225
226       not:
227
228               "if ('./cmd' =~ ms) { $cmd = $1; }"
229
230       See "extract_quotelike" for a (partial) solution to this problem.
231
232   "extract_bracketed"
233       Like "extract_delimited", the "extract_bracketed" function takes up to
234       three optional scalar arguments: a string to extract from, a delimiter
235       specifier, and a prefix pattern. As before, a missing prefix defaults
236       to optional whitespace and a missing text defaults to $_. However, a
237       missing delimiter specifier defaults to '{}()[]<>' (see below).
238
239       "extract_bracketed" extracts a balanced-bracket-delimited substring
240       (using any one (or more) of the user-specified delimiter brackets:
241       '(..)', '{..}', '[..]', or '<..>'). Optionally it will also respect
242       quoted unbalanced brackets (see below).
243
244       A "delimiter bracket" is a bracket in list of delimiters passed as
245       "extract_bracketed"'s second argument. Delimiter brackets are specified
246       by giving either the left or right (or both!) versions of the required
247       bracket(s). Note that the order in which two or more delimiter brackets
248       are specified is not significant.
249
250       A "balanced-bracket-delimited substring" is a substring bounded by
251       matched brackets, such that any other (left or right) delimiter bracket
252       within the substring is also matched by an opposite (right or left)
253       delimiter bracket at the same level of nesting. Any type of bracket not
254       in the delimiter list is treated as an ordinary character.
255
256       In other words, each type of bracket specified as a delimiter must be
257       balanced and correctly nested within the substring, and any other kind
258       of ("non-delimiter") bracket in the substring is ignored.
259
260       For example, given the string:
261
262               $text = "{ an '[irregularly :-(] {} parenthesized >:-)' string }";
263
264       then a call to "extract_bracketed" in a list context:
265
266               @result = extract_bracketed( $text, '{}' );
267
268       would return:
269
270               ( "{ an '[irregularly :-(] {} parenthesized >:-)' string }" , "" , "" )
271
272       since both sets of '{..}' brackets are properly nested and evenly
273       balanced.  (In a scalar context just the first element of the array
274       would be returned. In a void context, $text would be replaced by an
275       empty string.)
276
277       Likewise the call in:
278
279               @result = extract_bracketed( $text, '{[' );
280
281       would return the same result, since all sets of both types of specified
282       delimiter brackets are correctly nested and balanced.
283
284       However, the call in:
285
286               @result = extract_bracketed( $text, '{([<' );
287
288       would fail, returning:
289
290               ( undef , "{ an '[irregularly :-(] {} parenthesized >:-)' string }"  );
291
292       because the embedded pairs of '(..)'s and '[..]'s are "cross-nested"
293       and the embedded '>' is unbalanced. (In a scalar context, this call
294       would return an empty string. In a void context, $text would be
295       unchanged.)
296
297       Note that the embedded single-quotes in the string don't help in this
298       case, since they have not been specified as acceptable delimiters and
299       are therefore treated as non-delimiter characters (and ignored).
300
301       However, if a particular species of quote character is included in the
302       delimiter specification, then that type of quote will be correctly
303       handled.  for example, if $text is:
304
305               $text = '<A HREF=">>>>">link</A>';
306
307       then
308
309               @result = extract_bracketed( $text, '<">' );
310
311       returns:
312
313               ( '<A HREF=">>>>">', 'link</A>', "" )
314
315       as expected. Without the specification of """ as an embedded quoter:
316
317               @result = extract_bracketed( $text, '<>' );
318
319       the result would be:
320
321               ( '<A HREF=">', '>>>">link</A>', "" )
322
323       In addition to the quote delimiters "'", """, and "`", full Perl quote-
324       like quoting (i.e. q{string}, qq{string}, etc) can be specified by
325       including the letter 'q' as a delimiter. Hence:
326
327               @result = extract_bracketed( $text, '<q>' );
328
329       would correctly match something like this:
330
331               $text = '<leftop: conj /and/ conj>';
332
333       See also: "extract_quotelike" and "extract_codeblock".
334
335   "extract_variable"
336       "extract_variable" extracts any valid Perl variable or variable-
337       involved expression, including scalars, arrays, hashes, array accesses,
338       hash look-ups, method calls through objects, subroutine calls through
339       subroutine references, etc.
340
341       The subroutine takes up to two optional arguments:
342
343       1.  A string to be processed ($_ if the string is omitted or "undef")
344
345       2.  A string specifying a pattern to be matched as a prefix (which is
346           to be skipped). If omitted, optional whitespace is skipped.
347
348       On success in a list context, an array of 3 elements is returned. The
349       elements are:
350
351       [0] the extracted variable, or variablish expression
352
353       [1] the remainder of the input text,
354
355       [2] the prefix substring (if any),
356
357       On failure, all of these values (except the remaining text) are
358       "undef".
359
360       In a scalar context, "extract_variable" returns just the complete
361       substring that matched a variablish expression. "undef" is returned on
362       failure. In addition, the original input text has the returned
363       substring (and any prefix) removed from it.
364
365       In a void context, the input text just has the matched substring (and
366       any specified prefix) removed.
367
368   "extract_tagged"
369       "extract_tagged" extracts and segments text between (balanced)
370       specified tags.
371
372       The subroutine takes up to five optional arguments:
373
374       1.  A string to be processed ($_ if the string is omitted or "undef")
375
376       2.  A string specifying a pattern to be matched as the opening tag.  If
377           the pattern string is omitted (or "undef") then a pattern that
378           matches any standard XML tag is used.
379
380       3.  A string specifying a pattern to be matched at the closing tag.  If
381           the pattern string is omitted (or "undef") then the closing tag is
382           constructed by inserting a "/" after any leading bracket characters
383           in the actual opening tag that was matched (not the pattern that
384           matched the tag). For example, if the opening tag pattern is
385           specified as '{{\w+}}' and actually matched the opening tag
386           "{{DATA}}", then the constructed closing tag would be "{{/DATA}}".
387
388       4.  A string specifying a pattern to be matched as a prefix (which is
389           to be skipped). If omitted, optional whitespace is skipped.
390
391       5.  A hash reference containing various parsing options (see below)
392
393       The various options that can be specified are:
394
395       "reject => $listref"
396           The list reference contains one or more strings specifying patterns
397           that must not appear within the tagged text.
398
399           For example, to extract an HTML link (which should not contain
400           nested links) use:
401
402                   extract_tagged($text, '<A>', '</A>', undef, {reject => ['<A>']} );
403
404       "ignore => $listref"
405           The list reference contains one or more strings specifying patterns
406           that are not be be treated as nested tags within the tagged text
407           (even if they would match the start tag pattern).
408
409           For example, to extract an arbitrary XML tag, but ignore "empty"
410           elements:
411
412                   extract_tagged($text, undef, undef, undef, {ignore => ['<[^>]*/>']} );
413
414           (also see "gen_delimited_pat" below).
415
416       "fail => $str"
417           The "fail" option indicates the action to be taken if a matching
418           end tag is not encountered (i.e. before the end of the string or
419           some "reject" pattern matches). By default, a failure to match a
420           closing tag causes "extract_tagged" to immediately fail.
421
422           However, if the string value associated with <reject> is "MAX",
423           then "extract_tagged" returns the complete text up to the point of
424           failure.  If the string is "PARA", "extract_tagged" returns only
425           the first paragraph after the tag (up to the first line that is
426           either empty or contains only whitespace characters).  If the
427           string is "", the the default behaviour (i.e. failure) is
428           reinstated.
429
430           For example, suppose the start tag "/para" introduces a paragraph,
431           which then continues until the next "/endpara" tag or until another
432           "/para" tag is encountered:
433
434                   $text = "/para line 1\n\nline 3\n/para line 4";
435
436                   extract_tagged($text, '/para', '/endpara', undef,
437                                           {reject => '/para', fail => MAX );
438
439                   # EXTRACTED: "/para line 1\n\nline 3\n"
440
441           Suppose instead, that if no matching "/endpara" tag is found, the
442           "/para" tag refers only to the immediately following paragraph:
443
444                   $text = "/para line 1\n\nline 3\n/para line 4";
445
446                   extract_tagged($text, '/para', '/endpara', undef,
447                                   {reject => '/para', fail => MAX );
448
449                   # EXTRACTED: "/para line 1\n"
450
451           Note that the specified "fail" behaviour applies to nested tags as
452           well.
453
454       On success in a list context, an array of 6 elements is returned. The
455       elements are:
456
457       [0] the extracted tagged substring (including the outermost tags),
458
459       [1] the remainder of the input text,
460
461       [2] the prefix substring (if any),
462
463       [3] the opening tag
464
465       [4] the text between the opening and closing tags
466
467       [5] the closing tag (or "" if no closing tag was found)
468
469       On failure, all of these values (except the remaining text) are
470       "undef".
471
472       In a scalar context, "extract_tagged" returns just the complete
473       substring that matched a tagged text (including the start and end
474       tags). "undef" is returned on failure. In addition, the original input
475       text has the returned substring (and any prefix) removed from it.
476
477       In a void context, the input text just has the matched substring (and
478       any specified prefix) removed.
479
480   "gen_extract_tagged"
481       (Note: This subroutine is only available under Perl5.005)
482
483       "gen_extract_tagged" generates a new anonymous subroutine which
484       extracts text between (balanced) specified tags. In other words, it
485       generates a function identical in function to "extract_tagged".
486
487       The difference between "extract_tagged" and the anonymous subroutines
488       generated by "gen_extract_tagged", is that those generated subroutines:
489
490       ·   do not have to reparse tag specification or parsing options every
491           time they are called (whereas "extract_tagged" has to effectively
492           rebuild its tag parser on every call);
493
494       ·   make use of the new qr// construct to pre-compile the regexes they
495           use (whereas "extract_tagged" uses standard string variable
496           interpolation to create tag-matching patterns).
497
498       The subroutine takes up to four optional arguments (the same set as
499       "extract_tagged" except for the string to be processed). It returns a
500       reference to a subroutine which in turn takes a single argument (the
501       text to be extracted from).
502
503       In other words, the implementation of "extract_tagged" is exactly
504       equivalent to:
505
506               sub extract_tagged
507               {
508                       my $text = shift;
509                       $extractor = gen_extract_tagged(@_);
510                       return $extractor->($text);
511               }
512
513       (although "extract_tagged" is not currently implemented that way, in
514       order to preserve pre-5.005 compatibility).
515
516       Using "gen_extract_tagged" to create extraction functions for specific
517       tags is a good idea if those functions are going to be called more than
518       once, since their performance is typically twice as good as the more
519       general-purpose "extract_tagged".
520
521   "extract_quotelike"
522       "extract_quotelike" attempts to recognize, extract, and segment any one
523       of the various Perl quotes and quotelike operators (see perlop(3))
524       Nested backslashed delimiters, embedded balanced bracket delimiters
525       (for the quotelike operators), and trailing modifiers are all caught.
526       For example, in:
527
528               extract_quotelike 'q # an octothorpe: \# (not the end of the q!) #'
529
530               extract_quotelike '  "You said, \"Use sed\"."  '
531
532               extract_quotelike ' s{([A-Z]{1,8}\.[A-Z]{3})} /\L$1\E/; '
533
534               extract_quotelike ' tr/\\\/\\\\/\\\//ds; '
535
536       the full Perl quotelike operations are all extracted correctly.
537
538       Note too that, when using the /x modifier on a regex, any comment
539       containing the current pattern delimiter will cause the regex to be
540       immediately terminated. In other words:
541
542               'm /
543                       (?i)            # CASE INSENSITIVE
544                       [a-z_]          # LEADING ALPHABETIC/UNDERSCORE
545                       [a-z0-9]*       # FOLLOWED BY ANY NUMBER OF ALPHANUMERICS
546                  /x'
547
548       will be extracted as if it were:
549
550               'm /
551                       (?i)            # CASE INSENSITIVE
552                       [a-z_]          # LEADING ALPHABETIC/'
553
554       This behaviour is identical to that of the actual compiler.
555
556       "extract_quotelike" takes two arguments: the text to be processed and a
557       prefix to be matched at the very beginning of the text. If no prefix is
558       specified, optional whitespace is the default. If no text is given, $_
559       is used.
560
561       In a list context, an array of 11 elements is returned. The elements
562       are:
563
564       [0] the extracted quotelike substring (including trailing modifiers),
565
566       [1] the remainder of the input text,
567
568       [2] the prefix substring (if any),
569
570       [3] the name of the quotelike operator (if any),
571
572       [4] the left delimiter of the first block of the operation,
573
574       [5] the text of the first block of the operation (that is, the contents
575           of a quote, the regex of a match or substitution or the target list
576           of a translation),
577
578       [6] the right delimiter of the first block of the operation,
579
580       [7] the left delimiter of the second block of the operation (that is,
581           if it is a "s", "tr", or "y"),
582
583       [8] the text of the second block of the operation (that is, the
584           replacement of a substitution or the translation list of a
585           translation),
586
587       [9] the right delimiter of the second block of the operation (if any),
588
589       [10]
590           the trailing modifiers on the operation (if any).
591
592       For each of the fields marked "(if any)" the default value on success
593       is an empty string.  On failure, all of these values (except the
594       remaining text) are "undef".
595
596       In a scalar context, "extract_quotelike" returns just the complete
597       substring that matched a quotelike operation (or "undef" on failure).
598       In a scalar or void context, the input text has the same substring (and
599       any specified prefix) removed.
600
601       Examples:
602
603               # Remove the first quotelike literal that appears in text
604
605                       $quotelike = extract_quotelike($text,'.*?');
606
607               # Replace one or more leading whitespace-separated quotelike
608               # literals in $_ with "<QLL>"
609
610                       do { $_ = join '<QLL>', (extract_quotelike)[2,1] } until $@;
611
612
613               # Isolate the search pattern in a quotelike operation from $text
614
615                       ($op,$pat) = (extract_quotelike $text)[3,5];
616                       if ($op =~ /[ms]/)
617                       {
618                               print "search pattern: $pat\n";
619                       }
620                       else
621                       {
622                               print "$op is not a pattern matching operation\n";
623                       }
624
625   "extract_quotelike" and "here documents"
626       "extract_quotelike" can successfully extract "here documents" from an
627       input string, but with an important caveat in list contexts.
628
629       Unlike other types of quote-like literals, a here document is rarely a
630       contiguous substring. For example, a typical piece of code using here
631       document might look like this:
632
633               <<'EOMSG' || die;
634               This is the message.
635               EOMSG
636               exit;
637
638       Given this as an input string in a scalar context, "extract_quotelike"
639       would correctly return the string "<<'EOMSG'\nThis is the
640       message.\nEOMSG", leaving the string " || die;\nexit;" in the original
641       variable. In other words, the two separate pieces of the here document
642       are successfully extracted and concatenated.
643
644       In a list context, "extract_quotelike" would return the list
645
646       [0] "<<'EOMSG'\nThis is the message.\nEOMSG\n" (i.e. the full extracted
647           here document, including fore and aft delimiters),
648
649       [1] " || die;\nexit;" (i.e. the remainder of the input text,
650           concatenated),
651
652       [2] "" (i.e. the prefix substring -- trivial in this case),
653
654       [3] "<<" (i.e. the "name" of the quotelike operator)
655
656       [4] "'EOMSG'" (i.e. the left delimiter of the here document, including
657           any quotes),
658
659       [5] "This is the message.\n" (i.e. the text of the here document),
660
661       [6] "EOMSG" (i.e. the right delimiter of the here document),
662
663       [7..10]
664           "" (a here document has no second left delimiter, second text,
665           second right delimiter, or trailing modifiers).
666
667       However, the matching position of the input variable would be set to
668       "exit;" (i.e. after the closing delimiter of the here document), which
669       would cause the earlier " || die;\nexit;" to be skipped in any sequence
670       of code fragment extractions.
671
672       To avoid this problem, when it encounters a here document whilst
673       extracting from a modifiable string, "extract_quotelike" silently
674       rearranges the string to an equivalent piece of Perl:
675
676               <<'EOMSG'
677               This is the message.
678               EOMSG
679               || die;
680               exit;
681
682       in which the here document is contiguous. It still leaves the matching
683       position after the here document, but now the rest of the line on which
684       the here document starts is not skipped.
685
686       To prevent <extract_quotelike> from mucking about with the input in
687       this way (this is the only case where a list-context
688       "extract_quotelike" does so), you can pass the input variable as an
689       interpolated literal:
690
691               $quotelike = extract_quotelike("$var");
692
693   "extract_codeblock"
694       "extract_codeblock" attempts to recognize and extract a balanced
695       bracket delimited substring that may contain unbalanced brackets inside
696       Perl quotes or quotelike operations. That is, "extract_codeblock" is
697       like a combination of "extract_bracketed" and "extract_quotelike".
698
699       "extract_codeblock" takes the same initial three parameters as
700       "extract_bracketed": a text to process, a set of delimiter brackets to
701       look for, and a prefix to match first. It also takes an optional fourth
702       parameter, which allows the outermost delimiter brackets to be
703       specified separately (see below).
704
705       Omitting the first argument (input text) means process $_ instead.
706       Omitting the second argument (delimiter brackets) indicates that only
707       '{' is to be used.  Omitting the third argument (prefix argument)
708       implies optional whitespace at the start.  Omitting the fourth argument
709       (outermost delimiter brackets) indicates that the value of the second
710       argument is to be used for the outermost delimiters.
711
712       Once the prefix an dthe outermost opening delimiter bracket have been
713       recognized, code blocks are extracted by stepping through the input
714       text and trying the following alternatives in sequence:
715
716       1.  Try and match a closing delimiter bracket. If the bracket was the
717           same species as the last opening bracket, return the substring to
718           that point. If the bracket was mismatched, return an error.
719
720       2.  Try to match a quote or quotelike operator. If found, call
721           "extract_quotelike" to eat it. If "extract_quotelike" fails, return
722           the error it returned. Otherwise go back to step 1.
723
724       3.  Try to match an opening delimiter bracket. If found, call
725           "extract_codeblock" recursively to eat the embedded block. If the
726           recursive call fails, return an error. Otherwise, go back to step
727           1.
728
729       4.  Unconditionally match a bareword or any other single character, and
730           then go back to step 1.
731
732       Examples:
733
734               # Find a while loop in the text
735
736                       if ($text =~ s/.*?while\s*\{/{/)
737                       {
738                               $loop = "while " . extract_codeblock($text);
739                       }
740
741               # Remove the first round-bracketed list (which may include
742               # round- or curly-bracketed code blocks or quotelike operators)
743
744                       extract_codeblock $text, "(){}", '[^(]*';
745
746       The ability to specify a different outermost delimiter bracket is
747       useful in some circumstances. For example, in the Parse::RecDescent
748       module, parser actions which are to be performed only on a successful
749       parse are specified using a "<defer:...>" directive. For example:
750
751               sentence: subject verb object
752                               <defer: {$::theVerb = $item{verb}} >
753
754       Parse::RecDescent uses "extract_codeblock($text, '{}<>')" to extract
755       the code within the "<defer:...>" directive, but there's a problem.
756
757       A deferred action like this:
758
759                               <defer: {if ($count>10) {$count--}} >
760
761       will be incorrectly parsed as:
762
763                               <defer: {if ($count>
764
765       because the "less than" operator is interpreted as a closing delimiter.
766
767       But, by extracting the directive using
768       "extract_codeblock($text, '{}', undef, '<>')" the '>' character is only
769       treated as a delimited at the outermost level of the code block, so the
770       directive is parsed correctly.
771
772   "extract_multiple"
773       The "extract_multiple" subroutine takes a string to be processed and a
774       list of extractors (subroutines or regular expressions) to apply to
775       that string.
776
777       In an array context "extract_multiple" returns an array of substrings
778       of the original string, as extracted by the specified extractors.  In a
779       scalar context, "extract_multiple" returns the first substring
780       successfully extracted from the original string. In both scalar and
781       void contexts the original string has the first successfully extracted
782       substring removed from it. In all contexts "extract_multiple" starts at
783       the current "pos" of the string, and sets that "pos" appropriately
784       after it matches.
785
786       Hence, the aim of of a call to "extract_multiple" in a list context is
787       to split the processed string into as many non-overlapping fields as
788       possible, by repeatedly applying each of the specified extractors to
789       the remainder of the string. Thus "extract_multiple" is a generalized
790       form of Perl's "split" subroutine.
791
792       The subroutine takes up to four optional arguments:
793
794       1.  A string to be processed ($_ if the string is omitted or "undef")
795
796       2.  A reference to a list of subroutine references and/or qr// objects
797           and/or literal strings and/or hash references, specifying the
798           extractors to be used to split the string. If this argument is
799           omitted (or "undef") the list:
800
801                   [
802                           sub { extract_variable($_[0], '') },
803                           sub { extract_quotelike($_[0],'') },
804                           sub { extract_codeblock($_[0],'{}','') },
805                   ]
806
807           is used.
808
809       3.  An number specifying the maximum number of fields to return. If
810           this argument is omitted (or "undef"), split continues as long as
811           possible.
812
813           If the third argument is N, then extraction continues until N
814           fields have been successfully extracted, or until the string has
815           been completely processed.
816
817           Note that in scalar and void contexts the value of this argument is
818           automatically reset to 1 (under "-w", a warning is issued if the
819           argument has to be reset).
820
821       4.  A value indicating whether unmatched substrings (see below) within
822           the text should be skipped or returned as fields. If the value is
823           true, such substrings are skipped. Otherwise, they are returned.
824
825       The extraction process works by applying each extractor in sequence to
826       the text string.
827
828       If the extractor is a subroutine it is called in a list context and is
829       expected to return a list of a single element, namely the extracted
830       text. It may optionally also return two further arguments: a string
831       representing the text left after extraction (like $' for a pattern
832       match), and a string representing any prefix skipped before the
833       extraction (like $` in a pattern match). Note that this is designed to
834       facilitate the use of other Text::Balanced subroutines with
835       "extract_multiple". Note too that the value returned by an extractor
836       subroutine need not bear any relationship to the corresponding
837       substring of the original text (see examples below).
838
839       If the extractor is a precompiled regular expression or a string, it is
840       matched against the text in a scalar context with a leading '\G' and
841       the gc modifiers enabled. The extracted value is either $1 if that
842       variable is defined after the match, or else the complete match (i.e.
843       $&).
844
845       If the extractor is a hash reference, it must contain exactly one
846       element.  The value of that element is one of the above extractor types
847       (subroutine reference, regular expression, or string).  The key of that
848       element is the name of a class into which the successful return value
849       of the extractor will be blessed.
850
851       If an extractor returns a defined value, that value is immediately
852       treated as the next extracted field and pushed onto the list of fields.
853       If the extractor was specified in a hash reference, the field is also
854       blessed into the appropriate class,
855
856       If the extractor fails to match (in the case of a regex extractor), or
857       returns an empty list or an undefined value (in the case of a
858       subroutine extractor), it is assumed to have failed to extract.  If
859       none of the extractor subroutines succeeds, then one character is
860       extracted from the start of the text and the extraction subroutines
861       reapplied. Characters which are thus removed are accumulated and
862       eventually become the next field (unless the fourth argument is true,
863       in which case they are discarded).
864
865       For example, the following extracts substrings that are valid Perl
866       variables:
867
868               @fields = extract_multiple($text,
869                                          [ sub { extract_variable($_[0]) } ],
870                                          undef, 1);
871
872       This example separates a text into fields which are quote delimited,
873       curly bracketed, and anything else. The delimited and bracketed parts
874       are also blessed to identify them (the "anything else" is unblessed):
875
876               @fields = extract_multiple($text,
877                          [
878                               { Delim => sub { extract_delimited($_[0],q{'"}) } },
879                               { Brack => sub { extract_bracketed($_[0],'{}') } },
880                          ]);
881
882       This call extracts the next single substring that is a valid Perl
883       quotelike operator (and removes it from $text):
884
885               $quotelike = extract_multiple($text,
886                                             [
887                                               sub { extract_quotelike($_[0]) },
888                                             ], undef, 1);
889
890       Finally, here is yet another way to do comma-separated value parsing:
891
892               @fields = extract_multiple($csv_text,
893                                         [
894                                               sub { extract_delimited($_[0],q{'"}) },
895                                               qr/([^,]+)(.*)/,
896                                         ],
897                                         undef,1);
898
899       The list in the second argument means: "Try and extract a ' or "
900       delimited string, otherwise extract anything up to a comma...".  The
901       undef third argument means: "...as many times as possible...", and the
902       true value in the fourth argument means "...discarding anything else
903       that appears (i.e. the commas)".
904
905       If you wanted the commas preserved as separate fields (i.e. like split
906       does if your split pattern has capturing parentheses), you would just
907       make the last parameter undefined (or remove it).
908
909   "gen_delimited_pat"
910       The "gen_delimited_pat" subroutine takes a single (string) argument and
911          > builds a Friedl-style optimized regex that matches a string
912       delimited by any one of the characters in the single argument. For
913       example:
914
915               gen_delimited_pat(q{'"})
916
917       returns the regex:
918
919               (?:\"(?:\\\"|(?!\").)*\"|\'(?:\\\'|(?!\').)*\')
920
921       Note that the specified delimiters are automatically quotemeta'd.
922
923       A typical use of "gen_delimited_pat" would be to build special purpose
924       tags for "extract_tagged". For example, to properly ignore "empty" XML
925       elements (which might contain quoted strings):
926
927               my $empty_tag = '<(' . gen_delimited_pat(q{'"}) . '|.)+/>';
928
929               extract_tagged($text, undef, undef, undef, {ignore => [$empty_tag]} );
930
931       "gen_delimited_pat" may also be called with an optional second
932       argument, which specifies the "escape" character(s) to be used for each
933       delimiter.  For example to match a Pascal-style string (where ' is the
934       delimiter and '' is a literal ' within the string):
935
936               gen_delimited_pat(q{'},q{'});
937
938       Different escape characters can be specified for different delimiters.
939       For example, to specify that '/' is the escape for single quotes and
940       '%' is the escape for double quotes:
941
942               gen_delimited_pat(q{'"},q{/%});
943
944       If more delimiters than escape chars are specified, the last escape
945       char is used for the remaining delimiters.  If no escape char is
946       specified for a given specified delimiter, '\' is used.
947
948   "delimited_pat"
949       Note that "gen_delimited_pat" was previously called "delimited_pat".
950       That name may still be used, but is now deprecated.
951

DIAGNOSTICS

953       In a list context, all the functions return "(undef,$original_text)" on
954       failure. In a scalar context, failure is indicated by returning "undef"
955       (in this case the input text is not modified in any way).
956
957       In addition, on failure in any context, the $@ variable is set.
958       Accessing "$@->{error}" returns one of the error diagnostics listed
959       below.  Accessing "$@->{pos}" returns the offset into the original
960       string at which the error was detected (although not necessarily where
961       it occurred!)  Printing $@ directly produces the error message, with
962       the offset appended.  On success, the $@ variable is guaranteed to be
963       "undef".
964
965       The available diagnostics are:
966
967       "Did not find a suitable bracket: "%s""
968           The delimiter provided to "extract_bracketed" was not one of
969           '()[]<>{}'.
970
971       "Did not find prefix: /%s/"
972           A non-optional prefix was specified but wasn't found at the start
973           of the text.
974
975       "Did not find opening bracket after prefix: "%s""
976           "extract_bracketed" or "extract_codeblock" was expecting a
977           particular kind of bracket at the start of the text, and didn't
978           find it.
979
980       "No quotelike operator found after prefix: "%s""
981           "extract_quotelike" didn't find one of the quotelike operators "q",
982           "qq", "qw", "qx", "s", "tr" or "y" at the start of the substring it
983           was extracting.
984
985       "Unmatched closing bracket: "%c""
986           "extract_bracketed", "extract_quotelike" or "extract_codeblock"
987           encountered a closing bracket where none was expected.
988
989       "Unmatched opening bracket(s): "%s""
990           "extract_bracketed", "extract_quotelike" or "extract_codeblock" ran
991           out of characters in the text before closing one or more levels of
992           nested brackets.
993
994       "Unmatched embedded quote (%s)"
995           "extract_bracketed" attempted to match an embedded quoted
996           substring, but failed to find a closing quote to match it.
997
998       "Did not find closing delimiter to match '%s'"
999           "extract_quotelike" was unable to find a closing delimiter to match
1000           the one that opened the quote-like operation.
1001
1002       "Mismatched closing bracket: expected "%c" but found "%s""
1003           "extract_bracketed", "extract_quotelike" or "extract_codeblock"
1004           found a valid bracket delimiter, but it was the wrong species. This
1005           usually indicates a nesting error, but may indicate incorrect
1006           quoting or escaping.
1007
1008       "No block delimiter found after quotelike "%s""
1009           "extract_quotelike" or "extract_codeblock" found one of the
1010           quotelike operators "q", "qq", "qw", "qx", "s", "tr" or "y" without
1011           a suitable block after it.
1012
1013       "Did not find leading dereferencer"
1014           "extract_variable" was expecting one of '$', '@', or '%' at the
1015           start of a variable, but didn't find any of them.
1016
1017       "Bad identifier after dereferencer"
1018           "extract_variable" found a '$', '@', or '%' indicating a variable,
1019           but that character was not followed by a legal Perl identifier.
1020
1021       "Did not find expected opening bracket at %s"
1022           "extract_codeblock" failed to find any of the outermost opening
1023           brackets that were specified.
1024
1025       "Improperly nested codeblock at %s"
1026           A nested code block was found that started with a delimiter that
1027           was specified as being only to be used as an outermost bracket.
1028
1029       "Missing second block for quotelike "%s""
1030           "extract_codeblock" or "extract_quotelike" found one of the
1031           quotelike operators "s", "tr" or "y" followed by only one block.
1032
1033       "No match found for opening bracket"
1034           "extract_codeblock" failed to find a closing bracket to match the
1035           outermost opening bracket.
1036
1037       "Did not find opening tag: /%s/"
1038           "extract_tagged" did not find a suitable opening tag (after any
1039           specified prefix was removed).
1040
1041       "Unable to construct closing tag to match: /%s/"
1042           "extract_tagged" matched the specified opening tag and tried to
1043           modify the matched text to produce a matching closing tag (because
1044           none was specified). It failed to generate the closing tag, almost
1045           certainly because the opening tag did not start with a bracket of
1046           some kind.
1047
1048       "Found invalid nested tag: %s"
1049           "extract_tagged" found a nested tag that appeared in the "reject"
1050           list (and the failure mode was not "MAX" or "PARA").
1051
1052       "Found unbalanced nested tag: %s"
1053           "extract_tagged" found a nested opening tag that was not matched by
1054           a corresponding nested closing tag (and the failure mode was not
1055           "MAX" or "PARA").
1056
1057       "Did not find closing tag"
1058           "extract_tagged" reached the end of the text without finding a
1059           closing tag to match the original opening tag (and the failure mode
1060           was not "MAX" or "PARA").
1061

AUTHOR

1063       Damian Conway (damian@conway.org)
1064

BUGS AND IRRITATIONS

1066       There are undoubtedly serious bugs lurking somewhere in this code, if
1067       only because parts of it give the impression of understanding a great
1068       deal more about Perl than they really do.
1069
1070       Bug reports and other feedback are most welcome.
1071
1073       Copyright 1997 - 2001 Damian Conway. All Rights Reserved.
1074
1075       Some (minor) parts copyright 2009 Adam Kennedy.
1076
1077       This module is free software. It may be used, redistributed and/or
1078       modified under the same terms as Perl itself.
1079
1080
1081
1082perl v5.16.3                      2013-02-26               Text::Balanced(3pm)
Impressum