1Text::Balanced(3pm) Perl Programmers Reference Guide Text::Balanced(3pm)
2
6 Text::Balanced - Extract delimited text sequences from strings.
7
9 use Text::Balanced qw (
10 extract_delimited
11 extract_bracketed
12 extract_quotelike
13 extract_codeblock
14 extract_variable
15 extract_tagged
16 extract_multiple
17 gen_delimited_pat
18 gen_extract_tagged
19 );
20 # Extract the initial substring of $text that is delimited by
22 # two (unescaped) instances of the first character in $delim.
23 ($extracted, $remainder) = extract_delimited($text,$delim);
25 # Extract the initial substring of $text that is bracketed
28 # with a delimiter(s) specified by $delim (where the string
29 # in $delim contains one or more of '(){}[]<>').
30 ($extracted, $remainder) = extract_bracketed($text,$delim);
32 # Extract the initial substring of $text that is bounded by
35 # an XML tag.
36 ($extracted, $remainder) = extract_tagged($text);
38 # Extract the initial substring of $text that is bounded by
41 # a C<BEGIN>...C<END> pair. Don't allow nested C<BEGIN> tags
42 ($extracted, $remainder) =
44 extract_tagged($text,"BEGIN","END",undef,{bad=>["BEGIN"]});
45 # Extract the initial substring of $text that represents a
48 # Perl "quote or quote-like operation"
49 ($extracted, $remainder) = extract_quotelike($text);
51 # Extract the initial substring of $text that represents a block
54 # of Perl code, bracketed by any of character(s) specified by $delim
55 # (where the string $delim contains one or more of '(){}[]<>').
56 ($extracted, $remainder) = extract_codeblock($text,$delim);
58 # Extract the initial substrings of $text that would be extracted by
61 # one or more sequential applications of the specified functions
62 # or regular expressions
63 @extracted = extract_multiple($text,
65 [ \&extract_bracketed,
66 \&extract_quotelike,
67 \&some_other_extractor_sub,
68 qr/[xyz]*/,
69 'literal',
70 ]);
71 # Create a string representing an optimized pattern (a la Friedl) #
73 that matches a substring delimited by any of the specified characters #
74 (in this case: any type of quote or a slash)
75 $patstring = gen_delimited_pat(q{'"`/});
77 # Generate a reference to an anonymous sub that is just like
79 extract_tagged # but pre-compiled and optimized for a specific pair of
80 tags, and consequently # much faster (i.e. 3 times faster). It uses
81 qr// for better performance on # repeated calls, so it only works under
82 Perl 5.005 or later.
83 $extract_head = gen_extract_tagged('<HEAD>','</HEAD>');
85 ($extracted, $remainder) = $extract_head->($text);
87
89 The various "extract_..." subroutines may be used to extract a
90 delimited substring, possibly after skipping a specified prefix string.
91 By default, that prefix is optional whitespace ("/\s*/"), but you can
92 change it to whatever you wish (see below).
93 The substring to be extracted must appear at the current "pos" location
95 of the string's variable (or at index zero, if no "pos" position is
96 defined). In other words, the "extract_..." subroutines don't extract
97 the first occurrence of a substring anywhere in a string (like an
98 unanchored regex would). Rather, they extract an occurrence of the
99 substring appearing immediately at the current matching position in the
100 string (like a "\G"-anchored regex would).
101 General behaviour in list contexts
103 In a list context, all the subroutines return a list, the first three
104 elements of which are always:
105 [0] The extracted string, including the specified delimiters. If the
107 extraction fails "undef" is returned.
108 [1] The remainder of the input string (i.e. the characters after the
110 extracted string). On failure, the entire string is returned.
111 [2] The skipped prefix (i.e. the characters before the extracted
113 string). On failure, "undef" is returned.
114 Note that in a list context, the contents of the original input text
116 (the first argument) are not modified in any way.
117 However, if the input text was passed in a variable, that variable's
119 "pos" value is updated to point at the first character after the
120 extracted text. That means that in a list context the various
121 subroutines can be used much like regular expressions. For example:
122 while ( $next = (extract_quotelike($text))[0] )
124 {
125 # process next quote-like (in $next)
126 }
127 General behaviour in scalar and void contexts
129 In a scalar context, the extracted string is returned, having first
130 been removed from the input text. Thus, the following code also
131 processes each quote-like operation, but actually removes them from
132 $text:
133 while ( $next = extract_quotelike($text) )
135 {
136 # process next quote-like (in $next)
137 }
138 Note that if the input text is a read-only string (i.e. a literal), no
140 attempt is made to remove the extracted text.
141 In a void context the behaviour of the extraction subroutines is
143 exactly the same as in a scalar context, except (of course) that the
144 extracted substring is not returned.
145 A note about prefixes
147 Prefix patterns are matched without any trailing modifiers ("/gimsox"
148 etc.) This can bite you if you're expecting a prefix specification
149 like '.*?(?=<H1>)' to skip everything up to the first <H1> tag. Such a
150 prefix pattern will only succeed if the <H1> tag is on the current
151 line, since . normally doesn't match newlines.
152 To overcome this limitation, you need to turn on /s matching within the
154 prefix pattern, using the "(?s)" directive: '(?s).*?(?=<H1>)'
155 "extract_delimited"
157 The "extract_delimited" function formalizes the common idiom of
158 extracting a single-character-delimited substring from the start of a
159 string. For example, to extract a single-quote delimited string, the
160 following code is typically used:
161 ($remainder = $text) =~ s/\A('(\\.|[^'])*')//s;
163 $extracted = $1;
164 but with "extract_delimited" it can be simplified to:
166 ($extracted,$remainder) = extract_delimited($text, "'");
168 "extract_delimited" takes up to four scalars (the input text, the
170 delimiters, a prefix pattern to be skipped, and any escape characters)
171 and extracts the initial substring of the text that is appropriately
172 delimited. If the delimiter string has multiple characters, the first
173 one encountered in the text is taken to delimit the substring. The
174 third argument specifies a prefix pattern that is to be skipped (but
175 must be present!) before the substring is extracted. The final
176 argument specifies the escape character to be used for each delimiter.
177 All arguments are optional. If the escape characters are not specified,
179 every delimiter is escaped with a backslash ("\"). If the prefix is
180 not specified, the pattern '\s*' - optional whitespace - is used. If
181 the delimiter set is also not specified, the set "/["'`]/" is used. If
182 the text to be processed is not specified either, $_ is used.
183 In list context, "extract_delimited" returns a array of three elements,
185 the extracted substring (including the surrounding delimiters), the
186 remainder of the text, and the skipped prefix (if any). If a suitable
187 delimited substring is not found, the first element of the array is the
188 empty string, the second is the complete original text, and the prefix
189 returned in the third element is an empty string.
190 In a scalar context, just the extracted substring is returned. In a
192 void context, the extracted substring (and any prefix) are simply
193 removed from the beginning of the first argument.
194 Examples:
196 # Remove a single-quoted substring from the very beginning of $text:
198 $substring = extract_delimited($text, "'", '');
200 # Remove a single-quoted Pascalish substring (i.e. one in which
202 # doubling the quote character escapes it) from the very
203 # beginning of $text:
204 $substring = extract_delimited($text, "'", '', "'");
206 # Extract a single- or double- quoted substring from the
208 # beginning of $text, optionally after some whitespace
209 # (note the list context to protect $text from modification):
210 ($substring) = extract_delimited $text, q{"'};
212 # Delete the substring delimited by the first '/' in $text:
214 $text = join '', (extract_delimited($text,'/','[^/]*')[2,1];
216 Note that this last example is not the same as deleting the first
218 quote-like pattern. For instance, if $text contained the string:
219 "if ('./cmd' =~ m/$UNIXCMD/s) { $cmd = $1; }"
221 then after the deletion it would contain:
223 "if ('.$UNIXCMD/s) { $cmd = $1; }"
225 not:
227 "if ('./cmd' =~ ms) { $cmd = $1; }"
229 See "extract_quotelike" for a (partial) solution to this problem.
231 "extract_bracketed"
233 Like "extract_delimited", the "extract_bracketed" function takes up to
234 three optional scalar arguments: a string to extract from, a delimiter
235 specifier, and a prefix pattern. As before, a missing prefix defaults
236 to optional whitespace and a missing text defaults to $_. However, a
237 missing delimiter specifier defaults to '{}()[]<>' (see below).
238 "extract_bracketed" extracts a balanced-bracket-delimited substring
240 (using any one (or more) of the user-specified delimiter brackets:
241 '(..)', '{..}', '[..]', or '<..>'). Optionally it will also respect
242 quoted unbalanced brackets (see below).
243 A "delimiter bracket" is a bracket in list of delimiters passed as
245 "extract_bracketed"'s second argument. Delimiter brackets are specified
246 by giving either the left or right (or both!) versions of the required
247 bracket(s). Note that the order in which two or more delimiter brackets
248 are specified is not significant.
249 A "balanced-bracket-delimited substring" is a substring bounded by
251 matched brackets, such that any other (left or right) delimiter bracket
252 within the substring is also matched by an opposite (right or left)
253 delimiter bracket at the same level of nesting. Any type of bracket not
254 in the delimiter list is treated as an ordinary character.
255 In other words, each type of bracket specified as a delimiter must be
257 balanced and correctly nested within the substring, and any other kind
258 of ("non-delimiter") bracket in the substring is ignored.
259 For example, given the string:
261 $text = "{ an '[irregularly :-(] {} parenthesized >:-)' string }";
263 then a call to "extract_bracketed" in a list context:
265 @result = extract_bracketed( $text, '{}' );
267 would return:
269 ( "{ an '[irregularly :-(] {} parenthesized >:-)' string }" , "" , "" )
271 since both sets of '{..}' brackets are properly nested and evenly
273 balanced. (In a scalar context just the first element of the array
274 would be returned. In a void context, $text would be replaced by an
275 empty string.)
276 Likewise the call in:
278 @result = extract_bracketed( $text, '{[' );
280 would return the same result, since all sets of both types of specified
282 delimiter brackets are correctly nested and balanced.
283 However, the call in:
285 @result = extract_bracketed( $text, '{([<' );
287 would fail, returning:
289 ( undef , "{ an '[irregularly :-(] {} parenthesized >:-)' string }" );
291 because the embedded pairs of '(..)'s and '[..]'s are "cross-nested"
293 and the embedded '>' is unbalanced. (In a scalar context, this call
294 would return an empty string. In a void context, $text would be
295 unchanged.)
296 Note that the embedded single-quotes in the string don't help in this
298 case, since they have not been specified as acceptable delimiters and
299 are therefore treated as non-delimiter characters (and ignored).
300 However, if a particular species of quote character is included in the
302 delimiter specification, then that type of quote will be correctly
303 handled. for example, if $text is:
304 $text = '<A HREF=">>>>">link</A>';
306 then
308 @result = extract_bracketed( $text, '<">' );
310 returns:
312 ( '<A HREF=">>>>">', 'link</A>', "" )
314 as expected. Without the specification of """ as an embedded quoter:
316 @result = extract_bracketed( $text, '<>' );
318 the result would be:
320 ( '<A HREF=">', '>>>">link</A>', "" )
322 In addition to the quote delimiters "'", """, and "`", full Perl quote-
324 like quoting (i.e. q{string}, qq{string}, etc) can be specified by
325 including the letter 'q' as a delimiter. Hence:
326 @result = extract_bracketed( $text, '<q>' );
328 would correctly match something like this:
330 $text = '<leftop: conj /and/ conj>';
332 See also: "extract_quotelike" and "extract_codeblock".
334 "extract_variable"
336 "extract_variable" extracts any valid Perl variable or variable-
337 involved expression, including scalars, arrays, hashes, array accesses,
338 hash look-ups, method calls through objects, subroutine calls through
339 subroutine references, etc.
340 The subroutine takes up to two optional arguments:
342 1. A string to be processed ($_ if the string is omitted or "undef")
344 2. A string specifying a pattern to be matched as a prefix (which is
346 to be skipped). If omitted, optional whitespace is skipped.
347 On success in a list context, an array of 3 elements is returned. The
349 elements are:
350 [0] the extracted variable, or variablish expression
352 [1] the remainder of the input text,
354 [2] the prefix substring (if any),
356 On failure, all of these values (except the remaining text) are
358 "undef".
359 In a scalar context, "extract_variable" returns just the complete
361 substring that matched a variablish expression. "undef" is returned on
362 failure. In addition, the original input text has the returned
363 substring (and any prefix) removed from it.
364 In a void context, the input text just has the matched substring (and
366 any specified prefix) removed.
367 "extract_tagged"
369 "extract_tagged" extracts and segments text between (balanced)
370 specified tags.
371 The subroutine takes up to five optional arguments:
373 1. A string to be processed ($_ if the string is omitted or "undef")
375 2. A string specifying a pattern to be matched as the opening tag. If
377 the pattern string is omitted (or "undef") then a pattern that
378 matches any standard XML tag is used.
379 3. A string specifying a pattern to be matched at the closing tag. If
381 the pattern string is omitted (or "undef") then the closing tag is
382 constructed by inserting a "/" after any leading bracket characters
383 in the actual opening tag that was matched (not the pattern that
384 matched the tag). For example, if the opening tag pattern is
385 specified as '{{\w+}}' and actually matched the opening tag
386 "{{DATA}}", then the constructed closing tag would be "{{/DATA}}".
387 4. A string specifying a pattern to be matched as a prefix (which is
389 to be skipped). If omitted, optional whitespace is skipped.
390 5. A hash reference containing various parsing options (see below)
392 The various options that can be specified are:
394 "reject => $listref"
396 The list reference contains one or more strings specifying patterns
397 that must not appear within the tagged text.
398 For example, to extract an HTML link (which should not contain
400 nested links) use:
401 extract_tagged($text, '<A>', '</A>', undef, {reject => ['<A>']} );
403 "ignore => $listref"
405 The list reference contains one or more strings specifying patterns
406 that are not be be treated as nested tags within the tagged text
407 (even if they would match the start tag pattern).
408 For example, to extract an arbitrary XML tag, but ignore "empty"
410 elements:
411 extract_tagged($text, undef, undef, undef, {ignore => ['<[^>]*/>']} );
413 (also see "gen_delimited_pat" below).
415 "fail => $str"
417 The "fail" option indicates the action to be taken if a matching
418 end tag is not encountered (i.e. before the end of the string or
419 some "reject" pattern matches). By default, a failure to match a
420 closing tag causes "extract_tagged" to immediately fail.
421 However, if the string value associated with <reject> is "MAX",
423 then "extract_tagged" returns the complete text up to the point of
424 failure. If the string is "PARA", "extract_tagged" returns only
425 the first paragraph after the tag (up to the first line that is
426 either empty or contains only whitespace characters). If the
427 string is "", the the default behaviour (i.e. failure) is
428 reinstated.
429 For example, suppose the start tag "/para" introduces a paragraph,
431 which then continues until the next "/endpara" tag or until another
432 "/para" tag is encountered:
433 $text = "/para line 1\n\nline 3\n/para line 4";
435 extract_tagged($text, '/para', '/endpara', undef,
437 {reject => '/para', fail => MAX );
438 # EXTRACTED: "/para line 1\n\nline 3\n"
440 Suppose instead, that if no matching "/endpara" tag is found, the
442 "/para" tag refers only to the immediately following paragraph:
443 $text = "/para line 1\n\nline 3\n/para line 4";
445 extract_tagged($text, '/para', '/endpara', undef,
447 {reject => '/para', fail => MAX );
448 # EXTRACTED: "/para line 1\n"
450 Note that the specified "fail" behaviour applies to nested tags as
452 well.
453 On success in a list context, an array of 6 elements is returned. The
455 elements are:
456 [0] the extracted tagged substring (including the outermost tags),
458 [1] the remainder of the input text,
460 [2] the prefix substring (if any),
462 [3] the opening tag
464 [4] the text between the opening and closing tags
466 [5] the closing tag (or "" if no closing tag was found)
468 On failure, all of these values (except the remaining text) are
470 "undef".
471 In a scalar context, "extract_tagged" returns just the complete
473 substring that matched a tagged text (including the start and end
474 tags). "undef" is returned on failure. In addition, the original input
475 text has the returned substring (and any prefix) removed from it.
476 In a void context, the input text just has the matched substring (and
478 any specified prefix) removed.
479 "gen_extract_tagged"
481 (Note: This subroutine is only available under Perl5.005)
482 "gen_extract_tagged" generates a new anonymous subroutine which
484 extracts text between (balanced) specified tags. In other words, it
485 generates a function identical in function to "extract_tagged".
486 The difference between "extract_tagged" and the anonymous subroutines
488 generated by "gen_extract_tagged", is that those generated subroutines:
489 · do not have to reparse tag specification or parsing options every
491 time they are called (whereas "extract_tagged" has to effectively
492 rebuild its tag parser on every call);
493 · make use of the new qr// construct to pre-compile the regexes they
495 use (whereas "extract_tagged" uses standard string variable
496 interpolation to create tag-matching patterns).
497 The subroutine takes up to four optional arguments (the same set as
499 "extract_tagged" except for the string to be processed). It returns a
500 reference to a subroutine which in turn takes a single argument (the
501 text to be extracted from).
502 In other words, the implementation of "extract_tagged" is exactly
504 equivalent to:
505 sub extract_tagged
507 {
508 my $text = shift;
509 $extractor = gen_extract_tagged(@_);
510 return $extractor->($text);
511 }
512 (although "extract_tagged" is not currently implemented that way, in
514 order to preserve pre-5.005 compatibility).
515 Using "gen_extract_tagged" to create extraction functions for specific
517 tags is a good idea if those functions are going to be called more than
518 once, since their performance is typically twice as good as the more
519 general-purpose "extract_tagged".
520 "extract_quotelike"
522 "extract_quotelike" attempts to recognize, extract, and segment any one
523 of the various Perl quotes and quotelike operators (see perlop(3))
524 Nested backslashed delimiters, embedded balanced bracket delimiters
525 (for the quotelike operators), and trailing modifiers are all caught.
526 For example, in:
527 extract_quotelike 'q # an octothorpe: \# (not the end of the q!) #'
529 extract_quotelike ' "You said, \"Use sed\"." '
531 extract_quotelike ' s{([A-Z]{1,8}\.[A-Z]{3})} /\L$1\E/; '
533 extract_quotelike ' tr/\\\/\\\\/\\\//ds; '
535 the full Perl quotelike operations are all extracted correctly.
537 Note too that, when using the /x modifier on a regex, any comment
539 containing the current pattern delimiter will cause the regex to be
540 immediately terminated. In other words:
541 'm /
543 (?i) # CASE INSENSITIVE
544 [a-z_] # LEADING ALPHABETIC/UNDERSCORE
545 [a-z0-9]* # FOLLOWED BY ANY NUMBER OF ALPHANUMERICS
546 /x'
547 will be extracted as if it were:
549 'm /
551 (?i) # CASE INSENSITIVE
552 [a-z_] # LEADING ALPHABETIC/'
553 This behaviour is identical to that of the actual compiler.
555 "extract_quotelike" takes two arguments: the text to be processed and a
557 prefix to be matched at the very beginning of the text. If no prefix is
558 specified, optional whitespace is the default. If no text is given, $_
559 is used.
560 In a list context, an array of 11 elements is returned. The elements
562 are:
563 [0] the extracted quotelike substring (including trailing modifiers),
565 [1] the remainder of the input text,
567 [2] the prefix substring (if any),
569 [3] the name of the quotelike operator (if any),
571 [4] the left delimiter of the first block of the operation,
573 [5] the text of the first block of the operation (that is, the contents
575 of a quote, the regex of a match or substitution or the target list
576 of a translation),
577 [6] the right delimiter of the first block of the operation,
579 [7] the left delimiter of the second block of the operation (that is,
581 if it is a "s", "tr", or "y"),
582 [8] the text of the second block of the operation (that is, the
584 replacement of a substitution or the translation list of a
585 translation),
586 [9] the right delimiter of the second block of the operation (if any),
588 [10]
590 the trailing modifiers on the operation (if any).
591 For each of the fields marked "(if any)" the default value on success
593 is an empty string. On failure, all of these values (except the
594 remaining text) are "undef".
595 In a scalar context, "extract_quotelike" returns just the complete
597 substring that matched a quotelike operation (or "undef" on failure).
598 In a scalar or void context, the input text has the same substring (and
599 any specified prefix) removed.
600 Examples:
602 # Remove the first quotelike literal that appears in text
604 $quotelike = extract_quotelike($text,'.*?');
606 # Replace one or more leading whitespace-separated quotelike
608 # literals in $_ with "<QLL>"
609 do { $_ = join '<QLL>', (extract_quotelike)[2,1] } until $@;
611 # Isolate the search pattern in a quotelike operation from $text
614 ($op,$pat) = (extract_quotelike $text)[3,5];
616 if ($op =~ /[ms]/)
617 {
618 print "search pattern: $pat\n";
619 }
620 else
621 {
622 print "$op is not a pattern matching operation\n";
623 }
624 "extract_quotelike" and "here documents"
626 "extract_quotelike" can successfully extract "here documents" from an
627 input string, but with an important caveat in list contexts.
628 Unlike other types of quote-like literals, a here document is rarely a
630 contiguous substring. For example, a typical piece of code using here
631 document might look like this:
632 <<'EOMSG' || die;
634 This is the message.
635 EOMSG
636 exit;
637 Given this as an input string in a scalar context, "extract_quotelike"
639 would correctly return the string "<<'EOMSG'\nThis is the
640 message.\nEOMSG", leaving the string " || die;\nexit;" in the original
641 variable. In other words, the two separate pieces of the here document
642 are successfully extracted and concatenated.
643 In a list context, "extract_quotelike" would return the list
645 [0] "<<'EOMSG'\nThis is the message.\nEOMSG\n" (i.e. the full extracted
647 here document, including fore and aft delimiters),
648 [1] " || die;\nexit;" (i.e. the remainder of the input text,
650 concatenated),
651 [2] "" (i.e. the prefix substring -- trivial in this case),
653 [3] "<<" (i.e. the "name" of the quotelike operator)
655 [4] "'EOMSG'" (i.e. the left delimiter of the here document, including
657 any quotes),
658 [5] "This is the message.\n" (i.e. the text of the here document),
660 [6] "EOMSG" (i.e. the right delimiter of the here document),
662 [7..10]
664 "" (a here document has no second left delimiter, second text,
665 second right delimiter, or trailing modifiers).
666 However, the matching position of the input variable would be set to
668 "exit;" (i.e. after the closing delimiter of the here document), which
669 would cause the earlier " || die;\nexit;" to be skipped in any sequence
670 of code fragment extractions.
671 To avoid this problem, when it encounters a here document whilst
673 extracting from a modifiable string, "extract_quotelike" silently
674 rearranges the string to an equivalent piece of Perl:
675 <<'EOMSG'
677 This is the message.
678 EOMSG
679 || die;
680 exit;
681 in which the here document is contiguous. It still leaves the matching
683 position after the here document, but now the rest of the line on which
684 the here document starts is not skipped.
685 To prevent <extract_quotelike> from mucking about with the input in
687 this way (this is the only case where a list-context
688 "extract_quotelike" does so), you can pass the input variable as an
689 interpolated literal:
690 $quotelike = extract_quotelike("$var");
692 "extract_codeblock"
694 "extract_codeblock" attempts to recognize and extract a balanced
695 bracket delimited substring that may contain unbalanced brackets inside
696 Perl quotes or quotelike operations. That is, "extract_codeblock" is
697 like a combination of "extract_bracketed" and "extract_quotelike".
698 "extract_codeblock" takes the same initial three parameters as
700 "extract_bracketed": a text to process, a set of delimiter brackets to
701 look for, and a prefix to match first. It also takes an optional fourth
702 parameter, which allows the outermost delimiter brackets to be
703 specified separately (see below).
704 Omitting the first argument (input text) means process $_ instead.
706 Omitting the second argument (delimiter brackets) indicates that only
707 '{' is to be used. Omitting the third argument (prefix argument)
708 implies optional whitespace at the start. Omitting the fourth argument
709 (outermost delimiter brackets) indicates that the value of the second
710 argument is to be used for the outermost delimiters.
711 Once the prefix an dthe outermost opening delimiter bracket have been
713 recognized, code blocks are extracted by stepping through the input
714 text and trying the following alternatives in sequence:
715 1. Try and match a closing delimiter bracket. If the bracket was the
717 same species as the last opening bracket, return the substring to
718 that point. If the bracket was mismatched, return an error.
719 2. Try to match a quote or quotelike operator. If found, call
721 "extract_quotelike" to eat it. If "extract_quotelike" fails, return
722 the error it returned. Otherwise go back to step 1.
723 3. Try to match an opening delimiter bracket. If found, call
725 "extract_codeblock" recursively to eat the embedded block. If the
726 recursive call fails, return an error. Otherwise, go back to step
727 1.
728 4. Unconditionally match a bareword or any other single character, and
730 then go back to step 1.
731 Examples:
733 # Find a while loop in the text
735 if ($text =~ s/.*?while\s*\{/{/)
737 {
738 $loop = "while " . extract_codeblock($text);
739 }
740 # Remove the first round-bracketed list (which may include
742 # round- or curly-bracketed code blocks or quotelike operators)
743 extract_codeblock $text, "(){}", '[^(]*';
745 The ability to specify a different outermost delimiter bracket is
747 useful in some circumstances. For example, in the Parse::RecDescent
748 module, parser actions which are to be performed only on a successful
749 parse are specified using a "<defer:...>" directive. For example:
750 sentence: subject verb object
752 <defer: {$::theVerb = $item{verb}} >
753 Parse::RecDescent uses "extract_codeblock($text, '{}<>')" to extract
755 the code within the "<defer:...>" directive, but there's a problem.
756 A deferred action like this:
758 <defer: {if ($count>10) {$count--}} >
760 will be incorrectly parsed as:
762 <defer: {if ($count>
764 because the "less than" operator is interpreted as a closing delimiter.
766 But, by extracting the directive using
768 "extract_codeblock($text, '{}', undef, '<>')" the '>' character is only
769 treated as a delimited at the outermost level of the code block, so the
770 directive is parsed correctly.
771 "extract_multiple"
773 The "extract_multiple" subroutine takes a string to be processed and a
774 list of extractors (subroutines or regular expressions) to apply to
775 that string.
776 In an array context "extract_multiple" returns an array of substrings
778 of the original string, as extracted by the specified extractors. In a
779 scalar context, "extract_multiple" returns the first substring
780 successfully extracted from the original string. In both scalar and
781 void contexts the original string has the first successfully extracted
782 substring removed from it. In all contexts "extract_multiple" starts at
783 the current "pos" of the string, and sets that "pos" appropriately
784 after it matches.
785 Hence, the aim of of a call to "extract_multiple" in a list context is
787 to split the processed string into as many non-overlapping fields as
788 possible, by repeatedly applying each of the specified extractors to
789 the remainder of the string. Thus "extract_multiple" is a generalized
790 form of Perl's "split" subroutine.
791 The subroutine takes up to four optional arguments:
793 1. A string to be processed ($_ if the string is omitted or "undef")
795 2. A reference to a list of subroutine references and/or qr// objects
797 and/or literal strings and/or hash references, specifying the
798 extractors to be used to split the string. If this argument is
799 omitted (or "undef") the list:
800 [
802 sub { extract_variable($_[0], '') },
803 sub { extract_quotelike($_[0],'') },
804 sub { extract_codeblock($_[0],'{}','') },
805 ]
806 is used.
808 3. An number specifying the maximum number of fields to return. If
810 this argument is omitted (or "undef"), split continues as long as
811 possible.
812 If the third argument is N, then extraction continues until N
814 fields have been successfully extracted, or until the string has
815 been completely processed.
816 Note that in scalar and void contexts the value of this argument is
818 automatically reset to 1 (under "-w", a warning is issued if the
819 argument has to be reset).
820 4. A value indicating whether unmatched substrings (see below) within
822 the text should be skipped or returned as fields. If the value is
823 true, such substrings are skipped. Otherwise, they are returned.
824 The extraction process works by applying each extractor in sequence to
826 the text string.
827 If the extractor is a subroutine it is called in a list context and is
829 expected to return a list of a single element, namely the extracted
830 text. It may optionally also return two further arguments: a string
831 representing the text left after extraction (like $' for a pattern
832 match), and a string representing any prefix skipped before the
833 extraction (like $` in a pattern match). Note that this is designed to
834 facilitate the use of other Text::Balanced subroutines with
835 "extract_multiple". Note too that the value returned by an extractor
836 subroutine need not bear any relationship to the corresponding
837 substring of the original text (see examples below).
838 If the extractor is a precompiled regular expression or a string, it is
840 matched against the text in a scalar context with a leading '\G' and
841 the gc modifiers enabled. The extracted value is either $1 if that
842 variable is defined after the match, or else the complete match (i.e.
843 $&).
844 If the extractor is a hash reference, it must contain exactly one
846 element. The value of that element is one of the above extractor types
847 (subroutine reference, regular expression, or string). The key of that
848 element is the name of a class into which the successful return value
849 of the extractor will be blessed.
850 If an extractor returns a defined value, that value is immediately
852 treated as the next extracted field and pushed onto the list of fields.
853 If the extractor was specified in a hash reference, the field is also
854 blessed into the appropriate class,
855 If the extractor fails to match (in the case of a regex extractor), or
857 returns an empty list or an undefined value (in the case of a
858 subroutine extractor), it is assumed to have failed to extract. If
859 none of the extractor subroutines succeeds, then one character is
860 extracted from the start of the text and the extraction subroutines
861 reapplied. Characters which are thus removed are accumulated and
862 eventually become the next field (unless the fourth argument is true,
863 in which case they are discarded).
864 For example, the following extracts substrings that are valid Perl
866 variables:
867 @fields = extract_multiple($text,
869 [ sub { extract_variable($_[0]) } ],
870 undef, 1);
871 This example separates a text into fields which are quote delimited,
873 curly bracketed, and anything else. The delimited and bracketed parts
874 are also blessed to identify them (the "anything else" is unblessed):
875 @fields = extract_multiple($text,
877 [
878 { Delim => sub { extract_delimited($_[0],q{'"}) } },
879 { Brack => sub { extract_bracketed($_[0],'{}') } },
880 ]);
881 This call extracts the next single substring that is a valid Perl
883 quotelike operator (and removes it from $text):
884 $quotelike = extract_multiple($text,
886 [
887 sub { extract_quotelike($_[0]) },
888 ], undef, 1);
889 Finally, here is yet another way to do comma-separated value parsing:
891 @fields = extract_multiple($csv_text,
893 [
894 sub { extract_delimited($_[0],q{'"}) },
895 qr/([^,]+)(.*)/,
896 ],
897 undef,1);
898 The list in the second argument means: "Try and extract a ' or "
900 delimited string, otherwise extract anything up to a comma...". The
901 undef third argument means: "...as many times as possible...", and the
902 true value in the fourth argument means "...discarding anything else
903 that appears (i.e. the commas)".
904 If you wanted the commas preserved as separate fields (i.e. like split
906 does if your split pattern has capturing parentheses), you would just
907 make the last parameter undefined (or remove it).
908 "gen_delimited_pat"
910 The "gen_delimited_pat" subroutine takes a single (string) argument and
911 > builds a Friedl-style optimized regex that matches a string
912 delimited by any one of the characters in the single argument. For
913 example:
914 gen_delimited_pat(q{'"})
916 returns the regex:
918 (?:\"(?:\\\"|(?!\").)*\"|\'(?:\\\'|(?!\').)*\')
920 Note that the specified delimiters are automatically quotemeta'd.
922 A typical use of "gen_delimited_pat" would be to build special purpose
924 tags for "extract_tagged". For example, to properly ignore "empty" XML
925 elements (which might contain quoted strings):
926 my $empty_tag = '<(' . gen_delimited_pat(q{'"}) . '|.)+/>';
928 extract_tagged($text, undef, undef, undef, {ignore => [$empty_tag]} );
930 "gen_delimited_pat" may also be called with an optional second
932 argument, which specifies the "escape" character(s) to be used for each
933 delimiter. For example to match a Pascal-style string (where ' is the
934 delimiter and '' is a literal ' within the string):
935 gen_delimited_pat(q{'},q{'});
937 Different escape characters can be specified for different delimiters.
939 For example, to specify that '/' is the escape for single quotes and
940 '%' is the escape for double quotes:
941 gen_delimited_pat(q{'"},q{/%});
943 If more delimiters than escape chars are specified, the last escape
945 char is used for the remaining delimiters. If no escape char is
946 specified for a given specified delimiter, '\' is used.
947 "delimited_pat"
949 Note that "gen_delimited_pat" was previously called "delimited_pat".
950 That name may still be used, but is now deprecated.
951
953 In a list context, all the functions return "(undef,$original_text)" on
954 failure. In a scalar context, failure is indicated by returning "undef"
955 (in this case the input text is not modified in any way).
956 In addition, on failure in any context, the $@ variable is set.
958 Accessing "$@->{error}" returns one of the error diagnostics listed
959 below. Accessing "$@->{pos}" returns the offset into the original
960 string at which the error was detected (although not necessarily where
961 it occurred!) Printing $@ directly produces the error message, with
962 the offset appended. On success, the $@ variable is guaranteed to be
963 "undef".
964 The available diagnostics are:
966 "Did not find a suitable bracket: "%s""
968 The delimiter provided to "extract_bracketed" was not one of
969 '()[]<>{}'.
970 "Did not find prefix: /%s/"
972 A non-optional prefix was specified but wasn't found at the start
973 of the text.
974 "Did not find opening bracket after prefix: "%s""
976 "extract_bracketed" or "extract_codeblock" was expecting a
977 particular kind of bracket at the start of the text, and didn't
978 find it.
979 "No quotelike operator found after prefix: "%s""
981 "extract_quotelike" didn't find one of the quotelike operators "q",
982 "qq", "qw", "qx", "s", "tr" or "y" at the start of the substring it
983 was extracting.
984 "Unmatched closing bracket: "%c""
986 "extract_bracketed", "extract_quotelike" or "extract_codeblock"
987 encountered a closing bracket where none was expected.
988 "Unmatched opening bracket(s): "%s""
990 "extract_bracketed", "extract_quotelike" or "extract_codeblock" ran
991 out of characters in the text before closing one or more levels of
992 nested brackets.
993 "Unmatched embedded quote (%s)"
995 "extract_bracketed" attempted to match an embedded quoted
996 substring, but failed to find a closing quote to match it.
997 "Did not find closing delimiter to match '%s'"
999 "extract_quotelike" was unable to find a closing delimiter to match
1000 the one that opened the quote-like operation.
1001 "Mismatched closing bracket: expected "%c" but found "%s""
1003 "extract_bracketed", "extract_quotelike" or "extract_codeblock"
1004 found a valid bracket delimiter, but it was the wrong species. This
1005 usually indicates a nesting error, but may indicate incorrect
1006 quoting or escaping.
1007 "No block delimiter found after quotelike "%s""
1009 "extract_quotelike" or "extract_codeblock" found one of the
1010 quotelike operators "q", "qq", "qw", "qx", "s", "tr" or "y" without
1011 a suitable block after it.
1012 "Did not find leading dereferencer"
1014 "extract_variable" was expecting one of '$', '@', or '%' at the
1015 start of a variable, but didn't find any of them.
1016 "Bad identifier after dereferencer"
1018 "extract_variable" found a '$', '@', or '%' indicating a variable,
1019 but that character was not followed by a legal Perl identifier.
1020 "Did not find expected opening bracket at %s"
1022 "extract_codeblock" failed to find any of the outermost opening
1023 brackets that were specified.
1024 "Improperly nested codeblock at %s"
1026 A nested code block was found that started with a delimiter that
1027 was specified as being only to be used as an outermost bracket.
1028 "Missing second block for quotelike "%s""
1030 "extract_codeblock" or "extract_quotelike" found one of the
1031 quotelike operators "s", "tr" or "y" followed by only one block.
1032 "No match found for opening bracket"
1034 "extract_codeblock" failed to find a closing bracket to match the
1035 outermost opening bracket.
1036 "Did not find opening tag: /%s/"
1038 "extract_tagged" did not find a suitable opening tag (after any
1039 specified prefix was removed).
1040 "Unable to construct closing tag to match: /%s/"
1042 "extract_tagged" matched the specified opening tag and tried to
1043 modify the matched text to produce a matching closing tag (because
1044 none was specified). It failed to generate the closing tag, almost
1045 certainly because the opening tag did not start with a bracket of
1046 some kind.
1047 "Found invalid nested tag: %s"
1049 "extract_tagged" found a nested tag that appeared in the "reject"
1050 list (and the failure mode was not "MAX" or "PARA").
1051 "Found unbalanced nested tag: %s"
1053 "extract_tagged" found a nested opening tag that was not matched by
1054 a corresponding nested closing tag (and the failure mode was not
1055 "MAX" or "PARA").
1056 "Did not find closing tag"
1058 "extract_tagged" reached the end of the text without finding a
1059 closing tag to match the original opening tag (and the failure mode
1060 was not "MAX" or "PARA").
1061
1063 Damian Conway (damian@conway.org)
1064
1066 There are undoubtedly serious bugs lurking somewhere in this code, if
1067 only because parts of it give the impression of understanding a great
1068 deal more about Perl than they really do.
1069 Bug reports and other feedback are most welcome.
1071
1073 Copyright 1997 - 2001 Damian Conway. All Rights Reserved.
1074 Some (minor) parts copyright 2009 Adam Kennedy.
1076 This module is free software. It may be used, redistributed and/or
1078 modified under the same terms as Perl itself.
1079perl v5.16.3 2013-02-26 Text::Balanced(3pm)