1PPIx::Regexp::TokenizerU(s3e)r Contributed Perl DocumentaPtPiIoxn::Regexp::Tokenizer(3)
2
6 PPIx::Regexp::Tokenizer - Tokenize a regular expression
7
9 use PPIx::Regexp::Dumper;
10 PPIx::Regexp::Dumper->new( 'qr{foo}smx' )
11 ->print();
12
14 "PPIx::Regexp::Tokenizer" is a PPIx::Regexp::Support.
15 "PPIx::Regexp::Tokenizer" has no descendants.
17
19 This class provides tokenization of the regular expression.
20
22 This class provides the following public methods. Methods not
23 documented here (or documented below under "EXTERNAL TOKENIZERS") are
24 private, and unsupported in the sense that the author reserves the
25 right to change or remove them without notice.
26 new
28 my $tokenizer = PPIx::Regexp::Tokenizer->new( 'xyzzy' );
29 This static method instantiates the tokenizer. You must pass it the
31 regular expression to be parsed, either as a string or as a
32 PPI::Element of some sort. You can also pass optional name/value pairs
33 of arguments. The option names are specified without a leading dash.
34 Supported options are:
35 default_modifiers array_reference
37 This argument specifies default statement modifiers. It is
38 optional, but if specified must be an array reference. See the
39 PPIx::Regexp new() documentation for the details.
40 encoding name
42 This option specifies the encoding of the string to be tokenized.
43 If specified, an "Encode::decode" is done on the string (or the
44 "content" of the PPI class) before it is tokenized.
45 index_locations
47 This Boolean option specifies that the locations of the generated
48 tokens are to be computed.
49 postderef boolean
51 THIS ARGUMENT IS DEPRECATED. See DEPRECATION NOTICE in
52 PPIx::Regexp for the details.
53 This option specifies whether the tokenizer recognizes postfix
55 dereferencing. See the PPIx::Regexp new() documentation for the
56 details.
57 $PPIx::Regexp::Tokenizer::DEFAULT_POSTDEREF is not exported.
59 strict boolean
61 This option specifies whether tokenization should assume "use re
62 'strict';" is in effect.
63 The 'strict' pragma was introduced in Perl 5.22, and its
65 documentation says that it is experimental, and that there is no
66 commitment to backward compatibility. The same applies to the
67 tokenization produced when this option is asserted.
68 trace number
70 Specifying a positive value for this option causes a trace of the
71 tokenization. This option is unsupported in the sense that the
72 author reserves the right to alter it without notice.
73 If this option is unspecified, the value comes from environment
75 variable "PPIX_REGEXP_TOKENIZER_TRACE" (see "ENVIRONMENT
76 VARIABLES"). If this environment variable does not exist, the
77 default is 0.
78 Undocumented options are unsupported.
80 The returned value is the instantiated tokenizer, or "undef" if
82 instantiation failed. In the latter case a call to "errstr" will return
83 the reason.
84 content
86 print $tokenizer->content();
87 This method returns the string being tokenized. This will be the result
89 of the PPI::Element->content() method if the object was instantiated
90 with a PPI::Element.
91 default_modifiers
93 print join ', ', @{ $tokenizer->default_modifiers() };
94 This method returns a reference to a copy of the array passed to the
96 "default_modifiers" argument to new(). If this argument was not used to
97 instantiate the object, the return is a reference to an empty array.
98 encoding
100 This method returns the encoding of the data being parsed, if one was
101 set when the class was instantiated; otherwise it simply returns undef.
102 errstr
104 my $tokenizer = PPIx::Regexp::Tokenizer->new( 'xyzzy' )
105 or die PPIx::Regexp::Tokenizer->errstr();
106 This static method returns an error description if tokenizer
108 instantiation failed.
109 failures
111 print $tokenizer->failures(), " tokenization failures\n";
112 This method returns the number of tokenization failures encountered. A
114 tokenization failure is represented in the output token stream by a
115 PPIx::Regexp::Token::Unknown.
116 modifier
118 $tokenizer->modifier( 'x' )
119 and print "Tokenizing an extended regular expression\n";
120 This method returns true if the given modifier character was found on
122 the end of the regular expression, and false otherwise.
123 Starting with version 0.036_01, if the argument is a single-character
125 modifier followed by an asterisk (intended as a wild card character),
126 the return is the number of times that modifier appears. In this case
127 an exception will be thrown if you specify a multi-character modifier
128 (e.g. 'ee*'), or if you specify one of the match semantics modifiers
129 (e.g. 'a*').
130 If called by an external tokenizer, this method returns true if if the
132 given modifier was true at the current point in the tokenization.
133 next_token
135 my $token = $tokenizer->next_token();
136 This method returns the next token in the token stream, or nothing if
138 there are no more tokens.
139 significant
141 This method exists simply for the convenience of PPIx::Regexp::Dumper.
142 It always returns true.
143 tokens
145 my @tokens = $tokenizer->tokens();
146 This method returns all remaining tokens in the token stream.
148
150 This class does very little of its own tokenization. Instead the token
151 classes contain external tokenization routines, whose name is
152 '__PPIX_TOKENIZER__' concatenated with the current mode of the
153 tokenizer ('regexp' for regular expressions, 'repl' for the replacement
154 string).
155 These external tokenizers are called as static methods, and passed the
157 "PPIx::Regexp::Tokenizer" object and the current character in the
158 character stream.
159 If the external tokenizer wants to make one or more tokens, it returns
161 an array containing either length in characters for tokens of the
162 tokenizer's own class, or the results of one or more "make_token" calls
163 for tokens of an arbitrary class.
164 If the external tokenizer is not interested in the characters starting
166 at the current position it simply returns.
167 The following methods are for the use of external tokenizers, and are
169 not part of the public interface to this class.
170 capture
172 if ( $tokenizer->find_regexp( qr{ \A ( foo ) }smx ) ) {
173 foreach ( $tokenizer->capture() ) {
174 print "$_\n";
175 }
176 }
177 This method returns all the contents of any capture buffers from the
179 previous call to "find_regexp". The first element of the array (i.e.
180 element 0) corresponds to $1, and so on.
181 The captures are cleared by "make_token", as well as by another call to
183 "find_regexp".
184 cookie
186 $tokenizer->cookie( foo => sub { 1 } );
187 my $cookie = $tokenizer->cookie( 'foo' );
188 my $old_hint = $tokenizer->cookie( foo => undef );
189 This method either creates, deletes, or accesses a cookie.
191 A cookie is a code reference which is called whenever the tokenizer
193 makes a token. If it returns a false value, it is deleted. Explicitly
194 setting the cookie to "undef" also deletes it.
195 When you call "$tokenizer->cookie( 'foo' )", the current cookie is
197 returned. If you pass a new value of "undef" to delete the token, the
198 deleted cookie (if any) is returned.
199 When the "make_token" method calls a cookie, it passes it the tokenizer
201 and the token just made. If a token calls a cookie, it is recommended
202 that it merely pass the tokenizer, though of course the token can do
203 whatever it wants.
204 The cookie mechanism seems to be a bit of a crock, but it appeared to
206 be more work to fix things up in the lexer after the tokenizer got
207 something wrong.
208 The recommended way to write a cookie is to use a closure to store any
210 necessary data, and have a call to the cookie return the data;
211 otherwise the ultimate consumer of the cookie has no way to access the
212 data. Of course, it may be that the presence of the cookie at a certain
213 point in the parse is all that is required.
214 expect
216 $tokenizer->expect( 'PPIx::Regexp::Token::Code' );
217 This method inserts a given class at the head of the token scan, for
219 the next iteration only. More than one class can be specified. Class
220 names can be abbreviated by removing the leading 'PPIx::Regexp::'.
221 If no class is specified, this method does nothing.
223 The expectation lasts from the next time "get_token" is called until
225 the next time "make_token" makes a significant token, or until the next
226 "expect" call if that is done sooner.
227 find_regexp
229 my $end = $tokenizer->find_regexp( qr{ \A \w+ }smx );
230 my ( $begin, $end ) = $tokenizer->find_regexp(
231 qr{ \A \w+ }smx );
232 This method finds the given regular expression in the content, starting
234 at the current position. If called in scalar context, the offset from
235 the current position to the end of the matched string is returned. If
236 called in list context, the offsets to both the beginning and the end
237 of the matched string are returned.
238 find_matching_delimiter
240 my $offset = $tokenizer->find_matching_delimiter();
241 This method is used by tokenizers to find the delimiter matching the
243 character at the current position in the content string. If the
244 delimiter is an opening bracket of some sort, bracket nesting will be
245 taken into account.
246 When searching for the matching delimiter, the back slash character is
248 considered to escape the following character, so back-slashed
249 delimiters will be ignored. No other quoting mechanisms are recognized,
250 though, so delimiters inside quotes still count. This is actually the
251 way Perl works, as
252 $ perl -e 'qr<(?{ print "}" })>'
254 demonstrates.
256 This method returns the offset from the current position in the content
258 string to the matching delimiter (which will always be positive), or
259 undef if no match can be found.
260 get_mode
262 This method returns the name of the current mode of the tokenizer.
263 get_start_delimiter
265 my $start_delimiter = $tokenizer->get_start_delimiter();
266 This method is used by tokenizers to access the start delimiter for the
268 regular expression.
269 get_token
271 my $token = $tokenizer->make_token( 3 );
272 my @tokens = $tokenizer->get_token();
273 This method returns the next token that can be made from the input
275 stream. It is not part of the external interface, but is intended for
276 the use of an external tokenizer which calls it after making and
277 retaining its own token to look at the next token ( if any ) in the
278 input stream.
279 If any external tokenizer calls get_token without first calling
281 make_token, a fatal error occurs; this is better than the infinite
282 recursion which would occur if the condition were not trapped.
283 An external tokenizer must return anything returned by get_token;
285 otherwise tokens get lost.
286 interpolates
288 This method returns true if the top-level structure being tokenized
289 interpolates; that is, if the delimiter is not a single quote.
290 make_token
292 return $tokenizer->make_token( 3, 'PPIx::Regexp::Token::Unknown' );
293 This method is used by this class (and possibly by individual
295 tokenizers) to manufacture a token. Its arguments are the number of
296 characters to include in the token, and optionally the class of the
297 token. If no class name is given, the caller's class is used. Class
298 names may be shortened by removing the initial 'PPIx::Regexp::', which
299 will be restored by this method.
300 The token will be manufactured from the given number of characters
302 starting at the current cursor position, which will be adjusted.
303 If the given length would include characters past the end of the string
305 being tokenized, the length is reduced appropriately. If this means a
306 token with no characters, nothing is returned.
307 match
309 if ( $tokenizer->find_regexp( qr{ \A \w+ }smx ) ) {
310 print $tokenizer->match(), "\n";
311 }
312 This method returns the string matched by the previous call to
314 "find_regexp".
315 The match is set to "undef" by "make_token", as well as by another call
317 to "find_regexp".
318 modifier_duplicate
320 $tokenizer->modifier_duplicate();
321 This method duplicates the modifiers on the top of the modifier stack,
323 with the intent of creating a locally-scoped copy of the modifiers.
324 This should only be called by an external tokenizer that is actually
325 creating a modifier scope. In other words, only when creating a
326 PPIx::Regexp::Token::Structure token whose content is '('.
327 modifier_modify
329 $tokenizer->modifier_modify( name => $value ... );
330 This method sets new values for the modifiers in the local scope. Only
332 the modifiers whose names are actually passed have their values
333 changed.
334 This method is intended to be called after manufacturing a
336 PPIx::Regexp::Token::Modifier token, and passed the results of its
337 "modifiers" method.
338 modifier_pop
340 $tokenizer->modifier_pop();
341 This method removes the modifiers on the top of the modifier stack.
343 This should only be called by an external tokenizer that is ending a
344 modifier scope. In other words, only when creating a
345 PPIx::Regexp::Token::Structure token whose content is ')'.
346 Note that this method will never pop the last modifier item off the
348 stack, to guard against unmatched right parentheses.
349 modifier_seen
351 $tokenizer->modifier_seen( 'i' )
352 and print "/i was seen at some point.\n";
353 Unlike modifier(), this method returns a true value if the given
355 modifier has been seen in any scope visible from the current location
356 in the parse. There is no magic for group match semantics ( /a, /aa,
357 /d, /l, /u) or modifiers that can be repeated, like /x and /xx, or /e
358 and /ee.
359 peek
361 my $character = $tokenizer->peek();
362 my $next_char = $tokenizer->peek( 1 );
363 This method returns the character at the given non-negative offset from
365 the current position. If no offset is given, an offset of 0 is used.
366 If you ask for a negative offset or an offset off the end of the sting,
368 "undef" is returned.
369 ppi_document
371 This method makes a PPI document out of the remainder of the string,
372 and returns it.
373 prior_significant_token
375 $tokenizer->prior_significant_token( 'can_be_quantified' )
376 and print "The prior token can be quantified.\n";
377 This method calls the named method on the most-recently-instantiated
379 significant token, and returns the result. Any arguments subsequent to
380 the method name will be passed to the method.
381 Because this method is designed to be used within the tokenizing
383 system, it will die horribly if the named method does not exist.
384 If called with no arguments at all the most-recently-instantiated
386 significant token is returned.
387 strict
389 say 'Parse is ', $tokenizer->strict() ? 'strict' : 'lenient';
390 This method simply returns true or false, depending on whether the
392 'strict' option to "new()" was true or false.
393
395 A tokenizer trace can be requested by setting environment variable
396 PPIX_REGEXP_TOKENIZER_TRACE to a numeric value other than 0. Use of
397 this environment variable is unsupported in the same sense that the
398 "trace" option of "new" is unsupported. Explicitly specifying the
399 "trace" option to "new" overrides the environment variable.
400 The real reason this is documented is to give the user a way to
402 troubleshoot funny output from the tokenizer.
403
405 Support is by the author. Please file bug reports at
406 <https://rt.cpan.org/Public/Dist/Display.html?Name=PPIx-Regexp>,
407 <https://github.com/trwyant/perl-PPIx-Regexp/issues>, or in electronic
408 mail to the author.
409
411 Thomas R. Wyant, III wyant at cpan dot org
412
414 Copyright (C) 2009-2021 by Thomas R. Wyant, III
415 This program is free software; you can redistribute it and/or modify it
417 under the same terms as Perl 5.10.0. For more details, see the full
418 text of the licenses in the directory LICENSES.
419 This program is distributed in the hope that it will be useful, but
421 without any warranty; without even the implied warranty of
422 merchantability or fitness for a particular purpose.
423perl v5.34.0 2021-10-25 PPIx::Regexp::Tokenizer(3)