1PPIx::Regexp::TokenizerU(s3e)r Contributed Perl DocumentaPtPiIoxn::Regexp::Tokenizer(3)
2
6 PPIx::Regexp::Tokenizer - Tokenize a regular expression
7
9 use PPIx::Regexp::Dumper;
10 PPIx::Regexp::Dumper->new( 'qr{foo}smx' )
11 ->print();
12
14 "PPIx::Regexp::Tokenizer" is a PPIx::Regexp::Support.
15 "PPIx::Regexp::Tokenizer" has no descendants.
17
19 This class provides tokenization of the regular expression.
20
22 This class provides the following public methods. Methods not
23 documented here (or documented below under "EXTERNAL TOKENIZERS") are
24 private, and unsupported in the sense that the author reserves the
25 right to change or remove them without notice.
26 new
28 my $tokenizer = PPIx::Regexp::Tokenizer->new( 'xyzzy' );
29 This static method instantiates the tokenizer. You must pass it the
31 regular expression to be parsed, either as a string or as a
32 PPI::Element of some sort. You can also pass optional name/value pairs
33 of arguments. The option names are specified without a leading dash.
34 Supported options are:
35 default_modifiers array_reference
37 This argument specifies default statement modifiers. It is
38 optional, but if specified must be an array reference. See the
39 PPIx::Regexp new() documentation for the details.
40 encoding name
42 This option specifies the encoding of the string to be tokenized.
43 If specified, an "Encode::decode" is done on the string (or the
44 "content" of the PPI class) before it is tokenized.
45 index_locations
47 This Boolean option specifies that the locations of the generated
48 tokens are to be computed.
49 postderef boolean
51 This option specifies whether the tokenizer recognizes postfix
52 dereferencing. See the PPIx::Regexp new() documentation for the
53 details.
54 $PPIx::Regexp::Tokenizer::DEFAULT_POSTDEREF is not exported.
56 strict boolean
58 This option specifies whether tokenization should assume "use re
59 'strict';" is in effect.
60 The 'strict' pragma was introduced in Perl 5.22, and its
62 documentation says that it is experimental, and that there is no
63 commitment to backward compatibility. The same applies to the
64 tokenization produced when this option is asserted.
65 trace number
67 Specifying a positive value for this option causes a trace of the
68 tokenization. This option is unsupported in the sense that the
69 author reserves the right to alter it without notice.
70 If this option is unspecified, the value comes from environment
72 variable "PPIX_REGEXP_TOKENIZER_TRACE" (see "ENVIRONMENT
73 VARIABLES"). If this environment variable does not exist, the
74 default is 0.
75 Undocumented options are unsupported.
77 The returned value is the instantiated tokenizer, or "undef" if
79 instantiation failed. In the latter case a call to "errstr" will return
80 the reason.
81 content
83 print $tokenizer->content();
84 This method returns the string being tokenized. This will be the result
86 of the PPI::Element->content() method if the object was instantiated
87 with a PPI::Element.
88 default_modifiers
90 print join ', ', @{ $tokenizer->default_modifiers() };
91 This method returns a reference to a copy of the array passed to the
93 "default_modifiers" argument to new(). If this argument was not used to
94 instantiate the object, the return is a reference to an empty array.
95 encoding
97 This method returns the encoding of the data being parsed, if one was
98 set when the class was instantiated; otherwise it simply returns undef.
99 errstr
101 my $tokenizer = PPIx::Regexp::Tokenizer->new( 'xyzzy' )
102 or die PPIx::Regexp::Tokenizer->errstr();
103 This static method returns an error description if tokenizer
105 instantiation failed.
106 failures
108 print $tokenizer->failures(), " tokenization failures\n";
109 This method returns the number of tokenization failures encountered. A
111 tokenization failure is represented in the output token stream by a
112 PPIx::Regexp::Token::Unknown.
113 modifier
115 $tokenizer->modifier( 'x' )
116 and print "Tokenizing an extended regular expression\n";
117 This method returns true if the given modifier character was found on
119 the end of the regular expression, and false otherwise.
120 Starting with version 0.036_01, if the argument is a single-character
122 modifier followed by an asterisk (intended as a wild card character),
123 the return is the number of times that modifier appears. In this case
124 an exception will be thrown if you specify a multi-character modifier
125 (e.g. 'ee*'), or if you specify one of the match semantics modifiers
126 (e.g. 'a*').
127 If called by an external tokenizer, this method returns true if if the
129 given modifier was true at the current point in the tokenization.
130 next_token
132 my $token = $tokenizer->next_token();
133 This method returns the next token in the token stream, or nothing if
135 there are no more tokens.
136 significant
138 This method exists simply for the convenience of PPIx::Regexp::Dumper.
139 It always returns true.
140 tokens
142 my @tokens = $tokenizer->tokens();
143 This method returns all remaining tokens in the token stream.
145
147 This class does very little of its own tokenization. Instead the token
148 classes contain external tokenization routines, whose name is
149 '__PPIX_TOKENIZER__' concatenated with the current mode of the
150 tokenizer ('regexp' for regular expressions, 'repl' for the replacement
151 string).
152 These external tokenizers are called as static methods, and passed the
154 "PPIx::Regexp::Tokenizer" object and the current character in the
155 character stream.
156 If the external tokenizer wants to make one or more tokens, it returns
158 an array containing either length in characters for tokens of the
159 tokenizer's own class, or the results of one or more "make_token" calls
160 for tokens of an arbitrary class.
161 If the external tokenizer is not interested in the characters starting
163 at the current position it simply returns.
164 The following methods are for the use of external tokenizers, and are
166 not part of the public interface to this class.
167 capture
169 if ( $tokenizer->find_regexp( qr{ \A ( foo ) }smx ) ) {
170 foreach ( $tokenizer->capture() ) {
171 print "$_\n";
172 }
173 }
174 This method returns all the contents of any capture buffers from the
176 previous call to "find_regexp". The first element of the array (i.e.
177 element 0) corresponds to $1, and so on.
178 The captures are cleared by "make_token", as well as by another call to
180 "find_regexp".
181 cookie
183 $tokenizer->cookie( foo => sub { 1 } );
184 my $cookie = $tokenizer->cookie( 'foo' );
185 my $old_hint = $tokenizer->cookie( foo => undef );
186 This method either creates, deletes, or accesses a cookie.
188 A cookie is a code reference which is called whenever the tokenizer
190 makes a token. If it returns a false value, it is deleted. Explicitly
191 setting the cookie to "undef" also deletes it.
192 When you call "$tokenizer->cookie( 'foo' )", the current cookie is
194 returned. If you pass a new value of "undef" to delete the token, the
195 deleted cookie (if any) is returned.
196 When the "make_token" method calls a cookie, it passes it the tokenizer
198 and the token just made. If a token calls a cookie, it is recommended
199 that it merely pass the tokenizer, though of course the token can do
200 whatever it wants.
201 The cookie mechanism seems to be a bit of a crock, but it appeared to
203 be more work to fix things up in the lexer after the tokenizer got
204 something wrong.
205 The recommended way to write a cookie is to use a closure to store any
207 necessary data, and have a call to the cookie return the data;
208 otherwise the ultimate consumer of the cookie has no way to access the
209 data. Of course, it may be that the presence of the cookie at a certain
210 point in the parse is all that is required.
211 expect
213 $tokenizer->expect( 'PPIx::Regexp::Token::Code' );
214 This method inserts a given class at the head of the token scan, for
216 the next iteration only. More than one class can be specified. Class
217 names can be abbreviated by removing the leading 'PPIx::Regexp::'.
218 If no class is specified, this method does nothing.
220 The expectation lasts from the next time "get_token" is called until
222 the next time "make_token" makes a significant token, or until the next
223 "expect" call if that is done sooner.
224 find_regexp
226 my $end = $tokenizer->find_regexp( qr{ \A \w+ }smx );
227 my ( $begin, $end ) = $tokenizer->find_regexp(
228 qr{ \A \w+ }smx );
229 This method finds the given regular expression in the content, starting
231 at the current position. If called in scalar context, the offset from
232 the current position to the end of the matched string is returned. If
233 called in list context, the offsets to both the beginning and the end
234 of the matched string are returned.
235 find_matching_delimiter
237 my $offset = $tokenizer->find_matching_delimiter();
238 This method is used by tokenizers to find the delimiter matching the
240 character at the current position in the content string. If the
241 delimiter is an opening bracket of some sort, bracket nesting will be
242 taken into account.
243 When searching for the matching delimiter, the back slash character is
245 considered to escape the following character, so back-slashed
246 delimiters will be ignored. No other quoting mechanisms are recognized,
247 though, so delimiters inside quotes still count. This is actually the
248 way Perl works, as
249 $ perl -e 'qr<(?{ print "}" })>'
251 demonstrates.
253 This method returns the offset from the current position in the content
255 string to the matching delimiter (which will always be positive), or
256 undef if no match can be found.
257 get_mode
259 This method returns the name of the current mode of the tokenizer.
260 get_start_delimiter
262 my $start_delimiter = $tokenizer->get_start_delimiter();
263 This method is used by tokenizers to access the start delimiter for the
265 regular expression.
266 get_token
268 my $token = $tokenizer->make_token( 3 );
269 my @tokens = $tokenizer->get_token();
270 This method returns the next token that can be made from the input
272 stream. It is not part of the external interface, but is intended for
273 the use of an external tokenizer which calls it after making and
274 retaining its own token to look at the next token ( if any ) in the
275 input stream.
276 If any external tokenizer calls get_token without first calling
278 make_token, a fatal error occurs; this is better than the infinite
279 recursion which would occur if the condition were not trapped.
280 An external tokenizer must return anything returned by get_token;
282 otherwise tokens get lost.
283 interpolates
285 This method returns true if the top-level structure being tokenized
286 interpolates; that is, if the delimiter is not a single quote.
287 make_token
289 return $tokenizer->make_token( 3, 'PPIx::Regexp::Token::Unknown' );
290 This method is used by this class (and possibly by individual
292 tokenizers) to manufacture a token. Its arguments are the number of
293 characters to include in the token, and optionally the class of the
294 token. If no class name is given, the caller's class is used. Class
295 names may be shortened by removing the initial 'PPIx::Regexp::', which
296 will be restored by this method.
297 The token will be manufactured from the given number of characters
299 starting at the current cursor position, which will be adjusted.
300 If the given length would include characters past the end of the string
302 being tokenized, the length is reduced appropriately. If this means a
303 token with no characters, nothing is returned.
304 match
306 if ( $tokenizer->find_regexp( qr{ \A \w+ }smx ) ) {
307 print $tokenizer->match(), "\n";
308 }
309 This method returns the string matched by the previous call to
311 "find_regexp".
312 The match is set to "undef" by "make_token", as well as by another call
314 to "find_regexp".
315 modifier_duplicate
317 $tokenizer->modifier_duplicate();
318 This method duplicates the modifiers on the top of the modifier stack,
320 with the intent of creating a locally-scoped copy of the modifiers.
321 This should only be called by an external tokenizer that is actually
322 creating a modifier scope. In other words, only when creating a
323 PPIx::Regexp::Token::Structure token whose content is '('.
324 modifier_modify
326 $tokenizer->modifier_modify( name => $value ... );
327 This method sets new values for the modifiers in the local scope. Only
329 the modifiers whose names are actually passed have their values
330 changed.
331 This method is intended to be called after manufacturing a
333 PPIx::Regexp::Token::Modifier token, and passed the results of its
334 "modifiers" method.
335 modifier_pop
337 $tokenizer->modifier_pop();
338 This method removes the modifiers on the top of the modifier stack.
340 This should only be called by an external tokenizer that is ending a
341 modifier scope. In other words, only when creating a
342 PPIx::Regexp::Token::Structure token whose content is ')'.
343 Note that this method will never pop the last modifier item off the
345 stack, to guard against unmatched right parentheses.
346 modifier_seen
348 $tokenizer->modifier_seen( 'i' )
349 and print "/i was seen at some point.\n";
350 Unlike modifier(), this method returns a true value if the given
352 modifier has been seen in any scope visible from the current location
353 in the parse. There is no magic for group match semantics ( /a, /aa,
354 /d, /l, /u) or modifiers that can be repeated, like /x and /xx, or /e
355 and /ee.
356 peek
358 my $character = $tokenizer->peek();
359 my $next_char = $tokenizer->peek( 1 );
360 This method returns the character at the given non-negative offset from
362 the current position. If no offset is given, an offset of 0 is used.
363 If you ask for a negative offset or an offset off the end of the sting,
365 "undef" is returned.
366 ppi_document
368 This method makes a PPI document out of the remainder of the string,
369 and returns it.
370 prior_significant_token
372 $tokenizer->prior_significant_token( 'can_be_quantified' )
373 and print "The prior token can be quantified.\n";
374 This method calls the named method on the most-recently-instantiated
376 significant token, and returns the result. Any arguments subsequent to
377 the method name will be passed to the method.
378 Because this method is designed to be used within the tokenizing
380 system, it will die horribly if the named method does not exist.
381 If called with no arguments at all the most-recently-instantiated
383 significant token is returned.
384 strict
386 say 'Parse is ', $tokenizer->strict() ? 'strict' : 'lenient';
387 This method simply returns true or false, depending on whether the
389 'strict' option to "new()" was true or false.
390
392 A tokenizer trace can be requested by setting environment variable
393 PPIX_REGEXP_TOKENIZER_TRACE to a numeric value other than 0. Use of
394 this environment variable is unsupported in the same sense that the
395 "trace" option of "new" is unsupported. Explicitly specifying the
396 "trace" option to "new" overrides the environment variable.
397 The real reason this is documented is to give the user a way to
399 troubleshoot funny output from the tokenizer.
400
402 Support is by the author. Please file bug reports at
403 <https://rt.cpan.org>, or in electronic mail to the author.
404
406 Thomas R. Wyant, III wyant at cpan dot org
407
409 Copyright (C) 2009-2020 by Thomas R. Wyant, III
410 This program is free software; you can redistribute it and/or modify it
412 under the same terms as Perl 5.10.0. For more details, see the full
413 text of the licenses in the directory LICENSES.
414 This program is distributed in the hope that it will be useful, but
416 without any warranty; without even the implied warranty of
417 merchantability or fitness for a particular purpose.
418perl v5.32.0 2020-07-29 PPIx::Regexp::Tokenizer(3)