1PPIx::Regexp::TokenizerU(s3e)r Contributed Perl DocumentaPtPiIoxn::Regexp::Tokenizer(3)
2
6 PPIx::Regexp::Tokenizer - Tokenize a regular expression
7
9 use PPIx::Regexp::Dumper;
10 PPIx::Regexp::Dumper->new( 'qr{foo}smx' )
11 ->print();
12
14 "PPIx::Regexp::Tokenizer" is a PPIx::Regexp::Support.
15 "PPIx::Regexp::Tokenizer" has no descendants.
17
19 This class provides tokenization of the regular expression.
20
22 This class provides the following public methods. Methods not
23 documented here (or documented below under "EXTERNAL TOKENIZERS") are
24 private, and unsupported in the sense that the author reserves the
25 right to change or remove them without notice.
26 new
28 my $tokenizer = PPIx::Regexp::Tokenizer->new( 'xyzzy' );
29 This static method instantiates the tokenizer. You must pass it the
31 regular expression to be parsed, either as a string or as a
32 PPI::Element of some sort. You can also pass optional name/value pairs
33 of arguments. The option names are specified without a leading dash.
34 Supported options are:
35 default_modifiers array_reference
37 This argument specifies default statement modifiers. It is
38 optional, but if specified must be an array reference. See the
39 PPIx::Regexp new() documentation for the details.
40 encoding name
42 This option specifies the encoding of the string to be tokenized.
43 If specified, an "Encode::decode" is done on the string (or the
44 "content" of the PPI class) before it is tokenized.
45 postderef boolean
47 This option specifies whether the tokenizer recognizes postfix
48 dereferencing. See the PPIx::Regexp new() documentation for the
49 details.
50 $PPIx::Regexp::Tokenizer::DEFAULT_POSTDEREF is not exported.
52 strict boolean
54 This option specifies whether tokenization should assume "use re
55 'strict';" is in effect.
56 The 'strict' pragma was introduced in Perl 5.22, and its
58 documentation says that it is experimental, and that there is no
59 commitment to backward compatibility. The same applies to the
60 tokenization produced when this option is asserted.
61 trace number
63 Specifying a positive value for this option causes a trace of the
64 tokenization. This option is unsupported in the sense that the
65 author reserves the right to alter it without notice.
66 If this option is unspecified, the value comes from environment
68 variable "PPIX_REGEXP_TOKENIZER_TRACE" (see "ENVIRONMENT
69 VARIABLES"). If this environment variable does not exist, the
70 default is 0.
71 Undocumented options are unsupported.
73 The returned value is the instantiated tokenizer, or "undef" if
75 instantiation failed. In the latter case a call to "errstr" will return
76 the reason.
77 content
79 print $tokenizer->content();
80 This method returns the string being tokenized. This will be the result
82 of the PPI::Element->content() method if the object was instantiated
83 with a PPI::Element.
84 default_modifiers
86 print join ', ', @{ $tokenizer->default_modifiers() };
87 This method returns a reference to a copy of the array passed to the
89 "default_modifiers" argument to new(). If this argument was not used to
90 instantiate the object, the return is a reference to an empty array.
91 encoding
93 This method returns the encoding of the data being parsed, if one was
94 set when the class was instantiated; otherwise it simply returns undef.
95 errstr
97 my $tokenizer = PPIx::Regexp::Tokenizer->new( 'xyzzy' )
98 or die PPIx::Regexp::Tokenizer->errstr();
99 This static method returns an error description if tokenizer
101 instantiation failed.
102 failures
104 print $tokenizer->failures(), " tokenization failures\n";
105 This method returns the number of tokenization failures encountered. A
107 tokenization failure is represented in the output token stream by a
108 PPIx::Regexp::Token::Unknown.
109 modifier
111 $tokenizer->modifier( 'x' )
112 and print "Tokenizing an extended regular expression\n";
113 This method returns true if the given modifier character was found on
115 the end of the regular expression, and false otherwise.
116 Starting with version 0.036_01, if the argument is a single-character
118 modifier followed by an asterisk (intended as a wild card character),
119 the return is the number of times that modifier appears. In this case
120 an exception will be thrown if you specify a multi-character modifier
121 (e.g. 'ee*'), or if you specify one of the match semantics modifiers
122 (e.g. 'a*').
123 If called by an external tokenizer, this method returns true if if the
125 given modifier was true at the current point in the tokenization.
126 next_token
128 my $token = $tokenizer->next_token();
129 This method returns the next token in the token stream, or nothing if
131 there are no more tokens.
132 significant
134 This method exists simply for the convenience of PPIx::Regexp::Dumper.
135 It always returns true.
136 tokens
138 my @tokens = $tokenizer->tokens();
139 This method returns all remaining tokens in the token stream.
141
143 This class does very little of its own tokenization. Instead the token
144 classes contain external tokenization routines, whose name is
145 '__PPIX_TOKENIZER__' concatenated with the current mode of the
146 tokenizer ('regexp' for regular expressions, 'repl' for the replacement
147 string).
148 These external tokenizers are called as static methods, and passed the
150 "PPIx::Regexp::Tokenizer" object and the current character in the
151 character stream.
152 If the external tokenizer wants to make one or more tokens, it returns
154 an array containing either length in characters for tokens of the
155 tokenizer's own class, or the results of one or more "make_token" calls
156 for tokens of an arbitrary class.
157 If the external tokenizer is not interested in the characters starting
159 at the current position it simply returns.
160 The following methods are for the use of external tokenizers, and are
162 not part of the public interface to this class.
163 capture
165 if ( $tokenizer->find_regexp( qr{ \A ( foo ) }smx ) ) {
166 foreach ( $tokenizer->capture() ) {
167 print "$_\n";
168 }
169 }
170 This method returns all the contents of any capture buffers from the
172 previous call to "find_regexp". The first element of the array (i.e.
173 element 0) corresponds to $1, and so on.
174 The captures are cleared by "make_token", as well as by another call to
176 "find_regexp".
177 cookie
179 $tokenizer->cookie( foo => sub { 1 } );
180 my $cookie = $tokenizer->cookie( 'foo' );
181 my $old_hint = $tokenizer->cookie( foo => undef );
182 This method either creates, deletes, or accesses a cookie.
184 A cookie is a code reference which is called whenever the tokenizer
186 makes a token. If it returns a false value, it is deleted. Explicitly
187 setting the cookie to "undef" also deletes it.
188 When you call "$tokenizer->cookie( 'foo' )", the current cookie is
190 returned. If you pass a new value of "undef" to delete the token, the
191 deleted cookie (if any) is returned.
192 When the "make_token" method calls a cookie, it passes it the tokenizer
194 and the token just made. If a token calls a cookie, it is recommended
195 that it merely pass the tokenizer, though of course the token can do
196 whatever it wants.
197 The cookie mechanism seems to be a bit of a crock, but it appeared to
199 be more work to fix things up in the lexer after the tokenizer got
200 something wrong.
201 The recommended way to write a cookie is to use a closure to store any
203 necessary data, and have a call to the cookie return the data;
204 otherwise the ultimate consumer of the cookie has no way to access the
205 data. Of course, it may be that the presence of the cookie at a certain
206 point in the parse is all that is required.
207 expect
209 $tokenizer->expect( 'PPIx::Regexp::Token::Code' );
210 This method inserts a given class at the head of the token scan, for
212 the next iteration only. More than one class can be specified. Class
213 names can be abbreviated by removing the leading 'PPIx::Regexp::'.
214 If no class is specified, this method does nothing.
216 The expectation lasts from the next time "get_token" is called until
218 the next time "make_token" makes a significant token, or until the next
219 "expect" call if that is done sooner.
220 find_regexp
222 my $end = $tokenizer->find_regexp( qr{ \A \w+ }smx );
223 my ( $begin, $end ) = $tokenizer->find_regexp(
224 qr{ \A \w+ }smx );
225 This method finds the given regular expression in the content, starting
227 at the current position. If called in scalar context, the offset from
228 the current position to the end of the matched string is returned. If
229 called in list context, the offsets to both the beginning and the end
230 of the matched string are returned.
231 find_matching_delimiter
233 my $offset = $tokenizer->find_matching_delimiter();
234 This method is used by tokenizers to find the delimiter matching the
236 character at the current position in the content string. If the
237 delimiter is an opening bracket of some sort, bracket nesting will be
238 taken into account.
239 When searching for the matching delimiter, the back slash character is
241 considered to escape the following character, so back-slashed
242 delimiters will be ignored. No other quoting mechanisms are recognized,
243 though, so delimiters inside quotes still count. This is actually the
244 way Perl works, as
245 $ perl -e 'qr<(?{ print "}" })>'
247 demonstrates.
249 This method returns the offset from the current position in the content
251 string to the matching delimiter (which will always be positive), or
252 undef if no match can be found.
253 get_mode
255 This method returns the name of the current mode of the tokenizer.
256 get_start_delimiter
258 my $start_delimiter = $tokenizer->get_start_delimiter();
259 This method is used by tokenizers to access the start delimiter for the
261 regular expression.
262 get_token
264 my $token = $tokenizer->make_token( 3 );
265 my @tokens = $tokenizer->get_token();
266 This method returns the next token that can be made from the input
268 stream. It is not part of the external interface, but is intended for
269 the use of an external tokenizer which calls it after making and
270 retaining its own token to look at the next token ( if any ) in the
271 input stream.
272 If any external tokenizer calls get_token without first calling
274 make_token, a fatal error occurs; this is better than the infinite
275 recursion which would occur if the condition were not trapped.
276 An external tokenizer must return anything returned by get_token;
278 otherwise tokens get lost.
279 interpolates
281 This method returns true if the top-level structure being tokenized
282 interpolates; that is, if the delimiter is not a single quote.
283 make_token
285 return $tokenizer->make_token( 3, 'PPIx::Regexp::Token::Unknown' );
286 This method is used by this class (and possibly by individual
288 tokenizers) to manufacture a token. Its arguments are the number of
289 characters to include in the token, and optionally the class of the
290 token. If no class name is given, the caller's class is used. Class
291 names may be shortened by removing the initial 'PPIx::Regexp::', which
292 will be restored by this method.
293 The token will be manufactured from the given number of characters
295 starting at the current cursor position, which will be adjusted.
296 If the given length would include characters past the end of the string
298 being tokenized, the length is reduced appropriately. If this means a
299 token with no characters, nothing is returned.
300 match
302 if ( $tokenizer->find_regexp( qr{ \A \w+ }smx ) ) {
303 print $tokenizer->match(), "\n";
304 }
305 This method returns the string matched by the previous call to
307 "find_regexp".
308 The match is set to "undef" by "make_token", as well as by another call
310 to "find_regexp".
311 modifier_duplicate
313 $tokenizer->modifier_duplicate();
314 This method duplicates the modifiers on the top of the modifier stack,
316 with the intent of creating a locally-scoped copy of the modifiers.
317 This should only be called by an external tokenizer that is actually
318 creating a modifier scope. In other words, only when creating a
319 PPIx::Regexp::Token::Structure token whose content is '('.
320 modifier_modify
322 $tokenizer->modifier_modify( name => $value ... );
323 This method sets new values for the modifiers in the local scope. Only
325 the modifiers whose names are actually passed have their values
326 changed.
327 This method is intended to be called after manufacturing a
329 PPIx::Regexp::Token::Modifier token, and passed the results of its
330 "modifiers" method.
331 modifier_pop
333 $tokenizer->modifier_pop();
334 This method removes the modifiers on the top of the modifier stack.
336 This should only be called by an external tokenizer that is ending a
337 modifier scope. In other words, only when creating a
338 PPIx::Regexp::Token::Structure token whose content is ')'.
339 Note that this method will never pop the last modifier item off the
341 stack, to guard against unmatched right parentheses.
342 modifier_seen
344 $tokenizer->modifier_seen( 'i' )
345 and print "/i was seen at some point.\n";
346 Unlike modifier(), this method returns a true value if the given
348 modifier has been seen in any scope visible from the current location
349 in the parse. There is no magic for group match semantics ( /a, /aa,
350 /d, /l, /u) or modifiers that can be repeated, like /x and /xx, or /e
351 and /ee.
352 peek
354 my $character = $tokenizer->peek();
355 my $next_char = $tokenizer->peek( 1 );
356 This method returns the character at the given non-negative offset from
358 the current position. If no offset is given, an offset of 0 is used.
359 If you ask for a negative offset or an offset off the end of the sting,
361 "undef" is returned.
362 ppi_document
364 This method makes a PPI document out of the remainder of the string,
365 and returns it.
366 prior_significant_token
368 $tokenizer->prior_significant_token( 'can_be_quantified' )
369 and print "The prior token can be quantified.\n";
370 This method calls the named method on the most-recently-instantiated
372 significant token, and returns the result. Any arguments subsequent to
373 the method name will be passed to the method.
374 Because this method is designed to be used within the tokenizing
376 system, it will die horribly if the named method does not exist.
377 If called with no arguments at all the most-recently-instantiated
379 significant token is returned.
380 strict
382 say 'Parse is ', $tokenizer->strict() ? 'strict' : 'lenient';
383 This method simply returns true or false, depending on whether the
385 'strict' option to "new()" was true or false.
386
388 A tokenizer trace can be requested by setting environment variable
389 PPIX_REGEXP_TOKENIZER_TRACE to a numeric value other than 0. Use of
390 this environment variable is unsupported in the same sense that the
391 "trace" option of "new" is unsupported. Explicitly specifying the
392 "trace" option to "new" overrides the environment variable.
393 The real reason this is documented is to give the user a way to
395 troubleshoot funny output from the tokenizer.
396
398 Support is by the author. Please file bug reports at
399 <http://rt.cpan.org>, or in electronic mail to the author.
400
402 Thomas R. Wyant, III wyant at cpan dot org
403
405 Copyright (C) 2009-2019 by Thomas R. Wyant, III
406 This program is free software; you can redistribute it and/or modify it
408 under the same terms as Perl 5.10.0. For more details, see the full
409 text of the licenses in the directory LICENSES.
410 This program is distributed in the hope that it will be useful, but
412 without any warranty; without even the implied warranty of
413 merchantability or fitness for a particular purpose.
414perl v5.30.0 2019-09-02 PPIx::Regexp::Tokenizer(3)