1PPIx::Regexp::TokenizerU(s3e)r Contributed Perl DocumentaPtPiIoxn::Regexp::Tokenizer(3)
2
6 PPIx::Regexp::Tokenizer - Tokenize a regular expression
7
9 use PPIx::Regexp::Dumper;
10 PPIx::Regexp::Dumper->new( 'qr{foo}smx' )
11 ->print();
12
14 "PPIx::Regexp::Tokenizer" is a PPIx::Regexp::Support.
15 "PPIx::Regexp::Tokenizer" has no descendants.
17
19 This class provides tokenization of the regular expression.
20
22 This class provides the following public methods. Methods not
23 documented here (or documented below under "EXTERNAL TOKENIZERS") are
24 private, and unsupported in the sense that the author reserves the
25 right to change or remove them without notice.
26 new
28 my $tokenizer = PPIx::Regexp::Tokenizer->new( 'xyzzy' );
29 This static method instantiates the tokenizer. You must pass it the
31 regular expression to be parsed, either as a string or as a
32 PPI::Element of some sort. You can also pass optional name/value pairs
33 of arguments. The option names are specified without a leading dash.
34 Supported options are:
35 default_modifiers array_reference
37 This argument specifies default statement modifiers. It is
38 optional, but if specified must be an array reference. See the
39 PPIx::Regexp new() documentation for the details.
40 encoding name
42 This option specifies the encoding of the string to be tokenized.
43 If specified, an "Encode::decode" is done on the string (or the
44 "content" of the PPI class) before it is tokenized.
45 index_locations
47 This Boolean option specifies that the locations of the generated
48 tokens are to be computed.
49 strict boolean
51 This option specifies whether tokenization should assume "use re
52 'strict';" is in effect.
53 The 'strict' pragma was introduced in Perl 5.22, and its
55 documentation says that it is experimental, and that there is no
56 commitment to backward compatibility. The same applies to the
57 tokenization produced when this option is asserted.
58 trace number
60 Specifying a positive value for this option causes a trace of the
61 tokenization. This option is unsupported in the sense that the
62 author reserves the right to alter it without notice.
63 If this option is unspecified, the value comes from environment
65 variable "PPIX_REGEXP_TOKENIZER_TRACE" (see "ENVIRONMENT
66 VARIABLES"). If this environment variable does not exist, the
67 default is 0.
68 Undocumented options are unsupported.
70 The returned value is the instantiated tokenizer, or "undef" if
72 instantiation failed. In the latter case a call to "errstr" will return
73 the reason.
74 content
76 print $tokenizer->content();
77 This method returns the string being tokenized. This will be the result
79 of the PPI::Element->content() method if the object was instantiated
80 with a PPI::Element.
81 default_modifiers
83 print join ', ', @{ $tokenizer->default_modifiers() };
84 This method returns a reference to a copy of the array passed to the
86 "default_modifiers" argument to new(). If this argument was not used to
87 instantiate the object, the return is a reference to an empty array.
88 encoding
90 This method returns the encoding of the data being parsed, if one was
91 set when the class was instantiated; otherwise it simply returns undef.
92 errstr
94 my $tokenizer = PPIx::Regexp::Tokenizer->new( 'xyzzy' )
95 or die PPIx::Regexp::Tokenizer->errstr();
96 This static method returns an error description if tokenizer
98 instantiation failed.
99 failures
101 print $tokenizer->failures(), " tokenization failures\n";
102 This method returns the number of tokenization failures encountered. A
104 tokenization failure is represented in the output token stream by a
105 PPIx::Regexp::Token::Unknown.
106 modifier
108 $tokenizer->modifier( 'x' )
109 and print "Tokenizing an extended regular expression\n";
110 This method returns true if the given modifier character was found on
112 the end of the regular expression, and false otherwise.
113 Starting with version 0.036_01, if the argument is a single-character
115 modifier followed by an asterisk (intended as a wild card character),
116 the return is the number of times that modifier appears. In this case
117 an exception will be thrown if you specify a multi-character modifier
118 (e.g. 'ee*'), or if you specify one of the match semantics modifiers
119 (e.g. 'a*').
120 If called by an external tokenizer, this method returns true if if the
122 given modifier was true at the current point in the tokenization.
123 next_token
125 my $token = $tokenizer->next_token();
126 This method returns the next token in the token stream, or nothing if
128 there are no more tokens.
129 significant
131 This method exists simply for the convenience of PPIx::Regexp::Dumper.
132 It always returns true.
133 tokens
135 my @tokens = $tokenizer->tokens();
136 This method returns all remaining tokens in the token stream.
138
140 This class does very little of its own tokenization. Instead the token
141 classes contain external tokenization routines, whose name is
142 '__PPIX_TOKENIZER__' concatenated with the current mode of the
143 tokenizer ('regexp' for regular expressions, 'repl' for the replacement
144 string).
145 These external tokenizers are called as static methods, and passed the
147 "PPIx::Regexp::Tokenizer" object and the current character in the
148 character stream.
149 If the external tokenizer wants to make one or more tokens, it returns
151 an array containing either length in characters for tokens of the
152 tokenizer's own class, or the results of one or more "make_token" calls
153 for tokens of an arbitrary class.
154 If the external tokenizer is not interested in the characters starting
156 at the current position it simply returns.
157 The following methods are for the use of external tokenizers, and are
159 not part of the public interface to this class.
160 capture
162 if ( $tokenizer->find_regexp( qr{ \A ( foo ) }smx ) ) {
163 foreach ( $tokenizer->capture() ) {
164 print "$_\n";
165 }
166 }
167 This method returns all the contents of any capture buffers from the
169 previous call to "find_regexp". The first element of the array (i.e.
170 element 0) corresponds to $1, and so on.
171 The captures are cleared by "make_token", as well as by another call to
173 "find_regexp".
174 cookie
176 $tokenizer->cookie( foo => sub { 1 } );
177 my $cookie = $tokenizer->cookie( 'foo' );
178 my $old_hint = $tokenizer->cookie( foo => undef );
179 This method either creates, deletes, or accesses a cookie.
181 A cookie is a code reference which is called whenever the tokenizer
183 makes a token. If it returns a false value, it is deleted. Explicitly
184 setting the cookie to "undef" also deletes it.
185 When you call "$tokenizer->cookie( 'foo' )", the current cookie is
187 returned. If you pass a new value of "undef" to delete the token, the
188 deleted cookie (if any) is returned.
189 When the "make_token" method calls a cookie, it passes it the tokenizer
191 and the token just made. If a token calls a cookie, it is recommended
192 that it merely pass the tokenizer, though of course the token can do
193 whatever it wants.
194 The cookie mechanism seems to be a bit of a crock, but it appeared to
196 be more work to fix things up in the lexer after the tokenizer got
197 something wrong.
198 The recommended way to write a cookie is to use a closure to store any
200 necessary data, and have a call to the cookie return the data;
201 otherwise the ultimate consumer of the cookie has no way to access the
202 data. Of course, it may be that the presence of the cookie at a certain
203 point in the parse is all that is required.
204 expect
206 $tokenizer->expect( 'PPIx::Regexp::Token::Code' );
207 This method inserts a given class at the head of the token scan, for
209 the next iteration only. More than one class can be specified. Class
210 names can be abbreviated by removing the leading 'PPIx::Regexp::'.
211 If no class is specified, this method does nothing.
213 The expectation lasts from the next time "get_token" is called until
215 the next time "make_token" makes a significant token, or until the next
216 "expect" call if that is done sooner.
217 find_regexp
219 my $end = $tokenizer->find_regexp( qr{ \A \w+ }smx );
220 my ( $begin, $end ) = $tokenizer->find_regexp(
221 qr{ \A \w+ }smx );
222 This method finds the given regular expression in the content, starting
224 at the current position. If called in scalar context, the offset from
225 the current position to the end of the matched string is returned. If
226 called in list context, the offsets to both the beginning and the end
227 of the matched string are returned.
228 find_matching_delimiter
230 my $offset = $tokenizer->find_matching_delimiter();
231 This method is used by tokenizers to find the delimiter matching the
233 character at the current position in the content string. If the
234 delimiter is an opening bracket of some sort, bracket nesting will be
235 taken into account.
236 When searching for the matching delimiter, the back slash character is
238 considered to escape the following character, so back-slashed
239 delimiters will be ignored. No other quoting mechanisms are recognized,
240 though, so delimiters inside quotes still count. This is actually the
241 way Perl works, as
242 $ perl -e 'qr<(?{ print "}" })>'
244 demonstrates.
246 This method returns the offset from the current position in the content
248 string to the matching delimiter (which will always be positive), or
249 undef if no match can be found.
250 get_mode
252 This method returns the name of the current mode of the tokenizer.
253 get_start_delimiter
255 my $start_delimiter = $tokenizer->get_start_delimiter();
256 This method is used by tokenizers to access the start delimiter for the
258 regular expression.
259 get_token
261 my $token = $tokenizer->make_token( 3 );
262 my @tokens = $tokenizer->get_token();
263 This method returns the next token that can be made from the input
265 stream. It is not part of the external interface, but is intended for
266 the use of an external tokenizer which calls it after making and
267 retaining its own token to look at the next token ( if any ) in the
268 input stream.
269 If any external tokenizer calls get_token without first calling
271 make_token, a fatal error occurs; this is better than the infinite
272 recursion which would occur if the condition were not trapped.
273 An external tokenizer must return anything returned by get_token;
275 otherwise tokens get lost.
276 interpolates
278 This method returns true if the top-level structure being tokenized
279 interpolates; that is, if the delimiter is not a single quote.
280 make_token
282 return $tokenizer->make_token( 3, 'PPIx::Regexp::Token::Unknown' );
283 This method is used by this class (and possibly by individual
285 tokenizers) to manufacture a token. Its arguments are the number of
286 characters to include in the token, and optionally the class of the
287 token. If no class name is given, the caller's class is used. Class
288 names may be shortened by removing the initial 'PPIx::Regexp::', which
289 will be restored by this method.
290 The token will be manufactured from the given number of characters
292 starting at the current cursor position, which will be adjusted.
293 If the given length would include characters past the end of the string
295 being tokenized, the length is reduced appropriately. If this means a
296 token with no characters, nothing is returned.
297 match
299 if ( $tokenizer->find_regexp( qr{ \A \w+ }smx ) ) {
300 print $tokenizer->match(), "\n";
301 }
302 This method returns the string matched by the previous call to
304 "find_regexp".
305 The match is set to "undef" by "make_token", as well as by another call
307 to "find_regexp".
308 modifier_duplicate
310 $tokenizer->modifier_duplicate();
311 This method duplicates the modifiers on the top of the modifier stack,
313 with the intent of creating a locally-scoped copy of the modifiers.
314 This should only be called by an external tokenizer that is actually
315 creating a modifier scope. In other words, only when creating a
316 PPIx::Regexp::Token::Structure token whose content is '('.
317 modifier_modify
319 $tokenizer->modifier_modify( name => $value ... );
320 This method sets new values for the modifiers in the local scope. Only
322 the modifiers whose names are actually passed have their values
323 changed.
324 This method is intended to be called after manufacturing a
326 PPIx::Regexp::Token::Modifier token, and passed the results of its
327 "modifiers" method.
328 modifier_pop
330 $tokenizer->modifier_pop();
331 This method removes the modifiers on the top of the modifier stack.
333 This should only be called by an external tokenizer that is ending a
334 modifier scope. In other words, only when creating a
335 PPIx::Regexp::Token::Structure token whose content is ')'.
336 Note that this method will never pop the last modifier item off the
338 stack, to guard against unmatched right parentheses.
339 modifier_seen
341 $tokenizer->modifier_seen( 'i' )
342 and print "/i was seen at some point.\n";
343 Unlike modifier(), this method returns a true value if the given
345 modifier has been seen in any scope visible from the current location
346 in the parse. There is no magic for group match semantics ( /a, /aa,
347 /d, /l, /u) or modifiers that can be repeated, like /x and /xx, or /e
348 and /ee.
349 peek
351 my $character = $tokenizer->peek();
352 my $next_char = $tokenizer->peek( 1 );
353 This method returns the character at the given non-negative offset from
355 the current position. If no offset is given, an offset of 0 is used.
356 If you ask for a negative offset or an offset off the end of the sting,
358 "undef" is returned.
359 ppi_document
361 This method makes a PPI document out of the remainder of the string,
362 and returns it.
363 prior_significant_token
365 $tokenizer->prior_significant_token( 'can_be_quantified' )
366 and print "The prior token can be quantified.\n";
367 This method calls the named method on the most-recently-instantiated
369 significant token, and returns the result. Any arguments subsequent to
370 the method name will be passed to the method.
371 Because this method is designed to be used within the tokenizing
373 system, it will die horribly if the named method does not exist.
374 If called with no arguments at all the most-recently-instantiated
376 significant token is returned.
377 strict
379 say 'Parse is ', $tokenizer->strict() ? 'strict' : 'lenient';
380 This method simply returns true or false, depending on whether the
382 'strict' option to new() was true or false.
383
385 A tokenizer trace can be requested by setting environment variable
386 PPIX_REGEXP_TOKENIZER_TRACE to a numeric value other than 0. Use of
387 this environment variable is unsupported in the same sense that the
388 "trace" option of "new" is unsupported. Explicitly specifying the
389 "trace" option to "new" overrides the environment variable.
390 The real reason this is documented is to give the user a way to
392 troubleshoot funny output from the tokenizer.
393
395 Support is by the author. Please file bug reports at
396 <https://rt.cpan.org/Public/Dist/Display.html?Name=PPIx-Regexp>,
397 <https://github.com/trwyant/perl-PPIx-Regexp/issues>, or in electronic
398 mail to the author.
399
401 Thomas R. Wyant, III wyant at cpan dot org
402
404 Copyright (C) 2009-2023 by Thomas R. Wyant, III
405 This program is free software; you can redistribute it and/or modify it
407 under the same terms as Perl 5.10.0. For more details, see the full
408 text of the licenses in the directory LICENSES.
409 This program is distributed in the hope that it will be useful, but
411 without any warranty; without even the implied warranty of
412 merchantability or fitness for a particular purpose.
413perl v5.38.0 2023-07-21 PPIx::Regexp::Tokenizer(3)