1PPIx::Regexp::TokenizerU(s3e)r Contributed Perl DocumentaPtPiIoxn::Regexp::Tokenizer(3)
2
3
4

NAME

6       PPIx::Regexp::Tokenizer - Tokenize a regular expression
7

SYNOPSIS

9        use PPIx::Regexp::Dumper;
10        PPIx::Regexp::Dumper->new( 'qr{foo}smx' )
11            ->print();
12

INHERITANCE

14       "PPIx::Regexp::Tokenizer" is a PPIx::Regexp::Support.
15
16       "PPIx::Regexp::Tokenizer" has no descendants.
17

DESCRIPTION

19       This class provides tokenization of the regular expression.
20

METHODS

22       This class provides the following public methods. Methods not
23       documented here (or documented below under "EXTERNAL TOKENIZERS") are
24       private, and unsupported in the sense that the author reserves the
25       right to change or remove them without notice.
26
27   new
28        my $tokenizer = PPIx::Regexp::Tokenizer->new( 'xyzzy' );
29
30       This static method instantiates the tokenizer. You must pass it the
31       regular expression to be parsed, either as a string or as a
32       PPI::Element of some sort. You can also pass optional name/value pairs
33       of arguments. The option names are specified without a leading dash.
34       Supported options are:
35
36       default_modifiers array_reference
37           This argument specifies default statement modifiers. It is
38           optional, but if specified must be an array reference. See the
39           PPIx::Regexp new() documentation for the details.
40
41       encoding name
42           This option specifies the encoding of the string to be tokenized.
43           If specified, an "Encode::decode" is done on the string (or the
44           "content" of the PPI class) before it is tokenized.
45
46       postderef boolean
47           This option specifies whether the tokenizer recognizes postfix
48           dereferencing. See the PPIx::Regexp new() documentation for the
49           details.
50
51           $PPIx::Regexp::Tokenizer::DEFAULT_POSTDEREF is not exported.
52
53       strict boolean
54           This option specifies whether tokenization should assume "use re
55           'strict';" is in effect.
56
57           The 'strict' pragma was introduced in Perl 5.22, and its
58           documentation says that it is experimental, and that there is no
59           commitment to backward compatibility. The same applies to the
60           tokenization produced when this option is asserted.
61
62       trace number
63           Specifying a positive value for this option causes a trace of the
64           tokenization. This option is unsupported in the sense that the
65           author reserves the right to alter it without notice.
66
67           If this option is unspecified, the value comes from environment
68           variable "PPIX_REGEXP_TOKENIZER_TRACE" (see "ENVIRONMENT
69           VARIABLES"). If this environment variable does not exist, the
70           default is 0.
71
72       Undocumented options are unsupported.
73
74       The returned value is the instantiated tokenizer, or "undef" if
75       instantiation failed. In the latter case a call to "errstr" will return
76       the reason.
77
78   content
79        print $tokenizer->content();
80
81       This method returns the string being tokenized. This will be the result
82       of the PPI::Element->content() method if the object was instantiated
83       with a PPI::Element.
84
85   default_modifiers
86        print join ', ', @{ $tokenizer->default_modifiers() };
87
88       This method returns a reference to a copy of the array passed to the
89       "default_modifiers" argument to new(). If this argument was not used to
90       instantiate the object, the return is a reference to an empty array.
91
92   encoding
93       This method returns the encoding of the data being parsed, if one was
94       set when the class was instantiated; otherwise it simply returns undef.
95
96   errstr
97        my $tokenizer = PPIx::Regexp::Tokenizer->new( 'xyzzy' )
98            or die PPIx::Regexp::Tokenizer->errstr();
99
100       This static method returns an error description if tokenizer
101       instantiation failed.
102
103   failures
104        print $tokenizer->failures(), " tokenization failures\n";
105
106       This method returns the number of tokenization failures encountered. A
107       tokenization failure is represented in the output token stream by a
108       PPIx::Regexp::Token::Unknown.
109
110   modifier
111        $tokenizer->modifier( 'x' )
112            and print "Tokenizing an extended regular expression\n";
113
114       This method returns true if the given modifier character was found on
115       the end of the regular expression, and false otherwise.
116
117       Starting with version 0.036_01, if the argument is a single-character
118       modifier followed by an asterisk (intended as a wild card character),
119       the return is the number of times that modifier appears. In this case
120       an exception will be thrown if you specify a multi-character modifier
121       (e.g.  'ee*'), or if you specify one of the match semantics modifiers
122       (e.g.  'a*').
123
124       If called by an external tokenizer, this method returns true if if the
125       given modifier was true at the current point in the tokenization.
126
127   next_token
128        my $token = $tokenizer->next_token();
129
130       This method returns the next token in the token stream, or nothing if
131       there are no more tokens.
132
133   significant
134       This method exists simply for the convenience of PPIx::Regexp::Dumper.
135       It always returns true.
136
137   tokens
138        my @tokens = $tokenizer->tokens();
139
140       This method returns all remaining tokens in the token stream.
141

EXTERNAL TOKENIZERS

143       This class does very little of its own tokenization. Instead the token
144       classes contain external tokenization routines, whose name is
145       '__PPIX_TOKENIZER__' concatenated with the current mode of the
146       tokenizer ('regexp' for regular expressions, 'repl' for the replacement
147       string).
148
149       These external tokenizers are called as static methods, and passed the
150       "PPIx::Regexp::Tokenizer" object and the current character in the
151       character stream.
152
153       If the external tokenizer wants to make one or more tokens, it returns
154       an array containing either length in characters for tokens of the
155       tokenizer's own class, or the results of one or more "make_token" calls
156       for tokens of an arbitrary class.
157
158       If the external tokenizer is not interested in the characters starting
159       at the current position it simply returns.
160
161       The following methods are for the use of external tokenizers, and are
162       not part of the public interface to this class.
163
164   capture
165        if ( $tokenizer->find_regexp( qr{ \A ( foo ) }smx ) ) {
166            foreach ( $tokenizer->capture() ) {
167                print "$_\n";
168            }
169        }
170
171       This method returns all the contents of any capture buffers from the
172       previous call to "find_regexp". The first element of the array (i.e.
173       element 0) corresponds to $1, and so on.
174
175       The captures are cleared by "make_token", as well as by another call to
176       "find_regexp".
177
178   cookie
179        $tokenizer->cookie( foo => sub { 1 } );
180        my $cookie = $tokenizer->cookie( 'foo' );
181        my $old_hint = $tokenizer->cookie( foo => undef );
182
183       This method either creates, deletes, or accesses a cookie.
184
185       A cookie is a code reference which is called whenever the tokenizer
186       makes a token. If it returns a false value, it is deleted. Explicitly
187       setting the cookie to "undef" also deletes it.
188
189       When you call "$tokenizer->cookie( 'foo' )", the current cookie is
190       returned. If you pass a new value of "undef" to delete the token, the
191       deleted cookie (if any) is returned.
192
193       When the "make_token" method calls a cookie, it passes it the tokenizer
194       and the token just made. If a token calls a cookie, it is recommended
195       that it merely pass the tokenizer, though of course the token can do
196       whatever it wants.
197
198       The cookie mechanism seems to be a bit of a crock, but it appeared to
199       be more work to fix things up in the lexer after the tokenizer got
200       something wrong.
201
202       The recommended way to write a cookie is to use a closure to store any
203       necessary data, and have a call to the cookie return the data;
204       otherwise the ultimate consumer of the cookie has no way to access the
205       data. Of course, it may be that the presence of the cookie at a certain
206       point in the parse is all that is required.
207
208   expect
209        $tokenizer->expect( 'PPIx::Regexp::Token::Code' );
210
211       This method inserts a given class at the head of the token scan, for
212       the next iteration only. More than one class can be specified. Class
213       names can be abbreviated by removing the leading 'PPIx::Regexp::'.
214
215       If no class is specified, this method does nothing.
216
217       The expectation lasts from the next time "get_token" is called until
218       the next time "make_token" makes a significant token, or until the next
219       "expect" call if that is done sooner.
220
221   find_regexp
222        my $end = $tokenizer->find_regexp( qr{ \A \w+ }smx );
223        my ( $begin, $end ) = $tokenizer->find_regexp(
224            qr{ \A \w+ }smx );
225
226       This method finds the given regular expression in the content, starting
227       at the current position. If called in scalar context, the offset from
228       the current position to the end of the matched string is returned. If
229       called in list context, the offsets to both the beginning and the end
230       of the matched string are returned.
231
232   find_matching_delimiter
233        my $offset = $tokenizer->find_matching_delimiter();
234
235       This method is used by tokenizers to find the delimiter matching the
236       character at the current position in the content string. If the
237       delimiter is an opening bracket of some sort, bracket nesting will be
238       taken into account.
239
240       When searching for the matching delimiter, the back slash character is
241       considered to escape the following character, so back-slashed
242       delimiters will be ignored. No other quoting mechanisms are recognized,
243       though, so delimiters inside quotes still count. This is actually the
244       way Perl works, as
245
246        $ perl -e 'qr<(?{ print "}" })>'
247
248       demonstrates.
249
250       This method returns the offset from the current position in the content
251       string to the matching delimiter (which will always be positive), or
252       undef if no match can be found.
253
254   get_mode
255       This method returns the name of the current mode of the tokenizer.
256
257   get_start_delimiter
258        my $start_delimiter = $tokenizer->get_start_delimiter();
259
260       This method is used by tokenizers to access the start delimiter for the
261       regular expression.
262
263   get_token
264        my $token = $tokenizer->make_token( 3 );
265        my @tokens = $tokenizer->get_token();
266
267       This method returns the next token that can be made from the input
268       stream. It is not part of the external interface, but is intended for
269       the use of an external tokenizer which calls it after making and
270       retaining its own token to look at the next token ( if any ) in the
271       input stream.
272
273       If any external tokenizer calls get_token without first calling
274       make_token, a fatal error occurs; this is better than the infinite
275       recursion which would occur if the condition were not trapped.
276
277       An external tokenizer must return anything returned by get_token;
278       otherwise tokens get lost.
279
280   interpolates
281       This method returns true if the top-level structure being tokenized
282       interpolates; that is, if the delimiter is not a single quote.
283
284   make_token
285        return $tokenizer->make_token( 3, 'PPIx::Regexp::Token::Unknown' );
286
287       This method is used by this class (and possibly by individual
288       tokenizers) to manufacture a token. Its arguments are the number of
289       characters to include in the token, and optionally the class of the
290       token. If no class name is given, the caller's class is used. Class
291       names may be shortened by removing the initial 'PPIx::Regexp::', which
292       will be restored by this method.
293
294       The token will be manufactured from the given number of characters
295       starting at the current cursor position, which will be adjusted.
296
297       If the given length would include characters past the end of the string
298       being tokenized, the length is reduced appropriately. If this means a
299       token with no characters, nothing is returned.
300
301   match
302        if ( $tokenizer->find_regexp( qr{ \A \w+ }smx ) ) {
303            print $tokenizer->match(), "\n";
304        }
305
306       This method returns the string matched by the previous call to
307       "find_regexp".
308
309       The match is set to "undef" by "make_token", as well as by another call
310       to "find_regexp".
311
312   modifier_duplicate
313        $tokenizer->modifier_duplicate();
314
315       This method duplicates the modifiers on the top of the modifier stack,
316       with the intent of creating a locally-scoped copy of the modifiers.
317       This should only be called by an external tokenizer that is actually
318       creating a modifier scope. In other words, only when creating a
319       PPIx::Regexp::Token::Structure token whose content is '('.
320
321   modifier_modify
322        $tokenizer->modifier_modify( name => $value ... );
323
324       This method sets new values for the modifiers in the local scope. Only
325       the modifiers whose names are actually passed have their values
326       changed.
327
328       This method is intended to be called after manufacturing a
329       PPIx::Regexp::Token::Modifier token, and passed the results of its
330       "modifiers" method.
331
332   modifier_pop
333        $tokenizer->modifier_pop();
334
335       This method removes the modifiers on the top of the modifier stack.
336       This should only be called by an external tokenizer that is ending a
337       modifier scope. In other words, only when creating a
338       PPIx::Regexp::Token::Structure token whose content is ')'.
339
340       Note that this method will never pop the last modifier item off the
341       stack, to guard against unmatched right parentheses.
342
343   modifier_seen
344        $tokenizer->modifier_seen( 'i' )
345            and print "/i was seen at some point.\n";
346
347       Unlike modifier(), this method returns a true value if the given
348       modifier has been seen in any scope visible from the current location
349       in the parse. There is no magic for group match semantics ( /a, /aa,
350       /d, /l, /u) or modifiers that can be repeated, like /x and /xx, or /e
351       and /ee.
352
353   peek
354        my $character = $tokenizer->peek();
355        my $next_char = $tokenizer->peek( 1 );
356
357       This method returns the character at the given non-negative offset from
358       the current position. If no offset is given, an offset of 0 is used.
359
360       If you ask for a negative offset or an offset off the end of the sting,
361       "undef" is returned.
362
363   ppi_document
364       This method makes a PPI document out of the remainder of the string,
365       and returns it.
366
367   prior_significant_token
368        $tokenizer->prior_significant_token( 'can_be_quantified' )
369           and print "The prior token can be quantified.\n";
370
371       This method calls the named method on the most-recently-instantiated
372       significant token, and returns the result. Any arguments subsequent to
373       the method name will be passed to the method.
374
375       Because this method is designed to be used within the tokenizing
376       system, it will die horribly if the named method does not exist.
377
378       If called with no arguments at all the most-recently-instantiated
379       significant token is returned.
380
381   strict
382        say 'Parse is ', $tokenizer->strict() ? 'strict' : 'lenient';
383
384       This method simply returns true or false, depending on whether the
385       'strict' option to "new()" was true or false.
386

ENVIRONMENT VARIABLES

388       A tokenizer trace can be requested by setting environment variable
389       PPIX_REGEXP_TOKENIZER_TRACE to a numeric value other than 0. Use of
390       this environment variable is unsupported in the same sense that the
391       "trace" option of "new" is unsupported. Explicitly specifying the
392       "trace" option to "new" overrides the environment variable.
393
394       The real reason this is documented is to give the user a way to
395       troubleshoot funny output from the tokenizer.
396

SUPPORT

398       Support is by the author. Please file bug reports at
399       <https://rt.cpan.org>, or in electronic mail to the author.
400

AUTHOR

402       Thomas R. Wyant, III wyant at cpan dot org
403
405       Copyright (C) 2009-2020 by Thomas R. Wyant, III
406
407       This program is free software; you can redistribute it and/or modify it
408       under the same terms as Perl 5.10.0. For more details, see the full
409       text of the licenses in the directory LICENSES.
410
411       This program is distributed in the hope that it will be useful, but
412       without any warranty; without even the implied warranty of
413       merchantability or fitness for a particular purpose.
414
415
416
417perl v5.30.1                      2020-02-10        PPIx::Regexp::Tokenizer(3)
Impressum