1PPIx::Regexp::TokenizerU(s3e)r Contributed Perl DocumentaPtPiIoxn::Regexp::Tokenizer(3)
2
3
4

NAME

6       PPIx::Regexp::Tokenizer - Tokenize a regular expression
7

SYNOPSIS

9        use PPIx::Regexp::Dumper;
10        PPIx::Regexp::Dumper->new( 'qr{foo}smx' )
11            ->print();
12

INHERITANCE

14       "PPIx::Regexp::Tokenizer" is a PPIx::Regexp::Support.
15
16       "PPIx::Regexp::Tokenizer" has no descendants.
17

DESCRIPTION

19       This class provides tokenization of the regular expression.
20

METHODS

22       This class provides the following public methods. Methods not
23       documented here (or documented below under "EXTERNAL TOKENIZERS") are
24       private, and unsupported in the sense that the author reserves the
25       right to change or remove them without notice.
26
27   new
28        my $tokenizer = PPIx::Regexp::Tokenizer->new( 'xyzzy' );
29
30       This static method instantiates the tokenizer. You must pass it the
31       regular expression to be parsed, either as a string or as a
32       PPI::Element of some sort. You can also pass optional name/value pairs
33       of arguments. The option names are specified without a leading dash.
34       Supported options are:
35
36       default_modifiers array_reference
37           This argument specifies default statement modifiers. It is
38           optional, but if specified must be an array reference. See the
39           PPIx::Regexp new() documentation for the details.
40
41       encoding name
42           This option specifies the encoding of the string to be tokenized.
43           If specified, an "Encode::decode" is done on the string (or the
44           "content" of the PPI class) before it is tokenized.
45
46       index_locations
47           This Boolean option specifies that the locations of the generated
48           tokens are to be computed.
49
50       strict boolean
51           This option specifies whether tokenization should assume "use re
52           'strict';" is in effect.
53
54           The 'strict' pragma was introduced in Perl 5.22, and its
55           documentation says that it is experimental, and that there is no
56           commitment to backward compatibility. The same applies to the
57           tokenization produced when this option is asserted.
58
59       trace number
60           Specifying a positive value for this option causes a trace of the
61           tokenization. This option is unsupported in the sense that the
62           author reserves the right to alter it without notice.
63
64           If this option is unspecified, the value comes from environment
65           variable "PPIX_REGEXP_TOKENIZER_TRACE" (see "ENVIRONMENT
66           VARIABLES"). If this environment variable does not exist, the
67           default is 0.
68
69       Undocumented options are unsupported.
70
71       The returned value is the instantiated tokenizer, or "undef" if
72       instantiation failed. In the latter case a call to "errstr" will return
73       the reason.
74
75   content
76        print $tokenizer->content();
77
78       This method returns the string being tokenized. This will be the result
79       of the PPI::Element->content() method if the object was instantiated
80       with a PPI::Element.
81
82   default_modifiers
83        print join ', ', @{ $tokenizer->default_modifiers() };
84
85       This method returns a reference to a copy of the array passed to the
86       "default_modifiers" argument to new(). If this argument was not used to
87       instantiate the object, the return is a reference to an empty array.
88
89   encoding
90       This method returns the encoding of the data being parsed, if one was
91       set when the class was instantiated; otherwise it simply returns undef.
92
93   errstr
94        my $tokenizer = PPIx::Regexp::Tokenizer->new( 'xyzzy' )
95            or die PPIx::Regexp::Tokenizer->errstr();
96
97       This static method returns an error description if tokenizer
98       instantiation failed.
99
100   failures
101        print $tokenizer->failures(), " tokenization failures\n";
102
103       This method returns the number of tokenization failures encountered. A
104       tokenization failure is represented in the output token stream by a
105       PPIx::Regexp::Token::Unknown.
106
107   modifier
108        $tokenizer->modifier( 'x' )
109            and print "Tokenizing an extended regular expression\n";
110
111       This method returns true if the given modifier character was found on
112       the end of the regular expression, and false otherwise.
113
114       Starting with version 0.036_01, if the argument is a single-character
115       modifier followed by an asterisk (intended as a wild card character),
116       the return is the number of times that modifier appears. In this case
117       an exception will be thrown if you specify a multi-character modifier
118       (e.g.  'ee*'), or if you specify one of the match semantics modifiers
119       (e.g.  'a*').
120
121       If called by an external tokenizer, this method returns true if if the
122       given modifier was true at the current point in the tokenization.
123
124   next_token
125        my $token = $tokenizer->next_token();
126
127       This method returns the next token in the token stream, or nothing if
128       there are no more tokens.
129
130   significant
131       This method exists simply for the convenience of PPIx::Regexp::Dumper.
132       It always returns true.
133
134   tokens
135        my @tokens = $tokenizer->tokens();
136
137       This method returns all remaining tokens in the token stream.
138

EXTERNAL TOKENIZERS

140       This class does very little of its own tokenization. Instead the token
141       classes contain external tokenization routines, whose name is
142       '__PPIX_TOKENIZER__' concatenated with the current mode of the
143       tokenizer ('regexp' for regular expressions, 'repl' for the replacement
144       string).
145
146       These external tokenizers are called as static methods, and passed the
147       "PPIx::Regexp::Tokenizer" object and the current character in the
148       character stream.
149
150       If the external tokenizer wants to make one or more tokens, it returns
151       an array containing either length in characters for tokens of the
152       tokenizer's own class, or the results of one or more "make_token" calls
153       for tokens of an arbitrary class.
154
155       If the external tokenizer is not interested in the characters starting
156       at the current position it simply returns.
157
158       The following methods are for the use of external tokenizers, and are
159       not part of the public interface to this class.
160
161   capture
162        if ( $tokenizer->find_regexp( qr{ \A ( foo ) }smx ) ) {
163            foreach ( $tokenizer->capture() ) {
164                print "$_\n";
165            }
166        }
167
168       This method returns all the contents of any capture buffers from the
169       previous call to "find_regexp". The first element of the array (i.e.
170       element 0) corresponds to $1, and so on.
171
172       The captures are cleared by "make_token", as well as by another call to
173       "find_regexp".
174
175   cookie
176        $tokenizer->cookie( foo => sub { 1 } );
177        my $cookie = $tokenizer->cookie( 'foo' );
178        my $old_hint = $tokenizer->cookie( foo => undef );
179
180       This method either creates, deletes, or accesses a cookie.
181
182       A cookie is a code reference which is called whenever the tokenizer
183       makes a token. If it returns a false value, it is deleted. Explicitly
184       setting the cookie to "undef" also deletes it.
185
186       When you call "$tokenizer->cookie( 'foo' )", the current cookie is
187       returned. If you pass a new value of "undef" to delete the token, the
188       deleted cookie (if any) is returned.
189
190       When the "make_token" method calls a cookie, it passes it the tokenizer
191       and the token just made. If a token calls a cookie, it is recommended
192       that it merely pass the tokenizer, though of course the token can do
193       whatever it wants.
194
195       The cookie mechanism seems to be a bit of a crock, but it appeared to
196       be more work to fix things up in the lexer after the tokenizer got
197       something wrong.
198
199       The recommended way to write a cookie is to use a closure to store any
200       necessary data, and have a call to the cookie return the data;
201       otherwise the ultimate consumer of the cookie has no way to access the
202       data. Of course, it may be that the presence of the cookie at a certain
203       point in the parse is all that is required.
204
205   expect
206        $tokenizer->expect( 'PPIx::Regexp::Token::Code' );
207
208       This method inserts a given class at the head of the token scan, for
209       the next iteration only. More than one class can be specified. Class
210       names can be abbreviated by removing the leading 'PPIx::Regexp::'.
211
212       If no class is specified, this method does nothing.
213
214       The expectation lasts from the next time "get_token" is called until
215       the next time "make_token" makes a significant token, or until the next
216       "expect" call if that is done sooner.
217
218   find_regexp
219        my $end = $tokenizer->find_regexp( qr{ \A \w+ }smx );
220        my ( $begin, $end ) = $tokenizer->find_regexp(
221            qr{ \A \w+ }smx );
222
223       This method finds the given regular expression in the content, starting
224       at the current position. If called in scalar context, the offset from
225       the current position to the end of the matched string is returned. If
226       called in list context, the offsets to both the beginning and the end
227       of the matched string are returned.
228
229   find_matching_delimiter
230        my $offset = $tokenizer->find_matching_delimiter();
231
232       This method is used by tokenizers to find the delimiter matching the
233       character at the current position in the content string. If the
234       delimiter is an opening bracket of some sort, bracket nesting will be
235       taken into account.
236
237       When searching for the matching delimiter, the back slash character is
238       considered to escape the following character, so back-slashed
239       delimiters will be ignored. No other quoting mechanisms are recognized,
240       though, so delimiters inside quotes still count. This is actually the
241       way Perl works, as
242
243        $ perl -e 'qr<(?{ print "}" })>'
244
245       demonstrates.
246
247       This method returns the offset from the current position in the content
248       string to the matching delimiter (which will always be positive), or
249       undef if no match can be found.
250
251   get_mode
252       This method returns the name of the current mode of the tokenizer.
253
254   get_start_delimiter
255        my $start_delimiter = $tokenizer->get_start_delimiter();
256
257       This method is used by tokenizers to access the start delimiter for the
258       regular expression.
259
260   get_token
261        my $token = $tokenizer->make_token( 3 );
262        my @tokens = $tokenizer->get_token();
263
264       This method returns the next token that can be made from the input
265       stream. It is not part of the external interface, but is intended for
266       the use of an external tokenizer which calls it after making and
267       retaining its own token to look at the next token ( if any ) in the
268       input stream.
269
270       If any external tokenizer calls get_token without first calling
271       make_token, a fatal error occurs; this is better than the infinite
272       recursion which would occur if the condition were not trapped.
273
274       An external tokenizer must return anything returned by get_token;
275       otherwise tokens get lost.
276
277   interpolates
278       This method returns true if the top-level structure being tokenized
279       interpolates; that is, if the delimiter is not a single quote.
280
281   make_token
282        return $tokenizer->make_token( 3, 'PPIx::Regexp::Token::Unknown' );
283
284       This method is used by this class (and possibly by individual
285       tokenizers) to manufacture a token. Its arguments are the number of
286       characters to include in the token, and optionally the class of the
287       token. If no class name is given, the caller's class is used. Class
288       names may be shortened by removing the initial 'PPIx::Regexp::', which
289       will be restored by this method.
290
291       The token will be manufactured from the given number of characters
292       starting at the current cursor position, which will be adjusted.
293
294       If the given length would include characters past the end of the string
295       being tokenized, the length is reduced appropriately. If this means a
296       token with no characters, nothing is returned.
297
298   match
299        if ( $tokenizer->find_regexp( qr{ \A \w+ }smx ) ) {
300            print $tokenizer->match(), "\n";
301        }
302
303       This method returns the string matched by the previous call to
304       "find_regexp".
305
306       The match is set to "undef" by "make_token", as well as by another call
307       to "find_regexp".
308
309   modifier_duplicate
310        $tokenizer->modifier_duplicate();
311
312       This method duplicates the modifiers on the top of the modifier stack,
313       with the intent of creating a locally-scoped copy of the modifiers.
314       This should only be called by an external tokenizer that is actually
315       creating a modifier scope. In other words, only when creating a
316       PPIx::Regexp::Token::Structure token whose content is '('.
317
318   modifier_modify
319        $tokenizer->modifier_modify( name => $value ... );
320
321       This method sets new values for the modifiers in the local scope. Only
322       the modifiers whose names are actually passed have their values
323       changed.
324
325       This method is intended to be called after manufacturing a
326       PPIx::Regexp::Token::Modifier token, and passed the results of its
327       "modifiers" method.
328
329   modifier_pop
330        $tokenizer->modifier_pop();
331
332       This method removes the modifiers on the top of the modifier stack.
333       This should only be called by an external tokenizer that is ending a
334       modifier scope. In other words, only when creating a
335       PPIx::Regexp::Token::Structure token whose content is ')'.
336
337       Note that this method will never pop the last modifier item off the
338       stack, to guard against unmatched right parentheses.
339
340   modifier_seen
341        $tokenizer->modifier_seen( 'i' )
342            and print "/i was seen at some point.\n";
343
344       Unlike modifier(), this method returns a true value if the given
345       modifier has been seen in any scope visible from the current location
346       in the parse. There is no magic for group match semantics ( /a, /aa,
347       /d, /l, /u) or modifiers that can be repeated, like /x and /xx, or /e
348       and /ee.
349
350   peek
351        my $character = $tokenizer->peek();
352        my $next_char = $tokenizer->peek( 1 );
353
354       This method returns the character at the given non-negative offset from
355       the current position. If no offset is given, an offset of 0 is used.
356
357       If you ask for a negative offset or an offset off the end of the sting,
358       "undef" is returned.
359
360   ppi_document
361       This method makes a PPI document out of the remainder of the string,
362       and returns it.
363
364   prior_significant_token
365        $tokenizer->prior_significant_token( 'can_be_quantified' )
366           and print "The prior token can be quantified.\n";
367
368       This method calls the named method on the most-recently-instantiated
369       significant token, and returns the result. Any arguments subsequent to
370       the method name will be passed to the method.
371
372       Because this method is designed to be used within the tokenizing
373       system, it will die horribly if the named method does not exist.
374
375       If called with no arguments at all the most-recently-instantiated
376       significant token is returned.
377
378   strict
379        say 'Parse is ', $tokenizer->strict() ? 'strict' : 'lenient';
380
381       This method simply returns true or false, depending on whether the
382       'strict' option to new() was true or false.
383

ENVIRONMENT VARIABLES

385       A tokenizer trace can be requested by setting environment variable
386       PPIX_REGEXP_TOKENIZER_TRACE to a numeric value other than 0. Use of
387       this environment variable is unsupported in the same sense that the
388       "trace" option of "new" is unsupported. Explicitly specifying the
389       "trace" option to "new" overrides the environment variable.
390
391       The real reason this is documented is to give the user a way to
392       troubleshoot funny output from the tokenizer.
393

SUPPORT

395       Support is by the author. Please file bug reports at
396       <https://rt.cpan.org/Public/Dist/Display.html?Name=PPIx-Regexp>,
397       <https://github.com/trwyant/perl-PPIx-Regexp/issues>, or in electronic
398       mail to the author.
399

AUTHOR

401       Thomas R. Wyant, III wyant at cpan dot org
402
404       Copyright (C) 2009-2023 by Thomas R. Wyant, III
405
406       This program is free software; you can redistribute it and/or modify it
407       under the same terms as Perl 5.10.0. For more details, see the full
408       text of the licenses in the directory LICENSES.
409
410       This program is distributed in the hope that it will be useful, but
411       without any warranty; without even the implied warranty of
412       merchantability or fitness for a particular purpose.
413
414
415
416perl v5.38.0                      2023-07-21        PPIx::Regexp::Tokenizer(3)
Impressum