1PERLREAPI(1) Perl Programmers Reference Guide PERLREAPI(1)
2
6 perlreapi - perl regular expression plugin interface
7
9 As of Perl 5.9.5 there is a new interface for plugging and using other
10 regular expression engines than the default one.
11 Each engine is supposed to provide access to a constant structure of
13 the following format:
14 typedef struct regexp_engine {
16 REGEXP* (*comp) (pTHX_ const SV * const pattern, const U32 flags);
17 I32 (*exec) (pTHX_ REGEXP * const rx, char* stringarg, char* strend,
18 char* strbeg, I32 minend, SV* screamer,
19 void* data, U32 flags);
20 char* (*intuit) (pTHX_ REGEXP * const rx, SV *sv, char *strpos,
21 char *strend, U32 flags,
22 struct re_scream_pos_data_s *data);
23 SV* (*checkstr) (pTHX_ REGEXP * const rx);
24 void (*free) (pTHX_ REGEXP * const rx);
25 void (*numbered_buff_FETCH) (pTHX_ REGEXP * const rx, const I32 paren,
26 SV * const sv);
27 void (*numbered_buff_STORE) (pTHX_ REGEXP * const rx, const I32 paren,
28 SV const * const value);
29 I32 (*numbered_buff_LENGTH) (pTHX_ REGEXP * const rx, const SV * const sv,
30 const I32 paren);
31 SV* (*named_buff) (pTHX_ REGEXP * const rx, SV * const key,
32 SV * const value, U32 flags);
33 SV* (*named_buff_iter) (pTHX_ REGEXP * const rx, const SV * const lastkey,
34 const U32 flags);
35 SV* (*qr_package)(pTHX_ REGEXP * const rx);
36 #ifdef USE_ITHREADS
37 void* (*dupe) (pTHX_ REGEXP * const rx, CLONE_PARAMS *param);
38 #endif
39 When a regexp is compiled, its "engine" field is then set to point at
41 the appropriate structure, so that when it needs to be used Perl can
42 find the right routines to do so.
43 In order to install a new regexp handler, $^H{regcomp} is set to an
45 integer which (when casted appropriately) resolves to one of these
46 structures. When compiling, the "comp" method is executed, and the
47 resulting regexp structure's engine field is expected to point back at
48 the same structure.
49 The pTHX_ symbol in the definition is a macro used by perl under
51 threading to provide an extra argument to the routine holding a pointer
52 back to the interpreter that is executing the regexp. So under
53 threading all routines get an extra argument.
54
56 comp
57 REGEXP* comp(pTHX_ const SV * const pattern, const U32 flags);
58 Compile the pattern stored in "pattern" using the given "flags" and
60 return a pointer to a prepared "REGEXP" structure that can perform the
61 match. See "The REGEXP structure" below for an explanation of the
62 individual fields in the REGEXP struct.
63 The "pattern" parameter is the scalar that was used as the pattern.
65 previous versions of perl would pass two "char*" indicating the start
66 and end of the stringified pattern, the following snippet can be used
67 to get the old parameters:
68 STRLEN plen;
70 char* exp = SvPV(pattern, plen);
71 char* xend = exp + plen;
72 Since any scalar can be passed as a pattern it's possible to implement
74 an engine that does something with an array (""ook" =~ [ qw/ eek hlagh
75 / ]") or with the non-stringified form of a compiled regular expression
76 (""ook" =~ qr/eek/"). perl's own engine will always stringify
77 everything using the snippet above but that doesn't mean other engines
78 have to.
79 The "flags" parameter is a bitfield which indicates which of the
81 "msixp" flags the regex was compiled with. It also contains additional
82 info such as whether "use locale" is in effect.
83 The "eogc" flags are stripped out before being passed to the comp
85 routine. The regex engine does not need to know whether any of these
86 are set as those flags should only affect what perl does with the
87 pattern and its match variables, not how it gets compiled and executed.
88 By the time the comp callback is called, some of these flags have
90 already had effect (noted below where applicable). However most of
91 their effect occurs after the comp callback has run in routines that
92 read the "rx->extflags" field which it populates.
93 In general the flags should be preserved in "rx->extflags" after
95 compilation, although the regex engine might want to add or delete some
96 of them to invoke or disable some special behavior in perl. The flags
97 along with any special behavior they cause are documented below:
98 The pattern modifiers:
100 "/m" - RXf_PMf_MULTILINE
102 If this is in "rx->extflags" it will be passed to "Perl_fbm_instr"
103 by "pp_split" which will treat the subject string as a multi-line
104 string.
105 "/s" - RXf_PMf_SINGLELINE
107 "/i" - RXf_PMf_FOLD
108 "/x" - RXf_PMf_EXTENDED
109 If present on a regex "#" comments will be handled differently by
110 the tokenizer in some cases.
111 TODO: Document those cases.
113 "/p" - RXf_PMf_KEEPCOPY
115 Additional flags:
117 RXf_PMf_LOCALE
119 Set if "use locale" is in effect. If present in "rx->extflags"
120 "split" will use the locale dependent definition of whitespace
121 under when RXf_SKIPWHITE or RXf_WHITE are in effect. Under ASCII
122 whitespace is defined as per isSPACE, and by the internal macros
123 "is_utf8_space" under UTF-8 and "isSPACE_LC" under "use locale".
124 RXf_UTF8
126 Set if the pattern is SvUTF8(), set by Perl_pmruntime.
127 A regex engine may want to set or disable this flag during
129 compilation. The perl engine for instance may upgrade non-UTF-8
130 strings to UTF-8 if the pattern includes constructs such as
131 "\x{...}" that can only match Unicode values.
132 RXf_SPLIT
134 If "split" is invoked as "split ' '" or with no arguments (which
135 really means "split(' ', $_)", see split), perl will set this flag.
136 The regex engine can then check for it and set the SKIPWHITE and
137 WHITE extflags. To do this the perl engine does:
138 if (flags & RXf_SPLIT && r->prelen == 1 && r->precomp[0] == ' ')
140 r->extflags |= (RXf_SKIPWHITE|RXf_WHITE);
141 These flags can be set during compilation to enable optimizations in
143 the "split" operator.
144 RXf_SKIPWHITE
146 If the flag is present in "rx->extflags" "split" will delete
147 whitespace from the start of the subject string before it's
148 operated on. What is considered whitespace depends on whether the
149 subject is a UTF-8 string and whether the "RXf_PMf_LOCALE" flag is
150 set.
151 If RXf_WHITE is set in addition to this flag "split" will behave
153 like "split " "" under the perl engine.
154 RXf_START_ONLY
156 Tells the split operator to split the target string on newlines
157 ("\n") without invoking the regex engine.
158 Perl's engine sets this if the pattern is "/^/" ("plen == 1 && *exp
160 == '^'"), even under "/^/s", see split. Of course a different regex
161 engine might want to use the same optimizations with a different
162 syntax.
163 RXf_WHITE
165 Tells the split operator to split the target string on whitespace
166 without invoking the regex engine. The definition of whitespace
167 varies depending on whether the target string is a UTF-8 string and
168 on whether RXf_PMf_LOCALE is set.
169 Perl's engine sets this flag if the pattern is "\s+".
171 RXf_NULL
173 Tells the split operator to split the target string on characters.
174 The definition of character varies depending on whether the target
175 string is a UTF-8 string.
176 Perl's engine sets this flag on empty patterns, this optimization
178 makes "split //" much faster than it would otherwise be. It's even
179 faster than "unpack".
180 exec
182 I32 exec(pTHX_ REGEXP * const rx,
183 char *stringarg, char* strend, char* strbeg,
184 I32 minend, SV* screamer,
185 void* data, U32 flags);
186 Execute a regexp.
188 intuit
190 char* intuit(pTHX_ REGEXP * const rx,
191 SV *sv, char *strpos, char *strend,
192 const U32 flags, struct re_scream_pos_data_s *data);
193 Find the start position where a regex match should be attempted, or
195 possibly whether the regex engine should not be run because the pattern
196 can't match. This is called as appropriate by the core depending on the
197 values of the extflags member of the regexp structure.
198 checkstr
200 SV* checkstr(pTHX_ REGEXP * const rx);
201 Return a SV containing a string that must appear in the pattern. Used
203 by "split" for optimising matches.
204 free
206 void free(pTHX_ REGEXP * const rx);
207 Called by perl when it is freeing a regexp pattern so that the engine
209 can release any resources pointed to by the "pprivate" member of the
210 regexp structure. This is only responsible for freeing private data;
211 perl will handle releasing anything else contained in the regexp
212 structure.
213 Numbered capture callbacks
215 Called to get/set the value of "$`", "$'", $& and their named
216 equivalents, ${^PREMATCH}, ${^POSTMATCH} and $^{MATCH}, as well as the
217 numbered capture buffers ($1, $2, ...).
218 The "paren" parameter will be "-2" for "$`", "-1" for "$'", 0 for $&, 1
220 for $1 and so forth.
221 The names have been chosen by analogy with Tie::Scalar methods names
223 with an additional LENGTH callback for efficiency. However named
224 capture variables are currently not tied internally but implemented via
225 magic.
226 numbered_buff_FETCH
228 void numbered_buff_FETCH(pTHX_ REGEXP * const rx, const I32 paren,
230 SV * const sv);
231 Fetch a specified numbered capture. "sv" should be set to the scalar to
233 return, the scalar is passed as an argument rather than being returned
234 from the function because when it's called perl already has a scalar to
235 store the value, creating another one would be redundant. The scalar
236 can be set with "sv_setsv", "sv_setpvn" and friends, see perlapi.
237 This callback is where perl untaints its own capture variables under
239 taint mode (see perlsec). See the "Perl_reg_numbered_buff_fetch"
240 function in regcomp.c for how to untaint capture variables if that's
241 something you'd like your engine to do as well.
242 numbered_buff_STORE
244 void (*numbered_buff_STORE) (pTHX_ REGEXP * const rx, const I32 paren,
246 SV const * const value);
247 Set the value of a numbered capture variable. "value" is the scalar
249 that is to be used as the new value. It's up to the engine to make sure
250 this is used as the new value (or reject it).
251 Example:
253 if ("ook" =~ /(o*)/) {
255 # `paren' will be `1' and `value' will be `ee'
256 $1 =~ tr/o/e/;
257 }
258 Perl's own engine will croak on any attempt to modify the capture
260 variables, to do this in another engine use the following callback
261 (copied from "Perl_reg_numbered_buff_store"):
262 void
264 Example_reg_numbered_buff_store(pTHX_ REGEXP * const rx, const I32 paren,
265 SV const * const value)
266 {
267 PERL_UNUSED_ARG(rx);
268 PERL_UNUSED_ARG(paren);
269 PERL_UNUSED_ARG(value);
270 if (!PL_localizing)
272 Perl_croak(aTHX_ PL_no_modify);
273 }
274 Actually perl will not always croak in a statement that looks like it
276 would modify a numbered capture variable. This is because the STORE
277 callback will not be called if perl can determine that it doesn't have
278 to modify the value. This is exactly how tied variables behave in the
279 same situation:
280 package CaptureVar;
282 use base 'Tie::Scalar';
283 sub TIESCALAR { bless [] }
285 sub FETCH { undef }
286 sub STORE { die "This doesn't get called" }
287 package main;
289 tie my $sv => "CatptureVar";
291 $sv =~ y/a/b/;
292 Because $sv is "undef" when the "y///" operator is applied to it the
294 transliteration won't actually execute and the program won't "die".
295 This is different to how 5.8 and earlier versions behaved since the
296 capture variables were READONLY variables then, now they'll just die
297 when assigned to in the default engine.
298 numbered_buff_LENGTH
300 I32 numbered_buff_LENGTH (pTHX_ REGEXP * const rx, const SV * const sv,
302 const I32 paren);
303 Get the "length" of a capture variable. There's a special callback for
305 this so that perl doesn't have to do a FETCH and run "length" on the
306 result, since the length is (in perl's case) known from an offset
307 stored in "rx->offs" this is much more efficient:
308 I32 s1 = rx->offs[paren].start;
310 I32 s2 = rx->offs[paren].end;
311 I32 len = t1 - s1;
312 This is a little bit more complex in the case of UTF-8, see what
314 "Perl_reg_numbered_buff_length" does with is_utf8_string_loclen.
315 Named capture callbacks
317 Called to get/set the value of "%+" and "%-" as well as by some utility
318 functions in re.
319 There are two callbacks, "named_buff" is called in all the cases the
321 FETCH, STORE, DELETE, CLEAR, EXISTS and SCALAR Tie::Hash callbacks
322 would be on changes to "%+" and "%-" and "named_buff_iter" in the same
323 cases as FIRSTKEY and NEXTKEY.
324 The "flags" parameter can be used to determine which of these
326 operations the callbacks should respond to, the following flags are
327 currently defined:
328 Which Tie::Hash operation is being performed from the Perl level on
330 "%+" or "%+", if any:
331 RXapif_FETCH
333 RXapif_STORE
334 RXapif_DELETE
335 RXapif_CLEAR
336 RXapif_EXISTS
337 RXapif_SCALAR
338 RXapif_FIRSTKEY
339 RXapif_NEXTKEY
340 Whether "%+" or "%-" is being operated on, if any.
342 RXapif_ONE /* %+ */
344 RXapif_ALL /* %- */
345 Whether this is being called as "re::regname", "re::regnames" or
347 "re::regnames_count", if any. The first two will be combined with
348 "RXapif_ONE" or "RXapif_ALL".
349 RXapif_REGNAME
351 RXapif_REGNAMES
352 RXapif_REGNAMES_COUNT
353 Internally "%+" and "%-" are implemented with a real tied interface via
355 Tie::Hash::NamedCapture. The methods in that package will call back
356 into these functions. However the usage of Tie::Hash::NamedCapture for
357 this purpose might change in future releases. For instance this might
358 be implemented by magic instead (would need an extension to mgvtbl).
359 named_buff
361 SV* (*named_buff) (pTHX_ REGEXP * const rx, SV * const key,
363 SV * const value, U32 flags);
364 named_buff_iter
366 SV* (*named_buff_iter) (pTHX_ REGEXP * const rx, const SV * const lastkey,
368 const U32 flags);
369 qr_package
371 SV* qr_package(pTHX_ REGEXP * const rx);
372 The package the qr// magic object is blessed into (as seen by "ref
374 qr//"). It is recommended that engines change this to their package
375 name for identification regardless of whether they implement methods on
376 the object.
377 The package this method returns should also have the internal "Regexp"
379 package in its @ISA. "qr//->isa("Regexp")" should always be true
380 regardless of what engine is being used.
381 Example implementation might be:
383 SV*
385 Example_qr_package(pTHX_ REGEXP * const rx)
386 {
387 PERL_UNUSED_ARG(rx);
388 return newSVpvs("re::engine::Example");
389 }
390 Any method calls on an object created with "qr//" will be dispatched to
392 the package as a normal object.
393 use re::engine::Example;
395 my $re = qr//;
396 $re->meth; # dispatched to re::engine::Example::meth()
397 To retrieve the "REGEXP" object from the scalar in an XS function use
399 the "SvRX" macro, see "REGEXP Functions" in perlapi.
400 void meth(SV * rv)
402 PPCODE:
403 REGEXP * re = SvRX(sv);
404 dupe
406 void* dupe(pTHX_ REGEXP * const rx, CLONE_PARAMS *param);
407 On threaded builds a regexp may need to be duplicated so that the
409 pattern can be used by multiple threads. This routine is expected to
410 handle the duplication of any private data pointed to by the "pprivate"
411 member of the regexp structure. It will be called with the
412 preconstructed new regexp structure as an argument, the "pprivate"
413 member will point at the old private structure, and it is this
414 routine's responsibility to construct a copy and return a pointer to it
415 (which perl will then use to overwrite the field as passed to this
416 routine.)
417 This allows the engine to dupe its private data but also if necessary
419 modify the final structure if it really must.
420 On unthreaded builds this field doesn't exist.
422
424 The REGEXP struct is defined in regexp.h. All regex engines must be
425 able to correctly build such a structure in their "comp" routine.
426 The REGEXP structure contains all the data that perl needs to be aware
428 of to properly work with the regular expression. It includes data about
429 optimisations that perl can use to determine if the regex engine should
430 really be used, and various other control info that is needed to
431 properly execute patterns in various contexts such as is the pattern
432 anchored in some way, or what flags were used during the compile, or
433 whether the program contains special constructs that perl needs to be
434 aware of.
435 In addition it contains two fields that are intended for the private
437 use of the regex engine that compiled the pattern. These are the
438 "intflags" and "pprivate" members. "pprivate" is a void pointer to an
439 arbitrary structure whose use and management is the responsibility of
440 the compiling engine. perl will never modify either of these values.
441 typedef struct regexp {
443 /* what engine created this regexp? */
444 const struct regexp_engine* engine;
445 /* what re is this a lightweight copy of? */
447 struct regexp* mother_re;
448 /* Information about the match that the perl core uses to manage things */
450 U32 extflags; /* Flags used both externally and internally */
451 I32 minlen; /* mininum possible length of string to match */
452 I32 minlenret; /* mininum possible length of $& */
453 U32 gofs; /* chars left of pos that we search from */
454 /* substring data about strings that must appear
456 in the final match, used for optimisations */
457 struct reg_substr_data *substrs;
458 U32 nparens; /* number of capture buffers */
460 /* private engine specific data */
462 U32 intflags; /* Engine Specific Internal flags */
463 void *pprivate; /* Data private to the regex engine which
464 created this object. */
465 /* Data about the last/current match. These are modified during matching*/
467 U32 lastparen; /* last open paren matched */
468 U32 lastcloseparen; /* last close paren matched */
469 regexp_paren_pair *swap; /* Swap copy of *offs */
470 regexp_paren_pair *offs; /* Array of offsets for (@-) and (@+) */
471 char *subbeg; /* saved or original string so \digit works forever. */
473 SV_SAVED_COPY /* If non-NULL, SV which is COW from original */
474 I32 sublen; /* Length of string pointed by subbeg */
475 /* Information about the match that isn't often used */
477 I32 prelen; /* length of precomp */
478 const char *precomp; /* pre-compilation regular expression */
479 char *wrapped; /* wrapped version of the pattern */
481 I32 wraplen; /* length of wrapped */
482 I32 seen_evals; /* number of eval groups in the pattern - for security checks */
484 HV *paren_names; /* Optional hash of paren names */
485 /* Refcount of this regexp */
487 I32 refcnt; /* Refcount of this regexp */
488 } regexp;
489 The fields are discussed in more detail below:
491 "engine"
493 This field points at a regexp_engine structure which contains pointers
494 to the subroutines that are to be used for performing a match. It is
495 the compiling routine's responsibility to populate this field before
496 returning the regexp object.
497 Internally this is set to "NULL" unless a custom engine is specified in
499 $^H{regcomp}, perl's own set of callbacks can be accessed in the struct
500 pointed to by "RE_ENGINE_PTR".
501 "mother_re"
503 TODO, see
504 http://www.mail-archive.com/perl5-changes@perl.org/msg17328.html
505 <http://www.mail-archive.com/perl5-changes@perl.org/msg17328.html>
506 "extflags"
508 This will be used by perl to see what flags the regexp was compiled
509 with, this will normally be set to the value of the flags parameter by
510 the comp callback. See the comp documentation for valid flags.
511 "minlen" "minlenret"
513 The minimum string length required for the pattern to match. This is
514 used to prune the search space by not bothering to match any closer to
515 the end of a string than would allow a match. For instance there is no
516 point in even starting the regex engine if the minlen is 10 but the
517 string is only 5 characters long. There is no way that the pattern can
518 match.
519 "minlenret" is the minimum length of the string that would be found in
521 $& after a match.
522 The difference between "minlen" and "minlenret" can be seen in the
524 following pattern:
525 /ns(?=\d)/
527 where the "minlen" would be 3 but "minlenret" would only be 2 as the \d
529 is required to match but is not actually included in the matched
530 content. This distinction is particularly important as the substitution
531 logic uses the "minlenret" to tell whether it can do in-place
532 substitution which can result in considerable speedup.
533 "gofs"
535 Left offset from pos() to start match at.
536 "substrs"
538 Substring data about strings that must appear in the final match. This
539 is currently only used internally by perl's engine for but might be
540 used in the future for all engines for optimisations.
541 "nparens", "lasparen", and "lastcloseparen"
543 These fields are used to keep track of how many paren groups could be
544 matched in the pattern, which was the last open paren to be entered,
545 and which was the last close paren to be entered.
546 "intflags"
548 The engine's private copy of the flags the pattern was compiled with.
549 Usually this is the same as "extflags" unless the engine chose to
550 modify one of them.
551 "pprivate"
553 A void* pointing to an engine-defined data structure. The perl engine
554 uses the "regexp_internal" structure (see "Base Structures" in
555 perlreguts) but a custom engine should use something else.
556 "swap"
558 Unused. Left in for compatibility with perl 5.10.0.
559 "offs"
561 A "regexp_paren_pair" structure which defines offsets into the string
562 being matched which correspond to the $& and $1, $2 etc. captures, the
563 "regexp_paren_pair" struct is defined as follows:
564 typedef struct regexp_paren_pair {
566 I32 start;
567 I32 end;
568 } regexp_paren_pair;
569 If "->offs[num].start" or "->offs[num].end" is "-1" then that capture
571 buffer did not match. "->offs[0].start/end" represents $& (or
572 "${^MATCH" under "//p") and "->offs[paren].end" matches $$paren where
573 $paren = 1>.
574 "precomp" "prelen"
576 Used for optimisations. "precomp" holds a copy of the pattern that was
577 compiled and "prelen" its length. When a new pattern is to be compiled
578 (such as inside a loop) the internal "regcomp" operator checks whether
579 the last compiled "REGEXP"'s "precomp" and "prelen" are equivalent to
580 the new one, and if so uses the old pattern instead of compiling a new
581 one.
582 The relevant snippet from "Perl_pp_regcomp":
584 if (!re || !re->precomp || re->prelen != (I32)len ||
586 memNE(re->precomp, t, len))
587 /* Compile a new pattern */
588 "paren_names"
590 This is a hash used internally to track named capture buffers and their
591 offsets. The keys are the names of the buffers the values are dualvars,
592 with the IV slot holding the number of buffers with the given name and
593 the pv being an embedded array of I32. The values may also be
594 contained independently in the data array in cases where named
595 backreferences are used.
596 "substrs"
598 Holds information on the longest string that must occur at a fixed
599 offset from the start of the pattern, and the longest string that must
600 occur at a floating offset from the start of the pattern. Used to do
601 Fast-Boyer-Moore searches on the string to find out if its worth using
602 the regex engine at all, and if so where in the string to search.
603 "subbeg" "sublen" "saved_copy"
605 Used during execution phase for managing search and replace patterns.
606 "wrapped" "wraplen"
608 Stores the string "qr//" stringifies to. The perl engine for example
609 stores "(?-xism:eek)" in the case of "qr/eek/".
610 When using a custom engine that doesn't support the "(?:)" construct
612 for inline modifiers, it's probably best to have "qr//" stringify to
613 the supplied pattern, note that this will create undesired patterns in
614 cases such as:
615 my $x = qr/a|b/; # "a|b"
617 my $y = qr/c/i; # "c"
618 my $z = qr/$x$y/; # "a|bc"
619 There's no solution for this problem other than making the custom
621 engine understand a construct like "(?:)".
622 "seen_evals"
624 This stores the number of eval groups in the pattern. This is used for
625 security purposes when embedding compiled regexes into larger patterns
626 with "qr//".
627 "refcnt"
629 The number of times the structure is referenced. When this falls to 0
630 the regexp is automatically freed by a call to pregfree. This should be
631 set to 1 in each engine's "comp" routine.
632
634 Originally part of perlreguts.
635
637 Originally written by Yves Orton, expanded by var Arnfjoerd` Bjarmason.
638
640 Copyright 2006 Yves Orton and 2007 var Arnfjoerd` Bjarmason.
641 This program is free software; you can redistribute it and/or modify it
643 under the same terms as Perl itself.
644perl v5.12.4 2011-06-07 PERLREAPI(1)