1PERLREAPI(1) Perl Programmers Reference Guide PERLREAPI(1)
2
6 perlreapi - perl regular expression plugin interface
7
9 As of Perl 5.9.5 there is a new interface for plugging and using other
10 regular expression engines than the default one.
11 Each engine is supposed to provide access to a constant structure of
13 the following format:
14 typedef struct regexp_engine {
16 REGEXP* (*comp) (pTHX_ const SV * const pattern, const U32 flags);
17 I32 (*exec) (pTHX_ REGEXP * const rx, char* stringarg, char* strend,
18 char* strbeg, I32 minend, SV* screamer,
19 void* data, U32 flags);
20 char* (*intuit) (pTHX_ REGEXP * const rx, SV *sv, char *strpos,
21 char *strend, U32 flags,
22 struct re_scream_pos_data_s *data);
23 SV* (*checkstr) (pTHX_ REGEXP * const rx);
24 void (*free) (pTHX_ REGEXP * const rx);
25 void (*numbered_buff_FETCH) (pTHX_ REGEXP * const rx, const I32 paren,
26 SV * const sv);
27 void (*numbered_buff_STORE) (pTHX_ REGEXP * const rx, const I32 paren,
28 SV const * const value);
29 I32 (*numbered_buff_LENGTH) (pTHX_ REGEXP * const rx, const SV * const sv,
30 const I32 paren);
31 SV* (*named_buff) (pTHX_ REGEXP * const rx, SV * const key,
32 SV * const value, U32 flags);
33 SV* (*named_buff_iter) (pTHX_ REGEXP * const rx, const SV * const lastkey,
34 const U32 flags);
35 SV* (*qr_package)(pTHX_ REGEXP * const rx);
36 #ifdef USE_ITHREADS
37 void* (*dupe) (pTHX_ REGEXP * const rx, CLONE_PARAMS *param);
38 #endif
39 When a regexp is compiled, its "engine" field is then set to point at
41 the appropriate structure, so that when it needs to be used Perl can
42 find the right routines to do so.
43 In order to install a new regexp handler, $^H{regcomp} is set to an
45 integer which (when casted appropriately) resolves to one of these
46 structures. When compiling, the "comp" method is executed, and the
47 resulting regexp structure's engine field is expected to point back at
48 the same structure.
49 The pTHX_ symbol in the definition is a macro used by perl under
51 threading to provide an extra argument to the routine holding a pointer
52 back to the interpreter that is executing the regexp. So under
53 threading all routines get an extra argument.
54
56 comp
57 REGEXP* comp(pTHX_ const SV * const pattern, const U32 flags);
58 Compile the pattern stored in "pattern" using the given "flags" and
60 return a pointer to a prepared "REGEXP" structure that can perform the
61 match. See "The REGEXP structure" below for an explanation of the
62 individual fields in the REGEXP struct.
63 The "pattern" parameter is the scalar that was used as the pattern.
65 previous versions of perl would pass two "char*" indicating the start
66 and end of the stringified pattern, the following snippet can be used
67 to get the old parameters:
68 STRLEN plen;
70 char* exp = SvPV(pattern, plen);
71 char* xend = exp + plen;
72 Since any scalar can be passed as a pattern it's possible to implement
74 an engine that does something with an array (""ook" =~ [ qw/ eek hlagh
75 / ]") or with the non-stringified form of a compiled regular expression
76 (""ook" =~ qr/eek/"). perl's own engine will always stringify
77 everything using the snippet above but that doesn't mean other engines
78 have to.
79 The "flags" parameter is a bitfield which indicates which of the
81 "msixp" flags the regex was compiled with. It also contains additional
82 info such as whether "use locale" is in effect.
83 The "eogc" flags are stripped out before being passed to the comp
85 routine. The regex engine does not need to know whether any of these
86 are set as those flags should only affect what perl does with the
87 pattern and its match variables, not how it gets compiled and executed.
88 By the time the comp callback is called, some of these flags have
90 already had effect (noted below where applicable). However most of
91 their effect occurs after the comp callback has run in routines that
92 read the "rx->extflags" field which it populates.
93 In general the flags should be preserved in "rx->extflags" after
95 compilation, although the regex engine might want to add or delete some
96 of them to invoke or disable some special behavior in perl. The flags
97 along with any special behavior they cause are documented below:
98 The pattern modifiers:
100 "/m" - RXf_PMf_MULTILINE
102 If this is in "rx->extflags" it will be passed to "Perl_fbm_instr"
103 by "pp_split" which will treat the subject string as a multi-line
104 string.
105 "/s" - RXf_PMf_SINGLELINE
107 "/i" - RXf_PMf_FOLD
108 "/x" - RXf_PMf_EXTENDED
109 If present on a regex "#" comments will be handled differently by
110 the tokenizer in some cases.
111 TODO: Document those cases.
113 "/p" - RXf_PMf_KEEPCOPY
115 TODO: Document this
116 Character set
118 The character set semantics are determined by an enum that is
119 contained in this field. This is still experimental and subject to
120 change, but the current interface returns the rules by use of the
121 in-line function "get_regex_charset(const U32 flags)". The only
122 currently documented value returned from it is
123 REGEX_LOCALE_CHARSET, which is set if "use locale" is in effect. If
124 present in "rx->extflags", "split" will use the locale dependent
125 definition of whitespace when RXf_SKIPWHITE or RXf_WHITE is in
126 effect. ASCII whitespace is defined as per isSPACE, and by the
127 internal macros "is_utf8_space" under UTF-8, and "isSPACE_LC" under
128 "use locale".
129 Additional flags:
131 RXf_UTF8
133 Set if the pattern is SvUTF8(), set by Perl_pmruntime.
134 A regex engine may want to set or disable this flag during
136 compilation. The perl engine for instance may upgrade non-UTF-8
137 strings to UTF-8 if the pattern includes constructs such as
138 "\x{...}" that can only match Unicode values.
139 RXf_SPLIT
141 If "split" is invoked as "split ' '" or with no arguments (which
142 really means "split(' ', $_)", see split), perl will set this flag.
143 The regex engine can then check for it and set the SKIPWHITE and
144 WHITE extflags. To do this the perl engine does:
145 if (flags & RXf_SPLIT && r->prelen == 1 && r->precomp[0] == ' ')
147 r->extflags |= (RXf_SKIPWHITE|RXf_WHITE);
148 These flags can be set during compilation to enable optimizations in
150 the "split" operator.
151 RXf_SKIPWHITE
153 If the flag is present in "rx->extflags" "split" will delete
154 whitespace from the start of the subject string before it's
155 operated on. What is considered whitespace depends on whether the
156 subject is a UTF-8 string and whether the "RXf_PMf_LOCALE" flag is
157 set.
158 If RXf_WHITE is set in addition to this flag "split" will behave
160 like "split " "" under the perl engine.
161 RXf_START_ONLY
163 Tells the split operator to split the target string on newlines
164 ("\n") without invoking the regex engine.
165 Perl's engine sets this if the pattern is "/^/" ("plen == 1 && *exp
167 == '^'"), even under "/^/s", see split. Of course a different regex
168 engine might want to use the same optimizations with a different
169 syntax.
170 RXf_WHITE
172 Tells the split operator to split the target string on whitespace
173 without invoking the regex engine. The definition of whitespace
174 varies depending on whether the target string is a UTF-8 string and
175 on whether RXf_PMf_LOCALE is set.
176 Perl's engine sets this flag if the pattern is "\s+".
178 RXf_NULL
180 Tells the split operator to split the target string on characters.
181 The definition of character varies depending on whether the target
182 string is a UTF-8 string.
183 Perl's engine sets this flag on empty patterns, this optimization
185 makes "split //" much faster than it would otherwise be. It's even
186 faster than "unpack".
187 exec
189 I32 exec(pTHX_ REGEXP * const rx,
190 char *stringarg, char* strend, char* strbeg,
191 I32 minend, SV* screamer,
192 void* data, U32 flags);
193 Execute a regexp.
195 intuit
197 char* intuit(pTHX_ REGEXP * const rx,
198 SV *sv, char *strpos, char *strend,
199 const U32 flags, struct re_scream_pos_data_s *data);
200 Find the start position where a regex match should be attempted, or
202 possibly whether the regex engine should not be run because the pattern
203 can't match. This is called as appropriate by the core depending on the
204 values of the extflags member of the regexp structure.
205 checkstr
207 SV* checkstr(pTHX_ REGEXP * const rx);
208 Return a SV containing a string that must appear in the pattern. Used
210 by "split" for optimising matches.
211 free
213 void free(pTHX_ REGEXP * const rx);
214 Called by perl when it is freeing a regexp pattern so that the engine
216 can release any resources pointed to by the "pprivate" member of the
217 regexp structure. This is only responsible for freeing private data;
218 perl will handle releasing anything else contained in the regexp
219 structure.
220 Numbered capture callbacks
222 Called to get/set the value of "$`", "$'", $& and their named
223 equivalents, ${^PREMATCH}, ${^POSTMATCH} and $^{MATCH}, as well as the
224 numbered capture groups ($1, $2, ...).
225 The "paren" parameter will be "-2" for "$`", "-1" for "$'", 0 for $&, 1
227 for $1 and so forth.
228 The names have been chosen by analogy with Tie::Scalar methods names
230 with an additional LENGTH callback for efficiency. However named
231 capture variables are currently not tied internally but implemented via
232 magic.
233 numbered_buff_FETCH
235 void numbered_buff_FETCH(pTHX_ REGEXP * const rx, const I32 paren,
237 SV * const sv);
238 Fetch a specified numbered capture. "sv" should be set to the scalar to
240 return, the scalar is passed as an argument rather than being returned
241 from the function because when it's called perl already has a scalar to
242 store the value, creating another one would be redundant. The scalar
243 can be set with "sv_setsv", "sv_setpvn" and friends, see perlapi.
244 This callback is where perl untaints its own capture variables under
246 taint mode (see perlsec). See the "Perl_reg_numbered_buff_fetch"
247 function in regcomp.c for how to untaint capture variables if that's
248 something you'd like your engine to do as well.
249 numbered_buff_STORE
251 void (*numbered_buff_STORE) (pTHX_ REGEXP * const rx, const I32 paren,
253 SV const * const value);
254 Set the value of a numbered capture variable. "value" is the scalar
256 that is to be used as the new value. It's up to the engine to make sure
257 this is used as the new value (or reject it).
258 Example:
260 if ("ook" =~ /(o*)/) {
262 # 'paren' will be '1' and 'value' will be 'ee'
263 $1 =~ tr/o/e/;
264 }
265 Perl's own engine will croak on any attempt to modify the capture
267 variables, to do this in another engine use the following callback
268 (copied from "Perl_reg_numbered_buff_store"):
269 void
271 Example_reg_numbered_buff_store(pTHX_ REGEXP * const rx, const I32 paren,
272 SV const * const value)
273 {
274 PERL_UNUSED_ARG(rx);
275 PERL_UNUSED_ARG(paren);
276 PERL_UNUSED_ARG(value);
277 if (!PL_localizing)
279 Perl_croak(aTHX_ PL_no_modify);
280 }
281 Actually perl will not always croak in a statement that looks like it
283 would modify a numbered capture variable. This is because the STORE
284 callback will not be called if perl can determine that it doesn't have
285 to modify the value. This is exactly how tied variables behave in the
286 same situation:
287 package CaptureVar;
289 use base 'Tie::Scalar';
290 sub TIESCALAR { bless [] }
292 sub FETCH { undef }
293 sub STORE { die "This doesn't get called" }
294 package main;
296 tie my $sv => "CaptureVar";
298 $sv =~ y/a/b/;
299 Because $sv is "undef" when the "y///" operator is applied to it the
301 transliteration won't actually execute and the program won't "die".
302 This is different to how 5.8 and earlier versions behaved since the
303 capture variables were READONLY variables then, now they'll just die
304 when assigned to in the default engine.
305 numbered_buff_LENGTH
307 I32 numbered_buff_LENGTH (pTHX_ REGEXP * const rx, const SV * const sv,
309 const I32 paren);
310 Get the "length" of a capture variable. There's a special callback for
312 this so that perl doesn't have to do a FETCH and run "length" on the
313 result, since the length is (in perl's case) known from an offset
314 stored in "rx->offs" this is much more efficient:
315 I32 s1 = rx->offs[paren].start;
317 I32 s2 = rx->offs[paren].end;
318 I32 len = t1 - s1;
319 This is a little bit more complex in the case of UTF-8, see what
321 "Perl_reg_numbered_buff_length" does with is_utf8_string_loclen.
322 Named capture callbacks
324 Called to get/set the value of "%+" and "%-" as well as by some utility
325 functions in re.
326 There are two callbacks, "named_buff" is called in all the cases the
328 FETCH, STORE, DELETE, CLEAR, EXISTS and SCALAR Tie::Hash callbacks
329 would be on changes to "%+" and "%-" and "named_buff_iter" in the same
330 cases as FIRSTKEY and NEXTKEY.
331 The "flags" parameter can be used to determine which of these
333 operations the callbacks should respond to, the following flags are
334 currently defined:
335 Which Tie::Hash operation is being performed from the Perl level on
337 "%+" or "%+", if any:
338 RXapif_FETCH
340 RXapif_STORE
341 RXapif_DELETE
342 RXapif_CLEAR
343 RXapif_EXISTS
344 RXapif_SCALAR
345 RXapif_FIRSTKEY
346 RXapif_NEXTKEY
347 Whether "%+" or "%-" is being operated on, if any.
349 RXapif_ONE /* %+ */
351 RXapif_ALL /* %- */
352 Whether this is being called as "re::regname", "re::regnames" or
354 "re::regnames_count", if any. The first two will be combined with
355 "RXapif_ONE" or "RXapif_ALL".
356 RXapif_REGNAME
358 RXapif_REGNAMES
359 RXapif_REGNAMES_COUNT
360 Internally "%+" and "%-" are implemented with a real tied interface via
362 Tie::Hash::NamedCapture. The methods in that package will call back
363 into these functions. However the usage of Tie::Hash::NamedCapture for
364 this purpose might change in future releases. For instance this might
365 be implemented by magic instead (would need an extension to mgvtbl).
366 named_buff
368 SV* (*named_buff) (pTHX_ REGEXP * const rx, SV * const key,
370 SV * const value, U32 flags);
371 named_buff_iter
373 SV* (*named_buff_iter) (pTHX_ REGEXP * const rx, const SV * const lastkey,
375 const U32 flags);
376 qr_package
378 SV* qr_package(pTHX_ REGEXP * const rx);
379 The package the qr// magic object is blessed into (as seen by "ref
381 qr//"). It is recommended that engines change this to their package
382 name for identification regardless of whether they implement methods on
383 the object.
384 The package this method returns should also have the internal "Regexp"
386 package in its @ISA. "qr//->isa("Regexp")" should always be true
387 regardless of what engine is being used.
388 Example implementation might be:
390 SV*
392 Example_qr_package(pTHX_ REGEXP * const rx)
393 {
394 PERL_UNUSED_ARG(rx);
395 return newSVpvs("re::engine::Example");
396 }
397 Any method calls on an object created with "qr//" will be dispatched to
399 the package as a normal object.
400 use re::engine::Example;
402 my $re = qr//;
403 $re->meth; # dispatched to re::engine::Example::meth()
404 To retrieve the "REGEXP" object from the scalar in an XS function use
406 the "SvRX" macro, see "REGEXP Functions" in perlapi.
407 void meth(SV * rv)
409 PPCODE:
410 REGEXP * re = SvRX(sv);
411 dupe
413 void* dupe(pTHX_ REGEXP * const rx, CLONE_PARAMS *param);
414 On threaded builds a regexp may need to be duplicated so that the
416 pattern can be used by multiple threads. This routine is expected to
417 handle the duplication of any private data pointed to by the "pprivate"
418 member of the regexp structure. It will be called with the
419 preconstructed new regexp structure as an argument, the "pprivate"
420 member will point at the old private structure, and it is this
421 routine's responsibility to construct a copy and return a pointer to it
422 (which perl will then use to overwrite the field as passed to this
423 routine.)
424 This allows the engine to dupe its private data but also if necessary
426 modify the final structure if it really must.
427 On unthreaded builds this field doesn't exist.
429
431 The REGEXP struct is defined in regexp.h. All regex engines must be
432 able to correctly build such a structure in their "comp" routine.
433 The REGEXP structure contains all the data that perl needs to be aware
435 of to properly work with the regular expression. It includes data about
436 optimisations that perl can use to determine if the regex engine should
437 really be used, and various other control info that is needed to
438 properly execute patterns in various contexts such as is the pattern
439 anchored in some way, or what flags were used during the compile, or
440 whether the program contains special constructs that perl needs to be
441 aware of.
442 In addition it contains two fields that are intended for the private
444 use of the regex engine that compiled the pattern. These are the
445 "intflags" and "pprivate" members. "pprivate" is a void pointer to an
446 arbitrary structure whose use and management is the responsibility of
447 the compiling engine. perl will never modify either of these values.
448 typedef struct regexp {
450 /* what engine created this regexp? */
451 const struct regexp_engine* engine;
452 /* what re is this a lightweight copy of? */
454 struct regexp* mother_re;
455 /* Information about the match that the perl core uses to manage things */
457 U32 extflags; /* Flags used both externally and internally */
458 I32 minlen; /* mininum possible length of string to match */
459 I32 minlenret; /* mininum possible length of $& */
460 U32 gofs; /* chars left of pos that we search from */
461 /* substring data about strings that must appear
463 in the final match, used for optimisations */
464 struct reg_substr_data *substrs;
465 U32 nparens; /* number of capture groups */
467 /* private engine specific data */
469 U32 intflags; /* Engine Specific Internal flags */
470 void *pprivate; /* Data private to the regex engine which
471 created this object. */
472 /* Data about the last/current match. These are modified during matching*/
474 U32 lastparen; /* last open paren matched */
475 U32 lastcloseparen; /* last close paren matched */
476 regexp_paren_pair *swap; /* Swap copy of *offs */
477 regexp_paren_pair *offs; /* Array of offsets for (@-) and (@+) */
478 char *subbeg; /* saved or original string so \digit works forever. */
480 SV_SAVED_COPY /* If non-NULL, SV which is COW from original */
481 I32 sublen; /* Length of string pointed by subbeg */
482 /* Information about the match that isn't often used */
484 I32 prelen; /* length of precomp */
485 const char *precomp; /* pre-compilation regular expression */
486 char *wrapped; /* wrapped version of the pattern */
488 I32 wraplen; /* length of wrapped */
489 I32 seen_evals; /* number of eval groups in the pattern - for security checks */
491 HV *paren_names; /* Optional hash of paren names */
492 /* Refcount of this regexp */
494 I32 refcnt; /* Refcount of this regexp */
495 } regexp;
496 The fields are discussed in more detail below:
498 "engine"
500 This field points at a regexp_engine structure which contains pointers
501 to the subroutines that are to be used for performing a match. It is
502 the compiling routine's responsibility to populate this field before
503 returning the regexp object.
504 Internally this is set to "NULL" unless a custom engine is specified in
506 $^H{regcomp}, perl's own set of callbacks can be accessed in the struct
507 pointed to by "RE_ENGINE_PTR".
508 "mother_re"
510 TODO, see
511 http://www.mail-archive.com/perl5-changes@perl.org/msg17328.html
512 <http://www.mail-archive.com/perl5-changes@perl.org/msg17328.html>
513 "extflags"
515 This will be used by perl to see what flags the regexp was compiled
516 with, this will normally be set to the value of the flags parameter by
517 the comp callback. See the comp documentation for valid flags.
518 "minlen" "minlenret"
520 The minimum string length required for the pattern to match. This is
521 used to prune the search space by not bothering to match any closer to
522 the end of a string than would allow a match. For instance there is no
523 point in even starting the regex engine if the minlen is 10 but the
524 string is only 5 characters long. There is no way that the pattern can
525 match.
526 "minlenret" is the minimum length of the string that would be found in
528 $& after a match.
529 The difference between "minlen" and "minlenret" can be seen in the
531 following pattern:
532 /ns(?=\d)/
534 where the "minlen" would be 3 but "minlenret" would only be 2 as the \d
536 is required to match but is not actually included in the matched
537 content. This distinction is particularly important as the substitution
538 logic uses the "minlenret" to tell whether it can do in-place
539 substitution which can result in considerable speedup.
540 "gofs"
542 Left offset from pos() to start match at.
543 "substrs"
545 Substring data about strings that must appear in the final match. This
546 is currently only used internally by perl's engine for but might be
547 used in the future for all engines for optimisations.
548 "nparens", "lastparen", and "lastcloseparen"
550 These fields are used to keep track of how many paren groups could be
551 matched in the pattern, which was the last open paren to be entered,
552 and which was the last close paren to be entered.
553 "intflags"
555 The engine's private copy of the flags the pattern was compiled with.
556 Usually this is the same as "extflags" unless the engine chose to
557 modify one of them.
558 "pprivate"
560 A void* pointing to an engine-defined data structure. The perl engine
561 uses the "regexp_internal" structure (see "Base Structures" in
562 perlreguts) but a custom engine should use something else.
563 "swap"
565 Unused. Left in for compatibility with perl 5.10.0.
566 "offs"
568 A "regexp_paren_pair" structure which defines offsets into the string
569 being matched which correspond to the $& and $1, $2 etc. captures, the
570 "regexp_paren_pair" struct is defined as follows:
571 typedef struct regexp_paren_pair {
573 I32 start;
574 I32 end;
575 } regexp_paren_pair;
576 If "->offs[num].start" or "->offs[num].end" is "-1" then that capture
578 group did not match. "->offs[0].start/end" represents $& (or "${^MATCH"
579 under "//p") and "->offs[paren].end" matches $$paren where $paren = 1>.
580 "precomp" "prelen"
582 Used for optimisations. "precomp" holds a copy of the pattern that was
583 compiled and "prelen" its length. When a new pattern is to be compiled
584 (such as inside a loop) the internal "regcomp" operator checks whether
585 the last compiled "REGEXP"'s "precomp" and "prelen" are equivalent to
586 the new one, and if so uses the old pattern instead of compiling a new
587 one.
588 The relevant snippet from "Perl_pp_regcomp":
590 if (!re || !re->precomp || re->prelen != (I32)len ||
592 memNE(re->precomp, t, len))
593 /* Compile a new pattern */
594 "paren_names"
596 This is a hash used internally to track named capture groups and their
597 offsets. The keys are the names of the buffers the values are dualvars,
598 with the IV slot holding the number of buffers with the given name and
599 the pv being an embedded array of I32. The values may also be
600 contained independently in the data array in cases where named
601 backreferences are used.
602 "substrs"
604 Holds information on the longest string that must occur at a fixed
605 offset from the start of the pattern, and the longest string that must
606 occur at a floating offset from the start of the pattern. Used to do
607 Fast-Boyer-Moore searches on the string to find out if its worth using
608 the regex engine at all, and if so where in the string to search.
609 "subbeg" "sublen" "saved_copy"
611 Used during execution phase for managing search and replace patterns.
612 "wrapped" "wraplen"
614 Stores the string "qr//" stringifies to. The perl engine for example
615 stores "(?^:eek)" in the case of "qr/eek/".
616 When using a custom engine that doesn't support the "(?:)" construct
618 for inline modifiers, it's probably best to have "qr//" stringify to
619 the supplied pattern, note that this will create undesired patterns in
620 cases such as:
621 my $x = qr/a|b/; # "a|b"
623 my $y = qr/c/i; # "c"
624 my $z = qr/$x$y/; # "a|bc"
625 There's no solution for this problem other than making the custom
627 engine understand a construct like "(?:)".
628 "seen_evals"
630 This stores the number of eval groups in the pattern. This is used for
631 security purposes when embedding compiled regexes into larger patterns
632 with "qr//".
633 "refcnt"
635 The number of times the structure is referenced. When this falls to 0
636 the regexp is automatically freed by a call to pregfree. This should be
637 set to 1 in each engine's "comp" routine.
638
640 Originally part of perlreguts.
641
643 Originally written by Yves Orton, expanded by var Arnfjoerd` Bjarmason.
644
646 Copyright 2006 Yves Orton and 2007 var Arnfjoerd` Bjarmason.
647 This program is free software; you can redistribute it and/or modify it
649 under the same terms as Perl itself.
650perl v5.16.3 2013-03-04 PERLREAPI(1)