1Path::Iterator::Rule(3)User Contributed Perl DocumentatioPnath::Iterator::Rule(3)
2
6 Path::Iterator::Rule - Iterative, recursive file finder
7
9 version 1.014
10
12 use Path::Iterator::Rule;
13 my $rule = Path::Iterator::Rule->new; # match anything
15 $rule->file->size(">10k"); # add/chain rules
16 # iterator interface
18 my $next = $rule->iter( @dirs );
19 while ( defined( my $file = $next->() ) ) {
20 ...
21 }
22 # list interface
24 for my $file ( $rule->all( @dirs ) ) {
25 ...
26 }
27
29 This module iterates over files and directories to identify ones
30 matching a user-defined set of rules. The API is based heavily on
31 File::Find::Rule, but with more explicit distinction between matching
32 rules and options that influence how directories are searched. A
33 "Path::Iterator::Rule" object is a collection of rules (match criteria)
34 with methods to add additional criteria. Options that control
35 directory traversal are given as arguments to the method that generates
36 an iterator.
37 Here is a summary of features for comparison to other file finding
39 modules:
40 · provides many "helper" methods for specifying rules
42 · offers (lazy) iterator and flattened list interfaces
44 · custom rules implemented with callbacks
46 · breadth-first (default) or pre- or post-order depth-first searching
48 · follows symlinks (by default, but can be disabled)
50 · directories visited only once (no infinite loop; can be disabled)
52 · doesn't chdir during operation
54 · provides an API for extensions
56 As a convenience, the PIR module is an empty subclass of this one that
58 is less arduous to type for one-liners.
59 Note: paths are constructed with unix-style forward-slashes for
61 efficiency rather than using File::Spec. If proper path separators are
62 needed, call canonpath on the search results.
63
65 Constructors
66 "new"
67 my $rule = Path::Iterator::Rule->new;
69 Creates a new rule object that matches any file or directory. It takes
71 no arguments. For convenience, it may also be called on an object, in
72 which case it still returns a new object that matches any file or
73 directory.
74 "clone"
76 my $common = Path::Iterator::Rule->new->file->not_empty;
78 my $big_files = $common->clone->size(">1M");
79 my $small_files = $common->clone->size("<10K");
80 Creates a copy of a rule object. Useful for customizing different rule
82 objects against a common base.
83 Matching and iteration
85 "iter"
86 my $next = $rule->iter( @dirs, \%options);
88 while ( defined( my $file = $next->() ) ) {
89 ...
90 }
91 Creates a subroutine reference iterator that returns a single result
93 when dereferenced. This iterator is "lazy" -- results are not pre-
94 computed.
95 It takes as arguments a list of directories to search and an optional
97 hash reference of control options. If no search directories are
98 provided, the current directory is used ("."). Valid options include:
99 · "depthfirst" -- Controls order of results. Valid values are "1"
101 (post-order, depth-first search), "0" (breadth-first search) or
102 "-1" (pre-order, depth-first search). Default is 0.
103 · "error_handler" -- Catches errors during execution of rule tests.
105 Default handler dies with the filename and error. If set to undef,
106 error handling is disabled.
107 · "follow_symlinks" -- Follow directory symlinks when true. Default
109 is 1.
110 · "report_symlinks" -- Includes symlinks in results when true.
112 Default is equal to "follow_symlinks".
113 · "loop_safe" -- Prevents visiting the same directory more than once
115 when true. Default is 1.
116 · "relative" -- Return matching items relative to the search
118 directory. Default is 0.
119 · "sorted" -- Whether entries in a directory are sorted before
121 processing. Default is 1.
122 · "visitor" -- An optional coderef that will be called on items
124 matching all rules.
125 Filesystem loops might exist from either hard or soft links. The
127 "loop_safe" option prevents infinite loops, but adds some overhead by
128 making "stat" calls. Because directories are visited only once when
129 "loop_safe" is true, matches could come from a symlinked directory
130 before the real directory depending on the search order.
131 To get only the real files, turn off "follow_symlinks". You can have
133 symlinks included in results, but not descend into symlink directories
134 if you turn off "follow_symlinks", but turn on "report_symlinks".
135 Turning "loop_safe" off and leaving "follow_symlinks" on avoids "stat"
137 calls and will be fastest, but with the risk of an infinite loop and
138 repeated files. The default is slow, but safe.
139 The "error_handler" parameter must be a subroutine reference. It will
141 be called when a rule test throws an exception. The first argument
142 will be the file name being inspected and the second argument will be
143 the exception.
144 The optional "visitor" parameter must be a subroutine reference. If
146 set, it will be called for any result that matches. It is called the
147 same way a custom rule would be (see "EXTENDING") but its return value
148 is ignored. It is called when an item is first inspected --
149 "postorder" is not respected.
150 The paths inspected and returned will be relative to the search
152 directories provided. If these are absolute, then the paths returned
153 will have absolute paths. If these are relative, then the paths
154 returned will have relative paths.
155 If the search directories are absolute and the "relative" option is
157 true, files returned will be relative to the search directory. Note
158 that if the search directories are not mutually exclusive (whether
159 containing subdirectories like @INC or symbolic links), files found
160 could be returned relative to different initial search directories
161 based on "depthfirst", "follow_symlinks" or "loop_safe".
162 When the iterator is exhausted, it will return undef.
164 "iter_fast"
166 This works just like "iter", except that it optimizes for speed over
168 safety. Don't do this unless you're sure you need it and accept the
169 consequences. See "PERFORMANCE" for details.
170 "all"
172 my @matches = $rule->all( @dir, \%options );
174 Returns a list of paths that match the rule. It takes the same
176 arguments and has the same behaviors as the "iter" method. The "all"
177 method uses "iter" internally to fetch all results.
178 In scalar context, it will return the count of matched paths.
180 In void context, it is optimized to iterate over everything, but not
182 store results. This is most useful with the "visitor" option:
183 $rule->all( $path, { visitor => \&callback } );
185 "all_fast"
187 This works just like "all", except that it optimizes for speed over
189 safety. Don't do this unless you're sure you need it and accept the
190 consequences. See "PERFORMANCE" for details.
191 "test"
193 if ( $rule->test( $path, $basename, $stash ) ) { ... }
195 Test a file path against a rule. Used internally, but provided should
197 someone want to create their own, custom iteration algorithm.
198 Logic operations
200 "Path::Iterator::Rule" provides three logic operations for adding rules
201 to the object. Rules may be either a subroutine reference with
202 specific semantics (described below in "EXTENDING") or another
203 "Path::Iterator::Rule" object.
204 "and"
206 $rule->and( sub { -r -w -x $_ } ); # stacked filetest example
208 $rule->and( @more_rules );
209 Adds one or more constraints to the current rule. E.g. "old rule AND
211 new1 AND new2 AND ...". Returns the object to allow method chaining.
212 "or"
214 $rule->or(
216 $rule->new->name("foo*"),
217 $rule->new->name("bar*"),
218 sub { -r -w -x $_ },
219 );
220 Takes one or more alternatives and adds them as a constraint to the
222 current rule. E.g. "old rule AND ( new1 OR new2 OR ... )". Returns the
223 object to allow method chaining.
224 "not"
226 $rule->not( sub { -r -w -x $_ } );
228 Takes one or more alternatives and adds them as a negative constraint
230 to the current rule. E.g. "old rule AND NOT ( new1 AND new2 AND ...)".
231 Returns the object to allow method chaining.
232 "skip"
234 $rule->skip(
236 $rule->new->dir->not_writeable,
237 $rule->new->dir->name("foo"),
238 );
239 Takes one or more alternatives and will prune a directory if any of the
241 criteria match or if any of the rules already indicate the directory
242 should be pruned. Pruning means the directory will not be returned by
243 the iterator and will not be searched.
244 For files, it is equivalent to "$rule->not($rule->or(@rules))".
246 Returns the object to allow method chaining.
247 This method should be called as early as possible in the rule chain.
249 See "skip_dirs" below for further explanation and an example.
250
252 Rule methods are helpers that add constraints. Internally, they
253 generate a closure to accomplish the desired logic and add it to the
254 rule object with the "and" method. Rule methods return the object to
255 allow for method chaining.
256 File name rules
258 "name"
259 $rule->name( "foo.txt" );
261 $rule->name( qr/foo/, "bar.*");
262 The "name" method takes one or more patterns and creates a rule that is
264 true if any of the patterns match the basename of the file or directory
265 path. Patterns may be regular expressions or glob expressions (or
266 literal names).
267 "iname"
269 $rule->iname( "foo.txt" );
271 $rule->iname( qr/foo/, "bar.*");
272 The "iname" method is just like the "name" method, but matches case-
274 insensitively.
275 "skip_dirs"
277 $rule->skip_dirs( @patterns );
279 The "skip_dirs" method skips directories that match one or more
281 patterns. Patterns may be regular expressions or globs (just like
282 "name"). Directories that match will not be returned from the iterator
283 and will be excluded from further search. This includes the starting
284 directories. If that isn't what you want, see "skip_subdirs" instead.
285 Note: this rule should be specified early so that it has a chance to
287 operate before a logical shortcut. E.g.
288 $rule->skip_dirs(".git")->file; # OK
290 $rule->file->skip_dirs(".git"); # Won't work
291 In the latter case, when a ".git" directory is seen, the "file" rule
293 shortcuts the rule before the "skip_dirs" rule has a chance to act.
294 "skip_subdirs"
296 $rule->skip_subdirs( @patterns );
298 This works just like "skip_dirs", except that the starting directories
300 (depth 0) are not skipped and may be returned from the iterator unless
301 excluded by other rules.
302 File test rules
304 Most of the "-X" style filetest are available as boolean rules. The
305 table below maps the filetest to its corresponding method name.
306 Test | Method Test | Method
308 ------|------------- ------|----------------
309 -r | readable -R | r_readable
310 -w | writeable -W | r_writeable
311 -w | writable -W | r_writable
312 -x | executable -X | r_executable
313 -o | owned -O | r_owned
314 | |
315 -e | exists -f | file
316 -z | empty -d | directory, dir
317 -s | nonempty -l | symlink
318 | -p | fifo
319 -u | setuid -S | socket
320 -g | setgid -b | block
321 -k | sticky -c | character
322 | -t | tty
323 -T | ascii
324 -B | binary
325 For example:
327 $rule->file->nonempty; # -f -s $file
329 The -X operators for timestamps take a single argument in a form that
331 Number::Compare can interpret.
332 Test | Method
334 ------|-------------
335 -A | accessed
336 -M | modified
337 -C | changed
338 For example:
340 $rule->modified(">1"); # -M $file > 1
342 Stat test rules
344 All of the "stat" elements have a method that takes a single argument
345 in a form understood by Number::Compare.
346 stat() | Method
348 --------------------
349 0 | dev
350 1 | ino
351 2 | mode
352 3 | nlink
353 4 | uid
354 5 | gid
355 6 | rdev
356 7 | size
357 8 | atime
358 9 | mtime
359 10 | ctime
360 11 | blksize
361 12 | blocks
362 For example:
364 $rule->size(">10K")
366 Depth rules
368 $rule->min_depth(3);
369 $rule->max_depth(5);
370 The "min_depth" and "max_depth" rule methods take a single argument and
372 limit the paths returned to a minimum or maximum depth (respectively)
373 from the starting search directory. A depth of 0 means the starting
374 directory itself. A depth of 1 means its children. (This is similar
375 to the Unix "find" utility.)
376 Perl file rules
378 # All perl rules
379 $rule->perl_file;
380 # Individual perl file rules
382 $rule->perl_module; # .pm files
383 $rule->perl_pod; # .pod files
384 $rule->perl_test; # .t files
385 $rule->perl_installer; # Makefile.PL or Build.PL
386 $rule->perl_script; # .pl or 'perl' in the shebang
387 These rule methods match file names (or a shebang line) that are
389 typical of Perl distribution files.
390 Version control file rules
392 # Skip all known VCS files
393 $rule->skip_vcs;
394 # Skip individual VCS files
396 $rule->skip_cvs;
397 $rule->skip_rcs;
398 $rule->skip_svn;
399 $rule->skip_git;
400 $rule->skip_bzr;
401 $rule->skip_hg;
402 $rule->skip_darcs;
403 Skips files and/or prunes directories related to a version control
405 system. Just like "skip_dirs", these rules should be specified early
406 to get the correct behavior.
407 File content rules
409 "contents_match"
410 $rule->contents_match(qr/BEGIN .* END/xs);
412 The "contents_match" rule takes a list of regular expressions and
414 returns files that match one of the expressions.
415 The expressions are applied to the file's contents as a single string.
417 For large files, this is likely to take significant time and memory.
418 Files are assumed to be encoded in UTF-8, but alternative Perl IO
420 layers can be passed as the first argument:
421 $rule->contents_match(":encoding(iso-8859-1)", qr/BEGIN .* END/xs);
423 See perlio for further details.
425 "line_match"
427 $rule->line_match(qr/^new/i, qr/^Addition/);
429 The "line_match" rule takes a list of regular expressions and returns
431 files with at least one line that matches one of the expressions.
432 Files are assumed to be encoded in UTF-8, but alternative Perl IO
434 layers can be passed as the first argument.
435 "shebang"
437 $rule->shebang(qr/#!.*\bperl\b/);
439 The "shebang" rule takes a list of regular expressions or glob patterns
441 and checks them against the first line of a file.
442 Other rules
444 "dangling"
445 $rule->symlink->dangling;
447 $rule->not_dangling;
448 The "dangling" rule method matches dangling symlinks. Use it or its
450 inverse to control how dangling symlinks should be treated.
451 Negated rules
453 Most rule methods have a negated form preceded by "not_".
454 $rule->not_name("foo.*")
456 Because this happens automatically, it includes somewhat silly ones
458 like "not_nonempty" (which is thus a less efficient way of saying
459 "empty").
460 Rules that skip directories or version control files do not have a
462 negated version.
463
465 Custom rule subroutines
466 Rules are implemented as (usually anonymous) subroutine callbacks that
467 return a value indicating whether or not the rule matches. These
468 callbacks are called with three arguments. The first argument is a
469 path, which is also locally aliased as the $_ global variable for
470 convenience in simple tests.
471 $rule->and( sub { -r -w -x $_ } ); # tests $_
473 The second argument is the basename of the path, which is useful for
475 certain types of name checks:
476 $rule->and( sub { $_[1] =~ /foo|bar/ } ); "foo" or "bar" in basename;
478 The third argument is a hash reference that can be used to maintain
480 state. Keys beginning with an underscore are reserved for
481 "Path::Iterator::Rule" to provide additional data about the search in
482 progress. For example, the "_depth" key is used to support minimum and
483 maximum depth checks.
484 The custom rule subroutine must return one of four values:
486 · A true value -- indicates the constraint is satisfied
488 · A false value -- indicates the constraint is not satisfied
490 · "\1" -- indicate the constraint is satisfied, and prune if it's a
492 directory
493 · "\0" -- indicate the constraint is not satisfied, and prune if it's
495 a directory
496 A reference is a special flag that signals that a directory should not
498 be searched recursively, regardless of whether the directory should be
499 returned by the iterator or not.
500 The legacy "0 but true" value used previously for pruning is no longer
502 valid and will throw an exception if it is detected.
503 Here is an example. This is equivalent to the "max_depth" rule method
505 with a depth of 3:
506 $rule->and(
508 sub {
509 my ($path, $basename, $stash) = @_;
510 return 1 if $stash->{_depth} < 3;
511 return \1 if $stash->{_depth} == 3;
512 return \0; # should never get here
513 }
514 );
515 Files and directories and directories up to depth 3 will be returned
517 and directories will be searched. Files of depth 3 will be returned.
518 Directories of depth 3 will be returned, but their contents will not be
519 added to the search.
520 Returning a reference is "sticky" -- they will propagate through "and"
522 and "or" logic.
523 0 && \0 = \0 \0 && 0 = \0 0 || \0 = \0 \0 || 0 = \0
525 0 && \1 = \0 \0 && 1 = \0 0 || \1 = \1 \0 || 1 = \1
526 1 && \0 = \0 \1 && 0 = \0 1 || \0 = \1 \1 || 0 = \1
527 1 && \1 = \1 \1 && 1 = \1 1 || \1 = \1 \1 || 1 = \1
528 Once a directory is flagged to be pruned, it will be pruned regardless
530 of subsequent rules.
531 $rule->max_depth(3)->name(qr/foo/);
533 This will return files or directories with "foo" in the name, but all
535 directories at depth 3 will be pruned, regardless of whether they match
536 the name rule.
537 Generally, if you want to do directory pruning, you are encouraged to
539 use the "skip" method instead of writing your own logic using "\0" and
540 "\1".
541 Extension modules and custom rule methods
543 One of the strengths of File::Find::Rule is the many CPAN modules that
544 extend it. "Path::Iterator::Rule" provides the "add_helper" method to
545 provide a similar mechanism for extensions.
546 The "add_helper" class method takes three arguments, a "name" for the
548 rule method, a closure-generating callback, and a flag for not
549 generating a negated form of the rule. Unless the flag is true, an
550 inverted "not_*" method is generated automatically. Extension classes
551 should call this as a class method to install new rule methods. For
552 example, this adds a "foo" method that checks if the filename is "foo":
553 package Path::Iterator::Rule::Foo;
555 use Path::Iterator::Rule;
557 Path::Iterator::Rule->add_helper(
559 foo => sub {
560 my @args = @_; # do this to customize closure with arguments
561 return sub {
562 my ($item, $basename) = @_;
563 return if -d "$item";
564 return $basename =~ /^foo$/;
565 }
566 }
567 );
568 1;
570 This allows the following rule methods:
572 $rule->foo;
574 $fule->not_foo;
575 The "add_helper" method will warn and ignore a helper with the same
577 name as an existing method.
578 Subclassing
580 Instead of processing and returning strings, this module may be
581 subclassed to operate on objects that represent files. Such objects
582 must stringify to a file path.
583 The following private implementation methods must be overridden:
585 · _objectify -- given a path, return an object
587 · _children -- given a directory, return an (unsorted) list of [
589 basename, full path ] entries within it, excluding "." and ".."
590 Note that "_children" should return a list of tuples, where the tuples
592 are array references containing basename and full path.
593 See Path::Class::Rule source for an example.
595
597 If you run with lexical warnings enabled, "Path::Iterator::Rule" will
598 issue warnings in certain circumstances (such as an unreadable
599 directory that must be skipped). To disable these categories, put the
600 following statement at the correct scope:
601 no warnings 'Path::Iterator::Rule';
603
605 By default, "Path::Iterator::Rule" iterator options are "slow but
606 safe". They ensure uniqueness, return files in sorted order, and throw
607 nice error messages if something goes wrong.
608 If you want speed over safety, set these options:
610 %options = (
612 loop_safe => 0,
613 sorted => 0,
614 depthfirst => -1,
615 error_handler => undef
616 );
617 Alternatively, use the "iter_fast" and "all_fast" methods instead,
619 which set these options for you.
620 $iter = $rule->iter( @dirs, \%options );
622 $iter = $rule->iter_fast( @dirs ); # same thing
624 Depending on the file structure being searched, "depthfirst => -1" may
626 or may not be a good choice. If you have lots of nested directories and
627 all the files at the bottom, a depth first search might do less work or
628 use less memory, particularly if the search will be halted early (e.g.
629 finding the first N matches.)
630 Rules will shortcut on failure, so be sure to put rules likely to fail
632 early in a rule chain.
633 Consider:
635 $r1 = Path::Iterator::Rule->new->name(qr/foo/)->file;
637 $r2 = Path::Iterator::Rule->new->file->name(qr/foo/);
638 If there are lots of files, but only a few containing "foo", then $r1
640 above will be faster.
641 Rules are implemented as code references, so long chains have some
643 overhead. Consider testing with a custom coderef that combines several
644 tests into one.
645 Consider:
647 $r3 = Path::Iterator::Rule->new->and( sub { -x -w -r $_ } );
649 $r4 = Path::Iterator::Rule->new->executable->writeable->readable;
650 Rule $r3 above will be much faster, not only because it stacks the file
652 tests, but because it requires only a single code reference.
653
655 Some features are still unimplemented:
656 · Untainting options
658 · Some File::Find::Rule helpers (e.g. "grep")
660 · Extension class loading via "import()"
662 Filetest operators and stat rules are subject to the usual portability
664 considerations. See perlport for details.
665
667 There are many other file finding modules out there. They all have
668 various features/deficiencies, depending on your preferences and needs.
669 Here is an (incomplete) list of alternatives, with some comparison
670 commentary.
671 Path::Class::Rule and IO::All::Rule are subclasses of
673 "Path::Iterator::Rule" and operate on Path::Class and IO::All objects,
674 respectively. Because of this, they are substantially slower on large
675 directory trees than just using this module directly.
676 File::Find is part of the Perl core. It requires the user to write a
678 callback function to process each node of the search. Callbacks must
679 use global variables to determine the current node. It only supports
680 depth-first search (both pre- and post-order). It supports pre- and
681 post-processing callbacks; the former is required for sorting files to
682 process in a directory. File::Find::Closures can be used to help
683 create a callback for File::Find.
684 File::Find::Rule is an object-oriented wrapper around File::Find. It
686 provides a number of helper functions and there are many more
687 "File::Find::Rule::*" modules on CPAN with additional helpers. It
688 provides an iterator interface, but precomputes all the results.
689 File::Next provides iterators for file, directories or "everything".
691 It takes two callbacks, one to match files and one to decide which
692 directories to descend. It does not allow control over breadth/depth
693 order, though it does provide means to sort files for processing within
694 a directory. Like File::Find, it requires callbacks to use global
695 variables.
696 Path::Class::Iterator walks a directory structure with an iterator. It
698 is implemented as Path::Class subclasses, which adds a degree of extra
699 complexity. It takes a single callback to define "interesting" paths to
700 return. The callback gets a Path::Class::Iterator::File or
701 Path::Class::Iterator::Dir object for evaluation.
702 File::Find::Object and companion File::Find::Object::Rule are like
704 File::Find and File::Find::Rule, but without File::Find inside. They
705 use an iterator that does not precompute results. They can return
706 File::Find::Object::Result objects, which give a subset of the utility
707 of Path::Class objects. File::Find::Object::Rule appears to be a
708 literal translation of File::Find::Rule, including oddities like making
709 "-M" into a boolean.
710 File::chdir::WalkDir recursively descends a tree, calling a callback on
712 each file. No iterator. Supports exclusion patterns. Depth-first
713 post-order by default, but offers pre-order option. Does not process
714 symlinks.
715 File::Find::Iterator is based on iterator patterns in Higher Order
717 Perl. It allows a filtering callback. Symlinks are followed
718 automatically without infinite loop protection. No control over order.
719 It offers a "state file" option for resuming interrupted work.
720 File::Find::Declare has declarative helper rules, no iterator, is
722 Moose-based and offers no control over ordering or following symlinks.
723 File::Find::Node has no iterator, does matching via callback and offers
725 no control over ordering.
726 File::Set builds up a set of files to operate on from a list of
728 directories to include or exclude, with control over recursion. A
729 callback is applied to each file (or directory) in the set. There is
730 no iterator. There is no control over ordering. Symlinks are not
731 followed. It has several extra features for checksumming the set and
732 creating tarballs with /bin/tar.
733
735 Thank you to Ricardo Signes (rjbs) for inspiring me to write yet
736 another file finder module, for writing file finder optimization
737 benchmarks, and tirelessly running my code over and over to see if it
738 got faster.
739 · See the speed of Perl file finders
741 <http://rjbs.manxome.org/rubric/entry/1981>
742
744 Bugs / Feature Requests
745 Please report any bugs or feature requests through the issue tracker at
746 <https://github.com/dagolden/Path-Iterator-Rule/issues>. You will be
747 notified automatically of any progress on your issue.
748 Source Code
750 This is open source software. The code repository is available for
751 public review and contribution under the terms of the license.
752 <https://github.com/dagolden/Path-Iterator-Rule>
754 git clone https://github.com/dagolden/Path-Iterator-Rule.git
756
758 David Golden <dagolden@cpan.org>
759
761 · David Steinbrunner <dsteinbrunner@pobox.com>
762 · Diab Jerius <djerius@cfa.harvard.edu>
764 · Edward Betts <edward@4angle.com>
766 · Gian Piero Carrubba <gpiero@butterfly.fdc.rm-rf.it>
768 · Graham Knop <haarg@cpan.org>
770 · Ricardo Signes <rjbs@cpan.org>
772 · Slaven Rezic <slaven.rezic@idealo.de>
774 · Toby Inkster <tobyink@cpan.org>
776
778 This software is Copyright (c) 2013 by David Golden.
779 This is free software, licensed under:
781 The Apache License, Version 2.0, January 2004
783perl v5.30.1 2020-01-30 Path::Iterator::Rule(3)