1Proc::Queue(3) User Contributed Perl Documentation Proc::Queue(3)
2
6 Proc::Queue - limit the number of child processes running
7
9 use Proc::Queue size => 4, debug => 1;
10 package other;
12 use POSIX ":sys_wait_h"; # imports WNOHANG
13 # this loop creates new children, but Proc::Queue makes it wait every
15 # time the limit (4) is reached until enough children exit
16 foreach (1..100) {
17 my $f=fork;
18 if(defined ($f) and $f==0) {
19 print "-- I'm a forked process $$\n";
20 sleep rand 5;
21 print "-- I'm tired, going away $$\n";
22 exit(0)
23 }
24 1 while waitpid(-1, WNOHANG)>0; # reaps children
25 }
26 Proc::Queue::size(10); # changing limit to 10 concurrent processes
28 Proc::Queue::trace(1); # trace mode on
29 Proc::Queue::debug(0); # debug is off
30 Proc::Queue::delay(0.2); # set 200 miliseconds as minimum
31 # delay between fork calls
32 package other; # just to test it works on any package
34 print "going again!\n";
36 # another loop with different settings for Proc::Queue
38 foreach (1..20) {
39 my $f=fork;
40 if(defined ($f) and $f==0) {
41 print "-- I'm a forked process $$\n";
42 sleep rand 5;
43 print "-- I'm tired, going away $$\n";
44 exit(0)
45 }
46 }
47 1 while wait != -1;
49
51 This module lets you parallelise a perl program using the "fork",
52 "exit", "wait" and "waitpid" calls as usual but without taking care of
53 creating too many processes and overloading the machine.
54 It redefines perl "fork", "exit", "wait" and "waitpid" core functions.
56 Old programs do not need to be reprogrammed, only the "use Proc::Queue
57 ..." sentence has to be added to them.
58 Additionally, the module has two debugging modes (debug and trace) that
60 seem too be very useful when developing parallel aplications:
61 debug mode:
63 when active, dumps lots of information about processes being
64 created, exiting, being caught by parent, etc.
65 trace mode:
67 prints a line every time one of the "fork", "exit", "wait" or
68 "waitpid" functions are called.
69 It is also possible to set a minimun delay time between fork calls to
71 stop too many processes for starting in a short time interval.
72 Child processes continue to use the modified functions, but their
74 queues are reset and the maximun process number for them is set to 1
75 (anyway, children can change their queue size themselves).
76 Proc::Queue doesn't work if CHLD signal handler is set to "IGNORE".
78 Internally, Proc::Queue, automatically catches zombies and stores their
80 exit status in a private hash. To avoid leaking memory in long running
81 programs you have to call "wait" or "waitpid" to delete entries from
82 that hash or alternatively active the "ignore_children" mode:
83 Proc::Queue::ignore_children(1)
85 or
87 use Proc::Queue ignore_children=>1, ...
89 EXPORT
91 This module redefines the "fork", "wait", "waitpid" and "exit" calls.
92 EXPORT_OK
94 Functions "fork_now", "waitpids", "run_back", "run_back_now",
95 "all_exit_ok", "running_now", "system_back" and "system_back_now" can
96 be imported. Tag ":all" is defined to import all of them.
97 FUNCTIONS
99 There are several not exported functions that can be used to configure
100 the module:
101 size(), size($number)
103 If an argument is given the maximun number of concurrent processes
104 is set to it and the number of maximun processes that were allowed
105 before is returned.
106 If no argument is given, the number of processes allowed is
108 returned.
109 delay(), delay($time)
111 lets you set a minimun time in seconds to elapse between
112 consecutive calls to fork. It is useful to avoid creating too many
113 processes in a short time (that could degrade performance).
114 If Time::HiRes module is available delays shorted that 1 second are
116 allowed.
117 If no arg is given, the current delay is returned.
119 To clear it use Proc::Queue::delay(0).
121 weight(), weight($weight)
123 by default any process forked count as 1 through the max number of
124 processes allowed to run simultaneously (the queue size). "weight"
125 allows to change this, i.e.:
126 Proc::Queue::weight(3);
128 run_back { ... heavy process here ... };
129 Proc::Queue::weight(1);
130 causes the "heavy process" to count as three normal processes.
132 Valid weight values are integers greater than zero.
134 Remember to reset the weight back to 1 (or whatever) after the
136 heavier processes have been forked!.
137 allow_excess(), allow_excess($allow_excess)
139 by default the next queued process will be started as soon as the
140 number of running processes is smaller than the queue size--this is
141 regardless of the weight of the next queued process, so the queue
142 could become overloaded. Setting "allow_excess" to false forces
143 the next queued process to wait until there is room for it in the
144 queue, that is, the size of the queue less the weighted number of
145 currently running processes must be no smaller than the weight of
146 the next queued process in order for the next process to start.
147 Setting "allow_excess" to any value greater than zero (default is
149 1) resets the default behavior.
150 ignore_children($on)
152 calling
153 Proc::Queue::ignore_children(1);
155 is the equivalent to
157 $SIG{CHLD}='IGNORE'
159 when using Proc::Queue.
161 debug(), debug($boolean), trace(), trace($boolean)
163 Change or return the status for the debug and trace modes.
164 Other utility subroutines that can be imported from Proc::Queue are:
166 fork_now()
168 Sometimes you would need to fork a new child without waiting for
169 other children to exit if the queue is full, "fork_now" does that.
170 It is exportable so you can do...
171 use Proc::Queue size => 5, qw(fork_now), debug =>1;
173 $f=fork_now;
175 if(defined $f and $f == 0) {
176 print "I'm the child\n"; exit;
177 }
178 waitpids(@pid)
180 waits for all the processes in @pid to exit. It returns an array
181 with pairs of pid and exit values (pid1, exit1, pid2, exit2, pid3,
182 exit3,...) as returned by individual waitpid calls.
183 run_back(\&code), run_back { code }
185 Runs the argument subrutine in a forked child process and returns
186 the pid number for the new process.
187 run_back_now(\&code), run_back_now { code }
189 A mix between run_back and fork_now.
190 system_back(@command)
192 Similar to the "system" call but runs the command in the background
193 and waits for other children to exit first if there are already too
194 many running.
195 Returns the pid of the forked process or undef if the program was
197 not found.
198 system_back_now(@command)
200 As "system_back" but without checking if the maximun number of
201 children allowed has been reached.
202 all_exit_ok(@pid)
204 Do a "waitpids" call and test that all the processes exit with code
205 0.
206 running_now()
208 Returns the number of child processes currently running.
209 import(pkg,opt,val,opt,val,...,fnt_name,fnt_name,...)
211 The import function is not usually explicitally called but by the
212 "use Proc::Queue" statement.
213 Options allowed are "size", "debug", "weight" and "trace", i.e:
215 use Proc::Queue size=>10, debug=>1;
217 Anything that is not "size", "debug", "weight" or "trace" is
219 expected to be a function name to be imported.
220 use Proc::Queue size=>10, ':all';
222 BUGS
224 Proc::Queue is a very stable module, no bugs have been reported for a
225 long time.
226 Support for Win32 OSs is still experimental.
228
230 perlfunc(1), perlipc(1), POSIX, perlfork(1), Time::HiRes,
231 Parallel::ForkManager. The "example.pl" script contained in the module
232 distribution.
233
235 Copyright 2001-2003, 2005-2008 by Salvador Fandiño <sfandino@yahoo.com>
236 This library is free software; you can redistribute it and/or modify it
238 under the same terms as Perl itself.
239perl v5.32.0 2020-07-28 Proc::Queue(3)