1TheSchwartz(3) User Contributed Perl Documentation TheSchwartz(3)
2
6 TheSchwartz - reliable job queue
7
9 # MyApp.pm
10 package MyApp;
11 sub work_asynchronously {
13 my %args = @_;
14 my $client = TheSchwartz->new( databases => $DATABASE_INFO );
16 $client->insert('MyWorker', \%args);
17 }
18 # myworker.pl
21 package MyWorker;
22 use base qw( TheSchwartz::Worker );
23 sub work {
25 my $class = shift;
26 my TheSchwartz::Job $job = shift;
27 print "Workin' hard or hardly workin'? Hyuk!!\n";
29 $job->completed();
31 }
32 package main;
34 my $client = TheSchwartz->new( databases => $DATABASE_INFO );
36 $client->can_do('MyWorker');
37 $client->work();
38
40 TheSchwartz is a reliable job queue system. Your application can put
41 jobs into the system, and your worker processes can pull jobs from the
42 queue atomically to perform. Failed jobs can be left in the queue to
43 retry later.
44 Abilities specify what jobs a worker process can perform. Abilities are
46 the names of "TheSchwartz::Worker" subclasses, as in the synopsis: the
47 "MyWorker" class name is used to specify that the worker script can
48 perform the job. When using the "TheSchwartz" client's "work"
49 functions, the class-ability duality is used to automatically dispatch
50 to the proper class to do the actual work.
51 TheSchwartz clients will also prefer to do jobs for unused abilities
53 before reusing a particular ability, to avoid exhausting the supply of
54 one kind of job while jobs of other types stack up.
55 Some jobs with high setup times can be performed more efficiently if a
57 group of related jobs are performed together. TheSchwartz offers a
58 facility to coalesce jobs into groups, which a properly constructed
59 worker can find and perform at once. For example, if your worker were
60 delivering email, you might store the domain name from the recipient's
61 address as the coalescing value. The worker that grabs that job could
62 then batch deliver all the mail for that domain once it connects to
63 that domain's mail server.
64
66 "TheSchwartz->new( %args )"
67 Optional members of %args are:
68 · "databases"
70 An arrayref of database information. TheSchwartz workers can use
72 multiple databases, such that if any of them are unavailable, the
73 worker will search for appropriate jobs in the other databases
74 automatically.
75 Each member of the "databases" value should be a hashref containing
77 either:
78 · "dsn"
80 The database DSN for this database.
82 · "user"
84 The username to use when connecting to this database.
86 · "pass"
88 The password to use when connecting to this database.
90 or
92 · "driver"
94 A "Data::ObjectDriver::Driver::DBI" object.
96 See note below.
98 · "verbose"
100 A value indicating whether to log debug messages. If "verbose" is a
102 coderef, it is called to log debug messages. If "verbose" is not a
103 coderef but is some other true value, debug messages will be sent
104 to "STDERR". Otherwise, debug messages will not be logged.
105 · "prioritize"
107 A value indicating whether to utilize the job 'priority' field when
109 selecting jobs to be processed. If unspecified, jobs will always be
110 executed in a randomized order.
111 · "driver_cache_expiration"
113 Optional value to control how long database connections are cached
115 for in seconds. By default, connections are not cached. To re-use
116 the same database connection for five minutes, pass
117 driver_cache_expiration => 300 to the constructor. Improves job
118 throughput in cases where the work to process a job is small
119 compared to the database connection set-up and tear-down time.
120 · "retry_seconds"
122 The number of seconds after which to try reconnecting to apparently
124 dead databases. If not given, TheSchwartz will retry connecting to
125 databases after 30 seconds.
126 "$client->list_jobs( %args )"
128 Returns a list of "TheSchwartz::Job" objects matching the given
129 arguments. The required members of %args are:
130 · "funcname"
132 the name of the function or a reference to an array of functions
134 · "run_after"
136 the value you want to check <= against on the run_after column
138 · "grabbed_until"
140 the value you want to check <= against on the grabbed_until column
142 · "coalesce_op"
144 defaults to '=', set it to whatever you want to compare the
146 coalesce field too if you want to search, you can use 'LIKE'
147 · "coalesce"
149 coalesce value to search for, if you set op to 'LIKE' you can use
151 '%' here, do remember that '%' searches anchored at the beginning
152 of the string are much faster since it is can do a btree index
153 lookup
154 · "want_handle"
156 if you want all your jobs to be set up using a handle. defaults to
158 true. this option might be removed, as you should always have this
159 on a Job object.
160 It is important to remember that this function doesnt lock anything, it
162 just returns as many jobs as there is up to amount of databases *
163 FIND_JOB_BATCH_SIZE
164 "$client->lookup_job( $handle_id )"
166 Returns a "TheSchwartz::Job" corresponding to the given handle ID.
167 "$client->set_verbose( $verbose )"
169 Sets the current logging function to $verbose if it's a coderef. If not
170 a coderef, enables debug logging to "STDERR" if $verbose is true;
171 otherwise, disables logging.
172
174 The methods of TheSchwartz clients used by applications posting jobs to
175 the queue are:
176 "$client->insert( $job )"
178 Adds the given "TheSchwartz::Job" to one of the client's job databases.
179 "$client->insert( $funcname, $arg )"
181 Adds a new job with funcname $funcname and arguments $arg to the queue.
182 "$client->insert_jobs( @jobs )"
184 Adds the given "TheSchwartz::Job" objects to one of the client's job
185 databases. All the given jobs are recorded in one job database.
186 "$client->set_prioritize( $prioritize )"
188 Set the "prioritize" value as described in the constructor.
189
191 The methods of TheSchwartz clients for use in worker processes are:
192 "$client->can_do( $ability )"
194 Adds $ability to the list of abilities $client is capable of
195 performing. Subsequent calls to that client's "work" methods will find
196 jobs requiring the given ability.
197 "$client->work_once()"
199 Find and perform one job $client can do.
200 "$client->work_until_done()"
202 Find and perform jobs $client can do until no more such jobs are found
203 in any of the client's job databases.
204 "$client->work( [$delay] )"
206 Find and perform any jobs $client can do, forever. When no job is
207 available, the working process will sleep for $delay seconds (or 5, if
208 not specified) before looking again.
209 "$client->work_on($handle)"
211 Given a job handle (a scalar string) $handle, runs the job, then
212 returns.
213 "$client->grab_and_work_on($handle)"
215 Similar to $client->work_on($handle), except that the job will be
216 grabbed before being run. It guarantees that only one worker will work
217 on it (at least in the "grab_for" interval).
218 Returns false if the worker couldn't grab the job, and true if the
220 worker worked on it.
221 "$client->find_job_for_workers( [$abilities] )"
223 Returns a "TheSchwartz::Job" for a random job that the client can do.
224 If specified, the job returned matches one of the abilities in the
225 arrayref $abilities, rather than $client's abilities.
226 "$client->find_job_with_coalescing_value( $ability, $coval )"
228 Returns a "TheSchwartz::Job" for a random job for a worker capable of
229 $ability and with a coalescing value of $coval.
230 "$client->find_job_with_coalescing_prefix( $ability, $coval )"
232 Returns a "TheSchwartz::Job" for a random job for a worker capable of
233 $ability and with a coalescing value beginning with $coval.
234 Note the "TheSchwartz" implementation of this function uses a "LIKE"
236 query to find matching jobs, with all the attendant performance
237 implications for your job databases.
238 "$client->get_server_time( $driver )"
240 Given an open driver $driver to a database, gets the current server
241 time from the database.
242
244 The scoreboards can be used to monitor what the TheSchwartz::Worker
245 subclasses are currently working on. Once the scoreboard has been
246 enabled in the workers with "set_scoreboard" method the "thetop"
247 utility (shipped with TheSchwartz distribuition in the "extras"
248 directory) can be used to list all current jobs being worked on.
249 "$client->set_scoreboard( $dir )"
251 Enables the scoreboard. Setting this to 1 or "on" will cause
252 TheSchwartz to create a scoreboard file in a location it determines is
253 optimal.
254 Passing in any other option sets the directory the TheSchwartz
256 scoreboard directory should be created in. For example, if you set
257 this to "/tmp" then this would create a directory called
258 "/tmp/theschwartz" and a scoreboard file
259 "/tmp/theschwartz/scoreboard.pid" in it (where pid is the current
260 process pid.)
261 "$client->scoreboard()"
263 Returns the path to the current scoreboard file.
264 "$client->start_scoreboard()"
266 Writes the current job information to the scoreboard file (called by
267 the worker in work_safely before it actually starts working)
268 "$client->end_scoreboard()"
270 Appends the current job duration to the end of the scoreboard file
271 (called by the worker in work_safely once work has been completed)
272 "$client->clean_scoreboard()"
274 Removes the scoreboard file (but not the scoreboard directory.)
275 Automatically called by TheSchwartz during object destruction (i.e.
276 when the instance goes out of scope)
277
279 You can pass in a existing "Data::Object::Driver::DBI" object which
280 also allows you to reuse exist Database handles like so:
281 my $dbh = DBI->connect( $dsn, "root", "", {
283 RaiseError => 1,
284 PrintError => 0,
285 AutoCommit => 1,
286 } ) or die $DBI::errstr;
287 my $driver = Data::ObjectDriver::Driver::DBI->new( dbh => $dbh);
288 return TheSchwartz->new(databases => [{ driver => $driver }]);
289 Note: it's important that the "RaiseError" and "AutoCommit" flags are
291 set on the handle for various bits of functionality to work.
292
294 This software is Copyright 2007, Six Apart Ltd, cpan@sixapart.com. All
295 rights reserved.
296 TheSchwartz is free software; you may redistribute it and/or modify it
298 under the same terms as Perl itself.
299 TheScwhartz comes with no warranty of any kind.
301perl v5.12.0 2010-03-15 TheSchwartz(3)