1Minion(3) User Contributed Perl Documentation Minion(3)
2
6 Minion - Job queue
7
9 use Minion;
10 # Connect to backend
12 my $minion = Minion->new(Pg => 'postgresql://postgres@/test');
13 # Add tasks
15 $minion->add_task(something_slow => sub {
16 my ($job, @args) = @_;
17 sleep 5;
18 say 'This is a background worker process.';
19 });
20 # Enqueue jobs
22 $minion->enqueue(something_slow => ['foo', 'bar']);
23 $minion->enqueue(something_slow => [1, 2, 3] => {priority => 5});
24 # Perform jobs for testing
26 $minion->enqueue(something_slow => ['foo', 'bar']);
27 $minion->perform_jobs;
28 # Start a worker to perform up to 12 jobs concurrently
30 my $worker = $minion->worker;
31 $worker->status->{jobs} = 12;
32 $worker->run;
33
35 Minion is a high performance job queue for the Perl programming
36 language, with support for multiple named queues, priorities, delayed
37 jobs, job dependencies, job progress, job results, retries with
38 backoff, rate limiting, unique jobs, statistics, distributed workers,
39 parallel processing, autoscaling, remote control, Mojolicious
40 <https://mojolicious.org> admin ui, resource leak protection and
41 multiple backends (such as PostgreSQL <https://www.postgresql.org>).
42 Job queues allow you to process time and/or computationally intensive
44 tasks in background processes, outside of the request/response
45 lifecycle of web applications. Among those tasks you'll commonly find
46 image resizing, spam filtering, HTTP downloads, building tarballs,
47 warming caches and basically everything else you can imagine that's not
48 super fast.
49
51 You can use Minion as a standalone job queue or integrate it into
52 Mojolicious applications with the plugin Mojolicious::Plugin::Minion.
53 use Mojolicious::Lite;
55 plugin Minion => {Pg => 'postgresql://sri:s3cret@localhost/test'};
57 # Slow task
59 app->minion->add_task(poke_mojo => sub {
60 my $job = shift;
61 $job->app->ua->get('mojolicious.org');
62 $job->app->log->debug('We have poked mojolicious.org for a visitor');
63 });
64 # Perform job in a background worker process
66 get '/' => sub {
67 my $c = shift;
68 $c->minion->enqueue('poke_mojo');
69 $c->render(text => 'We will poke mojolicious.org for you soon.');
70 };
71 app->start;
73 Background worker processes are usually started with the command
75 Minion::Command::minion::worker, which becomes automatically available
76 when an application loads Mojolicious::Plugin::Minion.
77 $ ./myapp.pl minion worker
79 Jobs can be managed right from the command line with
81 Minion::Command::minion::job.
82 $ ./myapp.pl minion job
84 You can also add an admin ui to your application by loading the plugin
86 Mojolicious::Plugin::Minion::Admin. Just make sure to secure access
87 before making your application publically accessible.
88 # Make admin ui available under "/minion"
90 plugin 'Minion::Admin';
91 To manage background worker processes with systemd, you can use a unit
93 configuration file like this.
94 [Unit]
96 Description=My Mojolicious application workers
97 After=postgresql.service
98 [Service]
100 Type=simple
101 ExecStart=/home/sri/myapp/myapp.pl minion worker -m production
102 KillMode=process
103 [Install]
105 WantedBy=multi-user.target
106 Every job can fail or succeed, but not get lost, the system is
108 eventually consistent and will preserve job results for as long as you
109 like, depending on "remove_after". While individual workers can fail in
110 the middle of processing a job, the system will detect this and ensure
111 that no job is left in an uncertain state, depending on
112 "missing_after".
113
115 And as your application grows, you can move tasks into application
116 specific plugins.
117 package MyApp::Task::PokeMojo;
119 use Mojo::Base 'Mojolicious::Plugin';
120 sub register {
122 my ($self, $app) = @_;
123 $app->minion->add_task(poke_mojo => sub {
124 my $job = shift;
125 $job->app->ua->get('mojolicious.org');
126 $job->app->log->debug('We have poked mojolicious.org for a visitor');
127 });
128 }
129 1;
131 Which are loaded like any other plugin from your application.
133 # Mojolicious
135 $app->plugin('MyApp::Task::PokeMojo');
136 # Mojolicious::Lite
138 plugin 'MyApp::Task::PokeMojo';
139
141 This distribution also contains a great example application you can use
142 for inspiration. The link checker
143 <https://github.com/mojolicious/minion/tree/master/examples/linkcheck>
144 will show you how to integrate background jobs into well-structured
145 Mojolicious applications.
146
148 Minion inherits all events from Mojo::EventEmitter and can emit the
149 following new ones.
150 enqueue
152 $minion->on(enqueue => sub {
153 my ($minion, $id) = @_;
154 ...
155 });
156 Emitted after a job has been enqueued, in the process that enqueued it.
158 $minion->on(enqueue => sub {
160 my ($minion, $id) = @_;
161 say "Job $id has been enqueued.";
162 });
163 worker
165 $minion->on(worker => sub {
166 my ($minion, $worker) = @_;
167 ...
168 });
169 Emitted in the worker process after it has been created.
171 $minion->on(worker => sub {
173 my ($minion, $worker) = @_;
174 my $id = $worker->id;
175 say "Worker $$:$id started.";
176 });
177
179 Minion implements the following attributes.
180 app
182 my $app = $minion->app;
183 $minion = $minion->app(MyApp->new);
184 Application for job queue, defaults to a Mojo::HelloWorld object. Note
186 that this attribute is weakened.
187 backend
189 my $backend = $minion->backend;
190 $minion = $minion->backend(Minion::Backend::Pg->new);
191 Backend, usually a Minion::Backend::Pg object.
193 backoff
195 my $cb = $minion->backoff;
196 $minion = $minion->backoff(sub {...});
197 A callback used to calculate the delay for automatically retried jobs,
199 defaults to "(retries ** 4) + 15" (15, 16, 31, 96, 271, 640...), which
200 means that roughly 25 attempts can be made in 21 days.
201 $minion->backoff(sub {
203 my $retries = shift;
204 return ($retries ** 4) + 15 + int(rand 30);
205 });
206 missing_after
208 my $after = $minion->missing_after;
209 $minion = $minion->missing_after(172800);
210 Amount of time in seconds after which workers without a heartbeat will
212 be considered missing and removed from the registry by "repair",
213 defaults to 1800 (30 minutes).
214 remove_after
216 my $after = $minion->remove_after;
217 $minion = $minion->remove_after(86400);
218 Amount of time in seconds after which jobs that have reached the state
220 "finished" and have no unresolved dependencies will be removed
221 automatically by "repair", defaults to 172800 (2 days). It is not
222 recommended to set this value below 2 days.
223 tasks
225 my $tasks = $minion->tasks;
226 $minion = $minion->tasks({foo => sub {...}});
227 Registered tasks.
229
231 Minion inherits all methods from Mojo::EventEmitter and implements the
232 following new ones.
233 add_task
235 $minion = $minion->add_task(foo => sub {...});
236 Register a task.
238 # Job with result
240 $minion->add_task(add => sub {
241 my ($job, $first, $second) = @_;
242 $job->finish($first + $second);
243 });
244 my $id = $minion->enqueue(add => [1, 1]);
245 my $result = $minion->job($id)->info->{result};
246 broadcast
248 my $bool = $minion->broadcast('some_command');
249 my $bool = $minion->broadcast('some_command', [@args]);
250 my $bool = $minion->broadcast('some_command', [@args], [$id1, $id2, $id3]);
251 Broadcast remote control command to one or more workers.
253 # Broadcast "stop" command to all workers to kill job 10025
255 $minion->broadcast('stop', [10025]);
256 # Broadcast "kill" command to all workers to interrupt job 10026
258 $minion->broadcast('kill', ['INT', 10026]);
259 # Broadcast "jobs" command to pause worker 23
261 $minion->broadcast('jobs', [0], [23]);
262 enqueue
264 my $id = $minion->enqueue('foo');
265 my $id = $minion->enqueue(foo => [@args]);
266 my $id = $minion->enqueue(foo => [@args] => {priority => 1});
267 Enqueue a new job with "inactive" state. Arguments get serialized by
269 the "backend" (often with Mojo::JSON), so you shouldn't send objects
270 and be careful with binary data, nested data structures with hash and
271 array references are fine though.
272 These options are currently available:
274 attempts
276 attempts => 25
277 Number of times performing this job will be attempted, with a delay
279 based on "backoff" after the first attempt, defaults to 1.
280 delay
282 delay => 10
283 Delay job for this many seconds (from now), defaults to 0.
285 notes
287 notes => {foo => 'bar', baz => [1, 2, 3]}
288 Hash reference with arbitrary metadata for this job that gets
290 serialized by the "backend" (often with Mojo::JSON), so you shouldn't
291 send objects and be careful with binary data, nested data structures
292 with hash and array references are fine though.
293 parents
295 parents => [$id1, $id2, $id3]
296 One or more existing jobs this job depends on, and that need to have
298 transitioned to the state "finished" before it can be processed.
299 priority
301 priority => 5
302 Job priority, defaults to 0. Jobs with a higher priority get
304 performed first.
305 queue
307 queue => 'important'
308 Queue to put job in, defaults to "default".
310 foreground
312 my $bool = $minion->foreground($id);
313 Retry job in "minion_foreground" queue, then perform it right away with
315 a temporary worker in this process, very useful for debugging.
316 guard
318 my $guard = $minion->guard('foo', 3600);
319 my $guard = $minion->guard('foo', 3600, {limit => 20});
320 Same as "lock", but returns a scope guard object that automatically
322 releases the lock as soon as the object is destroyed, or "undef" if
323 aquiring the lock failed.
324 # Only one job should run at a time (unique job)
326 $minion->add_task(do_unique_stuff => sub {
327 my ($job, @args) = @_;
328 return $job->finish('Previous job is still active')
329 unless my $guard = $minion->guard('fragile_backend_service', 7200);
330 ...
331 });
332 # Only five jobs should run at a time and we try again later if necessary
334 $minion->add_task(do_concurrent_stuff => sub {
335 my ($job, @args) = @_;
336 return $job->retry({delay => 30})
337 unless my $guard = $minion->guard('some_web_service', 60, {limit => 5});
338 ...
339 });
340 history
342 my $history = $minion->history;
343 Get history information for job queue. Note that this method is
345 EXPERIMENTAL and might change without warning!
346 These fields are currently available:
348 daily
350 daily => [{epoch => 12345, finished_jobs => 95, failed_jobs => 2}, ...]
351 Hourly counts for processed jobs from the past day.
353 job
355 my $job = $minion->job($id);
356 Get Minion::Job object without making any changes to the actual job or
358 return "undef" if job does not exist.
359 # Check job state
361 my $state = $minion->job($id)->info->{state};
362 # Get job metadata
364 my $progress = $minion->$job($id)->info->{notes}{progress};
365 # Get job result
367 my $result = $minion->job($id)->info->{result};
368 lock
370 my $bool = $minion->lock('foo', 3600);
371 my $bool = $minion->lock('foo', 3600, {limit => 20});
372 Try to acquire a named lock that will expire automatically after the
374 given amount of time in seconds. You can release the lock manually with
375 "unlock" to limit concurrency, or let it expire for rate limiting. For
376 convenience you can also use "guard" to release the lock automatically,
377 even if the job failed.
378 # Only one job should run at a time (unique job)
380 $minion->add_task(do_unique_stuff => sub {
381 my ($job, @args) = @_;
382 return $job->finish('Previous job is still active')
383 unless $minion->lock('fragile_backend_service', 7200);
384 ...
385 $minion->unlock('fragile_backend_service');
386 });
387 # Only five jobs should run at a time and we wait for our turn
389 $minion->add_task(do_concurrent_stuff => sub {
390 my ($job, @args) = @_;
391 sleep 1 until $minion->lock('some_web_service', 60, {limit => 5});
392 ...
393 $minion->unlock('some_web_service');
394 });
395 # Only a hundred jobs should run per hour and we try again later if necessary
397 $minion->add_task(do_rate_limited_stuff => sub {
398 my ($job, @args) = @_;
399 return $job->retry({delay => 3600})
400 unless $minion->lock('another_web_service', 3600, {limit => 100});
401 ...
402 });
403 An expiration time of 0 can be used to check if a named lock already
405 exists without creating one.
406 # Check if the lock "foo" already exists
408 say 'Lock exists' unless $minion->lock('foo', 0);
409 These options are currently available:
411 limit
413 limit => 20
414 Number of shared locks with the same name that can be active at the
416 same time, defaults to 1.
417 new
419 my $minion = Minion->new(Pg => 'postgresql://postgres@/test');
420 my $minion = Minion->new(Pg => Mojo::Pg->new);
421 Construct a new Minion object.
423 perform_jobs
425 $minion->perform_jobs;
426 $minion->perform_jobs({queues => ['important']});
427 Perform all jobs with a temporary worker, very useful for testing.
429 # Longer version
431 my $worker = $minion->worker;
432 while (my $job = $worker->register->dequeue(0)) { $job->perform }
433 $worker->unregister;
434 These options are currently available:
436 queues
438 queues => ['important']
439 One or more queues to dequeue jobs from, defaults to "default".
441 repair
443 $minion = $minion->repair;
444 Repair worker registry and job queue if necessary.
446 reset
448 $minion = $minion->reset;
449 Reset job queue.
451 result_p
453 my $promise = $minion->result_p($id);
454 my $promise = $minion->result_p($id, {interval => 5});
455 Return a Mojo::Promise object for the result of a job. The state
457 "finished" will result in the promise being "fullfilled", and the state
458 "failed" in the promise being "rejected". This operation can be
459 cancelled by resolving the promise manually at any time. Note that this
460 method is EXPERIMENTAL and might change without warning!
461 # Enqueue job and receive the result at some point in the future
463 my $id = $minion->enqueue('foo');
464 $minion->result_p($id)->then(sub {
465 my $info = shift;
466 my $result = ref $info ? $info->{result} : 'Job already removed';
467 say "Finished: $result";
468 })->catch(sub {
469 my $info = shift;
470 say "Failed: $info->{result}";
471 })->wait;
472 These options are currently available:
474 interval
476 interval => 5
477 Polling interval in seconds for checking if the state of the job has
479 changed, defaults to 3.
480 stats
482 my $stats = $minion->stats;
483 Get statistics for the job queue.
485 # Check idle workers
487 my $idle = $minion->stats->{inactive_workers};
488 These fields are currently available:
490 active_jobs
492 active_jobs => 100
493 Number of jobs in "active" state.
495 active_locks
497 active_locks => 100
498 Number of active named locks.
500 active_workers
502 active_workers => 100
503 Number of workers that are currently processing a job.
505 delayed_jobs
507 delayed_jobs => 100
508 Number of jobs in "inactive" state that are scheduled to run at
510 specific time in the future or have unresolved dependencies. Note
511 that this field is EXPERIMENTAL and might change without warning!
512 enqueued_jobs
514 enqueued_jobs => 100000
515 Rough estimate of how many jobs have ever been enqueued. Note that
517 this field is EXPERIMENTAL and might change without warning!
518 failed_jobs
520 failed_jobs => 100
521 Number of jobs in "failed" state.
523 finished_jobs
525 finished_jobs => 100
526 Number of jobs in "finished" state.
528 inactive_jobs
530 inactive_jobs => 100
531 Number of jobs in "inactive" state.
533 inactive_workers
535 inactive_workers => 100
536 Number of workers that are currently not processing a job.
538 uptime
540 uptime => 1000
541 Uptime in seconds.
543 unlock
545 my $bool = $minion->unlock('foo');
546 Release a named lock that has been previously acquired with "lock".
548 worker
550 my $worker = $minion->worker;
551 Build Minion::Worker object.
553 # Use the standard worker with all its features
555 my $worker = $minion->worker;
556 $worker->status->{jobs} = 12;
557 $worker->status->{queues} = ['important'];
558 $worker->run;
559 # Perform one job manually in a separate process
561 my $worker = $minion->repair->worker->register;
562 my $job = $worker->dequeue(5);
563 $job->perform;
564 $worker->unregister;
565 # Perform one job manually in this process
567 my $worker = $minion->repair->worker->register;
568 my $job = $worker->dequeue(5);
569 if (my $err = $job->execute) { $job->fail($err) }
570 else { $job->finish }
571 $worker->unregister;
572 # Build a custom worker performing multiple jobs at the same time
574 my %jobs;
575 my $worker = $minion->repair->worker->register;
576 do {
577 for my $id (keys %jobs) {
578 delete $jobs{$id} if $jobs{$id}->is_finished;
579 }
580 if (keys %jobs >= 4) { sleep 5 }
581 else {
582 my $job = $worker->dequeue(5);
583 $jobs{$job->id} = $job->start if $job;
584 }
585 } while keys %jobs;
586 $worker->unregister;
587
589 This is the class hierarchy of the Minion distribution.
590 · Minion
592 · Minion::Backend
594 · Minion::Backend::Pg
596 · Minion::Command::minion
598 · Minion::Command::minion::job
600 · Minion::Command::minion::worker
602 · Minion::Job
604 · Minion::Worker
606 · Mojolicious::Plugin::Minion
608 · Mojolicious::Plugin::Minion::Admin
610
612 The Minion distribution includes a few files with different licenses
613 that have been bundled for internal use.
614 Minion Artwork
616 Copyright (C) 2017, Sebastian Riedel.
617 Licensed under the CC-SA License, Version 4.0
619 <http://creativecommons.org/licenses/by-sa/4.0>.
620 Bootstrap
622 Copyright (C) 2011-2018 The Bootstrap Authors.
623 Licensed under the MIT License,
625 <http://creativecommons.org/licenses/MIT>.
626 D3.js
628 Copyright (C) 2010-2016, Michael Bostock.
629 Licensed under the 3-Clause BSD License,
631 <https://opensource.org/licenses/BSD-3-Clause>.
632 epoch.js
634 Copyright (C) 2014 Fastly, Inc.
635 Licensed under the MIT License,
637 <http://creativecommons.org/licenses/MIT>.
638 Font Awesome
640 Copyright (C) Dave Gandy.
641 Licensed under the MIT License,
643 <http://creativecommons.org/licenses/MIT>, and the SIL OFL 1.1,
644 <http://scripts.sil.org/OFL>.
645 moment.js
647 Copyright (C) JS Foundation and other contributors.
648 Licensed under the MIT License,
650 <http://creativecommons.org/licenses/MIT>.
651 popper.js
653 Copyright (C) Federico Zivolo 2017.
654 Licensed under the MIT License,
656 <http://creativecommons.org/licenses/MIT>.
657
659 Sebastian Riedel, "sri@cpan.org".
660
662 In alphabetical order:
663 Andrey Khozov
665 Brian Medley
667 Hubert "depesz" Lubaczewski
669 Joel Berger
671 Paul Williams
673 Stefan Adams
675
677 Copyright (C) 2014-2019, Sebastian Riedel and others.
678 This program is free software, you can redistribute it and/or modify it
680 under the terms of the Artistic License version 2.0.
681
683 <https://github.com/mojolicious/minion>, Mojolicious::Guides,
684 <https://mojolicious.org>.
685perl v5.28.1 2019-02-03 Minion(3)