1Minion(3) User Contributed Perl Documentation Minion(3)
2
6 Minion - Job queue
7
9 use Minion;
10 # Connect to backend
12 my $minion = Minion->new(Pg => 'postgresql://postgres@/test');
13 # Add tasks
15 $minion->add_task(something_slow => sub ($job, @args) {
16 sleep 5;
17 say 'This is a background worker process.';
18 });
19 # Enqueue jobs
21 $minion->enqueue(something_slow => ['foo', 'bar']);
22 $minion->enqueue(something_slow => [1, 2, 3] => {priority => 5});
23 # Perform jobs for testing
25 $minion->enqueue(something_slow => ['foo', 'bar']);
26 $minion->perform_jobs;
27 # Start a worker to perform up to 12 jobs concurrently
29 my $worker = $minion->worker;
30 $worker->status->{jobs} = 12;
31 $worker->run;
32
34 Minion is a high performance job queue for the Perl programming
35 language, with support for multiple named queues, priorities, high
36 priority fast lane, delayed jobs, job dependencies, job progress, job
37 results, retries with backoff, rate limiting, unique jobs, expiring
38 jobs, statistics, distributed workers, parallel processing,
39 autoscaling, remote control, Mojolicious <https://mojolicious.org>
40 admin ui, resource leak protection and multiple backends (such as
41 PostgreSQL <https://www.postgresql.org>).
42 Job queues allow you to process time and/or computationally intensive
44 tasks in background processes, outside of the request/response
45 lifecycle of web applications. Among those tasks you'll commonly find
46 image resizing, spam filtering, HTTP downloads, building tarballs,
47 warming caches and basically everything else you can imagine that's not
48 super fast.
49 Take a look at our excellent documentation in Minion::Guide!
51
53 This distribution also contains a great example application you can use
54 for inspiration. The link checker
55 <https://github.com/mojolicious/minion/tree/main/examples/linkcheck>
56 will show you how to integrate background jobs into well-structured
57 Mojolicious applications.
58
60 Minion inherits all events from Mojo::EventEmitter and can emit the
61 following new ones.
62 enqueue
64 $minion->on(enqueue => sub ($minion, $id) {
65 ...
66 });
67 Emitted after a job has been enqueued, in the process that enqueued it.
69 $minion->on(enqueue => sub ($minion, $id) {
71 say "Job $id has been enqueued.";
72 });
73 worker
75 $minion->on(worker => sub ($minion, $worker) {
76 ...
77 });
78 Emitted in the worker process after it has been created.
80 $minion->on(worker => sub ($minion, $worker) {
82 say "Worker $$ started.";
83 });
84
86 Minion implements the following attributes.
87 app
89 my $app = $minion->app;
90 $minion = $minion->app(MyApp->new);
91 Application for job queue, defaults to a Mojo::HelloWorld object. Note
93 that this attribute is weakened.
94 backend
96 my $backend = $minion->backend;
97 $minion = $minion->backend(Minion::Backend::Pg->new);
98 Backend, usually a Minion::Backend::Pg object.
100 backoff
102 my $cb = $minion->backoff;
103 $minion = $minion->backoff(sub {...});
104 A callback used to calculate the delay for automatically retried jobs,
106 defaults to "(retries ** 4) + 15" (15, 16, 31, 96, 271, 640...), which
107 means that roughly 25 attempts can be made in 21 days.
108 $minion->backoff(sub ($retries) {
110 return ($retries ** 4) + 15 + int(rand 30);
111 });
112 missing_after
114 my $after = $minion->missing_after;
115 $minion = $minion->missing_after(172800);
116 Amount of time in seconds after which workers without a heartbeat will
118 be considered missing and removed from the registry by "repair",
119 defaults to 1800 (30 minutes).
120 remove_after
122 my $after = $minion->remove_after;
123 $minion = $minion->remove_after(86400);
124 Amount of time in seconds after which jobs that have reached the state
126 "finished" and have no unresolved dependencies will be removed
127 automatically by "repair", defaults to 172800 (2 days). It is not
128 recommended to set this value below 2 days.
129 stuck_after
131 my $after = $minion->stuck_after;
132 $minion = $minion->stuck_after(86400);
133 Amount of time in seconds after which jobs that have not been processed
135 will be considered stuck by "repair" and transition to the "failed"
136 state, defaults to 172800 (2 days).
137 tasks
139 my $tasks = $minion->tasks;
140 $minion = $minion->tasks({foo => sub {...}});
141 Registered tasks.
143
145 Minion inherits all methods from Mojo::EventEmitter and implements the
146 following new ones.
147 add_task
149 $minion = $minion->add_task(foo => sub {...});
150 $minion = $minion->add_task(foo => 'MyApp::Task::Foo');
151 Register a task, which can be a closure or a custom Minion::Job
153 subclass. Note that support for custom task classes is EXPERIMENTAL and
154 might change without warning!
155 # Job with result
157 $minion->add_task(add => sub ($job, $first, $second) {
158 $job->finish($first + $second);
159 });
160 my $id = $minion->enqueue(add => [1, 1]);
161 my $result = $minion->job($id)->info->{result};
162 broadcast
164 my $bool = $minion->broadcast('some_command');
165 my $bool = $minion->broadcast('some_command', [@args]);
166 my $bool = $minion->broadcast('some_command', [@args], [$id1, $id2, $id3]);
167 Broadcast remote control command to one or more workers.
169 # Broadcast "stop" command to all workers to kill job 10025
171 $minion->broadcast('stop', [10025]);
172 # Broadcast "kill" command to all workers to interrupt job 10026
174 $minion->broadcast('kill', ['INT', 10026]);
175 # Broadcast "jobs" command to pause worker 23
177 $minion->broadcast('jobs', [0], [23]);
178 class_for_task
180 my $class = $minion->class_for_task('foo');
181 Return job class for task. Note that this method is EXPERIMENTAL and
183 might change without warning!
184 enqueue
186 my $id = $minion->enqueue('foo');
187 my $id = $minion->enqueue(foo => [@args]);
188 my $id = $minion->enqueue(foo => [@args] => {priority => 1});
189 Enqueue a new job with "inactive" state. Arguments get serialized by
191 the "backend" (often with Mojo::JSON), so you shouldn't send objects
192 and be careful with binary data, nested data structures with hash and
193 array references are fine though.
194 These options are currently available:
196 attempts
198 attempts => 25
199 Number of times performing this job will be attempted, with a delay
201 based on "backoff" after the first attempt, defaults to 1.
202 delay
204 delay => 10
205 Delay job for this many seconds (from now), defaults to 0.
207 expire
209 expire => 300
210 Job is valid for this many seconds (from now) before it expires.
212 lax
214 lax => 1
215 Existing jobs this job depends on may also have transitioned to the
217 "failed" state to allow for it to be processed, defaults to "false".
218 Note that this option is EXPERIMENTAL and might change without
219 warning!
220 notes
222 notes => {foo => 'bar', baz => [1, 2, 3]}
223 Hash reference with arbitrary metadata for this job that gets
225 serialized by the "backend" (often with Mojo::JSON), so you shouldn't
226 send objects and be careful with binary data, nested data structures
227 with hash and array references are fine though.
228 parents
230 parents => [$id1, $id2, $id3]
231 One or more existing jobs this job depends on, and that need to have
233 transitioned to the state "finished" before it can be processed.
234 priority
236 priority => 5
237 Job priority, defaults to 0. Jobs with a higher priority get
239 performed first. Priorities can be positive or negative, but should
240 be in the range between 100 and "-100".
241 queue
243 queue => 'important'
244 Queue to put job in, defaults to "default".
246 foreground
248 my $bool = $minion->foreground($id);
249 Retry job in "minion_foreground" queue, then perform it right away with
251 a temporary worker in this process, very useful for debugging.
252 guard
254 my $guard = $minion->guard('foo', 3600);
255 my $guard = $minion->guard('foo', 3600, {limit => 20});
256 Same as "lock", but returns a scope guard object that automatically
258 releases the lock as soon as the object is destroyed, or "undef" if
259 aquiring the lock failed.
260 # Only one job should run at a time (unique job)
262 $minion->add_task(do_unique_stuff => sub ($job, @args) {
263 return $job->finish('Previous job is still active')
264 unless my $guard = $minion->guard('fragile_backend_service', 7200);
265 ...
266 });
267 # Only five jobs should run at a time and we try again later if necessary
269 $minion->add_task(do_concurrent_stuff => sub ($job, @args) {
270 return $job->retry({delay => 30})
271 unless my $guard = $minion->guard('some_web_service', 60, {limit => 5});
272 ...
273 });
274 history
276 my $history = $minion->history;
277 Get history information for job queue.
279 These fields are currently available:
281 daily
283 daily => [{epoch => 12345, finished_jobs => 95, failed_jobs => 2}, ...]
284 Hourly counts for processed jobs from the past day.
286 is_locked
288 my $bool = $minion->is_locked('foo');
289 Check if a lock with that name is currently active.
291 job
293 my $job = $minion->job($id);
294 Get Minion::Job object without making any changes to the actual job or
296 return "undef" if job does not exist.
297 # Check job state
299 my $state = $minion->job($id)->info->{state};
300 # Get job metadata
302 my $progress = $minion->job($id)->info->{notes}{progress};
303 # Get job result
305 my $result = $minion->job($id)->info->{result};
306 jobs
308 my $jobs = $minion->jobs;
309 my $jobs = $minion->jobs({states => ['inactive']});
310 Return Minion::Iterator object to safely iterate through job
312 information.
313 # Iterate through jobs for two tasks
315 my $jobs = $minion->jobs({tasks => ['foo', 'bar']});
316 while (my $info = $jobs->next) {
317 say "$info->{id}: $info->{state}";
318 }
319 # Remove all failed jobs from a named queue
321 my $jobs = $minion->jobs({states => ['failed'], queues => ['unimportant']});
322 while (my $info = $jobs->next) {
323 $minion->job($info->{id})->remove;
324 }
325 # Count failed jobs for a task
327 say $minion->jobs({states => ['failed'], tasks => ['foo']})->total;
328 These options are currently available:
330 ids
332 ids => ['23', '24']
333 List only jobs with these ids.
335 notes
337 notes => ['foo', 'bar']
338 List only jobs with one of these notes.
340 queues
342 queues => ['important', 'unimportant']
343 List only jobs in these queues.
345 states
347 states => ['inactive', 'active']
348 List only jobs in these states.
350 tasks
352 tasks => ['foo', 'bar']
353 List only jobs for these tasks.
355 These fields are currently available:
357 args
359 args => ['foo', 'bar']
360 Job arguments.
362 attempts
364 attempts => 25
365 Number of times performing this job will be attempted.
367 children
369 children => ['10026', '10027', '10028']
370 Jobs depending on this job.
372 created
374 created => 784111777
375 Epoch time job was created.
377 delayed
379 delayed => 784111777
380 Epoch time job was delayed to.
382 expires
384 expires => 784111777
385 Epoch time job is valid until before it expires.
387 finished
389 finished => 784111777
390 Epoch time job was finished.
392 id
394 id => 10025
395 Job id.
397 lax
399 lax => 0
400 Existing jobs this job depends on may also have failed to allow for
402 it to be processed.
403 notes
405 notes => {foo => 'bar', baz => [1, 2, 3]}
406 Hash reference with arbitrary metadata for this job.
408 parents
410 parents => ['10023', '10024', '10025']
411 Jobs this job depends on.
413 priority
415 priority => 3
416 Job priority.
418 queue
420 queue => 'important'
421 Queue name.
423 result
425 result => 'All went well!'
426 Job result.
428 retried
430 retried => 784111777
431 Epoch time job has been retried.
433 retries
435 retries => 3
436 Number of times job has been retried.
438 started
440 started => 784111777
441 Epoch time job was started.
443 state
445 state => 'inactive'
446 Current job state, usually "active", "failed", "finished" or
448 "inactive".
449 task
451 task => 'foo'
452 Task name.
454 time
456 time => 78411177
457 Server time.
459 worker
461 worker => '154'
462 Id of worker that is processing the job.
464 lock
466 my $bool = $minion->lock('foo', 3600);
467 my $bool = $minion->lock('foo', 3600, {limit => 20});
468 Try to acquire a named lock that will expire automatically after the
470 given amount of time in seconds. You can release the lock manually with
471 "unlock" to limit concurrency, or let it expire for rate limiting. For
472 convenience you can also use "guard" to release the lock automatically,
473 even if the job failed.
474 # Only one job should run at a time (unique job)
476 $minion->add_task(do_unique_stuff => sub ($job, @args) {
477 return $job->finish('Previous job is still active')
478 unless $minion->lock('fragile_backend_service', 7200);
479 ...
480 $minion->unlock('fragile_backend_service');
481 });
482 # Only five jobs should run at a time and we wait for our turn
484 $minion->add_task(do_concurrent_stuff => sub ($job, @args) {
485 sleep 1 until $minion->lock('some_web_service', 60, {limit => 5});
486 ...
487 $minion->unlock('some_web_service');
488 });
489 # Only a hundred jobs should run per hour and we try again later if necessary
491 $minion->add_task(do_rate_limited_stuff => sub ($job, @args) {
492 return $job->retry({delay => 3600})
493 unless $minion->lock('another_web_service', 3600, {limit => 100});
494 ...
495 });
496 An expiration time of 0 can be used to check if a named lock could have
498 been acquired without creating one.
499 # Check if the lock "foo" could have been acquired
501 say 'Lock could have been acquired' unless $minion->lock('foo', 0);
502 Or to simply check if a named lock already exists you can also use
504 "is_locked".
505 These options are currently available:
507 limit
509 limit => 20
510 Number of shared locks with the same name that can be active at the
512 same time, defaults to 1.
513 new
515 my $minion = Minion->new(Pg => 'postgresql://postgres@/test');
516 my $minion = Minion->new(Pg => Mojo::Pg->new);
517 Construct a new Minion object.
519 perform_jobs
521 $minion->perform_jobs;
522 $minion->perform_jobs({queues => ['important']});
523 Perform all jobs with a temporary worker, very useful for testing.
525 # Longer version
527 my $worker = $minion->worker;
528 while (my $job = $worker->register->dequeue(0)) { $job->perform }
529 $worker->unregister;
530 These options are currently available:
532 queues
534 queues => ['important']
535 One or more queues to dequeue jobs from, defaults to "default".
537 repair
539 $minion = $minion->repair;
540 Repair worker registry and job queue if necessary.
542 reset
544 $minion = $minion->reset({all => 1});
545 Reset job queue.
547 These options are currently available:
549 all
551 all => 1
552 Reset everything.
554 locks
556 locks => 1
557 Reset only locks.
559 perform_jobs_in_foreground
561 $minion->perform_jobs_in_foreground;
562 $minion->perform_jobs_in_foreground({queues => ['important']});
563 Same as "perform_jobs", but all jobs are performed in the current
565 process, without spawning new processes.
566 result_p
568 my $promise = $minion->result_p($id);
569 my $promise = $minion->result_p($id, {interval => 5});
570 Return a Mojo::Promise object for the result of a job. The state
572 "finished" will result in the promise being "fullfilled", and the state
573 "failed" in the promise being "rejected". This operation can be
574 cancelled by resolving the promise manually at any time.
575 # Enqueue job and receive the result at some point in the future
577 my $id = $minion->enqueue('foo');
578 $minion->result_p($id)->then(sub ($info) {
579 my $result = ref $info ? $info->{result} : 'Job already removed';
580 say "Finished: $result";
581 })->catch(sub ($info) {
582 say "Failed: $info->{result}";
583 })->wait;
584 These options are currently available:
586 interval
588 interval => 5
589 Polling interval in seconds for checking if the state of the job has
591 changed, defaults to 3.
592 stats
594 my $stats = $minion->stats;
595 Get statistics for the job queue.
597 # Check idle workers
599 my $idle = $minion->stats->{inactive_workers};
600 These fields are currently available:
602 active_jobs
604 active_jobs => 100
605 Number of jobs in "active" state.
607 active_locks
609 active_locks => 100
610 Number of active named locks.
612 active_workers
614 active_workers => 100
615 Number of workers that are currently processing a job.
617 delayed_jobs
619 delayed_jobs => 100
620 Number of jobs in "inactive" state that are scheduled to run at
622 specific time in the future or have unresolved dependencies.
623 enqueued_jobs
625 enqueued_jobs => 100000
626 Rough estimate of how many jobs have ever been enqueued.
628 failed_jobs
630 failed_jobs => 100
631 Number of jobs in "failed" state.
633 finished_jobs
635 finished_jobs => 100
636 Number of jobs in "finished" state.
638 inactive_jobs
640 inactive_jobs => 100
641 Number of jobs in "inactive" state.
643 inactive_workers
645 inactive_workers => 100
646 Number of workers that are currently not processing a job.
648 uptime
650 uptime => 1000
651 Uptime in seconds.
653 unlock
655 my $bool = $minion->unlock('foo');
656 Release a named lock that has been previously acquired with "lock".
658 worker
660 my $worker = $minion->worker;
661 Build Minion::Worker object. Note that this method should only be used
663 to implement custom workers.
664 # Use the standard worker with all its features
666 my $worker = $minion->worker;
667 $worker->status->{jobs} = 12;
668 $worker->status->{queues} = ['important'];
669 $worker->run;
670 # Perform one job manually in a separate process
672 my $worker = $minion->repair->worker->register;
673 my $job = $worker->dequeue(5);
674 $job->perform;
675 $worker->unregister;
676 # Perform one job manually in this process
678 my $worker = $minion->repair->worker->register;
679 my $job = $worker->dequeue(5);
680 if (my $err = $job->execute) { $job->fail($err) }
681 else { $job->finish }
682 $worker->unregister;
683 # Build a custom worker performing multiple jobs at the same time
685 my %jobs;
686 my $worker = $minion->repair->worker->register;
687 do {
688 for my $id (keys %jobs) {
689 delete $jobs{$id} if $jobs{$id}->is_finished;
690 }
691 if (keys %jobs >= 4) { sleep 5 }
692 else {
693 my $job = $worker->dequeue(5);
694 $jobs{$job->id} = $job->start if $job;
695 }
696 } while keys %jobs;
697 $worker->unregister;
698 workers
700 my $workers = $minion->workers;
701 my $workers = $minion->workers({ids => [2, 3]});
702 Return Minion::Iterator object to safely iterate through worker
704 information.
705 # Iterate through workers
707 my $workers = $minion->workers;
708 while (my $info = $workers->next) {
709 say "$info->{id}: $info->{host}";
710 }
711 These options are currently available:
713 ids
715 ids => ['23', '24']
716 List only workers with these ids.
718 These fields are currently available:
720 id
722 id => 22
723 Worker id.
725 host
727 host => 'localhost'
728 Worker host.
730 jobs
732 jobs => ['10023', '10024', '10025', '10029']
733 Ids of jobs the worker is currently processing.
735 notified
737 notified => 784111777
738 Epoch time worker sent the last heartbeat.
740 pid
742 pid => 12345
743 Process id of worker.
745 started
747 started => 784111777
748 Epoch time worker was started.
750 status
752 status => {queues => ['default', 'important']}
753 Hash reference with whatever status information the worker would like
755 to share.
756
758 This is the class hierarchy of the Minion distribution.
759 • Minion
761 • Minion::Backend
763 • Minion::Backend::Pg
765 • Minion::Command::minion
767 • Minion::Command::minion::job
769 • Minion::Command::minion::worker
771 • Minion::Iterator
773 • Minion::Job
775 • Minion::Worker
777 • Mojolicious::Plugin::Minion
779 • Mojolicious::Plugin::Minion::Admin
781
783 The Minion distribution includes a few files with different licenses
784 that have been bundled for internal use.
785 Minion Artwork
787 Copyright (C) 2017, Sebastian Riedel.
788 Licensed under the CC-SA License, Version 4.0
790 <http://creativecommons.org/licenses/by-sa/4.0>.
791 Bootstrap
793 Copyright (C) 2011-2021 The Bootstrap Authors.
794 Licensed under the MIT License,
796 <http://creativecommons.org/licenses/MIT>.
797 D3.js
799 Copyright (C) 2010-2016, Michael Bostock.
800 Licensed under the 3-Clause BSD License,
802 <https://opensource.org/licenses/BSD-3-Clause>.
803 epoch.js
805 Copyright (C) 2014 Fastly, Inc.
806 Licensed under the MIT License,
808 <http://creativecommons.org/licenses/MIT>.
809 Font Awesome
811 Copyright (C) Dave Gandy.
812 Licensed under the MIT License,
814 <http://creativecommons.org/licenses/MIT>, and the SIL OFL 1.1,
815 <http://scripts.sil.org/OFL>.
816 moment.js
818 Copyright (C) JS Foundation and other contributors.
819 Licensed under the MIT License,
821 <http://creativecommons.org/licenses/MIT>.
822 popper.js
824 Copyright (C) Federico Zivolo 2017.
825 Licensed under the MIT License,
827 <http://creativecommons.org/licenses/MIT>.
828
830 Project Founder
831 Sebastian Riedel, "sri@cpan.org".
832 Contributors
834 In alphabetical order:
835 Andrey Khozov
837 Andrii Nikitin
839 Brian Medley
841 Franz Skale
843 Hubert "depesz" Lubaczewski
845 Joel Berger
847 Paul Williams
849 Stefan Adams
851
853 Copyright (C) 2014-2021, Sebastian Riedel and others.
854 This program is free software, you can redistribute it and/or modify it
856 under the terms of the Artistic License version 2.0.
857
859 <https://github.com/mojolicious/minion>, Minion::Guide,
860 <https://minion.pm>, Mojolicious::Guides, <https://mojolicious.org>.
861perl v5.34.0 2021-07-22 Minion(3)