1Minion(3) User Contributed Perl Documentation Minion(3)
2
6 Minion - Job queue
7
9 use Minion;
10 # Connect to backend
12 my $minion = Minion->new(Pg => 'postgresql://postgres@/test');
13 # Add tasks
15 $minion->add_task(something_slow => sub ($job, @args) {
16 sleep 5;
17 say 'This is a background worker process.';
18 });
19 # Enqueue jobs
21 $minion->enqueue(something_slow => ['foo', 'bar']);
22 $minion->enqueue(something_slow => [1, 2, 3] => {priority => 5});
23 # Perform jobs for testing
25 $minion->enqueue(something_slow => ['foo', 'bar']);
26 $minion->perform_jobs;
27 # Start a worker to perform up to 12 jobs concurrently
29 my $worker = $minion->worker;
30 $worker->status->{jobs} = 12;
31 $worker->run;
32
34 Minion is a high performance job queue for the Perl programming
35 language, with support for multiple named queues, priorities, high
36 priority fast lane, delayed jobs, job dependencies, job progress, job
37 results, retries with backoff, rate limiting, unique jobs, expiring
38 jobs, statistics, distributed workers, parallel processing,
39 autoscaling, remote control, Mojolicious <https://mojolicious.org>
40 admin ui, resource leak protection and multiple backends (such as
41 PostgreSQL <https://www.postgresql.org>).
42 Job queues allow you to process time and/or computationally intensive
44 tasks in background processes, outside of the request/response
45 lifecycle of web applications. Among those tasks you'll commonly find
46 image resizing, spam filtering, HTTP downloads, building tarballs,
47 warming caches and basically everything else you can imagine that's not
48 super fast.
49 Take a look at our excellent documentation in Minion::Guide!
51
53 This distribution also contains a great example application you can use
54 for inspiration. The link checker
55 <https://github.com/mojolicious/minion/tree/main/examples/linkcheck>
56 will show you how to integrate background jobs into well-structured
57 Mojolicious applications.
58
60 Minion inherits all events from Mojo::EventEmitter and can emit the
61 following new ones.
62 enqueue
64 $minion->on(enqueue => sub ($minion, $id) {
65 ...
66 });
67 Emitted after a job has been enqueued, in the process that enqueued it.
69 $minion->on(enqueue => sub ($minion, $id) {
71 say "Job $id has been enqueued.";
72 });
73 worker
75 $minion->on(worker => sub ($minion, $worker) {
76 ...
77 });
78 Emitted in the worker process after it has been created.
80 $minion->on(worker => sub ($minion, $worker) {
82 say "Worker $$ started.";
83 });
84
86 Minion implements the following attributes.
87 app
89 my $app = $minion->app;
90 $minion = $minion->app(MyApp->new);
91 Application for job queue, defaults to a Mojo::HelloWorld object. Note
93 that this attribute is weakened.
94 backend
96 my $backend = $minion->backend;
97 $minion = $minion->backend(Minion::Backend::Pg->new);
98 Backend, usually a Minion::Backend::Pg object.
100 backoff
102 my $cb = $minion->backoff;
103 $minion = $minion->backoff(sub {...});
104 A callback used to calculate the delay for automatically retried jobs,
106 defaults to "(retries ** 4) + 15" (15, 16, 31, 96, 271, 640...), which
107 means that roughly 25 attempts can be made in 21 days.
108 $minion->backoff(sub ($retries) {
110 return ($retries ** 4) + 15 + int(rand 30);
111 });
112 missing_after
114 my $after = $minion->missing_after;
115 $minion = $minion->missing_after(172800);
116 Amount of time in seconds after which workers without a heartbeat will
118 be considered missing and removed from the registry by "repair",
119 defaults to 1800 (30 minutes).
120 remove_after
122 my $after = $minion->remove_after;
123 $minion = $minion->remove_after(86400);
124 Amount of time in seconds after which jobs that have reached the state
126 "finished" and have no unresolved dependencies will be removed
127 automatically by "repair", defaults to 172800 (2 days). It is not
128 recommended to set this value below 2 days.
129 stuck_after
131 my $after = $minion->stuck_after;
132 $minion = $minion->stuck_after(86400);
133 Amount of time in seconds after which jobs that have not been processed
135 will be considered stuck by "repair" and transition to the "failed"
136 state, defaults to 172800 (2 days).
137 tasks
139 my $tasks = $minion->tasks;
140 $minion = $minion->tasks({foo => sub {...}});
141 Registered tasks.
143
145 Minion inherits all methods from Mojo::EventEmitter and implements the
146 following new ones.
147 add_task
149 $minion = $minion->add_task(foo => sub {...});
150 $minion = $minion->add_task(foo => 'MyApp::Task::Foo');
151 Register a task, which can be a closure or a custom Minion::Job
153 subclass. Note that support for custom task classes is EXPERIMENTAL and
154 might change without warning!
155 # Job with result
157 $minion->add_task(add => sub ($job, $first, $second) {
158 $job->finish($first + $second);
159 });
160 my $id = $minion->enqueue(add => [1, 1]);
161 my $result = $minion->job($id)->info->{result};
162 broadcast
164 my $bool = $minion->broadcast('some_command');
165 my $bool = $minion->broadcast('some_command', [@args]);
166 my $bool = $minion->broadcast('some_command', [@args], [$id1, $id2, $id3]);
167 Broadcast remote control command to one or more workers.
169 # Broadcast "stop" command to all workers to kill job 10025
171 $minion->broadcast('stop', [10025]);
172 # Broadcast "kill" command to all workers to interrupt job 10026
174 $minion->broadcast('kill', ['INT', 10026]);
175 # Broadcast "jobs" command to pause worker 23
177 $minion->broadcast('jobs', [0], [23]);
178 class_for_task
180 my $class = $minion->class_for_task('foo');
181 Return job class for task. Note that this method is EXPERIMENTAL and
183 might change without warning!
184 enqueue
186 my $id = $minion->enqueue('foo');
187 my $id = $minion->enqueue(foo => [@args]);
188 my $id = $minion->enqueue(foo => [@args] => {priority => 1});
189 Enqueue a new job with "inactive" state. Arguments get serialized by
191 the "backend" (often with Mojo::JSON), so you shouldn't send objects
192 and be careful with binary data, nested data structures with hash and
193 array references are fine though.
194 These options are currently available:
196 attempts
198 attempts => 25
199 Number of times performing this job will be attempted, with a delay
201 based on "backoff" after the first attempt, defaults to 1.
202 delay
204 delay => 10
205 Delay job for this many seconds (from now), defaults to 0.
207 expire
209 expire => 300
210 Job is valid for this many seconds (from now) before it expires.
212 lax
214 lax => 1
215 Existing jobs this job depends on may also have transitioned to the
217 "failed" state to allow for it to be processed, defaults to "false".
218 Note that this option is EXPERIMENTAL and might change without
219 warning!
220 notes
222 notes => {foo => 'bar', baz => [1, 2, 3]}
223 Hash reference with arbitrary metadata for this job that gets
225 serialized by the "backend" (often with Mojo::JSON), so you shouldn't
226 send objects and be careful with binary data, nested data structures
227 with hash and array references are fine though.
228 parents
230 parents => [$id1, $id2, $id3]
231 One or more existing jobs this job depends on, and that need to have
233 transitioned to the state "finished" before it can be processed.
234 priority
236 priority => 5
237 Job priority, defaults to 0. Jobs with a higher priority get
239 performed first. Priorities can be positive or negative, but should
240 be in the range between 100 and -100.
241 queue
243 queue => 'important'
244 Queue to put job in, defaults to "default".
246 foreground
248 my $bool = $minion->foreground($id);
249 Retry job in "minion_foreground" queue, then perform it right away with
251 a temporary worker in this process, very useful for debugging.
252 guard
254 my $guard = $minion->guard('foo', 3600);
255 my $guard = $minion->guard('foo', 3600, {limit => 20});
256 Same as "lock", but returns a scope guard object that automatically
258 releases the lock as soon as the object is destroyed, or "undef" if
259 aquiring the lock failed.
260 # Only one job should run at a time (unique job)
262 $minion->add_task(do_unique_stuff => sub ($job, @args) {
263 return $job->finish('Previous job is still active')
264 unless my $guard = $minion->guard('fragile_backend_service', 7200);
265 ...
266 });
267 # Only five jobs should run at a time and we try again later if necessary
269 $minion->add_task(do_concurrent_stuff => sub ($job, @args) {
270 return $job->retry({delay => 30})
271 unless my $guard = $minion->guard('some_web_service', 60, {limit => 5});
272 ...
273 });
274 history
276 my $history = $minion->history;
277 Get history information for job queue.
279 These fields are currently available:
281 daily
283 daily => [{epoch => 12345, finished_jobs => 95, failed_jobs => 2}, ...]
284 Hourly counts for processed jobs from the past day.
286 is_locked
288 my $bool = $minion->is_locked('foo');
289 Check if a lock with that name is currently active.
291 job
293 my $job = $minion->job($id);
294 Get Minion::Job object without making any changes to the actual job or
296 return "undef" if job does not exist.
297 # Check job state
299 my $state = $minion->job($id)->info->{state};
300 # Get job metadata
302 my $progress = $minion->job($id)->info->{notes}{progress};
303 # Get job result
305 my $result = $minion->job($id)->info->{result};
306 jobs
308 my $jobs = $minion->jobs;
309 my $jobs = $minion->jobs({states => ['inactive']});
310 Return Minion::Iterator object to safely iterate through job
312 information.
313 # Iterate through jobs for two tasks
315 my $jobs = $minion->jobs({tasks => ['foo', 'bar']});
316 while (my $info = $jobs->next) {
317 say "$info->{id}: $info->{state}";
318 }
319 # Remove all failed jobs from a named queue
321 my $jobs = $minion->jobs({states => ['failed'], queues => ['unimportant']});
322 while (my $info = $jobs->next) {
323 $minion->job($info->{id})->remove;
324 }
325 # Count failed jobs for a task
327 say $minion->jobs({states => ['failed'], tasks => ['foo']})->total;
328 These options are currently available:
330 ids
332 ids => ['23', '24']
333 List only jobs with these ids.
335 notes
337 notes => ['foo', 'bar']
338 List only jobs with one of these notes.
340 queues
342 queues => ['important', 'unimportant']
343 List only jobs in these queues.
345 states
347 states => ['inactive', 'active']
348 List only jobs in these states.
350 tasks
352 tasks => ['foo', 'bar']
353 List only jobs for these tasks.
355 These fields are currently available:
357 args
359 args => ['foo', 'bar']
360 Job arguments.
362 attempts
364 attempts => 25
365 Number of times performing this job will be attempted.
367 children
369 children => ['10026', '10027', '10028']
370 Jobs depending on this job.
372 created
374 created => 784111777
375 Epoch time job was created.
377 delayed
379 delayed => 784111777
380 Epoch time job was delayed to.
382 expires
384 expires => 784111777
385 Epoch time job is valid until before it expires.
387 finished
389 finished => 784111777
390 Epoch time job was finished.
392 id
394 id => 10025
395 Job id.
397 lax
399 lax => 0
400 Existing jobs this job depends on may also have failed to allow for
402 it to be processed.
403 notes
405 notes => {foo => 'bar', baz => [1, 2, 3]}
406 Hash reference with arbitrary metadata for this job.
408 parents
410 parents => ['10023', '10024', '10025']
411 Jobs this job depends on.
413 priority
415 priority => 3
416 Job priority.
418 queue
420 queue => 'important'
421 Queue name.
423 result
425 result => 'All went well!'
426 Job result.
428 retried
430 retried => 784111777
431 Epoch time job has been retried.
433 retries
435 retries => 3
436 Number of times job has been retried.
438 started
440 started => 784111777
441 Epoch time job was started.
443 state
445 state => 'inactive'
446 Current job state, usually "active", "failed", "finished" or
448 "inactive".
449 task
451 task => 'foo'
452 Task name.
454 time
456 time => 78411177
457 Server time.
459 worker
461 worker => '154'
462 Id of worker that is processing the job.
464 lock
466 my $bool = $minion->lock('foo', 3600);
467 my $bool = $minion->lock('foo', 3600, {limit => 20});
468 Try to acquire a named lock that will expire automatically after the
470 given amount of time in seconds. You can release the lock manually with
471 "unlock" to limit concurrency, or let it expire for rate limiting. For
472 convenience you can also use "guard" to release the lock automatically,
473 even if the job failed.
474 # Only one job should run at a time (unique job)
476 $minion->add_task(do_unique_stuff => sub ($job, @args) {
477 return $job->finish('Previous job is still active')
478 unless $minion->lock('fragile_backend_service', 7200);
479 ...
480 $minion->unlock('fragile_backend_service');
481 });
482 # Only five jobs should run at a time and we wait for our turn
484 $minion->add_task(do_concurrent_stuff => sub ($job, @args) {
485 sleep 1 until $minion->lock('some_web_service', 60, {limit => 5});
486 ...
487 $minion->unlock('some_web_service');
488 });
489 # Only a hundred jobs should run per hour and we try again later if necessary
491 $minion->add_task(do_rate_limited_stuff => sub ($job, @args) {
492 return $job->retry({delay => 3600})
493 unless $minion->lock('another_web_service', 3600, {limit => 100});
494 ...
495 });
496 An expiration time of 0 can be used to check if a named lock could have
498 been acquired without creating one.
499 # Check if the lock "foo" could have been acquired
501 say 'Lock could have been acquired' unless $minion->lock('foo', 0);
502 Or to simply check if a named lock already exists you can also use
504 "is_locked".
505 These options are currently available:
507 limit
509 limit => 20
510 Number of shared locks with the same name that can be active at the
512 same time, defaults to 1.
513 new
515 my $minion = Minion->new(Pg => 'postgresql://postgres@/test');
516 my $minion = Minion->new(Pg => Mojo::Pg->new);
517 Construct a new Minion object.
519 perform_jobs
521 $minion->perform_jobs;
522 $minion->perform_jobs({queues => ['important']});
523 Perform all jobs with a temporary worker, very useful for testing.
525 # Longer version
527 my $worker = $minion->worker;
528 while (my $job = $worker->register->dequeue(0)) { $job->perform }
529 $worker->unregister;
530 These options are currently available:
532 id
534 id => '10023'
535 Dequeue a specific job.
537 min_priority
539 min_priority => 3
540 Do not dequeue jobs with a lower priority.
542 queues
544 queues => ['important']
545 One or more queues to dequeue jobs from, defaults to "default".
547 perform_jobs_in_foreground
549 $minion->perform_jobs_in_foreground;
550 $minion->perform_jobs_in_foreground({queues => ['important']});
551 Same as "perform_jobs", but all jobs are performed in the current
553 process, without spawning new processes.
554 repair
556 $minion = $minion->repair;
557 Repair worker registry and job queue if necessary.
559 reset
561 $minion = $minion->reset({all => 1});
562 Reset job queue.
564 These options are currently available:
566 all
568 all => 1
569 Reset everything.
571 locks
573 locks => 1
574 Reset only locks.
576 result_p
578 my $promise = $minion->result_p($id);
579 my $promise = $minion->result_p($id, {interval => 5});
580 Return a Mojo::Promise object for the result of a job. The state
582 "finished" will result in the promise being "fullfilled", and the state
583 "failed" in the promise being "rejected". This operation can be
584 cancelled by resolving the promise manually at any time.
585 # Enqueue job and receive the result at some point in the future
587 my $id = $minion->enqueue('foo');
588 $minion->result_p($id)->then(sub ($info) {
589 my $result = ref $info ? $info->{result} : 'Job already removed';
590 say "Finished: $result";
591 })->catch(sub ($info) {
592 say "Failed: $info->{result}";
593 })->wait;
594 These options are currently available:
596 interval
598 interval => 5
599 Polling interval in seconds for checking if the state of the job has
601 changed, defaults to 3.
602 stats
604 my $stats = $minion->stats;
605 Get statistics for the job queue.
607 # Check idle workers
609 my $idle = $minion->stats->{inactive_workers};
610 These fields are currently available:
612 active_jobs
614 active_jobs => 100
615 Number of jobs in "active" state.
617 active_locks
619 active_locks => 100
620 Number of active named locks.
622 active_workers
624 active_workers => 100
625 Number of workers that are currently processing a job.
627 delayed_jobs
629 delayed_jobs => 100
630 Number of jobs in "inactive" state that are scheduled to run at
632 specific time in the future or have unresolved dependencies.
633 enqueued_jobs
635 enqueued_jobs => 100000
636 Rough estimate of how many jobs have ever been enqueued.
638 failed_jobs
640 failed_jobs => 100
641 Number of jobs in "failed" state.
643 finished_jobs
645 finished_jobs => 100
646 Number of jobs in "finished" state.
648 inactive_jobs
650 inactive_jobs => 100
651 Number of jobs in "inactive" state.
653 inactive_workers
655 inactive_workers => 100
656 Number of workers that are currently not processing a job.
658 uptime
660 uptime => 1000
661 Uptime in seconds.
663 workers
665 workers => 200;
666 Number of registered workers.
668 unlock
670 my $bool = $minion->unlock('foo');
671 Release a named lock that has been previously acquired with "lock".
673 worker
675 my $worker = $minion->worker;
676 Build Minion::Worker object. Note that this method should only be used
678 to implement custom workers.
679 # Use the standard worker with all its features
681 my $worker = $minion->worker;
682 $worker->status->{jobs} = 12;
683 $worker->status->{queues} = ['important'];
684 $worker->run;
685 # Perform one job manually in a separate process
687 my $worker = $minion->repair->worker->register;
688 my $job = $worker->dequeue(5);
689 $job->perform;
690 $worker->unregister;
691 # Perform one job manually in this process
693 my $worker = $minion->repair->worker->register;
694 my $job = $worker->dequeue(5);
695 if (my $err = $job->execute) { $job->fail($err) }
696 else { $job->finish }
697 $worker->unregister;
698 # Build a custom worker performing multiple jobs at the same time
700 my %jobs;
701 my $worker = $minion->repair->worker->register;
702 do {
703 for my $id (keys %jobs) {
704 delete $jobs{$id} if $jobs{$id}->is_finished;
705 }
706 if (keys %jobs >= 4) { sleep 5 }
707 else {
708 my $job = $worker->dequeue(5);
709 $jobs{$job->id} = $job->start if $job;
710 }
711 } while keys %jobs;
712 $worker->unregister;
713 workers
715 my $workers = $minion->workers;
716 my $workers = $minion->workers({ids => [2, 3]});
717 Return Minion::Iterator object to safely iterate through worker
719 information.
720 # Iterate through workers
722 my $workers = $minion->workers;
723 while (my $info = $workers->next) {
724 say "$info->{id}: $info->{host}";
725 }
726 These options are currently available:
728 ids
730 ids => ['23', '24']
731 List only workers with these ids.
733 These fields are currently available:
735 id
737 id => 22
738 Worker id.
740 host
742 host => 'localhost'
743 Worker host.
745 jobs
747 jobs => ['10023', '10024', '10025', '10029']
748 Ids of jobs the worker is currently processing.
750 notified
752 notified => 784111777
753 Epoch time worker sent the last heartbeat.
755 pid
757 pid => 12345
758 Process id of worker.
760 started
762 started => 784111777
763 Epoch time worker was started.
765 status
767 status => {queues => ['default', 'important']}
768 Hash reference with whatever status information the worker would like
770 to share.
771
773 This is the class hierarchy of the Minion distribution.
774 • Minion
776 • Minion::Backend
778 • Minion::Backend::Pg
780 • Minion::Command::minion
782 • Minion::Command::minion::job
784 • Minion::Command::minion::worker
786 • Minion::Iterator
788 • Minion::Job
790 • Minion::Worker
792 • Mojolicious::Plugin::Minion
794 • Mojolicious::Plugin::Minion::Admin
796
798 The Minion distribution includes a few files with different licenses
799 that have been bundled for internal use.
800 Minion Artwork
802 Copyright (C) 2017, Sebastian Riedel.
803 Licensed under the CC-SA License, Version 4.0
805 <http://creativecommons.org/licenses/by-sa/4.0>.
806 Bootstrap
808 Copyright (C) 2011-2021 The Bootstrap Authors.
809 Licensed under the MIT License,
811 <http://creativecommons.org/licenses/MIT>.
812 D3.js
814 Copyright (C) 2010-2016, Michael Bostock.
815 Licensed under the 3-Clause BSD License,
817 <https://opensource.org/licenses/BSD-3-Clause>.
818 epoch.js
820 Copyright (C) 2014 Fastly, Inc.
821 Licensed under the MIT License,
823 <http://creativecommons.org/licenses/MIT>.
824 Font Awesome
826 Copyright (C) Dave Gandy.
827 Licensed under the MIT License,
829 <http://creativecommons.org/licenses/MIT>, and the SIL OFL 1.1,
830 <http://scripts.sil.org/OFL>.
831 moment.js
833 Copyright (C) JS Foundation and other contributors.
834 Licensed under the MIT License,
836 <http://creativecommons.org/licenses/MIT>.
837
839 Project Founder
840 Sebastian Riedel, "sri@cpan.org".
841 Contributors
843 In alphabetical order:
844 Andrey Khozov
846 Andrii Nikitin
848 Brian Medley
850 Franz Skale
852 Hubert "depesz" Lubaczewski
854 Joel Berger
856 Paul Williams
858 Stefan Adams
860
862 Copyright (C) 2014-2022, Sebastian Riedel and others.
863 This program is free software, you can redistribute it and/or modify it
865 under the terms of the Artistic License version 2.0.
866
868 <https://github.com/mojolicious/minion>, Minion::Guide,
869 <https://minion.pm>, Mojolicious::Guides, <https://mojolicious.org>.
870perl v5.36.0 2023-01-20 Minion(3)