1Minion(3)             User Contributed Perl Documentation            Minion(3)
2
3
4

NAME

6       Minion - Job queue
7

SYNOPSIS

9         use Minion;
10
11         # Connect to backend
12         my $minion = Minion->new(Pg => 'postgresql://postgres@/test');
13
14         # Add tasks
15         $minion->add_task(something_slow => sub ($job, @args) {
16           sleep 5;
17           say 'This is a background worker process.';
18         });
19
20         # Enqueue jobs
21         $minion->enqueue(something_slow => ['foo', 'bar']);
22         $minion->enqueue(something_slow => [1, 2, 3] => {priority => 5});
23
24         # Perform jobs for testing
25         $minion->enqueue(something_slow => ['foo', 'bar']);
26         $minion->perform_jobs;
27
28         # Start a worker to perform up to 12 jobs concurrently
29         my $worker = $minion->worker;
30         $worker->status->{jobs} = 12;
31         $worker->run;
32

DESCRIPTION

34       Minion is a high performance job queue for the Perl programming
35       language, with support for multiple named queues, priorities, high
36       priority fast lane, delayed jobs, job dependencies, job progress, job
37       results, retries with backoff, rate limiting, unique jobs, expiring
38       jobs, statistics, distributed workers, parallel processing,
39       autoscaling, remote control, Mojolicious <https://mojolicious.org>
40       admin ui, resource leak protection and multiple backends (such as
41       PostgreSQL <https://www.postgresql.org>).
42
43       Job queues allow you to process time and/or computationally intensive
44       tasks in background processes, outside of the request/response
45       lifecycle of web applications. Among those tasks you'll commonly find
46       image resizing, spam filtering, HTTP downloads, building tarballs,
47       warming caches and basically everything else you can imagine that's not
48       super fast.
49
50       Take a look at our excellent documentation in Minion::Guide!
51

EXAMPLES

53       This distribution also contains a great example application you can use
54       for inspiration. The link checker
55       <https://github.com/mojolicious/minion/tree/main/examples/linkcheck>
56       will show you how to integrate background jobs into well-structured
57       Mojolicious applications.
58

EVENTS

60       Minion inherits all events from Mojo::EventEmitter and can emit the
61       following new ones.
62
63   enqueue
64         $minion->on(enqueue => sub ($minion, $id) {
65           ...
66         });
67
68       Emitted after a job has been enqueued, in the process that enqueued it.
69
70         $minion->on(enqueue => sub ($minion, $id) {
71           say "Job $id has been enqueued.";
72         });
73
74   worker
75         $minion->on(worker => sub ($minion, $worker) {
76           ...
77         });
78
79       Emitted in the worker process after it has been created.
80
81         $minion->on(worker => sub ($minion, $worker) {
82           say "Worker $$ started.";
83         });
84

ATTRIBUTES

86       Minion implements the following attributes.
87
88   app
89         my $app = $minion->app;
90         $minion = $minion->app(MyApp->new);
91
92       Application for job queue, defaults to a Mojo::HelloWorld object. Note
93       that this attribute is weakened.
94
95   backend
96         my $backend = $minion->backend;
97         $minion     = $minion->backend(Minion::Backend::Pg->new);
98
99       Backend, usually a Minion::Backend::Pg object.
100
101   backoff
102         my $cb  = $minion->backoff;
103         $minion = $minion->backoff(sub {...});
104
105       A callback used to calculate the delay for automatically retried jobs,
106       defaults to "(retries ** 4) + 15" (15, 16, 31, 96, 271, 640...), which
107       means that roughly 25 attempts can be made in 21 days.
108
109         $minion->backoff(sub ($retries) {
110           return ($retries ** 4) + 15 + int(rand 30);
111         });
112
113   missing_after
114         my $after = $minion->missing_after;
115         $minion   = $minion->missing_after(172800);
116
117       Amount of time in seconds after which workers without a heartbeat will
118       be considered missing and removed from the registry by "repair",
119       defaults to 1800 (30 minutes).
120
121   remove_after
122         my $after = $minion->remove_after;
123         $minion   = $minion->remove_after(86400);
124
125       Amount of time in seconds after which jobs that have reached the state
126       "finished" and have no unresolved dependencies will be removed
127       automatically by "repair", defaults to 172800 (2 days). It is not
128       recommended to set this value below 2 days.
129
130   stuck_after
131         my $after = $minion->stuck_after;
132         $minion   = $minion->stuck_after(86400);
133
134       Amount of time in seconds after which jobs that have not been processed
135       will be considered stuck by "repair" and transition to the "failed"
136       state, defaults to 172800 (2 days).
137
138   tasks
139         my $tasks = $minion->tasks;
140         $minion   = $minion->tasks({foo => sub {...}});
141
142       Registered tasks.
143

METHODS

145       Minion inherits all methods from Mojo::EventEmitter and implements the
146       following new ones.
147
148   add_task
149         $minion = $minion->add_task(foo => sub {...});
150         $minion = $minion->add_task(foo => 'MyApp::Task::Foo');
151
152       Register a task, which can be a closure or a custom Minion::Job
153       subclass. Note that support for custom task classes is EXPERIMENTAL and
154       might change without warning!
155
156         # Job with result
157         $minion->add_task(add => sub ($job, $first, $second) {
158           $job->finish($first + $second);
159         });
160         my $id = $minion->enqueue(add => [1, 1]);
161         my $result = $minion->job($id)->info->{result};
162
163   broadcast
164         my $bool = $minion->broadcast('some_command');
165         my $bool = $minion->broadcast('some_command', [@args]);
166         my $bool = $minion->broadcast('some_command', [@args], [$id1, $id2, $id3]);
167
168       Broadcast remote control command to one or more workers.
169
170         # Broadcast "stop" command to all workers to kill job 10025
171         $minion->broadcast('stop', [10025]);
172
173         # Broadcast "kill" command to all workers to interrupt job 10026
174         $minion->broadcast('kill', ['INT', 10026]);
175
176         # Broadcast "jobs" command to pause worker 23
177         $minion->broadcast('jobs', [0], [23]);
178
179   class_for_task
180         my $class = $minion->class_for_task('foo');
181
182       Return job class for task. Note that this method is EXPERIMENTAL and
183       might change without warning!
184
185   enqueue
186         my $id = $minion->enqueue('foo');
187         my $id = $minion->enqueue(foo => [@args]);
188         my $id = $minion->enqueue(foo => [@args] => {priority => 1});
189
190       Enqueue a new job with "inactive" state. Arguments get serialized by
191       the "backend" (often with Mojo::JSON), so you shouldn't send objects
192       and be careful with binary data, nested data structures with hash and
193       array references are fine though.
194
195       These options are currently available:
196
197       attempts
198           attempts => 25
199
200         Number of times performing this job will be attempted, with a delay
201         based on "backoff" after the first attempt, defaults to 1.
202
203       delay
204           delay => 10
205
206         Delay job for this many seconds (from now), defaults to 0.
207
208       expire
209           expire => 300
210
211         Job is valid for this many seconds (from now) before it expires.
212
213       lax
214           lax => 1
215
216         Existing jobs this job depends on may also have transitioned to the
217         "failed" state to allow for it to be processed, defaults to "false".
218         Note that this option is EXPERIMENTAL and might change without
219         warning!
220
221       notes
222           notes => {foo => 'bar', baz => [1, 2, 3]}
223
224         Hash reference with arbitrary metadata for this job that gets
225         serialized by the "backend" (often with Mojo::JSON), so you shouldn't
226         send objects and be careful with binary data, nested data structures
227         with hash and array references are fine though.
228
229       parents
230           parents => [$id1, $id2, $id3]
231
232         One or more existing jobs this job depends on, and that need to have
233         transitioned to the state "finished" before it can be processed.
234
235       priority
236           priority => 5
237
238         Job priority, defaults to 0. Jobs with a higher priority get
239         performed first. Priorities can be positive or negative, but should
240         be in the range between 100 and "-100".
241
242       queue
243           queue => 'important'
244
245         Queue to put job in, defaults to "default".
246
247   foreground
248         my $bool = $minion->foreground($id);
249
250       Retry job in "minion_foreground" queue, then perform it right away with
251       a temporary worker in this process, very useful for debugging.
252
253   guard
254         my $guard = $minion->guard('foo', 3600);
255         my $guard = $minion->guard('foo', 3600, {limit => 20});
256
257       Same as "lock", but returns a scope guard object that automatically
258       releases the lock as soon as the object is destroyed, or "undef" if
259       aquiring the lock failed.
260
261         # Only one job should run at a time (unique job)
262         $minion->add_task(do_unique_stuff => sub ($job, @args) {
263           return $job->finish('Previous job is still active')
264             unless my $guard = $minion->guard('fragile_backend_service', 7200);
265           ...
266         });
267
268         # Only five jobs should run at a time and we try again later if necessary
269         $minion->add_task(do_concurrent_stuff => sub ($job, @args) {
270           return $job->retry({delay => 30})
271             unless my $guard = $minion->guard('some_web_service', 60, {limit => 5});
272           ...
273         });
274
275   history
276         my $history = $minion->history;
277
278       Get history information for job queue.
279
280       These fields are currently available:
281
282       daily
283           daily => [{epoch => 12345, finished_jobs => 95, failed_jobs => 2}, ...]
284
285         Hourly counts for processed jobs from the past day.
286
287   is_locked
288         my $bool = $minion->is_locked('foo');
289
290       Check if a lock with that name is currently active.
291
292   job
293         my $job = $minion->job($id);
294
295       Get Minion::Job object without making any changes to the actual job or
296       return "undef" if job does not exist.
297
298         # Check job state
299         my $state = $minion->job($id)->info->{state};
300
301         # Get job metadata
302         my $progress = $minion->job($id)->info->{notes}{progress};
303
304         # Get job result
305         my $result = $minion->job($id)->info->{result};
306
307   jobs
308         my $jobs = $minion->jobs;
309         my $jobs = $minion->jobs({states => ['inactive']});
310
311       Return Minion::Iterator object to safely iterate through job
312       information.
313
314         # Iterate through jobs for two tasks
315         my $jobs = $minion->jobs({tasks => ['foo', 'bar']});
316         while (my $info = $jobs->next) {
317           say "$info->{id}: $info->{state}";
318         }
319
320         # Remove all failed jobs from a named queue
321         my $jobs = $minion->jobs({states => ['failed'], queues => ['unimportant']});
322         while (my $info = $jobs->next) {
323           $minion->job($info->{id})->remove;
324         }
325
326         # Count failed jobs for a task
327         say $minion->jobs({states => ['failed'], tasks => ['foo']})->total;
328
329       These options are currently available:
330
331       ids
332           ids => ['23', '24']
333
334         List only jobs with these ids.
335
336       notes
337           notes => ['foo', 'bar']
338
339         List only jobs with one of these notes.
340
341       queues
342           queues => ['important', 'unimportant']
343
344         List only jobs in these queues.
345
346       states
347           states => ['inactive', 'active']
348
349         List only jobs in these states.
350
351       tasks
352           tasks => ['foo', 'bar']
353
354         List only jobs for these tasks.
355
356       These fields are currently available:
357
358       args
359           args => ['foo', 'bar']
360
361         Job arguments.
362
363       attempts
364           attempts => 25
365
366         Number of times performing this job will be attempted.
367
368       children
369           children => ['10026', '10027', '10028']
370
371         Jobs depending on this job.
372
373       created
374           created => 784111777
375
376         Epoch time job was created.
377
378       delayed
379           delayed => 784111777
380
381         Epoch time job was delayed to.
382
383       expires
384           expires => 784111777
385
386         Epoch time job is valid until before it expires.
387
388       finished
389           finished => 784111777
390
391         Epoch time job was finished.
392
393       id
394           id => 10025
395
396         Job id.
397
398       lax
399           lax => 0
400
401         Existing jobs this job depends on may also have failed to allow for
402         it to be processed.
403
404       notes
405           notes => {foo => 'bar', baz => [1, 2, 3]}
406
407         Hash reference with arbitrary metadata for this job.
408
409       parents
410           parents => ['10023', '10024', '10025']
411
412         Jobs this job depends on.
413
414       priority
415           priority => 3
416
417         Job priority.
418
419       queue
420           queue => 'important'
421
422         Queue name.
423
424       result
425           result => 'All went well!'
426
427         Job result.
428
429       retried
430           retried => 784111777
431
432         Epoch time job has been retried.
433
434       retries
435           retries => 3
436
437         Number of times job has been retried.
438
439       started
440           started => 784111777
441
442         Epoch time job was started.
443
444       state
445           state => 'inactive'
446
447         Current job state, usually "active", "failed", "finished" or
448         "inactive".
449
450       task
451           task => 'foo'
452
453         Task name.
454
455       time
456           time => 78411177
457
458         Server time.
459
460       worker
461           worker => '154'
462
463         Id of worker that is processing the job.
464
465   lock
466         my $bool = $minion->lock('foo', 3600);
467         my $bool = $minion->lock('foo', 3600, {limit => 20});
468
469       Try to acquire a named lock that will expire automatically after the
470       given amount of time in seconds. You can release the lock manually with
471       "unlock" to limit concurrency, or let it expire for rate limiting. For
472       convenience you can also use "guard" to release the lock automatically,
473       even if the job failed.
474
475         # Only one job should run at a time (unique job)
476         $minion->add_task(do_unique_stuff => sub ($job, @args) {
477           return $job->finish('Previous job is still active')
478             unless $minion->lock('fragile_backend_service', 7200);
479           ...
480           $minion->unlock('fragile_backend_service');
481         });
482
483         # Only five jobs should run at a time and we wait for our turn
484         $minion->add_task(do_concurrent_stuff => sub ($job, @args) {
485           sleep 1 until $minion->lock('some_web_service', 60, {limit => 5});
486           ...
487           $minion->unlock('some_web_service');
488         });
489
490         # Only a hundred jobs should run per hour and we try again later if necessary
491         $minion->add_task(do_rate_limited_stuff => sub ($job, @args) {
492           return $job->retry({delay => 3600})
493             unless $minion->lock('another_web_service', 3600, {limit => 100});
494           ...
495         });
496
497       An expiration time of 0 can be used to check if a named lock could have
498       been acquired without creating one.
499
500         # Check if the lock "foo" could have been acquired
501         say 'Lock could have been acquired' unless $minion->lock('foo', 0);
502
503       Or to simply check if a named lock already exists you can also use
504       "is_locked".
505
506       These options are currently available:
507
508       limit
509           limit => 20
510
511         Number of shared locks with the same name that can be active at the
512         same time, defaults to 1.
513
514   new
515         my $minion = Minion->new(Pg => 'postgresql://postgres@/test');
516         my $minion = Minion->new(Pg => Mojo::Pg->new);
517
518       Construct a new Minion object.
519
520   perform_jobs
521         $minion->perform_jobs;
522         $minion->perform_jobs({queues => ['important']});
523
524       Perform all jobs with a temporary worker, very useful for testing.
525
526         # Longer version
527         my $worker = $minion->worker;
528         while (my $job = $worker->register->dequeue(0)) { $job->perform }
529         $worker->unregister;
530
531       These options are currently available:
532
533       id
534           id => '10023'
535
536         Dequeue a specific job.
537
538       min_priority
539           min_priority => 3
540
541         Do not dequeue jobs with a lower priority.
542
543       queues
544           queues => ['important']
545
546         One or more queues to dequeue jobs from, defaults to "default".
547
548   perform_jobs_in_foreground
549         $minion->perform_jobs_in_foreground;
550         $minion->perform_jobs_in_foreground({queues => ['important']});
551
552       Same as "perform_jobs", but all jobs are performed in the current
553       process, without spawning new processes.
554
555   repair
556         $minion = $minion->repair;
557
558       Repair worker registry and job queue if necessary.
559
560   reset
561         $minion = $minion->reset({all => 1});
562
563       Reset job queue.
564
565       These options are currently available:
566
567       all
568           all => 1
569
570         Reset everything.
571
572       locks
573           locks => 1
574
575         Reset only locks.
576
577   result_p
578         my $promise = $minion->result_p($id);
579         my $promise = $minion->result_p($id, {interval => 5});
580
581       Return a Mojo::Promise object for the result of a job. The state
582       "finished" will result in the promise being "fullfilled", and the state
583       "failed" in the promise being "rejected". This operation can be
584       cancelled by resolving the promise manually at any time.
585
586         # Enqueue job and receive the result at some point in the future
587         my $id = $minion->enqueue('foo');
588         $minion->result_p($id)->then(sub ($info) {
589           my $result = ref $info ? $info->{result} : 'Job already removed';
590           say "Finished: $result";
591         })->catch(sub ($info) {
592           say "Failed: $info->{result}";
593         })->wait;
594
595       These options are currently available:
596
597       interval
598           interval => 5
599
600         Polling interval in seconds for checking if the state of the job has
601         changed, defaults to 3.
602
603   stats
604         my $stats = $minion->stats;
605
606       Get statistics for the job queue.
607
608         # Check idle workers
609         my $idle = $minion->stats->{inactive_workers};
610
611       These fields are currently available:
612
613       active_jobs
614           active_jobs => 100
615
616         Number of jobs in "active" state.
617
618       active_locks
619           active_locks => 100
620
621         Number of active named locks.
622
623       active_workers
624           active_workers => 100
625
626         Number of workers that are currently processing a job.
627
628       delayed_jobs
629           delayed_jobs => 100
630
631         Number of jobs in "inactive" state that are scheduled to run at
632         specific time in the future or have unresolved dependencies.
633
634       enqueued_jobs
635           enqueued_jobs => 100000
636
637         Rough estimate of how many jobs have ever been enqueued.
638
639       failed_jobs
640           failed_jobs => 100
641
642         Number of jobs in "failed" state.
643
644       finished_jobs
645           finished_jobs => 100
646
647         Number of jobs in "finished" state.
648
649       inactive_jobs
650           inactive_jobs => 100
651
652         Number of jobs in "inactive" state.
653
654       inactive_workers
655           inactive_workers => 100
656
657         Number of workers that are currently not processing a job.
658
659       uptime
660           uptime => 1000
661
662         Uptime in seconds.
663
664       workers
665           workers => 200;
666
667         Number of registered workers.
668
669   unlock
670         my $bool = $minion->unlock('foo');
671
672       Release a named lock that has been previously acquired with "lock".
673
674   worker
675         my $worker = $minion->worker;
676
677       Build Minion::Worker object. Note that this method should only be used
678       to implement custom workers.
679
680         # Use the standard worker with all its features
681         my $worker = $minion->worker;
682         $worker->status->{jobs} = 12;
683         $worker->status->{queues} = ['important'];
684         $worker->run;
685
686         # Perform one job manually in a separate process
687         my $worker = $minion->repair->worker->register;
688         my $job    = $worker->dequeue(5);
689         $job->perform;
690         $worker->unregister;
691
692         # Perform one job manually in this process
693         my $worker = $minion->repair->worker->register;
694         my $job    = $worker->dequeue(5);
695         if (my $err = $job->execute) { $job->fail($err) }
696         else                         { $job->finish }
697         $worker->unregister;
698
699         # Build a custom worker performing multiple jobs at the same time
700         my %jobs;
701         my $worker = $minion->repair->worker->register;
702         do {
703           for my $id (keys %jobs) {
704             delete $jobs{$id} if $jobs{$id}->is_finished;
705           }
706           if (keys %jobs >= 4) { sleep 5 }
707           else {
708             my $job = $worker->dequeue(5);
709             $jobs{$job->id} = $job->start if $job;
710           }
711         } while keys %jobs;
712         $worker->unregister;
713
714   workers
715         my $workers = $minion->workers;
716         my $workers = $minion->workers({ids => [2, 3]});
717
718       Return Minion::Iterator object to safely iterate through worker
719       information.
720
721         # Iterate through workers
722         my $workers = $minion->workers;
723         while (my $info = $workers->next) {
724           say "$info->{id}: $info->{host}";
725         }
726
727       These options are currently available:
728
729       ids
730           ids => ['23', '24']
731
732         List only workers with these ids.
733
734       These fields are currently available:
735
736       id
737           id => 22
738
739         Worker id.
740
741       host
742           host => 'localhost'
743
744         Worker host.
745
746       jobs
747           jobs => ['10023', '10024', '10025', '10029']
748
749         Ids of jobs the worker is currently processing.
750
751       notified
752           notified => 784111777
753
754         Epoch time worker sent the last heartbeat.
755
756       pid
757           pid => 12345
758
759         Process id of worker.
760
761       started
762           started => 784111777
763
764         Epoch time worker was started.
765
766       status
767           status => {queues => ['default', 'important']}
768
769         Hash reference with whatever status information the worker would like
770         to share.
771

API

773       This is the class hierarchy of the Minion distribution.
774
775       • Minion
776
777       • Minion::Backend
778
779         • Minion::Backend::Pg
780
781       • Minion::Command::minion
782
783       • Minion::Command::minion::job
784
785       • Minion::Command::minion::worker
786
787       • Minion::Iterator
788
789       • Minion::Job
790
791       • Minion::Worker
792
793       • Mojolicious::Plugin::Minion
794
795       • Mojolicious::Plugin::Minion::Admin
796

BUNDLED FILES

798       The Minion distribution includes a few files with different licenses
799       that have been bundled for internal use.
800
801   Minion Artwork
802         Copyright (C) 2017, Sebastian Riedel.
803
804       Licensed under the CC-SA License, Version 4.0
805       <http://creativecommons.org/licenses/by-sa/4.0>.
806
807   Bootstrap
808         Copyright (C) 2011-2021 The Bootstrap Authors.
809
810       Licensed under the MIT License,
811       <http://creativecommons.org/licenses/MIT>.
812
813   D3.js
814         Copyright (C) 2010-2016, Michael Bostock.
815
816       Licensed under the 3-Clause BSD License,
817       <https://opensource.org/licenses/BSD-3-Clause>.
818
819   epoch.js
820         Copyright (C) 2014 Fastly, Inc.
821
822       Licensed under the MIT License,
823       <http://creativecommons.org/licenses/MIT>.
824
825   Font Awesome
826         Copyright (C) Dave Gandy.
827
828       Licensed under the MIT License,
829       <http://creativecommons.org/licenses/MIT>, and the SIL OFL 1.1,
830       <http://scripts.sil.org/OFL>.
831
832   moment.js
833         Copyright (C) JS Foundation and other contributors.
834
835       Licensed under the MIT License,
836       <http://creativecommons.org/licenses/MIT>.
837

AUTHORS

839   Project Founder
840       Sebastian Riedel, "sri@cpan.org".
841
842   Contributors
843       In alphabetical order:
844
845         Andrey Khozov
846
847         Andrii Nikitin
848
849         Brian Medley
850
851         Franz Skale
852
853         Hubert "depesz" Lubaczewski
854
855         Joel Berger
856
857         Paul Williams
858
859         Stefan Adams
860
862       Copyright (C) 2014-2022, Sebastian Riedel and others.
863
864       This program is free software, you can redistribute it and/or modify it
865       under the terms of the Artistic License version 2.0.
866

SEE ALSO

868       <https://github.com/mojolicious/minion>, Minion::Guide,
869       <https://minion.pm>, Mojolicious::Guides, <https://mojolicious.org>.
870
871
872
873perl v5.36.0                      2022-07-22                         Minion(3)
Impressum