1Minion(3)             User Contributed Perl Documentation            Minion(3)
2
3
4

NAME

6       Minion - Job queue
7

SYNOPSIS

9         use Minion;
10
11         # Connect to backend
12         my $minion = Minion->new(Pg => 'postgresql://postgres@/test');
13
14         # Add tasks
15         $minion->add_task(something_slow => sub {
16           my ($job, @args) = @_;
17           sleep 5;
18           say 'This is a background worker process.';
19         });
20
21         # Enqueue jobs
22         $minion->enqueue(something_slow => ['foo', 'bar']);
23         $minion->enqueue(something_slow => [1, 2, 3] => {priority => 5});
24
25         # Perform jobs for testing
26         $minion->enqueue(something_slow => ['foo', 'bar']);
27         $minion->perform_jobs;
28
29         # Start a worker to perform up to 12 jobs concurrently
30         my $worker = $minion->worker;
31         $worker->status->{jobs} = 12;
32         $worker->run;
33

DESCRIPTION

35       Minion is a high performance job queue for the Perl programming
36       language, with support for multiple named queues, priorities, delayed
37       jobs, job dependencies, job progress, job results, retries with
38       backoff, rate limiting, unique jobs, statistics, distributed workers,
39       parallel processing, autoscaling, remote control, Mojolicious
40       <https://mojolicious.org> admin ui, resource leak protection and
41       multiple backends (such as PostgreSQL <https://www.postgresql.org>).
42
43       Job queues allow you to process time and/or computationally intensive
44       tasks in background processes, outside of the request/response
45       lifecycle of web applications. Among those tasks you'll commonly find
46       image resizing, spam filtering, HTTP downloads, building tarballs,
47       warming caches and basically everything else you can imagine that's not
48       super fast.
49

BASICS

51       You can use Minion as a standalone job queue or integrate it into
52       Mojolicious applications with the plugin Mojolicious::Plugin::Minion.
53
54         use Mojolicious::Lite;
55
56         plugin Minion => {Pg => 'postgresql://sri:s3cret@localhost/test'};
57
58         # Slow task
59         app->minion->add_task(poke_mojo => sub {
60           my $job = shift;
61           $job->app->ua->get('mojolicious.org');
62           $job->app->log->debug('We have poked mojolicious.org for a visitor');
63         });
64
65         # Perform job in a background worker process
66         get '/' => sub {
67           my $c = shift;
68           $c->minion->enqueue('poke_mojo');
69           $c->render(text => 'We will poke mojolicious.org for you soon.');
70         };
71
72         app->start;
73
74       Background worker processes are usually started with the command
75       Minion::Command::minion::worker, which becomes automatically available
76       when an application loads Mojolicious::Plugin::Minion.
77
78         $ ./myapp.pl minion worker
79
80       The worker process will fork a new process for every job that is being
81       processed. This allows for resources such as memory to be returned to
82       the operating system once a job is finished. Perl fork is very fast, so
83       don't worry about the overhead.
84
85         Minion::Worker
86         |- Minion::Job [1]
87         |- Minion::Job [2]
88         +- ...
89
90       By default up to four jobs will be processed in parallel, but that can
91       be changed with configuration options or on demand with signals.
92
93         $ ./myapp.pl minion worker -j 12
94
95       Jobs can be managed right from the command line with
96       Minion::Command::minion::job.
97
98         $ ./myapp.pl minion job
99
100       You can also add an admin ui to your application by loading the plugin
101       Mojolicious::Plugin::Minion::Admin. Just make sure to secure access
102       before making your application publically accessible.
103
104         # Make admin ui available under "/minion"
105         plugin 'Minion::Admin';
106
107       To manage background worker processes with systemd, you can use a unit
108       configuration file like this.
109
110         [Unit]
111         Description=My Mojolicious application workers
112         After=postgresql.service
113
114         [Service]
115         Type=simple
116         ExecStart=/home/sri/myapp/myapp.pl minion worker -m production
117         KillMode=process
118
119         [Install]
120         WantedBy=multi-user.target
121
122       Every job can fail or succeed, but not get lost, the system is
123       eventually consistent and will preserve job results for as long as you
124       like, depending on "remove_after". While individual workers can fail in
125       the middle of processing a job, the system will detect this and ensure
126       that no job is left in an uncertain state, depending on
127       "missing_after".
128

GROWING

130       And as your application grows, you can move tasks into application
131       specific plugins.
132
133         package MyApp::Task::PokeMojo;
134         use Mojo::Base 'Mojolicious::Plugin';
135
136         sub register {
137           my ($self, $app) = @_;
138           $app->minion->add_task(poke_mojo => sub {
139             my $job = shift;
140             $job->app->ua->get('mojolicious.org');
141             $job->app->log->debug('We have poked mojolicious.org for a visitor');
142           });
143         }
144
145         1;
146
147       Which are loaded like any other plugin from your application.
148
149         # Mojolicious
150         $app->plugin('MyApp::Task::PokeMojo');
151
152         # Mojolicious::Lite
153         plugin 'MyApp::Task::PokeMojo';
154

EXAMPLES

156       This distribution also contains a great example application you can use
157       for inspiration. The link checker
158       <https://github.com/mojolicious/minion/tree/master/examples/linkcheck>
159       will show you how to integrate background jobs into well-structured
160       Mojolicious applications.
161

EVENTS

163       Minion inherits all events from Mojo::EventEmitter and can emit the
164       following new ones.
165
166   enqueue
167         $minion->on(enqueue => sub {
168           my ($minion, $id) = @_;
169           ...
170         });
171
172       Emitted after a job has been enqueued, in the process that enqueued it.
173
174         $minion->on(enqueue => sub {
175           my ($minion, $id) = @_;
176           say "Job $id has been enqueued.";
177         });
178
179   worker
180         $minion->on(worker => sub {
181           my ($minion, $worker) = @_;
182           ...
183         });
184
185       Emitted in the worker process after it has been created.
186
187         $minion->on(worker => sub {
188           my ($minion, $worker) = @_;
189           my $id = $worker->id;
190           say "Worker $$:$id started.";
191         });
192

ATTRIBUTES

194       Minion implements the following attributes.
195
196   app
197         my $app = $minion->app;
198         $minion = $minion->app(MyApp->new);
199
200       Application for job queue, defaults to a Mojo::HelloWorld object. Note
201       that this attribute is weakened.
202
203   backend
204         my $backend = $minion->backend;
205         $minion     = $minion->backend(Minion::Backend::Pg->new);
206
207       Backend, usually a Minion::Backend::Pg object.
208
209   backoff
210         my $cb  = $minion->backoff;
211         $minion = $minion->backoff(sub {...});
212
213       A callback used to calculate the delay for automatically retried jobs,
214       defaults to "(retries ** 4) + 15" (15, 16, 31, 96, 271, 640...), which
215       means that roughly 25 attempts can be made in 21 days.
216
217         $minion->backoff(sub {
218           my $retries = shift;
219           return ($retries ** 4) + 15 + int(rand 30);
220         });
221
222   missing_after
223         my $after = $minion->missing_after;
224         $minion   = $minion->missing_after(172800);
225
226       Amount of time in seconds after which workers without a heartbeat will
227       be considered missing and removed from the registry by "repair",
228       defaults to 1800 (30 minutes).
229
230   remove_after
231         my $after = $minion->remove_after;
232         $minion   = $minion->remove_after(86400);
233
234       Amount of time in seconds after which jobs that have reached the state
235       "finished" and have no unresolved dependencies will be removed
236       automatically by "repair", defaults to 172800 (2 days). It is not
237       recommended to set this value below 2 days.
238
239   tasks
240         my $tasks = $minion->tasks;
241         $minion   = $minion->tasks({foo => sub {...}});
242
243       Registered tasks.
244

METHODS

246       Minion inherits all methods from Mojo::EventEmitter and implements the
247       following new ones.
248
249   add_task
250         $minion = $minion->add_task(foo => sub {...});
251
252       Register a task.
253
254         # Job with result
255         $minion->add_task(add => sub {
256           my ($job, $first, $second) = @_;
257           $job->finish($first + $second);
258         });
259         my $id = $minion->enqueue(add => [1, 1]);
260         my $result = $minion->job($id)->info->{result};
261
262   broadcast
263         my $bool = $minion->broadcast('some_command');
264         my $bool = $minion->broadcast('some_command', [@args]);
265         my $bool = $minion->broadcast('some_command', [@args], [$id1, $id2, $id3]);
266
267       Broadcast remote control command to one or more workers.
268
269         # Broadcast "stop" command to all workers to kill job 10025
270         $minion->broadcast('stop', [10025]);
271
272         # Broadcast "kill" command to all workers to interrupt job 10026
273         $minion->broadcast('kill', ['INT', 10026]);
274
275         # Broadcast "jobs" command to pause worker 23
276         $minion->broadcast('jobs', [0], [23]);
277
278   enqueue
279         my $id = $minion->enqueue('foo');
280         my $id = $minion->enqueue(foo => [@args]);
281         my $id = $minion->enqueue(foo => [@args] => {priority => 1});
282
283       Enqueue a new job with "inactive" state. Arguments get serialized by
284       the "backend" (often with Mojo::JSON), so you shouldn't send objects
285       and be careful with binary data, nested data structures with hash and
286       array references are fine though.
287
288       These options are currently available:
289
290       attempts
291           attempts => 25
292
293         Number of times performing this job will be attempted, with a delay
294         based on "backoff" after the first attempt, defaults to 1.
295
296       delay
297           delay => 10
298
299         Delay job for this many seconds (from now), defaults to 0.
300
301       notes
302           notes => {foo => 'bar', baz => [1, 2, 3]}
303
304         Hash reference with arbitrary metadata for this job that gets
305         serialized by the "backend" (often with Mojo::JSON), so you shouldn't
306         send objects and be careful with binary data, nested data structures
307         with hash and array references are fine though.
308
309       parents
310           parents => [$id1, $id2, $id3]
311
312         One or more existing jobs this job depends on, and that need to have
313         transitioned to the state "finished" before it can be processed.
314
315       priority
316           priority => 5
317
318         Job priority, defaults to 0. Jobs with a higher priority get
319         performed first.
320
321       queue
322           queue => 'important'
323
324         Queue to put job in, defaults to "default".
325
326   foreground
327         my $bool = $minion->foreground($id);
328
329       Retry job in "minion_foreground" queue, then perform it right away with
330       a temporary worker in this process, very useful for debugging.
331
332   guard
333         my $guard = $minion->guard('foo', 3600);
334         my $guard = $minion->guard('foo', 3600, {limit => 20});
335
336       Same as "lock", but returns a scope guard object that automatically
337       releases the lock as soon as the object is destroyed, or "undef" if
338       aquiring the lock failed.
339
340         # Only one job should run at a time (unique job)
341         $minion->add_task(do_unique_stuff => sub {
342           my ($job, @args) = @_;
343           return $job->finish('Previous job is still active')
344             unless my $guard = $minion->guard('fragile_backend_service', 7200);
345           ...
346         });
347
348         # Only five jobs should run at a time and we try again later if necessary
349         $minion->add_task(do_concurrent_stuff => sub {
350           my ($job, @args) = @_;
351           return $job->retry({delay => 30})
352             unless my $guard = $minion->guard('some_web_service', 60, {limit => 5});
353           ...
354         });
355
356   history
357         my $history = $minion->history;
358
359       Get history information for job queue. Note that this method is
360       EXPERIMENTAL and might change without warning!
361
362       These fields are currently available:
363
364       daily
365           daily => [{epoch => 12345, finished_jobs => 95, failed_jobs => 2}, ...]
366
367         Hourly counts for processed jobs from the past day.
368
369   job
370         my $job = $minion->job($id);
371
372       Get Minion::Job object without making any changes to the actual job or
373       return "undef" if job does not exist.
374
375         # Check job state
376         my $state = $minion->job($id)->info->{state};
377
378         # Get job metadata
379         my $progress = $minion->$job($id)->info->{notes}{progress};
380
381         # Get job result
382         my $result = $minion->job($id)->info->{result};
383
384   lock
385         my $bool = $minion->lock('foo', 3600);
386         my $bool = $minion->lock('foo', 3600, {limit => 20});
387
388       Try to acquire a named lock that will expire automatically after the
389       given amount of time in seconds. You can release the lock manually with
390       "unlock" to limit concurrency, or let it expire for rate limiting. For
391       convenience you can also use "guard" to release the lock automatically,
392       even if the job failed.
393
394         # Only one job should run at a time (unique job)
395         $minion->add_task(do_unique_stuff => sub {
396           my ($job, @args) = @_;
397           return $job->finish('Previous job is still active')
398             unless $minion->lock('fragile_backend_service', 7200);
399           ...
400           $minion->unlock('fragile_backend_service');
401         });
402
403         # Only five jobs should run at a time and we wait for our turn
404         $minion->add_task(do_concurrent_stuff => sub {
405           my ($job, @args) = @_;
406           sleep 1 until $minion->lock('some_web_service', 60, {limit => 5});
407           ...
408           $minion->unlock('some_web_service');
409         });
410
411         # Only a hundred jobs should run per hour and we try again later if necessary
412         $minion->add_task(do_rate_limited_stuff => sub {
413           my ($job, @args) = @_;
414           return $job->retry({delay => 3600})
415             unless $minion->lock('another_web_service', 3600, {limit => 100});
416           ...
417         });
418
419       An expiration time of 0 can be used to check if a named lock already
420       exists without creating one.
421
422         # Check if the lock "foo" already exists
423         say 'Lock exists' unless $minion->lock('foo', 0);
424
425       These options are currently available:
426
427       limit
428           limit => 20
429
430         Number of shared locks with the same name that can be active at the
431         same time, defaults to 1.
432
433   new
434         my $minion = Minion->new(Pg => 'postgresql://postgres@/test');
435         my $minion = Minion->new(Pg => Mojo::Pg->new);
436
437       Construct a new Minion object.
438
439   perform_jobs
440         $minion->perform_jobs;
441         $minion->perform_jobs({queues => ['important']});
442
443       Perform all jobs with a temporary worker, very useful for testing.
444
445         # Longer version
446         my $worker = $minion->worker;
447         while (my $job = $worker->register->dequeue(0)) { $job->perform }
448         $worker->unregister;
449
450       These options are currently available:
451
452       queues
453           queues => ['important']
454
455         One or more queues to dequeue jobs from, defaults to "default".
456
457   repair
458         $minion = $minion->repair;
459
460       Repair worker registry and job queue if necessary.
461
462   reset
463         $minion = $minion->reset;
464
465       Reset job queue.
466
467   result_p
468         my $promise = $minion->result_p($id);
469         my $promise = $minion->result_p($id, {interval => 5});
470
471       Return a Mojo::Promise object for the result of a job. The state
472       "finished" will result in the promise being "fullfilled", and the state
473       "failed" in the promise being "rejected". This operation can be
474       cancelled by resolving the promise manually at any time. Note that this
475       method is EXPERIMENTAL and might change without warning!
476
477         # Enqueue job and receive the result at some point in the future
478         my $id = $minion->enqueue('foo');
479         $minion->result_p($id)->then(sub {
480           my $info   = shift;
481           my $result = ref $info ? $info->{result} : 'Job already removed';
482           say "Finished: $result";
483         })->catch(sub {
484           my $info = shift;
485           say "Failed: $info->{result}";
486         })->wait;
487
488       These options are currently available:
489
490       interval
491           interval => 5
492
493         Polling interval in seconds for checking if the state of the job has
494         changed, defaults to 3.
495
496   stats
497         my $stats = $minion->stats;
498
499       Get statistics for the job queue.
500
501         # Check idle workers
502         my $idle = $minion->stats->{inactive_workers};
503
504       These fields are currently available:
505
506       active_jobs
507           active_jobs => 100
508
509         Number of jobs in "active" state.
510
511       active_locks
512           active_locks => 100
513
514         Number of active named locks.
515
516       active_workers
517           active_workers => 100
518
519         Number of workers that are currently processing a job.
520
521       delayed_jobs
522           delayed_jobs => 100
523
524         Number of jobs in "inactive" state that are scheduled to run at
525         specific time in the future or have unresolved dependencies. Note
526         that this field is EXPERIMENTAL and might change without warning!
527
528       enqueued_jobs
529           enqueued_jobs => 100000
530
531         Rough estimate of how many jobs have ever been enqueued. Note that
532         this field is EXPERIMENTAL and might change without warning!
533
534       failed_jobs
535           failed_jobs => 100
536
537         Number of jobs in "failed" state.
538
539       finished_jobs
540           finished_jobs => 100
541
542         Number of jobs in "finished" state.
543
544       inactive_jobs
545           inactive_jobs => 100
546
547         Number of jobs in "inactive" state.
548
549       inactive_workers
550           inactive_workers => 100
551
552         Number of workers that are currently not processing a job.
553
554       uptime
555           uptime => 1000
556
557         Uptime in seconds.
558
559   unlock
560         my $bool = $minion->unlock('foo');
561
562       Release a named lock that has been previously acquired with "lock".
563
564   worker
565         my $worker = $minion->worker;
566
567       Build Minion::Worker object. Note that this method should only be used
568       to implement custom workers.
569
570         # Use the standard worker with all its features
571         my $worker = $minion->worker;
572         $worker->status->{jobs} = 12;
573         $worker->status->{queues} = ['important'];
574         $worker->run;
575
576         # Perform one job manually in a separate process
577         my $worker = $minion->repair->worker->register;
578         my $job    = $worker->dequeue(5);
579         $job->perform;
580         $worker->unregister;
581
582         # Perform one job manually in this process
583         my $worker = $minion->repair->worker->register;
584         my $job    = $worker->dequeue(5);
585         if (my $err = $job->execute) { $job->fail($err) }
586         else                         { $job->finish }
587         $worker->unregister;
588
589         # Build a custom worker performing multiple jobs at the same time
590         my %jobs;
591         my $worker = $minion->repair->worker->register;
592         do {
593           for my $id (keys %jobs) {
594             delete $jobs{$id} if $jobs{$id}->is_finished;
595           }
596           if (keys %jobs >= 4) { sleep 5 }
597           else {
598             my $job = $worker->dequeue(5);
599             $jobs{$job->id} = $job->start if $job;
600           }
601         } while keys %jobs;
602         $worker->unregister;
603

REFERENCE

605       This is the class hierarchy of the Minion distribution.
606
607       · Minion
608
609       · Minion::Backend
610
611         · Minion::Backend::Pg
612
613       · Minion::Command::minion
614
615       · Minion::Command::minion::job
616
617       · Minion::Command::minion::worker
618
619       · Minion::Job
620
621       · Minion::Worker
622
623       · Mojolicious::Plugin::Minion
624
625       · Mojolicious::Plugin::Minion::Admin
626

BUNDLED FILES

628       The Minion distribution includes a few files with different licenses
629       that have been bundled for internal use.
630
631   Minion Artwork
632         Copyright (C) 2017, Sebastian Riedel.
633
634       Licensed under the CC-SA License, Version 4.0
635       <http://creativecommons.org/licenses/by-sa/4.0>.
636
637   Bootstrap
638         Copyright (C) 2011-2018 The Bootstrap Authors.
639
640       Licensed under the MIT License,
641       <http://creativecommons.org/licenses/MIT>.
642
643   D3.js
644         Copyright (C) 2010-2016, Michael Bostock.
645
646       Licensed under the 3-Clause BSD License,
647       <https://opensource.org/licenses/BSD-3-Clause>.
648
649   epoch.js
650         Copyright (C) 2014 Fastly, Inc.
651
652       Licensed under the MIT License,
653       <http://creativecommons.org/licenses/MIT>.
654
655   Font Awesome
656         Copyright (C) Dave Gandy.
657
658       Licensed under the MIT License,
659       <http://creativecommons.org/licenses/MIT>, and the SIL OFL 1.1,
660       <http://scripts.sil.org/OFL>.
661
662   moment.js
663         Copyright (C) JS Foundation and other contributors.
664
665       Licensed under the MIT License,
666       <http://creativecommons.org/licenses/MIT>.
667
668   popper.js
669         Copyright (C) Federico Zivolo 2017.
670
671       Licensed under the MIT License,
672       <http://creativecommons.org/licenses/MIT>.
673

AUTHOR

675       Sebastian Riedel, "sri@cpan.org".
676

CREDITS

678       In alphabetical order:
679
680         Andrey Khozov
681
682         Brian Medley
683
684         Franz Skale
685
686         Hubert "depesz" Lubaczewski
687
688         Joel Berger
689
690         Paul Williams
691
692         Stefan Adams
693
695       Copyright (C) 2014-2019, Sebastian Riedel and others.
696
697       This program is free software, you can redistribute it and/or modify it
698       under the terms of the Artistic License version 2.0.
699

SEE ALSO

701       <https://github.com/mojolicious/minion>, Mojolicious::Guides,
702       <https://mojolicious.org>.
703
704
705
706perl v5.30.0                      2019-09-01                         Minion(3)
Impressum