Minion(3pm)

1Minion(3)             User Contributed Perl Documentation            Minion(3)
2
3
4

NAME

6       Minion - Job queue
7

SYNOPSIS

9         use Minion;
10
11         # Connect to backend
12         my $minion = Minion->new(Pg => 'postgresql://postgres@/test');
13
14         # Add tasks
15         $minion->add_task(something_slow => sub {
16           my ($job, @args) = @_;
17           sleep 5;
18           say 'This is a background worker process.';
19         });
20
21         # Enqueue jobs
22         $minion->enqueue(something_slow => ['foo', 'bar']);
23         $minion->enqueue(something_slow => [1, 2, 3] => {priority => 5});
24
25         # Perform jobs for testing
26         $minion->enqueue(something_slow => ['foo', 'bar']);
27         $minion->perform_jobs;
28
29         # Start a worker to perform up to 12 jobs concurrently
30         my $worker = $minion->worker;
31         $worker->status->{jobs} = 12;
32         $worker->run;
33

DESCRIPTION

35       Minion is a high performance job queue for the Perl programming
36       language, with support for multiple named queues, priorities, delayed
37       jobs, job dependencies, job progress, job results, retries with
38       backoff, rate limiting, unique jobs, statistics, distributed workers,
39       parallel processing, autoscaling, remote control, Mojolicious
40       <http://mojolicious.org> admin ui, resource leak protection and
41       multiple backends (such as PostgreSQL <http://www.postgresql.org>).
42
43       Job queues allow you to process time and/or computationally intensive
44       tasks in background processes, outside of the request/response
45       lifecycle of web applications. Among those tasks you'll commonly find
46       image resizing, spam filtering, HTTP downloads, building tarballs,
47       warming caches and basically everything else you can imagine that's not
48       super fast.
49

BASICS

51       You can use Minion as a standalone job queue or integrate it into
52       Mojolicious applications with the plugin Mojolicious::Plugin::Minion.
53
54         use Mojolicious::Lite;
55
56         plugin Minion => {Pg => 'postgresql://sri:s3cret@localhost/test'};
57
58         # Slow task
59         app->minion->add_task(poke_mojo => sub {
60           my $job = shift;
61           $job->app->ua->get('mojolicious.org');
62           $job->app->log->debug('We have poked mojolicious.org for a visitor');
63         });
64
65         # Perform job in a background worker process
66         get '/' => sub {
67           my $c = shift;
68           $c->minion->enqueue('poke_mojo');
69           $c->render(text => 'We will poke mojolicious.org for you soon.');
70         };
71
72         app->start;
73
74       Background worker processes are usually started with the command
75       Minion::Command::minion::worker, which becomes automatically available
76       when an application loads Mojolicious::Plugin::Minion.
77
78         $ ./myapp.pl minion worker
79
80       Jobs can be managed right from the command line with
81       Minion::Command::minion::job.
82
83         $ ./myapp.pl minion job
84
85       You can also add an admin ui to your application by loading the plugin
86       Mojolicious::Plugin::Minion::Admin. Just make sure to secure access
87       before making your application publically accessible.
88
89         # Make admin ui available under "/minion"
90         plugin 'Minion::Admin';
91
92       To manage background worker processes with systemd, you can use a unit
93       configuration file like this.
94
95         [Unit]
96         Description=My Mojolicious application workers
97         After=postgresql.service
98
99         [Service]
100         Type=simple
101         ExecStart=/home/sri/myapp/myapp.pl minion worker -m production
102         KillMode=process
103
104         [Install]
105         WantedBy=multi-user.target
106
107       Every job can fail or succeed, but not get lost, the system is
108       eventually consistent and will preserve job results for as long as you
109       like, depending on "remove_after". While individual workers can fail in
110       the middle of processing a job, the system will detect this and ensure
111       that no job is left in an uncertain state, depending on
112       "missing_after".
113

GROWING

115       And as your application grows, you can move tasks into application
116       specific plugins.
117
118         package MyApp::Task::PokeMojo;
119         use Mojo::Base 'Mojolicious::Plugin';
120
121         sub register {
122           my ($self, $app) = @_;
123           $app->minion->add_task(poke_mojo => sub {
124             my $job = shift;
125             $job->app->ua->get('mojolicious.org');
126             $job->app->log->debug('We have poked mojolicious.org for a visitor');
127           });
128         }
129
130         1;
131
132       Which are loaded like any other plugin from your application.
133
134         # Mojolicious
135         $app->plugin('MyApp::Task::PokeMojo');
136
137         # Mojolicious::Lite
138         plugin 'MyApp::Task::PokeMojo';
139

EXAMPLES

141       This distribution also contains a great example application you can use
142       for inspiration. The link checker
143       <https://github.com/kraih/minion/tree/master/examples/linkcheck> will
144       show you how to integrate background jobs into well-structured
145       Mojolicious applications.
146

EVENTS

148       Minion inherits all events from Mojo::EventEmitter and can emit the
149       following new ones.
150
151   enqueue
152         $minion->on(enqueue => sub {
153           my ($minion, $id) = @_;
154           ...
155         });
156
157       Emitted after a job has been enqueued, in the process that enqueued it.
158
159         $minion->on(enqueue => sub {
160           my ($minion, $id) = @_;
161           say "Job $id has been enqueued.";
162         });
163
164   worker
165         $minion->on(worker => sub {
166           my ($minion, $worker) = @_;
167           ...
168         });
169
170       Emitted in the worker process after it has been created.
171
172         $minion->on(worker => sub {
173           my ($minion, $worker) = @_;
174           my $id = $worker->id;
175           say "Worker $$:$id started.";
176         });
177

ATTRIBUTES

179       Minion implements the following attributes.
180
181   app
182         my $app = $minion->app;
183         $minion = $minion->app(MyApp->new);
184
185       Application for job queue, defaults to a Mojo::HelloWorld object.
186
187   backend
188         my $backend = $minion->backend;
189         $minion     = $minion->backend(Minion::Backend::Pg->new);
190
191       Backend, usually a Minion::Backend::Pg object.
192
193   backoff
194         my $cb  = $minion->backoff;
195         $minion = $minion->backoff(sub {...});
196
197       A callback used to calculate the delay for automatically retried jobs,
198       defaults to "(retries ** 4) + 15" (15, 16, 31, 96, 271, 640...), which
199       means that roughly 25 attempts can be made in 21 days.
200
201         $minion->backoff(sub {
202           my $retries = shift;
203           return ($retries ** 4) + 15 + int(rand 30);
204         });
205
206   missing_after
207         my $after = $minion->missing_after;
208         $minion   = $minion->missing_after(172800);
209
210       Amount of time in seconds after which workers without a heartbeat will
211       be considered missing and removed from the registry by "repair",
212       defaults to 1800 (30 minutes).
213
214   remove_after
215         my $after = $minion->remove_after;
216         $minion   = $minion->remove_after(86400);
217
218       Amount of time in seconds after which jobs that have reached the state
219       "finished" and have no unresolved dependencies will be removed
220       automatically by "repair", defaults to 172800 (2 days). It is not
221       recommended to set this value below 2 days.
222
223   tasks
224         my $tasks = $minion->tasks;
225         $minion   = $minion->tasks({foo => sub {...}});
226
227       Registered tasks.
228

METHODS

230       Minion inherits all methods from Mojo::EventEmitter and implements the
231       following new ones.
232
233   add_task
234         $minion = $minion->add_task(foo => sub {...});
235
236       Register a task.
237
238         # Job with result
239         $minion->add_task(add => sub {
240           my ($job, $first, $second) = @_;
241           $job->finish($first + $second);
242         });
243         my $id = $minion->enqueue(add => [1, 1]);
244         my $result = $minion->job($id)->info->{result};
245
246   broadcast
247         my $bool = $minion->broadcast('some_command');
248         my $bool = $minion->broadcast('some_command', [@args]);
249         my $bool = $minion->broadcast('some_command', [@args], [$id1, $id2, $id3]);
250
251       Broadcast remote control command to one or more workers.
252
253         # Broadcast "stop" command to all workers to kill job 10025
254         $minion->broadcast('stop', [10025]);
255
256         # Broadcast "jobs" command to pause worker 23
257         $minion->broadcast('jobs', [0], [23]);
258
259   enqueue
260         my $id = $minion->enqueue('foo');
261         my $id = $minion->enqueue(foo => [@args]);
262         my $id = $minion->enqueue(foo => [@args] => {priority => 1});
263
264       Enqueue a new job with "inactive" state. Arguments get serialized by
265       the "backend" (often with Mojo::JSON), so you shouldn't send objects
266       and be careful with binary data, nested data structures with hash and
267       array references are fine though.
268
269       These options are currently available:
270
271       attempts
272           attempts => 25
273
274         Number of times performing this job will be attempted, with a delay
275         based on "backoff" after the first attempt, defaults to 1.
276
277       delay
278           delay => 10
279
280         Delay job for this many seconds (from now), defaults to 0.
281
282       notes
283           notes => {foo => 'bar', baz => [1, 2, 3]}
284
285         Hash reference with arbitrary metadata for this job that gets
286         serialized by the "backend" (often with Mojo::JSON), so you shouldn't
287         send objects and be careful with binary data, nested data structures
288         with hash and array references are fine though.
289
290       parents
291           parents => [$id1, $id2, $id3]
292
293         One or more existing jobs this job depends on, and that need to have
294         transitioned to the state "finished" before it can be processed.
295
296       priority
297           priority => 5
298
299         Job priority, defaults to 0. Jobs with a higher priority get
300         performed first.
301
302       queue
303           queue => 'important'
304
305         Queue to put job in, defaults to "default".
306
307   foreground
308         my $bool = $minion->foreground($id);
309
310       Retry job in "minion_foreground" queue, then perform it right away with
311       a temporary worker in this process, very useful for debugging.
312
313   guard
314         my $guard = $minion->guard('foo', 3600);
315         my $guard = $minion->guard('foo', 3600, {limit => 20});
316
317       Same as "lock", but returns a scope guard object that automatically
318       releases the lock as soon as the object is destroyed, or "undef" if
319       aquiring the lock failed.
320
321         # Only one job should run at a time (unique job)
322         $minion->add_task(do_unique_stuff => sub {
323           my ($job, @args) = @_;
324           return $job->finish('Previous job is still active')
325             unless my $guard = $minion->guard('fragile_backend_service', 7200);
326           ...
327         });
328
329         # Only five jobs should run at a time and we try again later if necessary
330         $minion->add_task(do_concurrent_stuff => sub {
331           my ($job, @args) = @_;
332           return $job->retry({delay => 30})
333             unless my $guard = $minion->guard('some_web_service', 60, {limit => 5});
334           ...
335         });
336
337   history
338         my $history = $minion->history;
339
340       Get history information for job queue. Note that this method is
341       EXPERIMENTAL and might change without warning!
342
343       These fields are currently available:
344
345       daily
346           daily => [{epoch => 12345, finished_jobs => 95, failed_jobs => 2}, ...]
347
348         Hourly counts for processed jobs from the past day.
349
350   job
351         my $job = $minion->job($id);
352
353       Get Minion::Job object without making any changes to the actual job or
354       return "undef" if job does not exist.
355
356         # Check job state
357         my $state = $minion->job($id)->info->{state};
358
359         # Get job metadata
360         my $progress = $minion->$job($id)->info->{notes}{progress};
361
362         # Get job result
363         my $result = $minion->job($id)->info->{result};
364
365   lock
366         my $bool = $minion->lock('foo', 3600);
367         my $bool = $minion->lock('foo', 3600, {limit => 20});
368
369       Try to acquire a named lock that will expire automatically after the
370       given amount of time in seconds. You can release the lock manually with
371       "unlock" to limit concurrency, or let it expire for rate limiting. For
372       convenience you can also use "guard" to release the lock automatically,
373       even if the job failed.
374
375         # Only one job should run at a time (unique job)
376         $minion->add_task(do_unique_stuff => sub {
377           my ($job, @args) = @_;
378           return $job->finish('Previous job is still active')
379             unless $minion->lock('fragile_backend_service', 7200);
380           ...
381           $minion->unlock('fragile_backend_service');
382         });
383
384         # Only five jobs should run at a time and we wait for our turn
385         $minion->add_task(do_concurrent_stuff => sub {
386           my ($job, @args) = @_;
387           sleep 1 until $minion->lock('some_web_service', 60, {limit => 5});
388           ...
389           $minion->unlock('some_web_service');
390         });
391
392         # Only a hundred jobs should run per hour and we try again later if necessary
393         $minion->add_task(do_rate_limited_stuff => sub {
394           my ($job, @args) = @_;
395           return $job->retry({delay => 3600})
396             unless $minion->lock('another_web_service', 3600, {limit => 100});
397           ...
398         });
399
400       An expiration time of 0 can be used to check if a named lock already
401       exists without creating one.
402
403         # Check if the lock "foo" already exists
404         say 'Lock exists' unless $minion->lock('foo', 0);
405
406       These options are currently available:
407
408       limit
409           limit => 20
410
411         Number of shared locks with the same name that can be active at the
412         same time, defaults to 1.
413
414   new
415         my $minion = Minion->new(Pg => 'postgresql://postgres@/test');
416         my $minion = Minion->new(Pg => Mojo::Pg->new);
417
418       Construct a new Minion object.
419
420   perform_jobs
421         $minion->perform_jobs;
422         $minion->perform_jobs({queues => ['important']});
423
424       Perform all jobs with a temporary worker, very useful for testing.
425
426         # Longer version
427         my $worker = $minion->worker;
428         while (my $job = $worker->register->dequeue(0)) { $job->perform }
429         $worker->unregister;
430
431       These options are currently available:
432
433       queues
434           queues => ['important']
435
436         One or more queues to dequeue jobs from, defaults to "default".
437
438   repair
439         $minion = $minion->repair;
440
441       Repair worker registry and job queue if necessary.
442
443   reset
444         $minion = $minion->reset;
445
446       Reset job queue.
447
448   stats
449         my $stats = $minion->stats;
450
451       Get statistics for the job queue.
452
453         # Check idle workers
454         my $idle = $minion->stats->{inactive_workers};
455
456       These fields are currently available:
457
458       active_jobs
459           active_jobs => 100
460
461         Number of jobs in "active" state.
462
463       active_locks
464           active_locks => 100
465
466         Number of active named locks.
467
468       active_workers
469           active_workers => 100
470
471         Number of workers that are currently processing a job.
472
473       delayed_jobs
474           delayed_jobs => 100
475
476         Number of jobs in "inactive" state that are scheduled to run at
477         specific time in the future or have unresolved dependencies. Note
478         that this field is EXPERIMENTAL and might change without warning!
479
480       enqueued_jobs
481           enqueued_jobs => 100000
482
483         Rough estimate of how many jobs have ever been enqueued. Note that
484         this field is EXPERIMENTAL and might change without warning!
485
486       failed_jobs
487           failed_jobs => 100
488
489         Number of jobs in "failed" state.
490
491       finished_jobs
492           finished_jobs => 100
493
494         Number of jobs in "finished" state.
495
496       inactive_jobs
497           inactive_jobs => 100
498
499         Number of jobs in "inactive" state.
500
501       inactive_workers
502           inactive_workers => 100
503
504         Number of workers that are currently not processing a job.
505
506       uptime
507           uptime => 1000
508
509         Uptime in seconds.
510
511   unlock
512         my $bool = $minion->unlock('foo');
513
514       Release a named lock that has been previously acquired with "lock".
515
516   worker
517         my $worker = $minion->worker;
518
519       Build Minion::Worker object.
520
521         # Use the standard worker with all its features
522         my $worker = $minion->worker;
523         $worker->status->{jobs} = 12;
524         $worker->status->{queues} = ['important'];
525         $worker->run;
526
527         # Perform one job manually in a separate process
528         my $worker = $minion->repair->worker->register;
529         my $job    = $worker->dequeue(5);
530         $job->perform;
531         $worker->unregister;
532
533         # Perform one job manually in this process
534         my $worker = $minion->repair->worker->register;
535         my $job    = $worker->dequeue(5);
536         if (my $err = $job->execute) { $job->fail($err) }
537         else                         { $job->finish }
538         $worker->unregister;
539
540         # Build a custom worker performing multiple jobs at the same time
541         my %jobs;
542         my $worker = $minion->repair->worker->register;
543         do {
544           for my $id (keys %jobs) {
545             delete $jobs{$id} if $jobs{$id}->is_finished;
546           }
547           if (keys %jobs >= 4) { sleep 5 }
548           else {
549             my $job = $worker->dequeue(5);
550             $jobs{$job->id} = $job->start if $job;
551           }
552         } while keys %jobs;
553         $worker->unregister;
554

REFERENCE

556       This is the class hierarchy of the Minion distribution.
557
558       · Minion
559
560       · Minion::Backend
561
562         · Minion::Backend::Pg
563
564       · Minion::Command::minion
565
566       · Minion::Command::minion::job
567
568       · Minion::Command::minion::worker
569
570       · Minion::Job
571
572       · Minion::Worker
573
574       · Mojolicious::Plugin::Minion
575
576       · Mojolicious::Plugin::Minion::Admin
577

BUNDLED FILES

579       The Minion distribution includes a few files with different licenses
580       that have been bundled for internal use.
581
582   Minion Artwork
583         Copyright (C) 2017, Sebastian Riedel.
584
585       Licensed under the CC-SA License, Version 4.0
586       <http://creativecommons.org/licenses/by-sa/4.0>.
587
588   Bootstrap
589         Copyright (C) 2011-2018 The Bootstrap Authors.
590
591       Licensed under the MIT License,
592       <http://creativecommons.org/licenses/MIT>.
593
594   D3.js
595         Copyright (C) 2010-2016, Michael Bostock.
596
597       Licensed under the 3-Clause BSD License,
598       <https://opensource.org/licenses/BSD-3-Clause>.
599
600   epoch.js
601         Copyright (C) 2014 Fastly, Inc.
602
603       Licensed under the MIT License,
604       <http://creativecommons.org/licenses/MIT>.
605
606   Font Awesome
607         Copyright (C) Dave Gandy.
608
609       Licensed under the MIT License,
610       <http://creativecommons.org/licenses/MIT>, and the SIL OFL 1.1,
611       <http://scripts.sil.org/OFL>.
612
613   moment.js
614         Copyright (C) JS Foundation and other contributors.
615
616       Licensed under the MIT License,
617       <http://creativecommons.org/licenses/MIT>.
618
619   popper.js
620         Copyright (C) Federico Zivolo 2017.
621
622       Licensed under the MIT License,
623       <http://creativecommons.org/licenses/MIT>.
624

AUTHOR

626       Sebastian Riedel, "sri@cpan.org".
627

CREDITS

629       In alphabetical order:
630
631         Andrey Khozov
632
633         Brian Medley
634
635         Hubert "depesz" Lubaczewski
636
637         Joel Berger
638
639         Paul Williams
640
641         Stefan Adams
642

COPYRIGHT AND LICENSE

644       Copyright (C) 2014-2018, Sebastian Riedel and others.
645
646       This program is free software, you can redistribute it and/or modify it
647       under the terms of the Artistic License version 2.0.
648