1Minion(3) User Contributed Perl Documentation Minion(3)
2
3
4
6 Minion - Job queue
7
9 use Minion;
10
11 # Connect to backend
12 my $minion = Minion->new(Pg => 'postgresql://postgres@/test');
13
14 # Add tasks
15 $minion->add_task(something_slow => sub {
16 my ($job, @args) = @_;
17 sleep 5;
18 say 'This is a background worker process.';
19 });
20
21 # Enqueue jobs
22 $minion->enqueue(something_slow => ['foo', 'bar']);
23 $minion->enqueue(something_slow => [1, 2, 3] => {priority => 5});
24
25 # Perform jobs for testing
26 $minion->enqueue(something_slow => ['foo', 'bar']);
27 $minion->perform_jobs;
28
29 # Start a worker to perform up to 12 jobs concurrently
30 my $worker = $minion->worker;
31 $worker->status->{jobs} = 12;
32 $worker->run;
33
35 Minion is a high performance job queue for the Perl programming
36 language, with support for multiple named queues, priorities, delayed
37 jobs, job dependencies, job progress, job results, retries with
38 backoff, rate limiting, unique jobs, statistics, distributed workers,
39 parallel processing, autoscaling, remote control, Mojolicious
40 <https://mojolicious.org> admin ui, resource leak protection and
41 multiple backends (such as PostgreSQL <https://www.postgresql.org>).
42
43 Job queues allow you to process time and/or computationally intensive
44 tasks in background processes, outside of the request/response
45 lifecycle of web applications. Among those tasks you'll commonly find
46 image resizing, spam filtering, HTTP downloads, building tarballs,
47 warming caches and basically everything else you can imagine that's not
48 super fast.
49
51 You can use Minion as a standalone job queue or integrate it into
52 Mojolicious applications with the plugin Mojolicious::Plugin::Minion.
53
54 use Mojolicious::Lite;
55
56 plugin Minion => {Pg => 'postgresql://sri:s3cret@localhost/test'};
57
58 # Slow task
59 app->minion->add_task(poke_mojo => sub {
60 my $job = shift;
61 $job->app->ua->get('mojolicious.org');
62 $job->app->log->debug('We have poked mojolicious.org for a visitor');
63 });
64
65 # Perform job in a background worker process
66 get '/' => sub {
67 my $c = shift;
68 $c->minion->enqueue('poke_mojo');
69 $c->render(text => 'We will poke mojolicious.org for you soon.');
70 };
71
72 app->start;
73
74 Background worker processes are usually started with the command
75 Minion::Command::minion::worker, which becomes automatically available
76 when an application loads Mojolicious::Plugin::Minion.
77
78 $ ./myapp.pl minion worker
79
80 The worker process will fork a new process for every job that is being
81 processed. This allows for resources such as memory to be returned to
82 the operating system once a job is finished. Perl fork is very fast, so
83 don't worry about the overhead.
84
85 Minion::Worker
86 |- Minion::Job [1]
87 |- Minion::Job [2]
88 +- ...
89
90 By default up to four jobs will be processed in parallel, but that can
91 be changed with configuration options or on demand with signals.
92
93 $ ./myapp.pl minion worker -j 12
94
95 Jobs can be managed right from the command line with
96 Minion::Command::minion::job.
97
98 $ ./myapp.pl minion job
99
100 You can also add an admin ui to your application by loading the plugin
101 Mojolicious::Plugin::Minion::Admin. Just make sure to secure access
102 before making your application publically accessible.
103
104 # Make admin ui available under "/minion"
105 plugin 'Minion::Admin';
106
107 To manage background worker processes with systemd, you can use a unit
108 configuration file like this.
109
110 [Unit]
111 Description=My Mojolicious application workers
112 After=postgresql.service
113
114 [Service]
115 Type=simple
116 ExecStart=/home/sri/myapp/myapp.pl minion worker -m production
117 KillMode=process
118
119 [Install]
120 WantedBy=multi-user.target
121
122 Every job can fail or succeed, but not get lost, the system is
123 eventually consistent and will preserve job results for as long as you
124 like, depending on "remove_after". While individual workers can fail in
125 the middle of processing a job, the system will detect this and ensure
126 that no job is left in an uncertain state, depending on
127 "missing_after".
128
130 And as your application grows, you can move tasks into application
131 specific plugins.
132
133 package MyApp::Task::PokeMojo;
134 use Mojo::Base 'Mojolicious::Plugin';
135
136 sub register {
137 my ($self, $app) = @_;
138 $app->minion->add_task(poke_mojo => sub {
139 my $job = shift;
140 $job->app->ua->get('mojolicious.org');
141 $job->app->log->debug('We have poked mojolicious.org for a visitor');
142 });
143 }
144
145 1;
146
147 Which are loaded like any other plugin from your application.
148
149 # Mojolicious
150 $app->plugin('MyApp::Task::PokeMojo');
151
152 # Mojolicious::Lite
153 plugin 'MyApp::Task::PokeMojo';
154
156 This distribution also contains a great example application you can use
157 for inspiration. The link checker
158 <https://github.com/mojolicious/minion/tree/master/examples/linkcheck>
159 will show you how to integrate background jobs into well-structured
160 Mojolicious applications.
161
163 Minion inherits all events from Mojo::EventEmitter and can emit the
164 following new ones.
165
166 enqueue
167 $minion->on(enqueue => sub {
168 my ($minion, $id) = @_;
169 ...
170 });
171
172 Emitted after a job has been enqueued, in the process that enqueued it.
173
174 $minion->on(enqueue => sub {
175 my ($minion, $id) = @_;
176 say "Job $id has been enqueued.";
177 });
178
179 worker
180 $minion->on(worker => sub {
181 my ($minion, $worker) = @_;
182 ...
183 });
184
185 Emitted in the worker process after it has been created.
186
187 $minion->on(worker => sub {
188 my ($minion, $worker) = @_;
189 my $id = $worker->id;
190 say "Worker $$:$id started.";
191 });
192
194 Minion implements the following attributes.
195
196 app
197 my $app = $minion->app;
198 $minion = $minion->app(MyApp->new);
199
200 Application for job queue, defaults to a Mojo::HelloWorld object. Note
201 that this attribute is weakened.
202
203 backend
204 my $backend = $minion->backend;
205 $minion = $minion->backend(Minion::Backend::Pg->new);
206
207 Backend, usually a Minion::Backend::Pg object.
208
209 backoff
210 my $cb = $minion->backoff;
211 $minion = $minion->backoff(sub {...});
212
213 A callback used to calculate the delay for automatically retried jobs,
214 defaults to "(retries ** 4) + 15" (15, 16, 31, 96, 271, 640...), which
215 means that roughly 25 attempts can be made in 21 days.
216
217 $minion->backoff(sub {
218 my $retries = shift;
219 return ($retries ** 4) + 15 + int(rand 30);
220 });
221
222 missing_after
223 my $after = $minion->missing_after;
224 $minion = $minion->missing_after(172800);
225
226 Amount of time in seconds after which workers without a heartbeat will
227 be considered missing and removed from the registry by "repair",
228 defaults to 1800 (30 minutes).
229
230 remove_after
231 my $after = $minion->remove_after;
232 $minion = $minion->remove_after(86400);
233
234 Amount of time in seconds after which jobs that have reached the state
235 "finished" and have no unresolved dependencies will be removed
236 automatically by "repair", defaults to 172800 (2 days). It is not
237 recommended to set this value below 2 days.
238
239 tasks
240 my $tasks = $minion->tasks;
241 $minion = $minion->tasks({foo => sub {...}});
242
243 Registered tasks.
244
246 Minion inherits all methods from Mojo::EventEmitter and implements the
247 following new ones.
248
249 add_task
250 $minion = $minion->add_task(foo => sub {...});
251
252 Register a task.
253
254 # Job with result
255 $minion->add_task(add => sub {
256 my ($job, $first, $second) = @_;
257 $job->finish($first + $second);
258 });
259 my $id = $minion->enqueue(add => [1, 1]);
260 my $result = $minion->job($id)->info->{result};
261
262 broadcast
263 my $bool = $minion->broadcast('some_command');
264 my $bool = $minion->broadcast('some_command', [@args]);
265 my $bool = $minion->broadcast('some_command', [@args], [$id1, $id2, $id3]);
266
267 Broadcast remote control command to one or more workers.
268
269 # Broadcast "stop" command to all workers to kill job 10025
270 $minion->broadcast('stop', [10025]);
271
272 # Broadcast "kill" command to all workers to interrupt job 10026
273 $minion->broadcast('kill', ['INT', 10026]);
274
275 # Broadcast "jobs" command to pause worker 23
276 $minion->broadcast('jobs', [0], [23]);
277
278 enqueue
279 my $id = $minion->enqueue('foo');
280 my $id = $minion->enqueue(foo => [@args]);
281 my $id = $minion->enqueue(foo => [@args] => {priority => 1});
282
283 Enqueue a new job with "inactive" state. Arguments get serialized by
284 the "backend" (often with Mojo::JSON), so you shouldn't send objects
285 and be careful with binary data, nested data structures with hash and
286 array references are fine though.
287
288 These options are currently available:
289
290 attempts
291 attempts => 25
292
293 Number of times performing this job will be attempted, with a delay
294 based on "backoff" after the first attempt, defaults to 1.
295
296 delay
297 delay => 10
298
299 Delay job for this many seconds (from now), defaults to 0.
300
301 notes
302 notes => {foo => 'bar', baz => [1, 2, 3]}
303
304 Hash reference with arbitrary metadata for this job that gets
305 serialized by the "backend" (often with Mojo::JSON), so you shouldn't
306 send objects and be careful with binary data, nested data structures
307 with hash and array references are fine though.
308
309 parents
310 parents => [$id1, $id2, $id3]
311
312 One or more existing jobs this job depends on, and that need to have
313 transitioned to the state "finished" before it can be processed.
314
315 priority
316 priority => 5
317
318 Job priority, defaults to 0. Jobs with a higher priority get
319 performed first.
320
321 queue
322 queue => 'important'
323
324 Queue to put job in, defaults to "default".
325
326 foreground
327 my $bool = $minion->foreground($id);
328
329 Retry job in "minion_foreground" queue, then perform it right away with
330 a temporary worker in this process, very useful for debugging.
331
332 guard
333 my $guard = $minion->guard('foo', 3600);
334 my $guard = $minion->guard('foo', 3600, {limit => 20});
335
336 Same as "lock", but returns a scope guard object that automatically
337 releases the lock as soon as the object is destroyed, or "undef" if
338 aquiring the lock failed.
339
340 # Only one job should run at a time (unique job)
341 $minion->add_task(do_unique_stuff => sub {
342 my ($job, @args) = @_;
343 return $job->finish('Previous job is still active')
344 unless my $guard = $minion->guard('fragile_backend_service', 7200);
345 ...
346 });
347
348 # Only five jobs should run at a time and we try again later if necessary
349 $minion->add_task(do_concurrent_stuff => sub {
350 my ($job, @args) = @_;
351 return $job->retry({delay => 30})
352 unless my $guard = $minion->guard('some_web_service', 60, {limit => 5});
353 ...
354 });
355
356 history
357 my $history = $minion->history;
358
359 Get history information for job queue. Note that this method is
360 EXPERIMENTAL and might change without warning!
361
362 These fields are currently available:
363
364 daily
365 daily => [{epoch => 12345, finished_jobs => 95, failed_jobs => 2}, ...]
366
367 Hourly counts for processed jobs from the past day.
368
369 job
370 my $job = $minion->job($id);
371
372 Get Minion::Job object without making any changes to the actual job or
373 return "undef" if job does not exist.
374
375 # Check job state
376 my $state = $minion->job($id)->info->{state};
377
378 # Get job metadata
379 my $progress = $minion->$job($id)->info->{notes}{progress};
380
381 # Get job result
382 my $result = $minion->job($id)->info->{result};
383
384 lock
385 my $bool = $minion->lock('foo', 3600);
386 my $bool = $minion->lock('foo', 3600, {limit => 20});
387
388 Try to acquire a named lock that will expire automatically after the
389 given amount of time in seconds. You can release the lock manually with
390 "unlock" to limit concurrency, or let it expire for rate limiting. For
391 convenience you can also use "guard" to release the lock automatically,
392 even if the job failed.
393
394 # Only one job should run at a time (unique job)
395 $minion->add_task(do_unique_stuff => sub {
396 my ($job, @args) = @_;
397 return $job->finish('Previous job is still active')
398 unless $minion->lock('fragile_backend_service', 7200);
399 ...
400 $minion->unlock('fragile_backend_service');
401 });
402
403 # Only five jobs should run at a time and we wait for our turn
404 $minion->add_task(do_concurrent_stuff => sub {
405 my ($job, @args) = @_;
406 sleep 1 until $minion->lock('some_web_service', 60, {limit => 5});
407 ...
408 $minion->unlock('some_web_service');
409 });
410
411 # Only a hundred jobs should run per hour and we try again later if necessary
412 $minion->add_task(do_rate_limited_stuff => sub {
413 my ($job, @args) = @_;
414 return $job->retry({delay => 3600})
415 unless $minion->lock('another_web_service', 3600, {limit => 100});
416 ...
417 });
418
419 An expiration time of 0 can be used to check if a named lock already
420 exists without creating one.
421
422 # Check if the lock "foo" already exists
423 say 'Lock exists' unless $minion->lock('foo', 0);
424
425 These options are currently available:
426
427 limit
428 limit => 20
429
430 Number of shared locks with the same name that can be active at the
431 same time, defaults to 1.
432
433 new
434 my $minion = Minion->new(Pg => 'postgresql://postgres@/test');
435 my $minion = Minion->new(Pg => Mojo::Pg->new);
436
437 Construct a new Minion object.
438
439 perform_jobs
440 $minion->perform_jobs;
441 $minion->perform_jobs({queues => ['important']});
442
443 Perform all jobs with a temporary worker, very useful for testing.
444
445 # Longer version
446 my $worker = $minion->worker;
447 while (my $job = $worker->register->dequeue(0)) { $job->perform }
448 $worker->unregister;
449
450 These options are currently available:
451
452 queues
453 queues => ['important']
454
455 One or more queues to dequeue jobs from, defaults to "default".
456
457 repair
458 $minion = $minion->repair;
459
460 Repair worker registry and job queue if necessary.
461
462 reset
463 $minion = $minion->reset;
464
465 Reset job queue.
466
467 result_p
468 my $promise = $minion->result_p($id);
469 my $promise = $minion->result_p($id, {interval => 5});
470
471 Return a Mojo::Promise object for the result of a job. The state
472 "finished" will result in the promise being "fullfilled", and the state
473 "failed" in the promise being "rejected". This operation can be
474 cancelled by resolving the promise manually at any time. Note that this
475 method is EXPERIMENTAL and might change without warning!
476
477 # Enqueue job and receive the result at some point in the future
478 my $id = $minion->enqueue('foo');
479 $minion->result_p($id)->then(sub {
480 my $info = shift;
481 my $result = ref $info ? $info->{result} : 'Job already removed';
482 say "Finished: $result";
483 })->catch(sub {
484 my $info = shift;
485 say "Failed: $info->{result}";
486 })->wait;
487
488 These options are currently available:
489
490 interval
491 interval => 5
492
493 Polling interval in seconds for checking if the state of the job has
494 changed, defaults to 3.
495
496 stats
497 my $stats = $minion->stats;
498
499 Get statistics for the job queue.
500
501 # Check idle workers
502 my $idle = $minion->stats->{inactive_workers};
503
504 These fields are currently available:
505
506 active_jobs
507 active_jobs => 100
508
509 Number of jobs in "active" state.
510
511 active_locks
512 active_locks => 100
513
514 Number of active named locks.
515
516 active_workers
517 active_workers => 100
518
519 Number of workers that are currently processing a job.
520
521 delayed_jobs
522 delayed_jobs => 100
523
524 Number of jobs in "inactive" state that are scheduled to run at
525 specific time in the future or have unresolved dependencies. Note
526 that this field is EXPERIMENTAL and might change without warning!
527
528 enqueued_jobs
529 enqueued_jobs => 100000
530
531 Rough estimate of how many jobs have ever been enqueued. Note that
532 this field is EXPERIMENTAL and might change without warning!
533
534 failed_jobs
535 failed_jobs => 100
536
537 Number of jobs in "failed" state.
538
539 finished_jobs
540 finished_jobs => 100
541
542 Number of jobs in "finished" state.
543
544 inactive_jobs
545 inactive_jobs => 100
546
547 Number of jobs in "inactive" state.
548
549 inactive_workers
550 inactive_workers => 100
551
552 Number of workers that are currently not processing a job.
553
554 uptime
555 uptime => 1000
556
557 Uptime in seconds.
558
559 unlock
560 my $bool = $minion->unlock('foo');
561
562 Release a named lock that has been previously acquired with "lock".
563
564 worker
565 my $worker = $minion->worker;
566
567 Build Minion::Worker object. Note that this method should only be used
568 to implement custom workers.
569
570 # Use the standard worker with all its features
571 my $worker = $minion->worker;
572 $worker->status->{jobs} = 12;
573 $worker->status->{queues} = ['important'];
574 $worker->run;
575
576 # Perform one job manually in a separate process
577 my $worker = $minion->repair->worker->register;
578 my $job = $worker->dequeue(5);
579 $job->perform;
580 $worker->unregister;
581
582 # Perform one job manually in this process
583 my $worker = $minion->repair->worker->register;
584 my $job = $worker->dequeue(5);
585 if (my $err = $job->execute) { $job->fail($err) }
586 else { $job->finish }
587 $worker->unregister;
588
589 # Build a custom worker performing multiple jobs at the same time
590 my %jobs;
591 my $worker = $minion->repair->worker->register;
592 do {
593 for my $id (keys %jobs) {
594 delete $jobs{$id} if $jobs{$id}->is_finished;
595 }
596 if (keys %jobs >= 4) { sleep 5 }
597 else {
598 my $job = $worker->dequeue(5);
599 $jobs{$job->id} = $job->start if $job;
600 }
601 } while keys %jobs;
602 $worker->unregister;
603
605 This is the class hierarchy of the Minion distribution.
606
607 · Minion
608
609 · Minion::Backend
610
611 · Minion::Backend::Pg
612
613 · Minion::Command::minion
614
615 · Minion::Command::minion::job
616
617 · Minion::Command::minion::worker
618
619 · Minion::Job
620
621 · Minion::Worker
622
623 · Mojolicious::Plugin::Minion
624
625 · Mojolicious::Plugin::Minion::Admin
626
628 The Minion distribution includes a few files with different licenses
629 that have been bundled for internal use.
630
631 Minion Artwork
632 Copyright (C) 2017, Sebastian Riedel.
633
634 Licensed under the CC-SA License, Version 4.0
635 <http://creativecommons.org/licenses/by-sa/4.0>.
636
637 Bootstrap
638 Copyright (C) 2011-2018 The Bootstrap Authors.
639
640 Licensed under the MIT License,
641 <http://creativecommons.org/licenses/MIT>.
642
643 D3.js
644 Copyright (C) 2010-2016, Michael Bostock.
645
646 Licensed under the 3-Clause BSD License,
647 <https://opensource.org/licenses/BSD-3-Clause>.
648
649 epoch.js
650 Copyright (C) 2014 Fastly, Inc.
651
652 Licensed under the MIT License,
653 <http://creativecommons.org/licenses/MIT>.
654
655 Font Awesome
656 Copyright (C) Dave Gandy.
657
658 Licensed under the MIT License,
659 <http://creativecommons.org/licenses/MIT>, and the SIL OFL 1.1,
660 <http://scripts.sil.org/OFL>.
661
662 moment.js
663 Copyright (C) JS Foundation and other contributors.
664
665 Licensed under the MIT License,
666 <http://creativecommons.org/licenses/MIT>.
667
668 popper.js
669 Copyright (C) Federico Zivolo 2017.
670
671 Licensed under the MIT License,
672 <http://creativecommons.org/licenses/MIT>.
673
675 Sebastian Riedel, "sri@cpan.org".
676
678 In alphabetical order:
679
680 Andrey Khozov
681
682 Brian Medley
683
684 Franz Skale
685
686 Hubert "depesz" Lubaczewski
687
688 Joel Berger
689
690 Paul Williams
691
692 Stefan Adams
693
695 Copyright (C) 2014-2019, Sebastian Riedel and others.
696
697 This program is free software, you can redistribute it and/or modify it
698 under the terms of the Artistic License version 2.0.
699
701 <https://github.com/mojolicious/minion>, Mojolicious::Guides,
702 <https://mojolicious.org>.
703
704
705
706perl v5.30.0 2019-09-01 Minion(3)