1Minion::Guide(3)      User Contributed Perl Documentation     Minion::Guide(3)
2
3
4

NAME

6       Minion::Guide - An introducion to Minion
7

OVERVIEW

9       This document contains an introduction to Minion and explains the most
10       important features it has to offer.
11

INTRODUCTION

13       Essentials every Minion developer should know.
14
15   Job queue
16       Job queues allow you to process time and/or computationally intensive
17       tasks in background processes, outside of the request/response
18       lifecycle of web applications. Among those tasks you'll commonly find
19       image resizing, spam filtering, HTTP downloads, building tarballs,
20       warming caches and basically everything else you can imagine that's not
21       super fast.
22
23         Mojo::Server::Prefork                              +--------------+                     Minion::Worker
24         |- Mojo::Server::Daemon [1]       enqueue job ->   |              |   -> dequeue job    |- Minion::Job [1]
25         |- Mojo::Server::Daemon [2]                        |  PostgreSQL  |                     |- Minion::Job [2]
26         |- Mojo::Server::Daemon [3]   retrieve result <-   |              |   <- store result   |- Minion::Job [3]
27         +- Mojo::Server::Daemon [4]                        +--------------+                     |- Minion::Job [4]
28                                                                                                 +- Minion::Job [5]
29
30       They are not to be confused with time based job schedulers, such as
31       cron or systemd timers. Both serve very different purposes, and cron
32       jobs are in fact commonly used to enqueue Minion jobs that need to
33       follow a schedule. For example to perform regular maintenance tasks.
34
35   Mojolicious
36       You can use Minion as a standalone job queue or integrate it into
37       Mojolicious applications with the plugin Mojolicious::Plugin::Minion.
38
39         use Mojolicious::Lite -signatures;
40
41         plugin Minion => {Pg => 'postgresql://sri:s3cret@localhost/test'};
42
43         # Slow task
44         app->minion->add_task(poke_mojo => sub ($job, @args) {
45           $job->app->ua->get('mojolicious.org');
46           $job->app->log->debug('We have poked mojolicious.org for a visitor');
47         });
48
49         # Perform job in a background worker process
50         get '/' => sub ($c) {
51           $c->minion->enqueue('poke_mojo');
52           $c->render(text => 'We will poke mojolicious.org for you soon.');
53         };
54
55         app->start;
56
57       Background worker processes are usually started with the command
58       Minion::Command::minion::worker, which becomes automatically available
59       when an application loads Mojolicious::Plugin::Minion.
60
61         $ ./myapp.pl minion worker
62
63       The worker process will fork a new process for every job that is being
64       processed. This allows for resources such as memory to be returned to
65       the operating system once a job is finished. Perl fork is very fast, so
66       don't worry about the overhead.
67
68         Minion::Worker
69         |- Minion::Job [1]
70         |- Minion::Job [2]
71         +- ...
72
73       By default up to four jobs will be processed in parallel, but that can
74       be changed with configuration options or on demand with signals.
75
76         $ ./myapp.pl minion worker -j 12
77
78       Jobs can be managed right from the command line with
79       Minion::Command::minion::job.
80
81         $ ./myapp.pl minion job
82
83       You can also add an admin ui to your application by loading the plugin
84       Mojolicious::Plugin::Minion::Admin. Just make sure to secure access
85       before making your application publically accessible.
86
87         # Make admin ui available under "/minion"
88         plugin 'Minion::Admin';
89
90   Deployment
91       To manage background worker processes with systemd, you can use a unit
92       configuration file like this.
93
94         [Unit]
95         Description=My Mojolicious application workers
96         After=postgresql.service
97
98         [Service]
99         Type=simple
100         ExecStart=/home/sri/myapp/myapp.pl minion worker -m production
101         KillMode=process
102
103         [Install]
104         WantedBy=multi-user.target
105
106   Consistency
107       Every new job starts out as "incative", then progresses to "active"
108       when it is dequeued by a worker, and finally ends up as "finished" or
109       "failed", depending on its result. Every "failed" job can then be
110       retried to progress back to the "inactive" state and start all over
111       again.
112
113                                                             +----------+
114                                                             |          |
115                                                    +----->  | finished |
116          +----------+            +--------+        |        |          |
117          |          |            |        |        |        +----------+
118          | inactive |  ------->  | active |  ------+
119          |          |            |        |        |        +----------+
120          +----------+            +--------+        |        |          |
121                                                    +----->  |  failed  |  -----+
122               ^                                             |          |       |
123               |                                             +----------+       |
124               |                                                                |
125               +----------------------------------------------------------------+
126
127       The system is eventually consistent and will preserve job results for
128       as long as you like, depending on "remove_after" in Minion. But be
129       aware that "failed" results are presrved indefinitely, and need to be
130       manually removed by an administrator if they are out of automatic
131       retries.
132
133       While individual workers can fail in the middle of processing a job,
134       the system will detect this and ensure that no job is left in an
135       uncertain state, depending on "missing_after" in Minion. Jobs that do
136       not get processed after a certain amount of time, depending on
137       "stuck_after" in Minion, will be considered stuck and fail
138       automatically. So an admin can take a look and resolve the issue.
139

FEATURES

141       Minion has many great features. This section is still very incomplete,
142       but will be expanded over time.
143
144   Priorities
145       Every job enqueued with "enqueue" in Minion has a priority. Jobs with a
146       higher priority get performed first, the default priority is 0.
147       Priorities can be positive or negative, but should be in the range
148       between 100 and "-100".
149
150         # Default priority
151         $minion->enqueue('check_links', ['https://mojolicious.org']);
152
153         # High priority
154         $minion->enqueue('check_links', ['https://mojolicious.org'], {priority => 30});
155
156         # Low priority
157         $minion->enqueue('check_links', ['https://mojolicious.org'], {priority => -30});
158
159       You can use "retry" in Minion::Job to raise or lower the priority of a
160       job.
161
162         $job->retry({priority => 50});
163
164   Job results
165       The result of a job has two parts. First there is its state, which can
166       be "finished" for a successfully processed job, and "failed" for the
167       opposite. And second there's a "result" data structure, that may be
168       "undef", a scalar, a hash reference, or an array reference. You can
169       check both at any time in the life cycle of a job with "job" in Minion,
170       all you need is the job id.
171
172         # Check job state
173         my $state = $minion->job($job_id)->info->{state};
174
175         # Get job result
176         my $result = $minion->job($job_id)->info->{result};
177
178       While the "state" will be assigned automatically by Minion, the
179       "result" for "finished" jobs is usually assigned manually with "finish"
180       in Minion::Job.
181
182         $minion->add_task(job_with_result => sub ($job) {
183           sleep 5;
184           $job->finish({message => 'This job should have taken about 5 seconds'});
185         });
186
187       For jobs that "failed" due to an exception, that exception will be
188       assigned as "result".
189
190         $minion->add_task(job_that_fails => sub ($job) {
191           sleep 5;
192           die 'This job should always fail after 5 seconds';
193         });
194
195       But jobs can also fail manually with "fail" in Minion::Job.
196
197         $minion->add_task(job_that_fails_with_result => sub ($job) {
198           sleep 5;
199           $job->fail({errors => ['This job should fail after 5 seconds']});
200         });
201
202       Retrieving job results is of course completely optional, and it is very
203       common to have jobs where the result is unimportant.
204
205   Named queues
206       Each job can be enqueued with "enqueue" in Minion into arbitrarily
207       named queues, independent of all their other properties. This is
208       commonly used to have separate classes of workers, for example to
209       ensure that free customers of your web service do not negatively affect
210       your service level agreements with paying customers. The default named
211       queue is "default", but aside from that it has no special properties.
212
213         # Use "default" queue
214         $minion->enqueue('check_links', ['https://mojolicious.org']);
215
216         # Use custom "important" queue
217         $minion->enqueue('check_links', ['https://mojolicious.org'], {queue => 'important'});
218
219       For every named queue you can start as many workers as you like with
220       the command Minion::Command::minion::worker. And each worker can
221       process jobs from multiple named queues. So your workers can have
222       overlapping responsibilities.
223
224         $ ./myapp.pl minion worker -q default -q important
225
226       There is one special named queue called "minion_foreground" that you
227       should avoid using directly. It is reserved for debugging jobs with
228       "foreground" in Minion.
229
230   Job progress
231       Progress information and other job metadata can be stored in notes at
232       any time during the life cycle of a job with "note" in Minion::Job. The
233       metadata can be arbitrary data strucutres constructed with scalars,
234       hash references and array references.
235
236         $minion->add_task(job_with_progress => sub ($job) {
237           sleep 1;
238           $job->note(progress => '25%');
239           sleep 1;
240           $job->note(progress => '50%');
241           sleep 1;
242           $job->note(progress => '75%');
243           sleep 1;
244           $job->note(progress => '100%');
245         });
246
247       Notes, similar to job results, can be retrieved with "job" in Minion,
248       all you need is the job id.
249
250         # Get job metadata
251         my $progress = $minion->job($job_id)->info->{notes}{progress};
252
253       You can also use notes to store arbitrary metadata with new jobs when
254       you create them with "enqueue" in Minion.
255
256         # Create job with metadata
257         $minion->enqueue('job_with_progress', [], {notes => {progress => 0, something_else => [1, 2, 3]}});
258
259       The admin ui provided by Mojolicious::Plugin::Minion::Admin allows
260       searching for jobs containing a certain note, so you can also use them
261       to tag jobs.
262
263   Delayed jobs
264       The "delay" option of "enqueue" in Minion can be used to delay the
265       processing of a job by a certain amount of seconds (from now).
266
267         # Job will not be processed for 60 seconds
268         $minion->enqueue('check_links', ['https://mojolicious.org'], {delay => 20});
269
270       You can use "retry" in Minion::Job to change the delay.
271
272         $job->retry({delay => 10});
273
274   Expiring jobs
275       The "expire" option of "enqueue" in Minion can be used to limit for how
276       many seconds (from now) a job should be valid before it expires and
277       gets deleted from the queue.
278
279         # Job will vanish if it is not dequeued within 60 seconds
280         $minion->enqueue('check_links', ['https://mojolicious.org'], {expire => 60});
281
282       You can use "retry" in Minion::Job to reset the expiration time.
283
284         $job->retry({expire => 30});
285
286   Custom workers
287       In cases where you don't want to use Minion together with Mojolicious,
288       you can just skip the plugins and write your own worker scripts.
289
290         #!/usr/bin/perl
291         use strict;
292         use warnings;
293
294         use Minion;
295
296         # Connect to backend
297         my $minion = Minion->new(Pg => 'postgresql://postgres@/test');
298
299         # Add tasks
300         $minion->add_task(something_slow => sub ($job, @args) {
301           sleep 5;
302           say 'This is a background worker process.';
303         });
304
305         # Start a worker to perform up to 12 jobs concurrently
306         my $worker = $minion->worker;
307         $worker->status->{jobs} = 12;
308         $worker->run;
309
310       The method "run" in Minion::Worker contains all features you would
311       expect from a Minion worker and can be easily configured with "status"
312       in Minion::Worker. For even more customization options Minion::Worker
313       also has a very rich low level API you could for example use to build
314       workers that do not fork at all.
315
316   Task plugins
317       As your Mojolicious application grows, you can move tasks into
318       application specific plugins.
319
320         package MyApp::Task::PokeMojo;
321         use Mojo::Base 'Mojolicious::Plugin', -signatures;
322
323         sub register ($self, $app, $config) {
324           $app->minion->add_task(poke_mojo => sub ($job, @args) {
325             $job->app->ua->get('mojolicious.org');
326             $job->app->log->debug('We have poked mojolicious.org for a visitor');
327           });
328         }
329
330         1;
331
332       Which are loaded like any other plugin from your application.
333
334         # Mojolicious
335         $app->plugin('MyApp::Task::PokeMojo');
336
337         # Mojolicious::Lite
338         plugin 'MyApp::Task::PokeMojo';
339
340   Task classes
341       For more flexibility, or if you are using Minion as a standalone job
342       queue, you can also move tasks into dedicated classes. Allowing the use
343       of Perl features such as inheritance and roles. But be aware that
344       support for task classes is still EXPERIMENTAL and might change without
345       warning!
346
347         package MyApp::Task::PokeMojo;
348         use Mojo::Base 'Minion::Job', -signatures;
349
350         sub run ($self, @args) {
351           $self->app->ua->get('mojolicious.org');
352           $self->app->log->debug('We have poked mojolicious.org for a visitor');
353         }
354
355         1;
356
357       Task classes are registered just like any other task with "add_task" in
358       Minion and you can even register the same class with multiple names.
359
360         $minion->add_task(poke_mojo => 'MyApp::Task::PokeMojo');
361

MORE

363       You can continue with Mojolicious::Guides now or take a look at the
364       Mojolicious wiki <https://github.com/mojolicious/mojo/wiki>, which
365       contains a lot more documentation and examples by many different
366       authors.
367

SUPPORT

369       If you have any questions the documentation might not yet answer, don't
370       hesitate to ask in the Forum <https://forum.mojolicious.org> or the
371       official IRC channel "#mojo" on "irc.libera.chat" (chat now!
372       <https://web.libera.chat/#mojo>).
373
374
375
376perl v5.34.0                      2021-07-22                  Minion::Guide(3)
Impressum