1Minion::Guide(3) User Contributed Perl Documentation Minion::Guide(3)
2
6 Minion::Guide - An introducion to Minion
7
9 This document contains an introduction to Minion and explains the most
10 important features it has to offer.
11
13 Essentials every Minion developer should know.
14 Job queue
16 Job queues allow you to process time and/or computationally intensive
17 tasks in background processes, outside of the request/response
18 lifecycle of web applications. Among those tasks you'll commonly find
19 image resizing, spam filtering, HTTP downloads, building tarballs,
20 warming caches and basically everything else you can imagine that's not
21 super fast.
22 Mojo::Server::Prefork +--------------+ Minion::Worker
24 |- Mojo::Server::Daemon [1] enqueue job -> | | -> dequeue job |- Minion::Job [1]
25 |- Mojo::Server::Daemon [2] | PostgreSQL | |- Minion::Job [2]
26 |- Mojo::Server::Daemon [3] retrieve result <- | | <- store result |- Minion::Job [3]
27 +- Mojo::Server::Daemon [4] +--------------+ |- Minion::Job [4]
28 +- Minion::Job [5]
29 They are not to be confused with time based job schedulers, such as
31 cron or systemd timers. Both serve very different purposes, and cron
32 jobs are in fact commonly used to enqueue Minion jobs that need to
33 follow a schedule. For example to perform regular maintenance tasks.
34 Mojolicious
36 You can use Minion as a standalone job queue or integrate it into
37 Mojolicious applications with the plugin Mojolicious::Plugin::Minion.
38 use Mojolicious::Lite -signatures;
40 plugin Minion => {Pg => 'postgresql://sri:s3cret@localhost/test'};
42 # Slow task
44 app->minion->add_task(poke_mojo => sub ($job, @args) {
45 $job->app->ua->get('mojolicious.org');
46 $job->app->log->debug('We have poked mojolicious.org for a visitor');
47 });
48 # Perform job in a background worker process
50 get '/' => sub ($c) {
51 $c->minion->enqueue('poke_mojo');
52 $c->render(text => 'We will poke mojolicious.org for you soon.');
53 };
54 app->start;
56 Background worker processes are usually started with the command
58 Minion::Command::minion::worker, which becomes automatically available
59 when an application loads Mojolicious::Plugin::Minion.
60 $ ./myapp.pl minion worker
62 The worker process will fork a new process for every job that is being
64 processed. This allows for resources such as memory to be returned to
65 the operating system once a job is finished. Perl fork is very fast, so
66 don't worry about the overhead.
67 Minion::Worker
69 |- Minion::Job [1]
70 |- Minion::Job [2]
71 +- ...
72 By default up to four jobs will be processed in parallel, but that can
74 be changed with configuration options or on demand with signals.
75 $ ./myapp.pl minion worker -j 12
77 Jobs can be managed right from the command line with
79 Minion::Command::minion::job.
80 $ ./myapp.pl minion job
82 You can also add an admin ui to your application by loading the plugin
84 Mojolicious::Plugin::Minion::Admin. Just make sure to secure access
85 before making your application publically accessible.
86 # Make admin ui available under "/minion"
88 plugin 'Minion::Admin';
89 Deployment
91 To manage background worker processes with systemd, you can use a unit
92 configuration file like this.
93 [Unit]
95 Description=My Mojolicious application workers
96 After=postgresql.service
97 [Service]
99 Type=simple
100 ExecStart=/home/sri/myapp/myapp.pl minion worker -m production
101 KillMode=process
102 [Install]
104 WantedBy=multi-user.target
105 Consistency
107 Every new job starts out as "incative", then progresses to "active"
108 when it is dequeued by a worker, and finally ends up as "finished" or
109 "failed", depending on its result. Every "failed" job can then be
110 retried to progress back to the "inactive" state and start all over
111 again.
112 +----------+
114 | |
115 +-----> | finished |
116 +----------+ +--------+ | | |
117 | | | | | +----------+
118 | inactive | -------> | active | ------+
119 | | | | | +----------+
120 +----------+ +--------+ | | |
121 +-----> | failed | -----+
122 ^ | | |
123 | +----------+ |
124 | |
125 +----------------------------------------------------------------+
126 The system is eventually consistent and will preserve job results for
128 as long as you like, depending on "remove_after" in Minion. But be
129 aware that "failed" results are presrved indefinitely, and need to be
130 manually removed by an administrator if they are out of automatic
131 retries.
132 While individual workers can fail in the middle of processing a job,
134 the system will detect this and ensure that no job is left in an
135 uncertain state, depending on "missing_after" in Minion. Jobs that do
136 not get processed after a certain amount of time, depending on
137 "stuck_after" in Minion, will be considered stuck and fail
138 automatically. So an admin can take a look and resolve the issue.
139
141 Minion has many great features. This section is still very incomplete,
142 but will be expanded over time.
143 Priorities
145 Every job enqueued with "enqueue" in Minion has a priority. Jobs with a
146 higher priority get performed first, the default priority is 0.
147 Priorities can be positive or negative, but should be in the range
148 between 100 and "-100".
149 # Default priority
151 $minion->enqueue('check_links', ['https://mojolicious.org']);
152 # High priority
154 $minion->enqueue('check_links', ['https://mojolicious.org'], {priority => 30});
155 # Low priority
157 $minion->enqueue('check_links', ['https://mojolicious.org'], {priority => -30});
158 You can use "retry" in Minion::Job to raise or lower the priority of a
160 job.
161 $job->retry({priority => 50});
163 Job results
165 The result of a job has two parts. First there is its state, which can
166 be "finished" for a successfully processed job, and "failed" for the
167 opposite. And second there's a "result" data structure, that may be
168 "undef", a scalar, a hash reference, or an array reference. You can
169 check both at any time in the life cycle of a job with "job" in Minion,
170 all you need is the job id.
171 # Check job state
173 my $state = $minion->job($job_id)->info->{state};
174 # Get job result
176 my $result = $minion->job($job_id)->info->{result};
177 While the "state" will be assigned automatically by Minion, the
179 "result" for "finished" jobs is usually assigned manually with "finish"
180 in Minion::Job.
181 $minion->add_task(job_with_result => sub ($job) {
183 sleep 5;
184 $job->finish({message => 'This job should have taken about 5 seconds'});
185 });
186 For jobs that "failed" due to an exception, that exception will be
188 assigned as "result".
189 $minion->add_task(job_that_fails => sub ($job) {
191 sleep 5;
192 die 'This job should always fail after 5 seconds';
193 });
194 But jobs can also fail manually with "fail" in Minion::Job.
196 $minion->add_task(job_that_fails_with_result => sub ($job) {
198 sleep 5;
199 $job->fail({errors => ['This job should fail after 5 seconds']});
200 });
201 Retrieving job results is of course completely optional, and it is very
203 common to have jobs where the result is unimportant.
204 Named queues
206 Each job can be enqueued with "enqueue" in Minion into arbitrarily
207 named queues, independent of all their other properties. This is
208 commonly used to have separate classes of workers, for example to
209 ensure that free customers of your web service do not negatively affect
210 your service level agreements with paying customers. The default named
211 queue is "default", but aside from that it has no special properties.
212 # Use "default" queue
214 $minion->enqueue('check_links', ['https://mojolicious.org']);
215 # Use custom "important" queue
217 $minion->enqueue('check_links', ['https://mojolicious.org'], {queue => 'important'});
218 For every named queue you can start as many workers as you like with
220 the command Minion::Command::minion::worker. And each worker can
221 process jobs from multiple named queues. So your workers can have
222 overlapping responsibilities.
223 $ ./myapp.pl minion worker -q default -q important
225 There is one special named queue called "minion_foreground" that you
227 should avoid using directly. It is reserved for debugging jobs with
228 "foreground" in Minion.
229 Job progress
231 Progress information and other job metadata can be stored in notes at
232 any time during the life cycle of a job with "note" in Minion::Job. The
233 metadata can be arbitrary data strucutres constructed with scalars,
234 hash references and array references.
235 $minion->add_task(job_with_progress => sub ($job) {
237 sleep 1;
238 $job->note(progress => '25%');
239 sleep 1;
240 $job->note(progress => '50%');
241 sleep 1;
242 $job->note(progress => '75%');
243 sleep 1;
244 $job->note(progress => '100%');
245 });
246 Notes, similar to job results, can be retrieved with "job" in Minion,
248 all you need is the job id.
249 # Get job metadata
251 my $progress = $minion->job($job_id)->info->{notes}{progress};
252 You can also use notes to store arbitrary metadata with new jobs when
254 you create them with "enqueue" in Minion.
255 # Create job with metadata
257 $minion->enqueue('job_with_progress', [], {notes => {progress => 0, something_else => [1, 2, 3]}});
258 The admin ui provided by Mojolicious::Plugin::Minion::Admin allows
260 searching for jobs containing a certain note, so you can also use them
261 to tag jobs.
262 Delayed jobs
264 The "delay" option of "enqueue" in Minion can be used to delay the
265 processing of a job by a certain amount of seconds (from now).
266 # Job will not be processed for 60 seconds
268 $minion->enqueue('check_links', ['https://mojolicious.org'], {delay => 20});
269 You can use "retry" in Minion::Job to change the delay.
271 $job->retry({delay => 10});
273 Expiring jobs
275 The "expire" option of "enqueue" in Minion can be used to limit for how
276 many seconds (from now) a job should be valid before it expires and
277 gets deleted from the queue.
278 # Job will vanish if it is not dequeued within 60 seconds
280 $minion->enqueue('check_links', ['https://mojolicious.org'], {expire => 60});
281 You can use "retry" in Minion::Job to reset the expiration time.
283 $job->retry({expire => 30});
285 Custom workers
287 In cases where you don't want to use Minion together with Mojolicious,
288 you can just skip the plugins and write your own worker scripts.
289 #!/usr/bin/perl
291 use strict;
292 use warnings;
293 use Minion;
295 # Connect to backend
297 my $minion = Minion->new(Pg => 'postgresql://postgres@/test');
298 # Add tasks
300 $minion->add_task(something_slow => sub ($job, @args) {
301 sleep 5;
302 say 'This is a background worker process.';
303 });
304 # Start a worker to perform up to 12 jobs concurrently
306 my $worker = $minion->worker;
307 $worker->status->{jobs} = 12;
308 $worker->run;
309 The method "run" in Minion::Worker contains all features you would
311 expect from a Minion worker and can be easily configured with "status"
312 in Minion::Worker. For even more customization options Minion::Worker
313 also has a very rich low level API you could for example use to build
314 workers that do not fork at all.
315 Task plugins
317 As your Mojolicious application grows, you can move tasks into
318 application specific plugins.
319 package MyApp::Task::PokeMojo;
321 use Mojo::Base 'Mojolicious::Plugin', -signatures;
322 sub register ($self, $app, $config) {
324 $app->minion->add_task(poke_mojo => sub ($job, @args) {
325 $job->app->ua->get('mojolicious.org');
326 $job->app->log->debug('We have poked mojolicious.org for a visitor');
327 });
328 }
329 1;
331 Which are loaded like any other plugin from your application.
333 # Mojolicious
335 $app->plugin('MyApp::Task::PokeMojo');
336 # Mojolicious::Lite
338 plugin 'MyApp::Task::PokeMojo';
339 Task classes
341 For more flexibility, or if you are using Minion as a standalone job
342 queue, you can also move tasks into dedicated classes. Allowing the use
343 of Perl features such as inheritance and roles. But be aware that
344 support for task classes is still EXPERIMENTAL and might change without
345 warning!
346 package MyApp::Task::PokeMojo;
348 use Mojo::Base 'Minion::Job', -signatures;
349 sub run ($self, @args) {
351 $self->app->ua->get('mojolicious.org');
352 $self->app->log->debug('We have poked mojolicious.org for a visitor');
353 }
354 1;
356 Task classes are registered just like any other task with "add_task" in
358 Minion and you can even register the same class with multiple names.
359 $minion->add_task(poke_mojo => 'MyApp::Task::PokeMojo');
361
363 You can continue with Mojolicious::Guides now or take a look at the
364 Mojolicious wiki <https://github.com/mojolicious/mojo/wiki>, which
365 contains a lot more documentation and examples by many different
366 authors.
367
369 If you have any questions the documentation might not yet answer, don't
370 hesitate to ask in the Forum <https://forum.mojolicious.org> or the
371 official IRC channel "#mojo" on "irc.libera.chat" (chat now!
372 <https://web.libera.chat/#mojo>).
373perl v5.34.0 2021-07-22 Minion::Guide(3)