1GEARMAND(8)                        Gearmand                        GEARMAND(8)
2
3
4

NAME

6       gearmand - Gearmand Documentation, http://gearman.info/
7

SYNOPSIS

9       General options
10
11       -b [ --backlog ] arg (=32)
12              Number of backlog connections for listen.
13
14       --check-args
15              Check  command  line  and  configuration  file argments and then
16              exit.
17
18       -d [ --daemon ]
19              Daemon, detach and run in the background.
20
21       -f [ --file-descriptors ] arg
22              Number of file descriptors to allow for the process (total  con‐
23              nections  will  be  slightly  less).  Default is max allowed for
24              user.
25
26       -h [ --help ]
27              Print this help menu.
28
29       -j [ --job-retries ] arg (=0)
30              Number of attempts to run the job before the job server  removes
31              it.  This  is  helpful  to  ensure  a bad job does not crash all
32              available workers. Default is no limit.
33
34       -l [ --log-file ] arg
35              Log file to write errors and information to.  Turning  this  op‐
36              tion on also forces the first verbose level to be enabled.
37
38       -L [ --listen ] arg
39              Address the server should listen on. Default is INADDR_ANY.
40
41       -p [ --port ] arg (=4730)
42              Port the server should listen on.
43
44       -P [ --pid-file ] arg
45              File to write process ID out to.
46
47       -r [ --protocol ] arg
48              Load protocol module.
49
50       -R [ --round-robin ]
51              Assign  work in round-robin order per worker connection. The de‐
52              fault is to assign work in the order of functions added  by  the
53              worker.
54
55       -q [ --queue-type ] arg
56              Persistent queue type to use.
57
58       -t [ --threads ] arg (=4)
59              Number of I/O threads to use. Default=4.
60
61       -u [ --user ] arg
62              Switch to given user after startup.
63
64       -v [ --verbose ] arg (=v)
65              Set  verbosity  level.  Defaults to ERROR. Accepts FATAL, ALERT,
66              CRITICAL, ERROR, WARNING, NOTICE, INFO, and DEBUG.
67
68       -V [ --version ]
69              Display the version of gearmand and exit.
70
71       -w [ --worker-wakeup ] arg (=0)
72              Number of workers to wakeup for each job received.  The  default
73              is to wakeup all available workers.
74
75       --keepalive
76              Enable keepalive on sockets.
77
78       --keepalive-idle arg (=-1)
79              If keepalive is enabled, set the value for TCP_KEEPIDLE for sys‐
80              tems that support it. A value of -1 means that either the system
81              does not support it or an error occurred when trying to retrieve
82              the default value.
83
84       --keepalive-interval arg (=-1)
85              If keepalive is enabled, set the  value  for  TCP_KEEPINTVL  for
86              systems  that  support  it.  A value of -1 means that either the
87              system does not support it or an error occurred when  trying  to
88              retrieve the default value.
89
90       --keepalive-count arg (=-1)
91              If  keepalive is enabled, set the value for TCP_KEEPCNT for sys‐
92              tems that support it. A value of -1 means that either the system
93              does not support it or an error occurred when trying to retrieve
94              the default value.
95
96       HTTP:
97
98       --http-port arg (=8080)
99              Port to listen on.
100
101       sqlite
102
103       --libsqlite3-db arg
104              Database file to use.
105
106       --libsqlite3-table arg (=gearman_queue)
107              Table to use.
108
109       Memcached(libmemcached)
110
111       --libmemcached-servers arg
112              List of Memcached servers to use.
113
114       Drizzle/MySQL(libdrizzle)
115
116       --libdrizzle-host arg
117              Host of server.
118
119       --libdrizzle-port arg
120              Port of server. (by default Drizzle)
121
122       --libdrizzle-uds arg
123              Unix domain socket for server.
124
125       --libdrizzle-user arg
126              User name for authentication.
127
128       --libdrizzle-password arg
129              Password for authentication.
130
131       --libdrizzle-db arg
132              Schema/Database to use.
133
134       --libdrizzle-table arg
135              Table to use.
136
137       --libdrizzle-mysql arg
138              Use MySQL protocol.
139
140       Postgres
141
142       --libpq-conninfo arg
143              PostgreSQL connection information string.
144
145       --libpq-table arg (=queue)
146              Table to use.
147
148       tokyocabinet
149
150       --libtokyocabinet-file arg
151              File name of the database. [see: man tcadb, tcadbopen() for name
152              guidelines]
153
154       --libtokyocabinet-optimize
155              Optimize database on open. [default=true]
156

DESCRIPTION

158       Gearman  provides  a  generic application framework to farm out work to
159       other machines or processes that are better suited to do the  work.  It
160       allows  you  to do work in parallel, to load balance processing, and to
161       call functions between languages. It can be used in a variety of appli‐
162       cations,  from high-availability web sites to the transport of database
163       replication events. In other words, it is the nervous  system  for  how
164       distributed processing communicates. A few strong points about Gearman:
165
166       • Open  Source  - It's free! (in both meanings of the word) Gearman has
167         an active open source community that is easy to get involved with  if
168         you need help or want to contribute.
169
170       • Multi-language  - There are interfaces for a number of languages, and
171         this list is growing. You also have the option to write heterogeneous
172         applications with clients submitting work in one language and workers
173         performing that work in another.
174
175       • Flexible - You are not tied to any specific design pattern.  You  can
176         quickly  put  together  distributed  applications using any model you
177         choose, one of those options being Map/Reduce.
178
179       • Fast - Gearman has a simple protocol and interface with a  new  opti‐
180         mized server in C to minimize your application overhead.
181
182       • Embeddable  -  Since Gearman is fast and lightweight, it is great for
183         applications of all sizes. It is also easy to introduce into existing
184         applications with minimal overhead.
185
186       • No single point of failure - Gearman can not only help scale systems,
187         but can do it in a fault tolerant way.
188
189   Thread Model
190       The -t option to gearmand allows you to specify multiple  I/O  threads,
191       this  is enabled by default. There are currently three types of threads
192       in the job server:
193
194       Listening and management thread - only one I/O thread - can  have  many
195       Processing thread - only one
196
197       When  no -t option is given or -t 0 is given, all of three thread types
198       happen within a single thread. When -t 1 is given, there  is  a  thread
199       for listening/management and a thread for I/O and processing. When -t 2
200       is given, there is a thread for each type of thread above. For  all  -t
201       option values above 2, more I/O threads are created.
202
203       The listening and management thread is mainly responsible for accepting
204       new connections and assigning those connections to an  I/O  thread  (if
205       there  are  many).  It also coordinates startup and shutdown within the
206       server. This thread will have an  instance  of  libevent  for  managing
207       socket  events  and  signals  on an internal pipe. This pipe is used to
208       wakeup the thread or to coordinate shutdown.
209
210       The I/O thread is responsible for doing the read and write system calls
211       on  the  sockets  and  initial packet parsing. Once the packet has been
212       parsed it it put into an asynchronous queue for the  processing  thread
213       (each  thread  has  it's own queue so there is very little contention).
214       Each I/O thread has it's own instance of libevent for  managing  socket
215       events and signals on an internal pipe like the listening thread.
216
217       The processing thread should have no system calls within it (except for
218       the occasional brk() for more memory), and manages  the  various  lists
219       and  hash tables used for tracking unique keys, job handles, functions,
220       and job queues. All packets that need to be sent  back  to  connections
221       are  put  into an asynchronous queue for the I/O thread. The I/O thread
222       will pick these up and send them back over the  connected  socket.  All
223       packets flow through the processing thread since it contains the infor‐
224       mation needed to process the packets. This is due to the complex nature
225       of  the various lists and hash tables. If multiple threads were modify‐
226       ing them the locking overhead would most likely cause worse performance
227       than having it in a single thread (and would also complicate the code).
228       In the future more work may be pushed to the I/O threads, and the  pro‐
229       cessing  thread can retain minimal functionality to manage those tables
230       and lists. So far this has not been a significant bottleneck, a 16 core
231       Intel machine is able to process upwards of 50k jobs per second.
232
233       For  thread safety to work when UUID are generated, you must be running
234       the uuidd daemon.
235
236   Persistent Queues
237       Inside the Gearman job server, all job queues  are  stored  in  memory.
238       This means if a server restarts or crashes with pending jobs, they will
239       be lost and are never run by a worker. Persistent queues were added  to
240       allow background jobs to be stored in an external durable queue so they
241       may live between server restarts and crashes. The persistent  queue  is
242       only  enabled  for  background jobs because foreground jobs have an at‐
243       tached client. If a job server goes away, the client  can  detect  this
244       and  restart the foreground job somewhere else (or report an error back
245       to the original caller). Background jobs on the other hand have no  at‐
246       tached client and are simply expected to be run when submitted.
247
248       The  persistent queue works by calling a module callback function right
249       before putting a new job in the internal queue for pending jobs  to  be
250       run.  This  allows  the module to store the job about to be run in some
251       persistent way so that it can later be replayed during a restart.  Once
252       it  is  stored  through  the  module,  the  job  is put onto the active
253       runnable queue, waking up available workers if needed. Once the job has
254       been  successfully completed by a worker, another module callback func‐
255       tion is called to notify the module the job is done and can be removed.
256       If  a  job server crashes or is restarted between these two calls for a
257       job, the jobs are reloaded during the next job server start.  When  the
258       job  server  starts  up, it will call a replay callback function in the
259       module to provide a list of all jobs that were not  complete.  This  is
260       used to populate the internal memory queue of jobs to be run. Once this
261       replay is complete, the job server finishes its initialization and  the
262       jobs  are now runnable once workers connect (the queue should be in the
263       same state as when it crashed). These jobs are removed from the persis‐
264       tent  queue when completed as normal. NOTE: Deleting jobs from the per‐
265       sistent queue storage will not remove them  from  the  in-memory  queue
266       while the server is running.
267
268       The  queues  are implemented using a modular interface so it is easy to
269       add new data stores for the persistent queue.
270
271       A persistent queue module is enabled by passing the -q  or  –queue-type
272       option  to  gearmand. Run gearmand –help to see which queue modules are
273       supported on your system. If you are missing options for one you  would
274       like  to use, you will need to install any dependencies and then recom‐
275       pile the gearmand package.
276
277   Extended Protocols
278       The protocol plugin interface allows you to take over the  packet  send
279       and  receive functions, allowing you to pack the buffers as required by
280       the protocol. The core read and write functions  can  (and  should)  be
281       used by the protocol plugin.
282
283   HTTP
284       This  protocol  plugin allows you to map HTTP requests to Gearman jobs.
285       It only provides client job submission currently, but  it  may  be  ex‐
286       tended  to  support  other  request types in the future. The plugin can
287       handle both GET and POST data, the latter being used to send a workload
288       to the job server. The URL being requested is translated into the func‐
289       tion being called.
290
291       For example, the request:
292
293          POST /reverse HTTP/1.1
294          Content-Length: 12
295
296          Hello world!
297
298       Is translated into a job submission request for the function  “reverse”
299       and workload “Hello world!”. This will respond with:
300
301          HTTP/1.0 200 OK
302          X-Gearman-Job-Handle: H:lap:4
303          Content-Length: 12
304          Server: Gearman/0.8
305
306          !dlrow olleH
307
308       The following headers can be passed to change the behavior of the job:
309
310          * X-Gearman-Unique: <unique key>
311          * X-Gearman-Background: true
312          * X-Gearman-Priority: <high|low>
313
314       For  example,  to  run a low priority background job, the following re‐
315       quest can be sent:
316
317          POST /reverse HTTP/1.1
318          Content-Length: 12
319          X-Gearman-Background: true
320          X-Gearman-Priority: low
321
322          Hello world!
323
324       The response for this request will not have any data associated with it
325       since it was a background job:
326
327          HTTP/1.0 200 OK
328          X-Gearman-Job-Handle: H:lap:6
329          Content-Length: 0
330          Server: Gearman/0.8
331
332       The HTTP protocol should be considered experimental.
333

HOME

335       To find out more information please check: http://gearman.info/
336

SEE ALSO

338       gearman(1) gearadmin(1) libgearmand(3)
339

AUTHOR

341       Data Differential http://www.datadifferential.com/
342
344       2011-2014, Data Differential, http://www.datadifferential.com/
345
346
347
348
3491.1.20                           Nov 19, 2022                      GEARMAND(8)
Impressum