1GEARMAND(8)                        Gearmand                        GEARMAND(8)
2
3
4

NAME

6       gearmand - Gearmand Documentation, http://gearman.info/
7

SYNOPSIS

9       General options
10
11       -b [ --backlog ] arg (=32)
12              Number of backlog connections for listen.
13
14       --check-args
15              Check  command  line  and  configuration  file argments and then
16              exit.
17
18       -d [ --daemon ]
19              Daemon, detach and run in the background.
20
21       -f [ --file-descriptors ] arg
22              Number of file descriptors to allow for the process (total  con‐
23              nections  will  be  slightly  less).  Default is max allowed for
24              user.
25
26       -h [ --help ]
27              Print this help menu.
28
29       -j [ --job-retries ] arg (=0)
30              Number of attempts to run the job before the job server  removes
31              it.  This  is  helpful  to  ensure  a bad job does not crash all
32              available workers. Default is no limit.
33
34       -l [ --log-file ] arg
35              Log file to write  errors  and  information  to.   Turning  this
36              option on also forces the first verbose level to be enabled.
37
38       -L [ --listen ] arg
39              Address the server should listen on. Default is INADDR_ANY.
40
41       -p [ --port ] arg (=4730)
42              Port the server should listen on.
43
44       -P [ --pid-file ] arg
45              File to write process ID out to.
46
47       -r [ --protocol ] arg
48              Load protocol module.
49
50       -R [ --round-robin ]
51              Assign  work  in  round-robin  order  per worker connection. The
52              default is to assign work in the order of functions added by the
53              worker.
54
55       -q [ --queue-type ] arg
56              Persistent queue type to use.
57
58       -t [ --threads ] arg (=4)
59              Number of I/O threads to use. Default=4.
60
61       -u [ --user ] arg
62              Switch to given user after startup.
63
64       -v [ --verbose ] arg (=v)
65              Increase verbosity level by one.
66
67       -V [ --version ]
68              Display the version of gearmand and exit.
69
70       -w [ --worker-wakeup ] arg (=0)
71              Number  of  workers to wakeup for each job received. The default
72              is to wakeup all available workers.
73
74       --keepalive
75              Enable keepalive on sockets.
76
77       --keepalive-idle arg (=-1)
78              If keepalive is enabled, set the value for TCP_KEEPIDLE for sys‐
79              tems that support it. A value of -1 means that either the system
80              does not support it or an error occurred when trying to retrieve
81              the default value.
82
83       --keepalive-interval arg (=-1)
84              If  keepalive  is  enabled,  set the value for TCP_KEEPINTVL for
85              systems that support it. A value of -1  means  that  either  the
86              system  does  not support it or an error occurred when trying to
87              retrieve the default value.
88
89       --keepalive-count arg (=-1)
90              If keepalive is enabled, set the value for TCP_KEEPCNT for  sys‐
91              tems that support it. A value of -1 means that either the system
92              does not support it or an error occurred when trying to retrieve
93              the default value.
94
95       HTTP:
96
97       --http-port arg (=8080)
98              Port to listen on.
99
100       sqlite
101
102       --libsqlite3-db arg
103              Database file to use.
104
105       --libsqlite3-table arg (=gearman_queue)
106              Table to use.
107
108       Memcached(libmemcached)
109
110       --libmemcached-servers arg
111              List of Memcached servers to use.
112
113       Drizzle/MySQL(libdrizzle)
114
115       --libdrizzle-host arg
116              Host of server.
117
118       --libdrizzle-port arg
119              Port of server. (by default Drizzle)
120
121       --libdrizzle-uds arg
122              Unix domain socket for server.
123
124       --libdrizzle-user arg
125              User name for authentication.
126
127       --libdrizzle-password arg
128              Password for authentication.
129
130       --libdrizzle-db arg
131              Schema/Database to use.
132
133       --libdrizzle-table arg
134              Table to use.
135
136       --libdrizzle-mysql arg
137              Use MySQL protocol.
138
139       Postgres
140
141       --libpq-conninfo arg
142              PostgreSQL connection information string.
143
144       --libpq-table arg (=queue)
145              Table to use.
146
147       tokyocabinet
148
149       --libtokyocabinet-file arg
150              File name of the database. [see: man tcadb, tcadbopen() for name
151              guidelines]
152
153       --libtokyocabinet-optimize
154              Optimize database on open. [default=true]
155

DESCRIPTION

157       Gearman provides a generic application framework to farm  out  work  to
158       other  machines  or processes that are better suited to do the work. It
159       allows you to do work in parallel, to load balance processing,  and  to
160       call functions between languages. It can be used in a variety of appli‐
161       cations, from high-availability web sites to the transport of  database
162       replication  events.  In  other words, it is the nervous system for how
163       distributed processing communicates. A few strong points about Gearman:
164
165       · Open Source - It's free! (in both meanings of the word)  Gearman  has
166         an  active open source community that is easy to get involved with if
167         you need help or want to contribute.
168
169       · Multi-language - There are interfaces for a number of languages,  and
170         this list is growing. You also have the option to write heterogeneous
171         applications with clients submitting work in one language and workers
172         performing that work in another.
173
174       · Flexible  -  You are not tied to any specific design pattern. You can
175         quickly put together distributed applications  using  any  model  you
176         choose, one of those options being Map/Reduce.
177
178       · Fast  -  Gearman has a simple protocol and interface with a new opti‐
179         mized server in C to minimize your application overhead.
180
181       · Embeddable - Since Gearman is fast and lightweight, it is  great  for
182         applications of all sizes. It is also easy to introduce into existing
183         applications with minimal overhead.
184
185       · No single point of failure - Gearman can not only help scale systems,
186         but can do it in a fault tolerant way.
187
188   Thread Model
189       The  -t  option to gearmand allows you to specify multiple I/O threads,
190       this is enabled by default. There are currently three types of  threads
191       in the job server:
192
193       Listening  and  management thread - only one I/O thread - can have many
194       Processing thread - only one
195
196       When no -t option is given or -t 0 is given, all of three thread  types
197       happen  within  a  single thread. When -t 1 is given, there is a thread
198       for listening/management and a thread for I/O and processing. When -t 2
199       is  given,  there is a thread for each type of thread above. For all -t
200       option values above 2, more I/O threads are created.
201
202       The listening and management thread is mainly responsible for accepting
203       new  connections  and  assigning those connections to an I/O thread (if
204       there are many). It also coordinates startup and  shutdown  within  the
205       server.  This  thread  will  have  an instance of libevent for managing
206       socket events and signals on an internal pipe. This  pipe  is  used  to
207       wakeup the thread or to coordinate shutdown.
208
209       The I/O thread is responsible for doing the read and write system calls
210       on the sockets and initial packet parsing. Once  the  packet  has  been
211       parsed  it  it put into an asynchronous queue for the processing thread
212       (each thread has it's own queue so there is  very  little  contention).
213       Each  I/O  thread has it's own instance of libevent for managing socket
214       events and signals on an internal pipe like the listening thread.
215
216       The processing thread should have no system calls within it (except for
217       the  occasional  brk()  for more memory), and manages the various lists
218       and hash tables used for tracking unique keys, job handles,  functions,
219       and  job  queues.  All packets that need to be sent back to connections
220       are put into an asynchronous queue for the I/O thread. The  I/O  thread
221       will  pick  these  up and send them back over the connected socket. All
222       packets flow through the processing thread since it contains the infor‐
223       mation needed to process the packets. This is due to the complex nature
224       of the various lists and hash tables. If multiple threads were  modify‐
225       ing them the locking overhead would most likely cause worse performance
226       than having it in a single thread (and would also complicate the code).
227       In  the future more work may be pushed to the I/O threads, and the pro‐
228       cessing thread can retain minimal functionality to manage those  tables
229       and lists. So far this has not been a significant bottleneck, a 16 core
230       Intel machine is able to process upwards of 50k jobs per second.
231
232       For thread safety to work when UUID are generated, you must be  running
233       the uuidd daemon.
234
235   Persistent Queues
236       Inside  the  Gearman  job  server, all job queues are stored in memory.
237       This means if a server restarts or crashes with pending jobs, they will
238       be  lost and are never run by a worker. Persistent queues were added to
239       allow background jobs to be stored in an external durable queue so they
240       may  live  between server restarts and crashes. The persistent queue is
241       only enabled for  background  jobs  because  foreground  jobs  have  an
242       attached  client. If a job server goes away, the client can detect this
243       and restart the foreground job somewhere else (or report an error  back
244       to  the  original  caller).  Background  jobs on the other hand have no
245       attached client and are simply expected to be run when submitted.
246
247       The persistent queue works by calling a module callback function  right
248       before  putting  a new job in the internal queue for pending jobs to be
249       run. This allows the module to store the job about to be  run  in  some
250       persistent  way so that it can later be replayed during a restart. Once
251       it is stored through the  module,  the  job  is  put  onto  the  active
252       runnable queue, waking up available workers if needed. Once the job has
253       been successfully completed by a worker, another module callback  func‐
254       tion is called to notify the module the job is done and can be removed.
255       If a job server crashes or is restarted between these two calls  for  a
256       job,  the  jobs are reloaded during the next job server start. When the
257       job server starts up, it will call a replay callback  function  in  the
258       module  to  provide  a list of all jobs that were not complete. This is
259       used to populate the internal memory queue of jobs to be run. Once this
260       replay  is complete, the job server finishes its initialization and the
261       jobs are now runnable once workers connect (the queue should be in  the
262       same state as when it crashed). These jobs are removed from the persis‐
263       tent queue when completed as normal. NOTE: Deleting jobs from the  per‐
264       sistent  queue  storage  will  not remove them from the in-memory queue
265       while the server is running.
266
267       The queues are implemented using a modular interface so it is  easy  to
268       add new data stores for the persistent queue.
269
270       A  persistent  queue module is enabled by passing the -q or –queue-type
271       option to gearmand. Run gearmand –help to see which queue  modules  are
272       supported  on your system. If you are missing options for one you would
273       like to use, you will need to install any dependencies and then  recom‐
274       pile the gearmand package.
275
276   Extended Protocols
277       The  protocol  plugin interface allows you to take over the packet send
278       and receive functions, allowing you to pack the buffers as required  by
279       the  protocol.  The  core  read and write functions can (and should) be
280       used by the protocol plugin.
281
282   HTTP
283       This protocol plugin allows you to map HTTP requests to  Gearman  jobs.
284       It  only  provides  client  job  submission  currently,  but  it may be
285       extended to support other request types in the future. The  plugin  can
286       handle both GET and POST data, the latter being used to send a workload
287       to the job server. The URL being requested is translated into the func‐
288       tion being called.
289
290       For example, the request:
291
292          POST /reverse HTTP/1.1
293          Content-Length: 12
294
295          Hello world!
296
297       Is  translated into a job submission request for the function “reverse”
298       and workload “Hello world!”. This will respond with:
299
300          HTTP/1.0 200 OK
301          X-Gearman-Job-Handle: H:lap:4
302          Content-Length: 12
303          Server: Gearman/0.8
304
305          !dlrow olleH
306
307       The following headers can be passed to change the behavior of the job:
308
309          * X-Gearman-Unique: <unique key>
310          * X-Gearman-Background: true
311          * X-Gearman-Priority: <high|low>
312
313       For example, to run  a  low  priority  background  job,  the  following
314       request can be sent:
315
316          POST /reverse HTTP/1.1
317          Content-Length: 12
318          X-Gearman-Background: true
319          X-Gearman-Priority: low
320
321          Hello world!
322
323       The response for this request will not have any data associated with it
324       since it was a background job:
325
326          HTTP/1.0 200 OK
327          X-Gearman-Job-Handle: H:lap:6
328          Content-Length: 0
329          Server: Gearman/0.8
330
331       The HTTP protocol should be considered experimental.
332

HOME

334       To find out more information please check: http://gearman.info/
335

SEE ALSO

337       gearman(1) gearadmin(1) libgearmand(3)
338

AUTHOR

340       Data Differential http://www.datadifferential.com/
341
343       2011-2014, Data Differential, http://www.datadifferential.com/
344
345
346
347
3481.1.18                         December 11, 2017                   GEARMAND(8)
Impressum