1GEARMAND(8) Gearmand GEARMAND(8)
2
6 gearmand - Gearmand Documentation, http://gearman.info/
7
9 General options
10 -b [ --backlog ] arg (=32)
12 Number of backlog connections for listen.
13 --check-args
15 Check command line and configuration file argments and then
16 exit.
17 -d [ --daemon ]
19 Daemon, detach and run in the background.
20 -f [ --file-descriptors ] arg
22 Number of file descriptors to allow for the process (total con‐
23 nections will be slightly less). Default is max allowed for
24 user.
25 -h [ --help ]
27 Print this help menu.
28 -j [ --job-retries ] arg (=0)
30 Number of attempts to run the job before the job server removes
31 it. This is helpful to ensure a bad job does not crash all
32 available workers. Default is no limit.
33 -l [ --log-file ] arg
35 Log file to write errors and information to. Turning this op‐
36 tion on also forces the first verbose level to be enabled.
37 -L [ --listen ] arg
39 Address the server should listen on. Default is INADDR_ANY.
40 -p [ --port ] arg (=4730)
42 Port the server should listen on.
43 -P [ --pid-file ] arg
45 File to write process ID out to.
46 -r [ --protocol ] arg
48 Load protocol module.
49 -R [ --round-robin ]
51 Assign work in round-robin order per worker connection. The de‐
52 fault is to assign work in the order of functions added by the
53 worker.
54 -q [ --queue-type ] arg
56 Persistent queue type to use.
57 -t [ --threads ] arg (=4)
59 Number of I/O threads to use. Default=4.
60 -u [ --user ] arg
62 Switch to given user after startup.
63 -v [ --verbose ] arg (=v)
65 Set verbosity level. Defaults to ERROR. Accepts FATAL, ALERT,
66 CRITICAL, ERROR, WARNING, NOTICE, INFO, and DEBUG.
67 -V [ --version ]
69 Display the version of gearmand and exit.
70 -w [ --worker-wakeup ] arg (=0)
72 Number of workers to wakeup for each job received. The default
73 is to wakeup all available workers.
74 --keepalive
76 Enable keepalive on sockets.
77 --keepalive-idle arg (=-1)
79 If keepalive is enabled, set the value for TCP_KEEPIDLE for sys‐
80 tems that support it. A value of -1 means that either the system
81 does not support it or an error occurred when trying to retrieve
82 the default value.
83 --keepalive-interval arg (=-1)
85 If keepalive is enabled, set the value for TCP_KEEPINTVL for
86 systems that support it. A value of -1 means that either the
87 system does not support it or an error occurred when trying to
88 retrieve the default value.
89 --keepalive-count arg (=-1)
91 If keepalive is enabled, set the value for TCP_KEEPCNT for sys‐
92 tems that support it. A value of -1 means that either the system
93 does not support it or an error occurred when trying to retrieve
94 the default value.
95 HTTP:
97 --http-port arg (=8080)
99 Port to listen on.
100 sqlite
102 --libsqlite3-db arg
104 Database file to use.
105 --libsqlite3-table arg (=gearman_queue)
107 Table to use.
108 Memcached(libmemcached)
110 --libmemcached-servers arg
112 List of Memcached servers to use.
113 Drizzle/MySQL(libdrizzle)
115 --libdrizzle-host arg
117 Host of server.
118 --libdrizzle-port arg
120 Port of server. (by default Drizzle)
121 --libdrizzle-uds arg
123 Unix domain socket for server.
124 --libdrizzle-user arg
126 User name for authentication.
127 --libdrizzle-password arg
129 Password for authentication.
130 --libdrizzle-db arg
132 Schema/Database to use.
133 --libdrizzle-table arg
135 Table to use.
136 --libdrizzle-mysql arg
138 Use MySQL protocol.
139 Postgres
141 --libpq-conninfo arg
143 PostgreSQL connection information string.
144 --libpq-table arg (=queue)
146 Table to use.
147 tokyocabinet
149 --libtokyocabinet-file arg
151 File name of the database. [see: man tcadb, tcadbopen() for name
152 guidelines]
153 --libtokyocabinet-optimize
155 Optimize database on open. [default=true]
156
158 Gearman provides a generic application framework to farm out work to
159 other machines or processes that are better suited to do the work. It
160 allows you to do work in parallel, to load balance processing, and to
161 call functions between languages. It can be used in a variety of appli‐
162 cations, from high-availability web sites to the transport of database
163 replication events. In other words, it is the nervous system for how
164 distributed processing communicates. A few strong points about Gearman:
165 • Open Source - It's free! (in both meanings of the word) Gearman has
167 an active open source community that is easy to get involved with if
168 you need help or want to contribute.
169 • Multi-language - There are interfaces for a number of languages, and
171 this list is growing. You also have the option to write heterogeneous
172 applications with clients submitting work in one language and workers
173 performing that work in another.
174 • Flexible - You are not tied to any specific design pattern. You can
176 quickly put together distributed applications using any model you
177 choose, one of those options being Map/Reduce.
178 • Fast - Gearman has a simple protocol and interface with a new opti‐
180 mized server in C to minimize your application overhead.
181 • Embeddable - Since Gearman is fast and lightweight, it is great for
183 applications of all sizes. It is also easy to introduce into existing
184 applications with minimal overhead.
185 • No single point of failure - Gearman can not only help scale systems,
187 but can do it in a fault tolerant way.
188 Thread Model
190 The -t option to gearmand allows you to specify multiple I/O threads,
191 this is enabled by default. There are currently three types of threads
192 in the job server:
193 Listening and management thread - only one I/O thread - can have many
195 Processing thread - only one
196 When no -t option is given or -t 0 is given, all of three thread types
198 happen within a single thread. When -t 1 is given, there is a thread
199 for listening/management and a thread for I/O and processing. When -t 2
200 is given, there is a thread for each type of thread above. For all -t
201 option values above 2, more I/O threads are created.
202 The listening and management thread is mainly responsible for accepting
204 new connections and assigning those connections to an I/O thread (if
205 there are many). It also coordinates startup and shutdown within the
206 server. This thread will have an instance of libevent for managing
207 socket events and signals on an internal pipe. This pipe is used to
208 wakeup the thread or to coordinate shutdown.
209 The I/O thread is responsible for doing the read and write system calls
211 on the sockets and initial packet parsing. Once the packet has been
212 parsed it it put into an asynchronous queue for the processing thread
213 (each thread has it's own queue so there is very little contention).
214 Each I/O thread has it's own instance of libevent for managing socket
215 events and signals on an internal pipe like the listening thread.
216 The processing thread should have no system calls within it (except for
218 the occasional brk() for more memory), and manages the various lists
219 and hash tables used for tracking unique keys, job handles, functions,
220 and job queues. All packets that need to be sent back to connections
221 are put into an asynchronous queue for the I/O thread. The I/O thread
222 will pick these up and send them back over the connected socket. All
223 packets flow through the processing thread since it contains the infor‐
224 mation needed to process the packets. This is due to the complex nature
225 of the various lists and hash tables. If multiple threads were modify‐
226 ing them the locking overhead would most likely cause worse performance
227 than having it in a single thread (and would also complicate the code).
228 In the future more work may be pushed to the I/O threads, and the pro‐
229 cessing thread can retain minimal functionality to manage those tables
230 and lists. So far this has not been a significant bottleneck, a 16 core
231 Intel machine is able to process upwards of 50k jobs per second.
232 For thread safety to work when UUID are generated, you must be running
234 the uuidd daemon.
235 Persistent Queues
237 Inside the Gearman job server, all job queues are stored in memory.
238 This means if a server restarts or crashes with pending jobs, they will
239 be lost and are never run by a worker. Persistent queues were added to
240 allow background jobs to be stored in an external durable queue so they
241 may live between server restarts and crashes. The persistent queue is
242 only enabled for background jobs because foreground jobs have an at‐
243 tached client. If a job server goes away, the client can detect this
244 and restart the foreground job somewhere else (or report an error back
245 to the original caller). Background jobs on the other hand have no at‐
246 tached client and are simply expected to be run when submitted.
247 The persistent queue works by calling a module callback function right
249 before putting a new job in the internal queue for pending jobs to be
250 run. This allows the module to store the job about to be run in some
251 persistent way so that it can later be replayed during a restart. Once
252 it is stored through the module, the job is put onto the active
253 runnable queue, waking up available workers if needed. Once the job has
254 been successfully completed by a worker, another module callback func‐
255 tion is called to notify the module the job is done and can be removed.
256 If a job server crashes or is restarted between these two calls for a
257 job, the jobs are reloaded during the next job server start. When the
258 job server starts up, it will call a replay callback function in the
259 module to provide a list of all jobs that were not complete. This is
260 used to populate the internal memory queue of jobs to be run. Once this
261 replay is complete, the job server finishes its initialization and the
262 jobs are now runnable once workers connect (the queue should be in the
263 same state as when it crashed). These jobs are removed from the persis‐
264 tent queue when completed as normal. NOTE: Deleting jobs from the per‐
265 sistent queue storage will not remove them from the in-memory queue
266 while the server is running.
267 The queues are implemented using a modular interface so it is easy to
269 add new data stores for the persistent queue.
270 A persistent queue module is enabled by passing the -q or –queue-type
272 option to gearmand. Run gearmand –help to see which queue modules are
273 supported on your system. If you are missing options for one you would
274 like to use, you will need to install any dependencies and then recom‐
275 pile the gearmand package.
276 Extended Protocols
278 The protocol plugin interface allows you to take over the packet send
279 and receive functions, allowing you to pack the buffers as required by
280 the protocol. The core read and write functions can (and should) be
281 used by the protocol plugin.
282 HTTP
284 This protocol plugin allows you to map HTTP requests to Gearman jobs.
285 It only provides client job submission currently, but it may be ex‐
286 tended to support other request types in the future. The plugin can
287 handle both GET and POST data, the latter being used to send a workload
288 to the job server. The URL being requested is translated into the func‐
289 tion being called.
290 For example, the request:
292 POST /reverse HTTP/1.1
294 Content-Length: 12
295 Hello world!
297 Is translated into a job submission request for the function “reverse”
299 and workload “Hello world!”. This will respond with:
300 HTTP/1.0 200 OK
302 X-Gearman-Job-Handle: H:lap:4
303 Content-Length: 12
304 Server: Gearman/0.8
305 !dlrow olleH
307 The following headers can be passed to change the behavior of the job:
309 * X-Gearman-Unique: <unique key>
311 * X-Gearman-Background: true
312 * X-Gearman-Priority: <high|low>
313 For example, to run a low priority background job, the following re‐
315 quest can be sent:
316 POST /reverse HTTP/1.1
318 Content-Length: 12
319 X-Gearman-Background: true
320 X-Gearman-Priority: low
321 Hello world!
323 The response for this request will not have any data associated with it
325 since it was a background job:
326 HTTP/1.0 200 OK
328 X-Gearman-Job-Handle: H:lap:6
329 Content-Length: 0
330 Server: Gearman/0.8
331 The HTTP protocol should be considered experimental.
333
335 To find out more information please check: http://gearman.info/
336
338 gearman(1) gearadmin(1) libgearmand(3)
339
341 Data Differential http://www.datadifferential.com/
342
344 2011-2014, Data Differential, http://www.datadifferential.com/
3451.1.20 Nov 19, 2022 GEARMAND(8)