1distcc(1) General Commands Manual distcc(1)
2
6 distcc - distributed C/C++/ObjC compiler
7
9 distcc <compiler> [COMPILER OPTIONS]
10 distcc [COMPILER OPTIONS]
12 <compiler> [COMPILER OPTIONS]
14
16 distcc distributes compilation of C code across several machines on a
17 network. distcc should always generate the same results as a local
18 compile, is simple to install and use, and is often much faster than a
19 local compile.
20 distcc sends the complete preprocessed source code and compiler argu‐
22 ments across the network for each job, so the machines do not need to
23 share a filesystem, have the same headers or libraries installed, or
24 have synchronized clocks.
25 Compilation is driven by a "client" machine, which is typically the
27 developer's workstation or laptop. The distcc client runs on this
28 machine, as does make, the preprocessor, the linker, and other stages
29 of the build process. Any number of "volunteer" machines help the
30 client to build the program, by running the distccd(1) daemon, C com‐
31 piler and assembler as required.
32 distcc can run across either TCP sockets (on port 3632 by default), or
34 through a tunnel command such as ssh(1). For TCP connections the vol‐
35 unteers must run the distccd(1) daemon either directly or from inetd.
36 For SSH connections distccd must be installed but should not be listen‐
37 ing for connections.
38 TCP connections should only be used on secure networks because there is
40 no user authentication or protection of source or object code. SSH
41 connections are typically 25% slower because of processor overhead for
42 encryption, although this can vary greatly depending on CPUs, network
43 and the program being built.
44 distcc is intended to be used with GNU Make's -j option, which runs
46 several compiler processes concurrently. distcc spreads the jobs
47 across both local and remote CPUs. Because distcc is able to distrib‐
48 ute most of the work across the network a higher concurrency level can
49 be used than for local builds. The -j value should normally be set to
50 about twice the total number of available CPUs, to allow for some tasks
51 being blocked waiting for disk or network IO. distcc can also work
52 with other build control tools such as SCons.
53 It is strongly recommended that you install the same compiler version
55 on all machines participating in a build. Incompatible compilers may
56 cause mysterious compile or link failures.
57
59 1 For each machine, download distcc, unpack, and install.
60 2 On each of the servers, run distccd --daemon optionally with
62 --allow options to restrict access.
63 3 Put the names of the servers in your environment:
65 $ export DISTCC_HOSTS='localhost red green blue'
66 4 Build!
68 $ make -j8 CC=distcc
69
71 distcc only ever runs the compiler and assembler remotely. The pre‐
72 processor must always run locally because it needs to access various
73 header files on the local machine which may not be present, or may not
74 be the same, on the volunteer. The linker similarly needs to examine
75 libraries and object files, and so must run locally.
76 The compiler and assembler take only a single input file (the prepro‐
78 cessed source) and produce a single output (the object file). distcc
79 ships these two files across the network and can therefore run the com‐
80 piler/assembler remotely.
81 Fortunately, for most programs running the preprocessor is relatively
83 cheap, and the linker is called relatively infrequent, so most of the
84 work can be distributed.
85 distcc examines its command line to determine which of these phases are
87 being invoked, and whether the job can be distributed.
88
90 Most options passed to distcc are interpreted as compiler options. Two
91 options are understood by distcc itself:
92 --help Displays summary instructions.
94 --version
96 Displays the distcc client version.
97
99 There are three different ways to call distcc, to suit different cir‐
100 cumstances:
101 distcc can be installed under the name of the real compiler, to
103 intercept calls to it and run them remotely. This "masqueraded"
104 compiler has the widest compatibility with existing source
105 trees, and is convenient when you want to use distcc for all
106 compilation. The fact that distcc is being used is transparent
107 to the makefiles.
108 distcc can be prepended to compiler command lines, such as
110 "distcc cc -c hello.c" or CC="distcc gcc". This is convenient
111 when you want to use distcc for only some compilations or to try
112 it out, but can cause trouble with some makefiles or versions of
113 libtool that assume $CC does not contain a space.
114 Finally, distcc can be used directly as a compiler. "cc" is
116 always used as the name of the real compiler in this "implicit"
117 mode. This can be convenient for interactive use when
118 "explicit" mode does not work but is not really recommended for
119 new use.
120 Remember that you should not use two methods for calling distcc at the
122 same time. If you are using a masquerade directory, don't change CC
123 and/or CXX, just put the dirirectory early on your PATH. If you're not
124 using a masquerade directory, you'll need to either change CC and/or
125 CXX, or modify the makefile(s) to call distcc explicitly.
126
128 The basic idea is to create a "masquerade directory" which contains
129 links from the name of the real compiler to the distcc binary. This
130 directory is inserted early on the PATH, so that calls to the compiler
131 are intercepted and distcc is run instead. distcc then removes itself
132 from the PATH to find the real compiler.
133 For example:
135 # mkdir /usr/lib/distcc/bin
137 # cd /usr/lib/distcc/bin
138 # ln -s ../../../bin/distcc gcc
139 # ln -s ../../../bin/distcc cc
140 # ln -s ../../../bin/distcc g++
141 # ln -s ../../../bin/distcc c++
142 Then, to use distcc, a user just needs to put the directory
144 /usr/lib/distcc/bin early in the PATH, and have set a host list in
145 DISTCC_HOSTS or a file. distcc will handle the rest.
146 Note that this masquerade directory must occur on the PATH earlier than
148 the directory that contains the actual compilers of the same names, and
149 that any auxiliary programs that these compilers call (such as as or
150 ld) must also be found on the PATH in a directory after the masquerade
151 directory since distcc calls out to the real compiler with a PATH value
152 that has all directory up to and including the masquerade directory
153 trimmed off.
154 It is possible to get a "recursion error" in masquerade mode, which
156 means that distcc is somehow finding itself again, not the real com‐
157 piler. This can indicate that you have two masquerade directories on
158 the PATH, possibly because of having two distcc installations in dif‐
159 ferent locations. It can also indicate that you're trying to mix "mas‐
160 queraded" and "explicit" operation.
161
163 ccache is a program that speeds software builds by caching the results
164 of compilations. ccache is normally called before distcc, so that
165 results are retrieved from a normal cache. Some experimentation may be
166 required for idiosyncratic makefiles to make everything work together.
167 The most reliable method is to set
169 CCACHE_PREFIX="distcc"
171 This tells ccache to run distcc as a wrapper around the real compiler.
173 ccache still uses the real compiler to detect compiler upgrades.
174 ccache can then be run using either a masquerade directory or by set‐
176 ting
177 CC="ccache gcc"
179 As of version 2.2, ccache does not cache compilation from preprocessed
181 source and so will never get a cache hit if it is run from distccd or
182 distcc. It must be run only on the client side and before distcc to be
183 any use.
184
186 A "host list" tells distcc which machines to use for compilation. In
187 order, distcc looks in the $DISTCC_HOSTS environment variable, the
188 user's $DISTCC_DIR/hosts file, and the system-wide host file. If no
189 host list can be found, distcc emits a warning and compiles locally.
190 The host list is a simple whitespace separated list of host specifica‐
192 tions. The simplest and most common form is a host names, such as
193 localhost red green blue
195 distcc prefers hosts towards the start of the list, so machines should
197 be listed in descending order of speed. In particular, when only a
198 single compilation can be run (such as from a configure script), the
199 first machine listed is used.
200 Placing localhost at the right point in the list is important to get‐
202 ting good performance. Because overhead for running jobs locally is
203 low, localhost should normally be first. However, it is important that
204 the client have enough cycles free to run the local jobs and the distcc
205 client. If the client is slower than the volunteers, or if there are
206 many volunteers, then the client should be put later in the list or not
207 at all. As a general rule, if the aggregate CPU speed of the client is
208 less than one fifth of the total, then the client should be left out of
209 the list.
210 Performance depends on the details of the source and makefiles used for
212 the project, and the machine and network speeds. Experimenting with
213 different settings for the host list and -j factor may improve perfor‐
214 mance.
215 The syntax is
217 DISTCC_HOSTS = HOSTSPEC ...
219 HOSTSPEC = LOCAL_HOST | SSH_HOST | TCP_HOST | OLDSTYLE_TCP_HOST
220 LOCAL_HOST = localhost[/LIMIT]
221 SSH_HOST = [USER]@HOSTID[/LIMIT][:COMMAND][OPTIONS]
222 TCP_HOST = HOSTID[:PORT][/LIMIT][OPTIONS]
223 OLDSTYLE_TCP_HOST = HOSTID[/LIMIT][:PORT][OPTIONS]
224 HOSTID = HOSTNAME | IPV4
225 OPTIONS = ,OPTION[OPTIONS]
226 OPTION = lzo
227 Here are some individual examples of the syntax:
229 localhost
231 The literal word "localhost" is interpreted specially to cause
232 compilations to be directly executed, rather than passed to a
233 daemon on the local machine. If you do want to connect to a
234 daemon on the local machine for testing, then give the machine's
235 IP address or real hostname. (This will be slower.)
236 IPV4 A literal IPv4 address, such as 10.0.0.1
238 HOSTNAME
240 A hostname to be looked up using the resolver.
241 :PORT Connect to a specified decimal port number, rather than the
243 default of 3632.
244 @HOSTID
246 Connect to the host over SSH, rather than TCP. Options for the
247 SSH connection can be set in ~/.ssh/config
248 USER@ Connect to the host over SSH as a specified username.
250 :COMMAND
252 Connect over SSH, and use a specified path to find the distccd
253 server. This is normally only needed if for some reason you
254 can't install distccd into a directory on the default PATH for
255 SSH connections. Use this if you get errors like "distccd: com‐
256 mand not found" in SSH mode.
257 /LIMIT A decimal limit can be added to any host specification to
259 restrict the number of jobs that this client will send to the
260 machine. The limit defaults to four per host (two for local‐
261 host), but may be further restricted by the server. You should
262 only need to increase this for servers with more than two pro‐
263 cessors.
264 ,lzo Enables LZO compression for this TCP or SSH host.
266 Here is an example demonstrating some possibilities:
268 localhost/2 @bigman/16:/opt/bin/distccd oldmachine:4200/1
270 # cartman is down
271 distant/3,lzo
272 Comments are allowed in host specifications. Comments start with a
274 hash/pound sign (#) and run to the end of the line.
275 If a host in the list is not reachable distcc will emit a warning and
277 ignore that host for about one minute.
278
280 The lzo host option specifies that LZO compression should be used for
281 data transfer, including preprocessed source, object code and error
282 messages. Compression is usually economical on networks slower than
283 100Mbps, but results may vary depending on the network, processors and
284 source tree.
285 Enabling compression makes the distcc client and server use more CPU
287 time, but less network traffic. The compression ratio is typically 4:1
288 for source and 2:1 for object code.
289 Using compression requires both client and server to use at least
291 release 2.9 of distcc. No server configuration is required: the server
292 always responds with compressed replies to compressed requests.
293
295 If the compiler name is an absolute path, it is passed verbatim to the
296 server and the compiler is run from that directory. For example:
297 distcc /usr/local/bin/gcc-3.1415 -c hello.c
299 If the compiler name is not absolute, or not fully qualified, distccd's
301 PATH is searched. When distcc is run from a masquerade directory, only
302 the base name of the compiler is used. The client's PATH is used only
303 to run the preprocessor and has no effect on the server's path.
304
306 Both the distcc client and server impose timeouts on transfer of data
307 across the network. This is intended to detect hosts which are down or
308 unreachable, and to prevent compiles hanging indefinitely if a server
309 is disconnected while in use. If a client-side timeout expires, the
310 job will be re-run locally.
311 The timeouts are not configurable at present.
313
315 Error messages or warnings from local or remote compilers are passed
316 through to diagnostic output on the client.
317 distcc can supply extensive debugging information when the verbose
319 option is used. This is controlled by the DISTCC_VERBOSE environment
320 variable on the client, and the --verbose option on the server. For
321 troubleshooting, examine both the client and server error messages.
322
324 The exit code of distcc is normally that of the compiler: zero for suc‐
325 cessful compilation and non-zero otherwise.
326 distcc distinguishes between "genuine" errors such as a syntax error in
328 the source, and "accidental" errors such as a networking problem con‐
329 necting to a volunteer. In the case of accidental errors, distcc will
330 retry the compilation locally unless the DISTCC_FALLBACK option has
331 been disabled.
332 If the compiler exits with a signal, distcc returns an exit code of 128
334 plus the signal number.
335 distcc internal errors cause an exit code between 100 and 127. In par‐
337 ticular
338 100 General distcc failure.
340 105 Out of memory.
342 110 Compiler not found.
344 111 Recursive call to distcc.
346 116 No hosts defined and fallbacks disabled.
348 (Others are listed in exitcode.h.)
350
352 If $DISTCC_HOSTS is not set, distcc reads a host list from either
353 $DISTCC_DIR/hosts or a system-wide configuration file set at compile
354 time. The file locations are shown in the output from distcc --help
355 distcc creates a number of temporary and lock files underneath the tem‐
357 porary directory.
358
360 distcc's behaviour is controlled by a number of environment variables.
361 For most cases nothing need be set if the host list is stored in a
362 file.
363 DISTCC_HOSTS
365 Space-separated list of volunteer host specifications.
366 DISTCC_VERBOSE
368 If set to 1, distcc produces explanatory messages on the stan‐
369 dard error stream or in the log file. This can be helpful in
370 debugging problems. Bug reports should include verbose output.
371 DISTCC_LOG
373 Log file to receive messages from distcc itself, rather than
374 stderr.
375 DISTCC_FALLBACK
377 By default distcc will compile locally if it fails to distribute
378 a job to the intended machine, or if no host list can be found.
379 If this variable is set to 0 then fallbacks are disabled and
380 those compilations will simply fail. Note that this does not
381 affect jobs which must always be local such as linking.
382 DISTCC_SAVE_TEMPS
384 If set to 1, temporary files are not deleted after use. Good
385 for debugging, or if your disks are too empty.
386 DISTCC_TCP_CORK
388 If set to 0, disable use of "TCP corks", even if they're present
389 on this system. Using corks normally helps pack requests into
390 fewer packets and aids performance. This should normally be
391 left enabled.
392 DISTCC_SSH
394 Specifies the command used for opening SSH connections.
395 Defaults to "ssh" but may be set to a different connection com‐
396 mand such as "lsh" or "tsocks-ssh" that accepts a similar com‐
397 mand line. The command is not split into words and is not exe‐
398 cuted through the shell.
399 DISTCC_DIR
401 Per-user configuration directory to store lock files and state
402 files. By default ~/.distcc/ is used.
403 TMPDIR Directory for temporary files such as preprocessor output. By
405 default /tmp/ is used.
406 UNCACHED_ERR_FD
408 If set and if DISTCC_LOG is not set, distcc errors are written
409 to the file descriptor identified by this variable. This vari‐
410 able is intended mainly for automatic use by ccache, which sets
411 it to avoid caching transient errors such as network problems.
412
414 Cross compilation means building programs to run on a machine with a
415 different processor, architecture, or operating system to where they
416 were compiled. distcc supports cross compilation, including teams of
417 mixed-architecture machines, although some changes to the compilation
418 commands may be required.
419 The compilation command passed to distcc must be one that will execute
421 properly on every volunteer machine to produce an object file of the
422 appropriate type. If the machines have different processors, then sim‐
423 ply using distcc cc will probably not work, because that will normally
424 invoke the volunteer's native compiler.
425 Machines with the same CPU but different operating systems may not nec‐
427 essarily generate compatible .o files.
428 Several different gcc configurations can be installed side-by-side on
430 any machine. If you build gcc from source, you should use the --pro‐
431 gram-suffix configuration options to cause it to be installed with a
432 name that encodes the gcc version and the target platform.
433 The recommended convention for the gcc name is TARGET-gcc-VERSION such
435 as i686-linux-gcc-3.2 . GCC 3.3 will install itself under this name,
436 in addition to TARGET-gcc and, if it's native, gcc-VERSION and gcc .
437 The compiler must be installed under the same name on the client and on
439 every volunteer machine.
440
442 If you think you have found a distcc bug, please see the file report‐
443 ing-bugs.txt in the documentation directory for information on how to
444 report it.
445 Some makefiles have missing or extra dependencies that cause incorrect
447 or slow parallel builds. Recursive make is inefficient and can leave
448 processors unnecessarily idle for long periods. (See Recursive Make
449 Considered Harmful by Peter Miller.) Makefile bugs are the most common
450 cause of trees failing to build under distcc. Alternatives to Make
451 such as SCons can give much faster builds for some projects.
452 Using different versions of gcc can cause confusing build problems
454 because the header files and binary interfaces have changed over time,
455 and some distributors have included incompatible patches without chang‐
456 ing the version number. distcc does not protect against using incom‐
457 patible versions. Compiler errors about link problems or declarations
458 in system header files are usually due to mismatched or incorrectly
459 installed compilers.
460 Due to limitations in gcc, gdb may not be able to automatically find
462 the source files for programs built using distcc in some circumstances.
463 The gdb directory command can be used. This should be fixed in gcc
464 3.4.
465 gcc's -MD option can produce output in the wrong directory if the
467 source and object files are in different directories and the -MF option
468 is not used. There is no perfect solution because of incompatible
469 changes between gcc versions. Explicitly specifying the dependency
470 output file with -MF will fix the problem.
471 TCP mode connections should only be used on trusted networks.
473 Including slow machines in the list of volunteer hosts can slow the
475 build down.
476 When distcc or ccache is used on NFS, the filesystem must be exported
478 with the no_subtree_check option to allow reliable renames between
479 directories.
480 The compiler can be invoked with a command line gcc hello.c to both
482 compile and link. distcc doesn't split this into separate parts, but
483 rather runs the whole thing locally.
484 Other known bugs may be documented on http://distcc.samba.org/
486
488 distcc was written by Martin Pool <mbp@sourcefrog.net>, with the co-
489 operation of many scholars including Wayne Davison, Frerich Raabe, Dim‐
490 itri Papadopoulos and others noted in the NEWS file. Please report
491 bugs to <distcc@lists.samba.org>.
492
494 You are free to use distcc. distcc (including this manual) may be
495 copied, modified or distributed only under the terms of the GNU General
496 Public Licence version 2 or later. distcc comes with absolutely no
497 warrany. A copy of the GPL is included in the file COPYING.
498
500 distccd(1), ccache(1), gcc(1), make(1) http://distcc.samba.org/
501 http://ccache.samba.org/
502 28 July 2004 distcc(1)