1include_server(1) General Commands Manual include_server(1)
2
6 include_server.py - conservative approximation of include dependencies
7 for C/C++
8
10 include_server --port INCLUDE_SERVER_PORT [OPTIONS]
11
13 include_server.py starts an include server process. This process
14 answers queries from distcc(1) clients about what files to include in
15 C/C++ compilations. The include_server.py command itself terminates as
16 soon as the include server has been spawned.
17 The INCLUDE_SERVER_PORT argument is the name of a socket used for all
19 communication between distcc clients and the include server. The
20 pump(1) command is responsible for creating the socket location, for
21 passing it to this script, and for passing it to all distcc clients via
22 the environment variable named INCLUDE_SERVER_PORT.
23 The protocol used by the include server uses distcc's RPC implementa‐
25 tion. Each distcc request consists of (1) the current directory and (2)
26 the list of arguments of the compilation command.
27 If the include server is able to process the request, then it answers
29 the distcc client by sending a list of filepaths. The filepaths are
30 those of the compressed source and header files found to be necessary
31 for compilation through include analysis. The list also comprises sym‐
32 bolic links and even dummy files needed for the compilation server to
33 construct an accurate replica of the parts of the filesystem needed for
34 compilation. In this way, a needed header file like /path/foo.h is
35 compressed, renamed, and stored in a temporary location, such as
36 /dev/shm/tmpiAvfGv.include_server-9368-1/path/foo.h.lzo. The distcc
37 client will pass these files on to a compilation server, where they
38 will be uncompressed and mounted temporarily.
39 If the include server is not able to process the request, then it
41 returns the empty list to the distcc client.
42 There are two kinds of failures that relate to the include server. The
44 include server may fail to compute the includes or fail in other ways,
45 see section INCLUDE SERVER SYMPTOMS. Also, the compilation on the
46 remove server may fail due to inadequacy of the calculated include clo‐
47 sure, but then succeed when locally retried, see section DISTCC DIS‐
48 CREPANCY SYMPTOMS.
49
52 The following options are understood by include_server.py.
53 -dPAT, --debug_pattern=PAT
55 Bit vector for turning on warnings and debugging
56 1 = warnings
57 2 = trace some functions
58 other powers of two: see include_server/basics.py.
59 -e, --email
61 Send email to 'distcc-pump-errors' or if defined, the value of
62 environment variable DISTCC_EMAILLOG_WHOM_TO_BLAME, when include
63 server gets in trouble. The default is to not send email.
64 --email_bound NUMBER
66 Maximal number of emails to send (in addition to a final email).
67 Default: 3.
68 --no-email
70 Do not send email. This is the default.
71 --path_observation_re=RE
73 Issue warning message whenever a filename is resolved to a real‐
74 path that is matched by RE, which is a regular expression in
75 Python syntax. This is useful for finding out where files
76 included actually come from. Use RE="" to find them all. Note:
77 warnings must be enabled with at least -d1.
78 --pid_file FILEPATH
80 The pid of the include server is written to file FILEPATH. This
81 allows a script such a pump to tear down the include server.
82 -s, --statistics
84 Print information to stdout about include analysis.
85 --stat_reset_triggers=LIST
87 Flush stat caches when the timestamp of any filepath in LIST
88 changes or the filepath comes in or out of existence. LIST is a
89 colon separated string of filepaths, possibly containing simple
90 globs (as allowed by Python's glob module). Print a warning
91 whenever such a change happens (if warnings are enabled). This
92 option allows limited exceptions to distcc_pump's normal assump‐
93 tion that source files are not modified during the build.
94 -t, --time
96 Print elapsed, user, and system time to stderr.
97 --unsafe_absolute_includes
99 Do preprocessing on the compilation server even if includes of
100 absolute filepaths are encountered. Normally the include-server
101 will fall back on local preprocessing if it detects any absolute
102 includes. Thus, this flag is useful for preventing such fall‐
103 backs when the absolute includes are a false alarm, either
104 because the absolute include is discarded during preprocessing
105 or because the absolutely included file exists on the compila‐
106 tion servers.
107 More precisely, with --unsafe_absolute_includes absolute
109 includes are ignored for the purposes of gathering the include
110 closure. Using this option may lead to incorrect results
111 because (1) the header may actually be included on the compila‐
112 tion server and it may not be the same as on the client, (2) the
113 include directives of the header are not further analyzed.
114 The option is useful for compiling code that has such hardcoded
116 absolute locations of header files inside conditional directives
117 (e.g. "#ifdef") that render the includes irrelevant. More pre‐
118 cisely, these includes must be eliminated during preprocessing
119 for the actual configuration. Then the question of existence of
120 the header file is moot and the remote compilation is sound.
121 This is often the case if such includes are meant for unusual
122 configurations different from the actual configuration.
123 --no_force_dirs
125 Do not force the creation of all directories used in an include
126 path. May improve performance for some cases, but will break
127 builds which use include structures like "<foo/../file.h>" with‐
128 out including other files in foo/.
129 -v, --verify
131 Verify that files in CPP closure are contained in closure calcu‐
132 lated by include processor.
133 -w, --write_include_closure
135 Write a .d_approx file which lists all the included files calcu‐
136 lated by the include server; with -x, additionally write the
137 included files as calculated by CPP to a .d_exact file.
138 -x, --exact_analysis
140 Use CPP instead, do not omit system headers files.
141
144 The most likely messages and warnings to come from the include proces‐
145 sor are listed below.
146 Preprocessing locally. Include server not covering: Couldn't determine
148 default system include directories
149 To determine the default system header directories, the include
150 server runs the compiler once for each language needed during
151 its session. This message indicates that the compiler specified
152 to distcc is not present on the client.
153 Preprocessing locally. Include server not covering: Bailing out because
155 include server spent more than ...s user time handling request
156 In uncommon situations, the include server fails to analyze very
157 complicated macro expressions. The distcc client will use plain
158 distcc mode.
159 Warning: Filepath must be relative but isn't
161 The include server does not accept absolute filepaths, such as
162 /usr/include/stdio.h, in include directives, because there is no
163 guarantee that this header on the compilation server machine
164 will be the same as that on the client. The include server
165 gives up analyzing the include closure. The distcc client cannot
166 use pump-mode.
167 To overcome this problem in a not always reliable way, set the
169 environment variable INCLUDE_SERVER_ARGS='--unsafe_abso‐
170 lute_includes' when invoking the pump script to pass the
171 --unsafe_absolute_includes option to the include server.
172 Warning: Absolute filepath ... was IGNORED
174 The --unsafe_absolute_includes is in use. This situation hap‐
175 pens under the same circumstances as when "Filepath must be rel‐
176 ative but isn't" is issued, but in this case the include will
177 provide an answer to the distcc client.
178 Warning: Path '/PATH/FILE' changed/came into existence/no longer exists
180 These warnings are issued when using stat reset triggers.
181 Because /PATH/FILE changed, the include server clears its
182 caches; the new version of the file (or the lack of it) renders
183 the include analysis invalid. This message can usually be
184 ignored; it does signify a somewhat precarious use of files by
185 the build system. It is recommended to fix the build system so
186 that files are not rewritten.
187 Warning: For translation unit ..., lookup of file ... resolved to ...
189 whose realpath is ...
190 This warning occurs with --path_observation_re when a new real‐
191 path matching a source or header file is observed.
192
195 The interactions between the build system, distcc, and the include
196 server is somewhat complex. When a distcc commands receives a failing
197 compilation from the remote server it retries the compilation locally.
198 This section discusses the causes of discrepancies between remote and
199 local compilation. These are flagged by the demotion message:
200 __________Warning: ... pump-mode compilation(s) failed on server, but
202 succeeded locally.
203 __________Distcc-pump was demoted to plain mode. See the Distcc Dis‐
204 crepancy Symptoms section in the include_server(1) man page.
205 The pump script issues this message at the end of the build. This means
207 that for at least one distcc invocation a local compilation succeeded
208 after the remote compilation failed. Each distcc invocation for which
209 such a discrepancy occurred in turn also issues a message such as:
210 Warning: remote compilation of '...' failed, retried locally and got a
212 different result.
213 The demotion makes subsequent distcc invocations use plain distcc mode.
215 Thus preprocessing will take place on the local machine for the remain‐
216 der of the build. This technique prevents very slow builds where all
217 compilations end up on the local machine after failing remotely.
218 Of course, if the local compilations fails after the remote failures,
220 then the distcc invocation exits with the non-zero status of the local
221 compilation. The error messages printed are also those of the local
222 compilation.
223 The fallback behavior for distcc-pump mode to local compilation can be
225 disabled by setting the environment variable DISTCC_FALLBACK to 0,
226 which makes the distcc command fail as soon as the remote compilation
227 has failed. This setting is very useful for debugging why the remote
228 compilation went wrong, because now the output from the server will be
229 printed.
230 Next we discuss the possible causes of discrepancies.
232 The user changed a source or header file during the build.
234 This yields inconsistent results of course.
235 A source or header file changed during the build.
237 The build system rewrites a file. For Linux kernel 2.6, this
238 happens for 'include/linux/compile.h' and 'include/asm/asm-off‐
239 sets.h'. This condition is fixed by letting the include server
240 know that it must reset its caches when a stat of any of the
241 files changes. Practically, this is done by gathering the files
242 in a colon-separated list and then setting the
243 INCLUDE_SERVER_ARGS environment variable when invoking the pump
244 script, so that it passes the --stat_reset_triggers option; for
245 example,
246 INCLUDE_SERVER_ARGS="--stat_reset_triggers=include/linux/com‐
248 pile.h:include/asm/asm-offsets.h"
249 A header file is potentially included, but does not exist, and is then
251 later included.
252 This occurs when some header foo.h includes another header file
253 trick.h, but the trick.h file has not yet been generated and the
254 inclusion is actually ignored because of preprocessing direc‐
255 tives. The include server will probe for the existence of
256 trick.h, because it overapproximates all possible ways direc‐
257 tives actually evaluate. The file trick.h is determined not to
258 exist. If it is later generated, and then really included, then
259 the include server will falsely believe that the file still does
260 not exist. The solution to this problem is to make the build
261 system generate trick.h before the first time any header file is
262 included that makes a syntactic reference to trick.h
263 The include server was started with --unsafe_absolute_includes.
265 This is a problem if there are header files locally that do not
266 exist remotely and that are actually used. Such includes are
267 often protected by conditional directives that evaluate so that
268 are actually used on only specific and often uncommon platforms.
269 If you are not compiling for such a platform, then it may be
270 correct to use --unsafe_absolute_include.
271 The include server has calculated the wrong includes.
273 We do not know of such a situation.
274
278 The exit code of include_server.py is usually 0. That the include
279 server has been started properly is communicated through the existence
280 of the pid_file.
281
284 DISTCC_EMAILLOG_WHOM_TO_BLAME The email address to use for include
285 server automated emails. The default is 'distcc-pump-errors' (which is
286 an email address that probably will not exist in your domain).
287 Additionally, the invocation of the compiler may use additional envi‐
289 ronment variables.
290
294 If you think you have found a distcc bug, please see the file report‐
295 ing-bugs.txt in the documentation directory for information on how to
296 report it.
297 In distcc-pump mode, the include server is unable to handle certain
300 very complicated computed includes as found in parts of the Boost
301 library. The include server will time out and distcc will revert to
302 plain mode.
303 Other known bugs may be documented on http://code.google.com/p/distcc/
305
308 The include server was written by Nils Klarlund, with assistance from
309 Fergus Henderson, Manos Renieris, and Craig Silverstein. Please report
310 bugs to <distcc@lists.samba.org>.
311
314 You are free to use distcc. distcc (including this manual) may be
315 copied, modified or distributed only under the terms of the GNU General
316 Public Licence version 2 or later. distcc comes with absolutely no
317 warrany. A copy of the GPL is included in the file COPYING.
318
321 distcc(1), distccd(1), include_server(1), and gcc(1).
322 http://code.google.com/p/distcc/ https://ccache.dev/
323 9 June 2008 include_server(1)