1ICECREAM(7) Icecream User's Manual ICECREAM(7)
2
6 Icecream - A distributed compile system
7
9 Icecream is a distributed compile system for C and C++.
10 Icecream is created by SUSE and is based on ideas and code by distcc.
13 Like distcc it takes compile jobs from your (KDE) build and distributes
14 it to remote machines allowing a parallel build on several machines
15 you've got. But unlike distcc Icecream uses a central server that
16 schedules the compile jobs to the fastest free server and is as this
17 dynamic. This advantage pays off mostly for shared computers, if you're
18 the only user on x machines, you have full control over them anyway.
19
22 You need:
23 · One machine that runs the scheduler ("./scheduler -d")
26 · Many machines that run the daemon ("./iceccd -d")
28 If you want to compile using icecream, make sure $prefix/bin is the
30 first first entry in your path, e.g. type export PATH=/opt/ice‐
31 cream/bin:$PATH (Hint: put this in ~/.bashrc or /etc/profile to not
32 have to type it in everytime)
33 Then you just compile with make -j <num>, where <num> is the amount of
36 jobs you want to compile in parallel. Don't exaggerate. Numbers greater
37 than 15 normally cause trouble.
38 WARNING: Never use icecream in untrusted environments. Run the deamons
41 and the scheduler as unpriviliged user in such networks if you have to!
42 But you will have to rely on homogeneous networks then (see below).
43 If you want funny stats, you might want to run "icemon".
46
49 If you are running icecream daemons (note: they _all_ must be running
50 as root. In the future icecream might gain the ability to know when
51 machines can't accept a different env, but for now it is all or nothing
52 ) in the same icecream network but on machines with incompatible com‐
53 piler versions you have to tell icecream which environment you are
54 using. Use icecc --build-native to create an archive file containing
55 all the files necessary to setup the compiler environment. The file
56 will have a random unique name like
57 "ddaea39ca1a7c88522b185eca04da2d8.tar.bz2" per default. Rename it to
58 something more expressive for your convenience, e.g.
59 "i386-3.3.1.tar.bz2". Set ICECC_VERSION=<filename_of_archive_contain‐
60 ing_your_environment> in the shell environment where you start the com‐
61 pile jobs and the file will be transfered to the daemons where your
62 compile jobs run and installed to a chroot environment for executing
63 the compile jobs in the environment fitting to the environment of the
64 client. This requires that the icecream deamon runs as root.
65 If you do not set ICECC_VERSION, the client will use a tar ball pro‐
68 vided by the daemon running on the same machine. So you can always be
69 sure you're not tricked by incompatible gcc versions - and you can
70 share your computer with users of other distributions (or different
71 versions of your beloved SUSE Linux :)
72
75 SUSE got quite some good machines not having a processor from Intel or
76 AMD, so icecream is pretty good in using cross-compiler environments
77 similiar to the above way of spreading compilers. There the ICECC_VER‐
78 SION varaible looks like <native_filename>(,<platform>:<cross_com‐
79 piler_filename>)*, for example like this:
80 /work/9.1-i386.tar.bz2,ia64:/work/9.1-cross-ia64.tar.bz2
81 How to package such a cross compiler is pretty straightforward if you
84 look what's inside the tarballs generated by icecc --build-native.
85
88 When building for embedded targets like ARM often you'll have a
89 toolchain that runs on your host and produces code for the target. In
90 these situations you can exploit the power of icecream as well.
91 Create symlinks from where icecc is to the name of your cross compilers
94 (e.g. arm-linux-g++ and arm-linux-gcc), make sure that these symlinks
95 are in the path and before the path of your toolchain, with $ICECC_CC
96 and $ICECC_CXX you need to tell icecream which compilers to use for
97 preprocessing and local compiling. e.g. set it to
98 ICECC_CC=arm-linux-gcc and ICECC_CXX=arm-linux-g++.
99 As the next step you need to create a .tar.bz2 of your cross compiler,
102 check the result of build-native to see what needs to be present.
103 Finally one needs to set ICECC_VERSION and point it to the tar.bz2
106 you've created. When you start compiling your toolchain will be used.
107 NOTE: with ICECC_VERSION you point out on which platforms your
110 toolchain runs, you do not indicate for which target code will be gen‐
111 erated.
112
115 The easiest way to use ccache with icecream is putting the symlink mas‐
116 querades into /opt/icream/bin and putting small wrapper scripts in
117 /opt/ccache/bin
118 cat /opt/ccache/bin/g++:
121 #! /bin/sh
123 export CCACHE_PATH=/opt/icecream/bin
125 export PATH=/opt/icecream/bin:/usr/bin:$PATH
126 ccache g++ "$@"
127 Then you can replace /opt/icecream/bin with /opt/ccache/bin in your
131 $PATH and all icecream calls will go through ccache (and Qt will com‐
132 pile in 62s :)
133 Note however that ccache isn't really worth the trouble if you're not
136 recompiling your KDE three times a day from scratch (it adds quite some
137 overhead in comparing the preprocessor output and uses quite some disc
138 space and I found a cache hit of 18% a bit too few, so I disabled it
139 again).
140
143 You can use the environment variable ICECC_DEBUG to control if icecream
144 gives debug output or not. Set it to debug to get debug output. The
145 other possible values are error, warning and info (the -v option for
146 daemon and scheduler raise the level per -v on the command line - so
147 use -vvv for full debug).
148
151 Numbers of my test case (some STL C++ genetic algorithm)
152 · g++ on my machine: 1.6s
155 · g++ on fast machine: 1.1s
157 · icecream using my machine as remote machine: 1.9s
159 · icecream using fast machine: 1.8s
161 The icecream overhead is quite huge as you might notice, but the com‐
163 piler can't interleave preprocessing with compilation and the file
164 needs to be read/written once more and in between the file is trans‐
165 fered.
166 But even if the other computer is faster, using g++ on my local machine
169 is faster. If you're (for whatever reason) alone in your network at
170 some point, you loose all advantages of distributed compiling and only
171 add the overhead. So icecream got a special case for local compilations
172 (the same special meaning that localhost got within $DISTCC_HOSTS).
173 This makes compiling on my machine using icecream down to 1.7s (the
174 overhead is actually less than 0.1s in average).
175 As the scheduler is aware of that meaning, it will prefer your own com‐
178 puter if it's free and got not less than 70% of the fastest available
179 computer.
180 Keep in mind, that this affects only the first compile job, the second
183 one is distributed anyway. So if I had to compile two of my files, I
184 would get
185 · g++ -j1 on my machine: 3.2s
188 · g++ -j1 on the fast machine: 2.2s
190 · using icecream -j2 on my machine: max(1.7,1.8)=1.8s
192 · (using icecream -j2 on the other machine: max(1.1,1.8)=1.8s)
194 The math is a bit tricky and depends a lot on the current state of the
196 compilation network, but make sure you're not blindly assuming make -j2
197 halfs your compilation time.
198
201 In most requirements icecream isn't special, e.g. it doesn't matter
202 what distributed compile system you use, you won't have fun if your
203 nodes are connected through than less or equal to 10MBit. Note that
204 icecream compresses input and output files (using lzo), so you can calc
205 with ~1MBit per compile job - i.e more than make -j10 won't be possible
206 without delays.
207 Remember that more machines are only good if you can use massive paral‐
210 lelization, but you will for sure get the best result if your submit‐
211 ting machine (the one you called g++ on) will be fast enough to feed
212 the others. Especially if your project consists of many easy to compile
213 files, the preprocessing and file IO will be job enough to need a quick
214 machine.
215 The scheduler will try to give you the fastest machines available, so
218 even if you add old machines, they will be used only in exceptional
219 situations, but still you can have bad luck - the scheduler doesn't
220 know how long a job will take before it started. So if you have 3
221 machines and two quick to compile and one long to compile source file,
222 you're not safe from a choice where everyone has to wait on the slow
223 machine. Keep that in mind.
224
227 A short overview of the ports icecream requires:
228 · TCP/10245 on the daemon computers (required)
231 · TCP/8765 for the the scheduler computer (required)
233 · TCP/8766 for the telnet interface to the scheduler (optional)
235 · UDP/8765 for broadcast to find the scheduler (optional)
237 Note that the SuSEfirewall2 on SUSE < 9.1 got some problems configuring
239 broadcast. So you might need the -s option for the daemon in any case
240 there. If the monitor can't find the scheduler, use USE_SCHED‐
241 ULER=<host> icemon (or send me a patch :)
242
245 icecream, scheduler, iceccd, icemon
246
249 Stephan Kulow <coolo@suse.de>
250 Michael Matz <matz@suse.de>
253 Cornelius Schumacher <cschum@suse.de>
256 ...and various other contributors.
259Icecream April 21th, 2005 ICECREAM(7)