1KLOGD(8) Linux System Administration KLOGD(8)
2
6 klogd - Kernel Log Daemon
7
9 klogd [ -c n ] [ -d ] [ -f fname ] [ -iI ] [ -n ] [ -o ] [ -p ] [ -s ]
10 [ -k fname ] [ -v ] [ -x ] [ -2 ]
11
13 klogd is a system daemon which intercepts and logs Linux kernel mes‐
14 sages.
15
17 -c n Sets the default log level of console messages to n.
18 -d Enable debugging mode. This will generate LOTS of output to
20 stderr.
21 -f file
23 Log messages to the specified filename rather than to the syslog
24 facility.
25 -i -I Signal the currently executing klogd daemon. Both of these
27 switches control the loading/reloading of symbol information.
28 The -i switch signals the daemon to reload the kernel module
29 symbols. The -I switch signals for a reload of both the static
30 kernel symbols and the kernel module symbols.
31 -n Avoid auto-backgrounding. This is needed especially if the
33 klogd is started and controlled by init(8).
34 -o Execute in 'one-shot' mode. This causes klogd to read and log
36 all the messages that are found in the kernel message buffers.
37 After a single read and log cycle the daemon exits.
38 -p Enable paranoia. This option controls when klogd loads kernel
40 module symbol information. Setting this switch causes klogd to
41 load the kernel module symbol information whenever an Oops
42 string is detected in the kernel message stream.
43 -s Force klogd to use the system call interface to the kernel mes‐
45 sage buffers.
46 -k file
48 Use the specified file as the source of kernel symbol informa‐
49 tion.
50 -v Print version and exit.
52 -x Omits EIP translation and therefore doesn't read the System.map
54 file.
55 -2 When symbols are expanded, print the line twice. Once with
57 addresses converted to symbols, once with the raw text. This
58 allows external programs such as ksymoops do their own process‐
59 ing on the original data.
60
62 The functionality of klogd has been typically incorporated into other
63 versions of syslogd but this seems to be a poor place for it. In the
64 modern Linux kernel a number of kernel messaging issues such as sourc‐
65 ing, prioritization and resolution of kernel addresses must be
66 addressed. Incorporating kernel logging into a separate process offers
67 a cleaner separation of services.
68 In Linux there are two potential sources of kernel log information: the
70 /proc file system and the syscall (sys_syslog) interface, although
71 ultimately they are one and the same. Klogd is designed to choose
72 whichever source of information is the most appropriate. It does this
73 by first checking for the presence of a mounted /proc file system. If
74 this is found the /proc/kmsg file is used as the source of kernel log
75 information. If the proc file system is not mounted klogd uses a sys‐
76 tem call to obtain kernel messages. The command line switch (-s) can
77 be used to force klogd to use the system call interface as its messag‐
78 ing source.
79 If kernel messages are directed through the syslogd daemon the klogd
81 daemon, as of version 1.1, has the ability to properly prioritize ker‐
82 nel messages. Prioritization of the kernel messages was added to it at
83 approximately version 0.99pl13 of the kernel. The raw kernel messages
84 are of the form:
85 <[0-7]>Something said by the kernel.
87 The priority of the kernel message is encoded as a single numeric digit
89 enclosed inside the <> pair. The definitions of these values is given
90 in the kernel include file kernel.h. When a message is received from
91 the kernel the klogd daemon reads this priority level and assigns the
92 appropriate priority level to the syslog message. If file output (-f)
93 is used the prioritization sequence is left pre-pended to the kernel
94 message.
95 The klogd daemon can also be used in a 'one-shot' mode for reading the
97 kernel message buffers. One shot mode is selected by specifying the -o
98 switch on the command line. Output will be directed to either the sys‐
99 logd daemon or to an alternate file specified by the -f switch.
100 For example, to read all the kernel messages after a system boot
102 and record them in a file called krnl.msg the following command
103 would be given.
104 klogd -o -f ./krnl.msg
106
108 If the kernel detects an internal error condition a general protection
109 fault will be triggered. As part of the GPF handling procedure the
110 kernel prints out a status report indicating the state of the processor
111 at the time of the fault. Included in this display are the contents of
112 the microprocessor's registers, the contents of the kernel stack and a
113 tracing of what functions were being executed at the time of the fault.
114 This information is EXTREMELY IMPORTANT in determining what caused the
116 internal error condition. The difficulty comes when a kernel developer
117 attempts to analyze this information. The raw numeric information
118 present in the protection fault printout is of very little use to the
119 developers. This is due to the fact that kernels are not identical and
120 the addresses of variable locations or functions will not be the same
121 in all kernels. In order to correctly diagnose the cause of failure a
122 kernel developer needs to know what specific kernel functions or vari‐
123 able locations were involved in the error.
124 As part of the kernel compilation process a listing is created which
126 specified the address locations of important variables and function in
127 the kernel being compiled. This listing is saved in a file called Sys‐
128 tem.map in the top of the kernel directory source tree. Using this
129 listing a kernel developer can determine exactly what the kernel was
130 doing when the error condition occurred.
131 The process of resolving the numeric addresses from the protection
133 fault printout can be done manually or by using the ksymoops program
134 which is included in the kernel sources.
135 As a convenience klogd will attempt to resolve kernel numeric addresses
137 to their symbolic forms if a kernel symbol table is available at execu‐
138 tion time. If you require the original address of the symbol, use the
139 -2 switch to preserve the numeric address. A symbol table may be spec‐
140 ified by using the -k switch on the command line. If a symbol file is
141 not explicitly specified the following filenames will be tried:
142 /boot/System.map
144 /System.map
145 /usr/src/linux/System.map
146 Version information is supplied in the system maps as of kernel 1.3.43.
148 This version information is used to direct an intelligent search of the
149 list of symbol tables. This feature is useful since it provides sup‐
150 port for both production and experimental kernels.
151 For example a production kernel may have its map file stored in
153 /boot/System.map. If an experimental or test kernel is compiled with
154 the sources in the 'standard' location of /usr/src/linux the system map
155 will be found in /usr/src/linux/System.map. When klogd starts under
156 the experimental kernel the map in /boot/System.map will be bypassed in
157 favor of the map in /usr/src/linux/System.map.
158 Modern kernels as of 1.3.43 properly format important kernel addresses
160 so that they will be recognized and translated by klogd. Earlier ker‐
161 nels require a source code patch be applied to the kernel sources.
162 This patch is supplied with the sysklogd sources.
163 The process of analyzing kernel protections faults works very well with
165 a static kernel. Additional difficulties are encountered when attempt‐
166 ing to diagnose errors which occur in loadable kernel modules. Load‐
167 able kernel modules are used to implement kernel functionality in a
168 form which can be loaded or unloaded at will. The use of loadable mod‐
169 ules is useful from a debugging standpoint and can also be useful in
170 decreasing the amount of memory required by a kernel.
171 The difficulty with diagnosing errors in loadable modules is due to the
173 dynamic nature of the kernel modules. When a module is loaded the ker‐
174 nel will allocate memory to hold the module, when the module is
175 unloaded this memory will be returned back to the kernel. This dynamic
176 memory allocation makes it impossible to produce a map file which
177 details the addresses of the variable and functions in a kernel load‐
178 able module. Without this location map it is not possible for a kernel
179 developer to determine what went wrong if a protection fault involves a
180 kernel module.
181 klogd has support for dealing with the problem of diagnosing protection
183 faults in kernel loadable modules. At program start time or in
184 response to a signal the daemon will interrogate the kernel for a list‐
185 ing of all modules loaded and the addresses in memory they are loaded
186 at. Individual modules can also register the locations of important
187 functions when the module is loaded. The addresses of these exported
188 symbols are also determined during this interrogation process.
189 When a protection fault occurs an attempt will be made to resolve ker‐
191 nel addresses from the static symbol table. If this fails the symbols
192 from the currently loaded modules are examined in an attempt to resolve
193 the addresses. At the very minimum this allows klogd to indicate which
194 loadable module was responsible for generating the protection fault.
195 Additional information may be available if the module developer chose
196 to export symbol information from the module.
197 Proper and accurate resolution of addresses in kernel modules requires
199 that klogd be informed whenever the kernel module status changes. The
200 -i and -I switches can be used to signal the currently executing daemon
201 that symbol information be reloaded. Of most importance to proper res‐
202 olution of module symbols is the -i switch. Each time a kernel module
203 is loaded or removed from the kernel the following command should be
204 executed:
205 klogd -i
207 The -p switch can also be used to insure that module symbol information
209 is up to date. This switch instructs klogd to reload the module symbol
210 information whenever a protection fault is detected. Caution should be
211 used before invoking the program in 'paranoid' mode. The stability of
212 the kernel and the operating environment is always under question when
213 a protection fault occurs. Since the klogd daemon must execute system
214 calls in order to read the module symbol information there is the pos‐
215 sibility that the system may be too unstable to capture useful informa‐
216 tion. A much better policy is to insure that klogd is updated whenever
217 a module is loaded or unloaded. Having uptodate symbol information
218 loaded increases the probability of properly resolving a protection
219 fault if it should occur.
220 Included in the sysklogd source distribution is a patch to the mod‐
222 ules-2.0.0 package which allows the insmod, rmmod and modprobe utili‐
223 ties to automatically signal klogd whenever a module is inserted or
224 removed from the kernel. Using this patch will insure that the symbol
225 information maintained in klogd is always consistent with the current
226 kernel state.
227
229 The klogd daemon allows the ability to alter the presentation of kernel
230 messages to the system console. Consequent with the prioritization of
231 kernel messages was the inclusion of default messaging levels for the
232 kernel. In a stock kernel the the default console log level is set to
233 7. Any messages with a priority level numerically lower than 7 (higher
234 priority) appear on the console.
235 Messages of priority level 7 are considered to be 'debug' messages and
237 will thus not appear on the console. Many administrators, particularly
238 in a multi-user environment, prefer that all kernel messages be handled
239 by klogd and either directed to a file or to the syslogd daemon. This
240 prevents 'nuisance' messages such as line printer out of paper or disk
241 change detected from cluttering the console.
242 When -c is given on the commandline the klogd daemon will execute a
244 system call to inhibit all kernel messages from being displayed on the
245 console. Former versions always issued this system call and defaulted
246 to all kernel messages except for panics. This is handled differently
247 currently so klogd doesn't need to set this value anymore. The argu‐
248 ment given to the -c switch specifies the priority level of messages
249 which will be directed to the console. Note that messages of a prior‐
250 ity value LOWER than the indicated number will be directed to the con‐
251 sole.
252 For example, to have the kernel display all messages with a pri‐
254 ority level of 3 (KERN_ERR) or more severe the following command
255 would be executed:
256 klogd -c 4
258 The definitions of the numeric values for kernel messages are given in
260 the file kernel.h which can be found in the /usr/include/linux direc‐
261 tory if the kernel sources are installed. These values parallel the
262 syslog priority values which are defined in the file syslog.h found in
263 the /usr/include/sys sub-directory.
264 The console log level is usually configured with the sysctl(8) program,
266 directly or via its configuration file /etc/sysctl.conf. In this file
267 the following line
268 kernel.printk = 4 4 1 7
270 corresponds to the sampe setting above.
272
274 The klogd will respond to eight signals: SIGHUP, SIGINT, SIGKILL,
275 SIGTERM, SIGTSTP, SIGUSR1, SIGUSR2 and SIGCONT. The SIGINT, SIGKILL,
276 SIGTERM and SIGHUP signals will cause the daemon to close its kernel
277 log sources and terminate gracefully.
278 The SIGTSTP and SIGCONT signals are used to start and stop kernel log‐
280 ging. Upon receipt of a SIGTSTP signal the daemon will close its log
281 sources and spin in an idle loop. Subsequent receipt of a SIGCONT sig‐
282 nal will cause the daemon to go through its initialization sequence and
283 re-choose an input source. Using SIGSTOP and SIGCONT in combination
284 the kernel log input can be re-chosen without stopping and restarting
285 the daemon. For example if the /proc file system is to be un-mounted
286 the following command sequence should be used:
287 # kill -TSTP pid
289 # umount /proc
290 # kill -CONT pid
291 Notations will be made in the system logs with LOG_INFO priority docu‐
293 menting the start/stop of logging.
294 The SIGUSR1 and SIGUSR2 signals are used to initiate loading/reloading
296 of kernel symbol information. Receipt of the SIGUSR1 signal will cause
297 the kernel module symbols to be reloaded. Signaling the daemon with
298 SIGUSR2 will cause both the static kernel symbols and the kernel module
299 symbols to be reloaded.
300 Provided that the System.map file is placed in an appropriate location
302 the signal of generally greatest usefulness is the SIGUSR1 signal.
303 This signal is designed to be used to signal the daemon when kernel
304 modules are loaded/unloaded. Sending this signal to the daemon after a
305 kernel module state change will insure that proper resolution of sym‐
306 bols will occur if a protection fault occurs in the address space occu‐
307 pied by a kernel module.
308
310 /proc/kmsg
311 One Source for kernel messages klogd
312 /var/run/klogd.pid
313 The file containing the process id of klogd
314 /boot/System.map, /System.map, /usr/src/linux/System.map
315 Default locations for kernel system maps.
316
318 Probably numerous. Well formed context diffs appreciated.
319
321 The kernel log daemon klogd was originally written by Steve Lord
322 <lord@cray.com>, Greg Wettstein made major improvements. Martin
323 Schulze <joey@infodrom.org> fixed some bugs and took over maintenance.
324Version 1.5 27 May 2007 KLOGD(8)