1KLOGD(8) Linux System Administration KLOGD(8)
2
6 klogd - Kernel Log Daemon
7
9 klogd [ -c n ] [ -d ] [ -f fname ] [ -iI ] [ -n ] [ -o ] [ -p ] [ -s ]
10 [ -k fname ] [ -v ] [ -x ] [ -2 ]
11
13 klogd is a system daemon which intercepts and logs Linux kernel mes‐
14 sages.
15
17 -c n Sets the default log level of console messages to n.
18 -d Enable debugging mode. This will generate LOTS of output to
20 stderr.
21 -f file
23 Log messages to the specified filename rather than to the syslog
24 facility.
25 -i -I Signal the currently executing klogd daemon. Both of these
27 switches control the loading/reloading of symbol information.
28 The -i switch signals the daemon to reload the kernel module
29 symbols. The -I switch signals for a reload of both the static
30 kernel symbols and the kernel module symbols.
31 -n Avoid auto-backgrounding. This is needed especially if the
33 klogd is started and controlled by init(8).
34 -o Execute in 'one-shot' mode. This causes klogd to read and log
36 all the messages that are found in the kernel message buffers.
37 After a single read and log cycle the daemon exits.
38 -p Enable paranoia. This option controls when klogd loads kernel
40 module symbol information. Setting this switch causes klogd to
41 load the kernel module symbol information whenever an Oops
42 string is detected in the kernel message stream.
43 -s Force klogd to use the system call interface to the kernel mes‐
45 sage buffers.
46 -k file
48 Use the specified file as the source of kernel symbol informa‐
49 tion.
50 -v Print version and exit.
52 -x Omits EIP translation and therefore doesn't read the System.map
54 file.
55 -2 When symbols are expanded, print the line twice. Once with
57 addresses converted to symbols, once with the raw text. This
58 allows external programs such as ksymoops do their own process‐
59 ing on the original data.
60
62 The functionality of klogd has been typically incorporated into other
63 versions of syslogd but this seems to be a poor place for it. In the
64 modern Linux kernel a number of kernel messaging issues such as sourc‐
65 ing, prioritization and resolution of kernel addresses must be
66 addressed. Incorporating kernel logging into a separate process offers
67 a cleaner separation of services.
68 In Linux there are two potential sources of kernel log information: the
70 /proc file system and the syscall (sys_syslog) interface, although
71 ultimately they are one and the same. Klogd is designed to choose
72 whichever source of information is the most appropriate. It does this
73 by first checking for the presence of a mounted /proc file system. If
74 this is found the /proc/kmsg file is used as the source of kernel log
75 information. If the proc file system is not mounted klogd uses a sys‐
76 tem call to obtain kernel messages. The command line switch (-s) can
77 be used to force klogd to use the system call interface as its messag‐
78 ing source.
79 If kernel messages are directed through the syslogd daemon the klogd
81 daemon, as of version 1.1, has the ability to properly prioritize ker‐
82 nel messages. Prioritization of the kernel messages was added to it at
83 approximately version 0.99pl13 of the kernel. The raw kernel messages
84 are of the form:
85 <[0-7]>Something said by the kernel.
87 The priority of the kernel message is encoded as a single numeric digit
89 enclosed inside the <> pair. The definitions of these values is given
90 in the kernel include file kernel.h. When a message is received from
91 the kernel the klogd daemon reads this priority level and assigns the
92 appropriate priority level to the syslog message. If file output (-f)
93 is used the prioritization sequence is left pre-pended to the kernel
94 message.
95 The klogd daemon also allows the ability to alter the presentation of
97 kernel messages to the system console. Consequent with the prioritiza‐
98 tion of kernel messages was the inclusion of default messaging levels
99 for the kernel. In a stock kernel the the default console log level is
100 set to 7. Any messages with a priority level numerically lower than 7
101 (higher priority) appear on the console.
102 Messages of priority level 7 are considered to be 'debug' messages and
104 will thus not appear on the console. Many administrators, particularly
105 in a multi-user environment, prefer that all kernel messages be handled
106 by klogd and either directed to a file or to the syslogd daemon. This
107 prevents 'nuisance' messages such as line printer out of paper or disk
108 change detected from cluttering the console.
109 When -c is given on the commandline the klogd daemon will execute a
111 system call to inhibit all kernel messages from being displayed on the
112 console. Former versions always issued this system call and defaulted
113 to all kernel messages except for panics. This is handled differently
114 nowardays so klogd doesn't need to set this value anymore. The argu‐
115 ment given to the -c switch specifies the priority level of messages
116 which will be directed to the console. Note that messages of a prior‐
117 ity value LOWER than the indicated number will be directed to the con‐
118 sole.
119 For example, to have the kernel display all messages with a pri‐
121 ority level of 3 (KERN_ERR) or more severe the following command
122 would be executed:
123 klogd -c 4
125 The definitions of the numeric values for kernel messages are given in
127 the file kernel.h which can be found in the /usr/include/linux direc‐
128 tory if the kernel sources are installed. These values parallel the
129 syslog priority values which are defined in the file syslog.h found in
130 the /usr/include/sys sub-directory.
131 The klogd daemon can also be used in a 'one-shot' mode for reading the
133 kernel message buffers. One shot mode is selected by specifying the -o
134 switch on the command line. Output will be directed to either the sys‐
135 logd daemon or to an alternate file specified by the -f switch.
136 For example, to read all the kernel messages after a system boot
138 and record them in a file called krnl.msg the following command
139 would be given.
140 klogd -o -f ./krnl.msg
142
144 If the kernel detects an internal error condition a general protection
145 fault will be triggered. As part of the GPF handling procedure the
146 kernel prints out a status report indicating the state of the processor
147 at the time of the fault. Included in this display are the contents of
148 the microprocessor's registers, the contents of the kernel stack and a
149 tracing of what functions were being executed at the time of the fault.
150 This information is EXTREMELY IMPORTANT in determining what caused the
152 internal error condition. The difficulty comes when a kernel developer
153 attempts to analyze this information. The raw numeric information
154 present in the protection fault printout is of very little use to the
155 developers. This is due to the fact that kernels are not identical and
156 the addresses of variable locations or functions will not be the same
157 in all kernels. In order to correctly diagnose the cause of failure a
158 kernel developer needs to know what specific kernel functions or vari‐
159 able locations were involved in the error.
160 As part of the kernel compilation process a listing is created which
162 specified the address locations of important variables and function in
163 the kernel being compiled. This listing is saved in a file called Sys‐
164 tem.map in the top of the kernel directory source tree. Using this
165 listing a kernel developer can determine exactly what the kernel was
166 doing when the error condition occurred.
167 The process of resolving the numeric addresses from the protection
169 fault printout can be done manually or by using the ksymoops program
170 which is included in the kernel sources.
171 As a convenience klogd will attempt to resolve kernel numeric addresses
173 to their symbolic forms if a kernel symbol table is available at execu‐
174 tion time. If you require the original address of the symbol, use the
175 -2 switch to preserve the numeric address. A symbol table may be spec‐
176 ified by using the -k switch on the command line. If a symbol file is
177 not explicitly specified the following filenames will be tried:
178 /boot/System.map
180 /System.map
181 /usr/src/linux/System.map
182 Version information is supplied in the system maps as of kernel 1.3.43.
184 This version information is used to direct an intelligent search of the
185 list of symbol tables. This feature is useful since it provides sup‐
186 port for both production and experimental kernels.
187 For example a production kernel may have its map file stored in
189 /boot/System.map. If an experimental or test kernel is compiled with
190 the sources in the 'standard' location of /usr/src/linux the system map
191 will be found in /usr/src/linux/System.map. When klogd starts under
192 the experimental kernel the map in /boot/System.map will be bypassed in
193 favor of the map in /usr/src/linux/System.map.
194 Modern kernels as of 1.3.43 properly format important kernel addresses
196 so that they will be recognized and translated by klogd. Earlier ker‐
197 nels require a source code patch be applied to the kernel sources.
198 This patch is supplied with the sysklogd sources.
199 The process of analyzing kernel protections faults works very well with
201 a static kernel. Additional difficulties are encountered when attempt‐
202 ing to diagnose errors which occur in loadable kernel modules. Load‐
203 able kernel modules are used to implement kernel functionality in a
204 form which can be loaded or unloaded at will. The use of loadable mod‐
205 ules is useful from a debugging standpoint and can also be useful in
206 decreasing the amount of memory required by a kernel.
207 The difficulty with diagnosing errors in loadable modules is due to the
209 dynamic nature of the kernel modules. When a module is loaded the ker‐
210 nel will allocate memory to hold the module, when the module is
211 unloaded this memory will be returned back to the kernel. This dynamic
212 memory allocation makes it impossible to produce a map file which
213 details the addresses of the variable and functions in a kernel load‐
214 able module. Without this location map it is not possible for a kernel
215 developer to determine what went wrong if a protection fault involves a
216 kernel module.
217 klogd has support for dealing with the problem of diagnosing protection
219 faults in kernel loadable modules. At program start time or in
220 response to a signal the daemon will interrogate the kernel for a list‐
221 ing of all modules loaded and the addresses in memory they are loaded
222 at. Individual modules can also register the locations of important
223 functions when the module is loaded. The addresses of these exported
224 symbols are also determined during this interrogation process.
225 When a protection fault occurs an attempt will be made to resolve ker‐
227 nel addresses from the static symbol table. If this fails the symbols
228 from the currently loaded modules are examined in an attempt to resolve
229 the addresses. At the very minimum this allows klogd to indicate which
230 loadable module was responsible for generating the protection fault.
231 Additional information may be available if the module developer chose
232 to export symbol information from the module.
233 Proper and accurate resolution of addresses in kernel modules requires
235 that klogd be informed whenever the kernel module status changes. The
236 -i and -I switches can be used to signal the currently executing daemon
237 that symbol information be reloaded. Of most importance to proper res‐
238 olution of module symbols is the -i switch. Each time a kernel module
239 is loaded or removed from the kernel the following command should be
240 executed:
241 klogd -i
243 The -p switch can also be used to insure that module symbol information
245 is up to date. This switch instructs klogd to reload the module symbol
246 information whenever a protection fault is detected. Caution should be
247 used before invoking the program in ´paranoid´ mode. The stability of
248 the kernel and the operating environment is always under question when
249 a protection fault occurs. Since the klogd daemon must execute system
250 calls in order to read the module symbol information there is the pos‐
251 sibility that the system may be too unstable to capture useful informa‐
252 tion. A much better policy is to insure that klogd is updated whenever
253 a module is loaded or unloaded. Having uptodate symbol information
254 loaded increases the probability of properly resolving a protection
255 fault if it should occur.
256 Included in the sysklogd source distribution is a patch to the mod‐
258 ules-2.0.0 package which allows the insmod, rmmod and modprobe utili‐
259 ties to automatically signal klogd whenever a module is inserted or
260 removed from the kernel. Using this patch will insure that the symbol
261 information maintained in klogd is always consistent with the current
262 kernel state.
263
265 The klogd will respond to eight signals: SIGHUP, SIGINT, SIGKILL,
266 SIGTERM, SIGTSTP, SIGUSR1, SIGUSR2 and SIGCONT. The SIGINT, SIGKILL,
267 SIGTERM and SIGHUP signals will cause the daemon to close its kernel
268 log sources and terminate gracefully.
269 The SIGTSTP and SIGCONT signals are used to start and stop kernel log‐
271 ging. Upon receipt of a SIGTSTP signal the daemon will close its log
272 sources and spin in an idle loop. Subsequent receipt of a SIGCONT sig‐
273 nal will cause the daemon to go through its initialization sequence and
274 re-choose an input source. Using SIGSTOP and SIGCONT in combination
275 the kernel log input can be re-chosen without stopping and restarting
276 the daemon. For example if the /proc file system is to be un-mounted
277 the following command sequence should be used:
278 # kill -TSTP pid
280 # umount /proc
281 # kill -CONT pid
282 Notations will be made in the system logs with LOG_INFO priority docu‐
284 menting the start/stop of logging.
285 The SIGUSR1 and SIGUSR2 signals are used to initiate loading/reloading
287 of kernel symbol information. Receipt of the SIGUSR1 signal will cause
288 the kernel module symbols to be reloaded. Signaling the daemon with
289 SIGUSR2 will cause both the static kernel symbols and the kernel module
290 symbols to be reloaded.
291 Provided that the System.map file is placed in an appropriate location
293 the signal of generally greatest usefulness is the SIGUSR1 signal.
294 This signal is designed to be used to signal the daemon when kernel
295 modules are loaded/unloaded. Sending this signal to the daemon after a
296 kernel module state change will insure that proper resolution of sym‐
297 bols will occur if a protection fault occurs in the address space occu‐
298 pied by a kernel module.
299
301 /proc/kmsg
302 One Source for kernel messages klogd
303 /var/run/klogd.pid
304 The file containing the process id of klogd
305 /boot/System.map, /System.map, /usr/src/linux/System.map
306 Default locations for kernel system maps.
307
309 Probably numerous. Well formed context diffs appreciated.
310
312 The klogd was originally written by Steve Lord (lord@cray.com), Greg
313 Wettstein made major improvements.
314 Dr. Greg Wettstein (greg@wind.enjellic.com)
316 Enjellic Systems Development
317 Oncology Research Divsion Computing Facility
319 Roger Maris Cancer Center
320 Fargo, ND 58122
321Version 1.4 21 August, 1999 KLOGD(8)