1CRASH(8) System Manager's Manual CRASH(8)
2
6 crash - Analyze Linux crash dump data or a live system
7
9 crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS] (dumpfile form)
10 crash [OPTION]... [NAMELIST] (live system form)
11
13 Crash is a tool for interactively analyzing the state of the Linux sys‐
14 tem while it is running, or after a kernel crash has occurred and a
15 core dump has been created by the netdump, diskdump, LKCD, kdump, xen‐
16 dump or kvmdump facilities. It is loosely based on the SVR4 UNIX crash
17 command, but has been significantly enhanced by completely merging it
18 with the gdb(1) debugger. The marriage of the two effectively combines
19 the kernel-specific nature of the traditional UNIX crash utility with
20 the source code level debugging capabilities of gdb(1).
21 In the dumpfile form, both a NAMELIST and a MEMORY-IMAGE argument must
23 be entered. In the live system form, the NAMELIST argument must be
24 entered if the kernel's vmlinux file is not located in a known loca‐
25 tion, such as the /usr/lib/debug/lib/modules/<kernel-version> direc‐
26 tory.
27 The crash utility has also been extended to support the analysis of
29 dumpfiles generated by a crash of the Xen hypervisor. In that case,
30 the NAMELIST argument must be that of the xen-syms binary. Live system
31 analysis is not supported for the Xen hypervisor.
32 The crash utility command set consists of common kernel core analysis
34 tools such as kernel stack back traces of all processes, source code
35 disassembly, formatted kernel structure and variable displays, virtual
36 memory data, dumps of linked-lists, etc., along with several commands
37 that delve deeper into specific kernel subsystems. Appropriate gdb
38 commands may also be entered, which in turn are passed on to the gdb
39 module for execution. If desired, commands may be placed in either a
40 $HOME/.crashrc file and/or in a .crashrc file in the current directory.
41 During initialization, the commands in $HOME/.crashrc are executed
42 first, followed by those in the ./.crashrc file.
43 The crash utility is designed to be independent of Linux version depen‐
45 dencies. When new kernel source code impacts the correct functionality
46 of crash and its command set, the utility will be updated to recognize
47 new kernel code changes, while maintaining backwards compatibility with
48 earlier releases.
49
51 NAMELIST
52 This is a pathname to an uncompressed kernel image (a vmlinux
53 file), or a Xen hypervisor image (a xen-syms file) which has
54 been compiled with the "-g" option. If using the dumpfile form,
55 a vmlinux file may be compressed in either gzip or bzip2 for‐
56 mats.
57 MEMORY-IMAGE[@ADDRESS]
59 A kernel core dump file created by the netdump, diskdump, LKCD
60 kdump, xendump or kvmdump facilities.
61 If a MEMORY-IMAGE argument is not entered, the session will be
63 invoked on the live system, which typically requires root privi‐
64 leges because of the device file used to access system RAM. By
65 default, /dev/crash will be used if it exists. If it does not
66 exist, then /dev/mem will be used; but if the kernel has been
67 configured with CONFIG_STRICT_DEVMEM, then /proc/kcore will be
68 used. It is permissible to explicitly enter /dev/crash,
69 /dev/mem or /proc/kcore.
70 An @ADDRESS value must be appended to the MEMORY-IMAGE if the
72 dumpfile is a raw RAM dumpfile that has no header information
73 describing the file contents. Multiple MEMORY-IMAGE@ADDRESS
74 ordered pairs may be entered, with each dumpfile containing a
75 contiguous block of RAM, where the ADDRESS value is the physical
76 start address of the block expressed in hexadecimal. The physi‐
77 cal address value(s) will be used to create a temporary ELF
78 header in /var/tmp, which will only exist during the crash ses‐
79 sion.
80 mapfile
82 If the NAMELIST file is not the same kernel that is running
83 (live system form), or the kernel that was running when the sys‐
84 tem crashed (dumpfile form), then the System.map file of the
85 original kernel should be entered on the command line.
86 -h [option]
88 --help [option]
89 Without an option argument, display a crash usage help message.
90 If the option argument is a crash command name, the help page
91 for that command is displayed. If it is the string "input", a
92 page describing the various crash command line input options is
93 displayed. If it is the string "output", a page describing com‐
94 mand line output options is displayed. If it is the string
95 "all", then all of the possible help messages are displayed.
96 After the help message is displayed, crash exits.
97 -s Silently proceed directly to the "crash>" prompt without dis‐
99 playing any version, GPL, or crash initialization data during
100 startup, and by default, runtime command output is not passed to
101 any scrolling command.
102 -i file
104 Execute the command(s) contained in file prior to displaying the
105 "crash>" prompt for interactive user input.
106 -d num Set the internal debug level. The higher the number, the more
108 debugging data will be printed when crash initializes and runs.
109 -S Use /boot/System.map as the mapfile.
111 -e vi | emacs
113 Set the readline(3) command line editing mode to "vi" or
114 "emacs". The default editing mode is "vi".
115 -f Force the usage of a compressed vmlinux file if its original
117 name does not start with "vmlinux".
118 -k Indicate that the NAMELIST file is an LKCD "Kerntypes" debuginfo
120 file.
121 -g [namelist]
123 Determine if a vmlinux or xen-syms namelist file contains debug‐
124 ging data.
125 -t Display the system-crash timestamp and exit.
127 -L Attempt to lock all of its virtual address space into memory by
129 calling mlockall(MCL_CURRENT|MCL_FUTURE) during initialization.
130 If the system call fails, an error message will be displayed,
131 but the session continues.
132 -c tty-device
134 Open the tty-device as the console used for debug messages.
135 -p page-size
137 If a processor's page size cannot be determined by the dumpfile,
138 and the processor default cannot be used, use page-size.
139 -o filename
141 Only used with the MEMORY-IMAGE@ADDRESS format for raw RAM dump‐
142 files, specifies a filename of a new ELF vmcore that will be
143 created and used as the dumpfile. It will be saved to allow
144 future use as a standalone vmcore, replacing the original raw
145 RAM dumpfile.
146 -m option=value
148 --machdep option=value
149 Pass an option and value pair to machine-dependent code. These
150 architecture-specific option/pairs should only be required in
151 very rare circumstances:
152 X86_64:
154 phys_base=<physical-address>
155 irq_eframe_link=<value>
156 max_physmem_bits=<value>
157 vm=orig (pre-2.6.11 virtual memory address ranges)
158 vm=2.6.11 (2.6.11 and later virtual memory address ranges)
159 vm=xen (Xen kernel virtual memory address ranges)
160 vm=xen-rhel4 (RHEL4 Xen kernel virtual address ranges)
161 PPC64:
162 vm=orig
163 vm=2.6.14 (4-level page tables)
164 IA64:
165 phys_start=<physical-address>
166 init_stack_size=<size>
167 vm=4l (4-level page tables)
168 ARM:
169 phys_base=<physical-address>
170 ARM64:
171 phys_offset=<physical-address>
172 -x Automatically load extension modules from a particular direc‐
174 tory. If a directory is specified in the CRASH_EXTENSIONS shell
175 environment variable, then that directory will be used. Other‐
176 wise /usr/lib64/crash/extensions (64-bit architectures) or
177 /usr/lib/crash/extensions (32-bit architectures) will be used;
178 if they do not exist, then the ./extensions directory will be
179 used.
180 --active
182 Track only the active task on each cpu.
183 --buildinfo
185 Display the crash binary's build date, the user ID of the
186 builder, the hostname of the machine where the build was done,
187 the target architecture, the version number, and the compiler
188 version.
189 --memory_module modname
191 Use the modname as an alternative kernel module to the crash.ko
192 module that creates the /dev/crash device.
193 --memory_device device
195 Use device as an alternative device to the /dev/crash, /dev/mem
196 or /proc/kcore devices.
197 --log dumpfile
199 Dump the contents of the kernel log buffer. A kernel namelist
200 argument is not necessary, but the dumpfile must contain the
201 VMCOREINFO data taken from the original /proc/vmcore ELF header.
202 --no_kallsyms
204 Do not use kallsyms-generated symbol information contained
205 within kernel module object files.
206 --no_modules
208 Do not access or display any kernel module related information.
209 --no_ikconf
211 Do not attempt to read configuration data that was built into
212 kernels configured with CONFIG_IKCONFIG.
213 --no_data_debug
215 Do not verify the validity of all structure member offsets and
216 structure sizes that it uses.
217 --no_kmem_cache
219 Do not initialize the kernel's slab cache infrastructure, and
220 commands that use kmem_cache-related data will not work.
221 --no_elf_notes
223 Do not use the registers from the ELF NT_PRSTATUS notes saved in
224 a compressed kdump header for backtraces.
225 --kmem_cache_delay
227 Delay the initialization of the kernel's slab cache infrastruc‐
228 ture until it is required by a run-time command.
229 --readnow
231 Pass this flag to the embedded gdb module, which will override
232 its two-stage strategy that it uses for reading symbol tables
233 from the NAMELIST.
234 --smp Specify that the system being analyzed is an SMP kernel.
236 -v
238 --version
239 Display the version of the crash utility, the version of the
240 embedded gdb module, GPL information, and copyright notices.
241 --cpus number
243 Specify the number of cpus in the SMP system being analyzed.
244 --osrelease dumpfile
246 Display the OSRELEASE vmcoreinfo string from a kdump dumpfile
247 header.
248 --hyper
250 Force the session to be that of a Xen hypervisor.
251 --p2m_mfn pfn
253 When a Xen Hypervisor or its dom0 kernel crashes, the dumpfile
254 is typically analyzed with either the Xen hypervisor or the dom0
255 kernel. It is also possible to analyze any of the guest domU
256 kernels if the pfn_to_mfn_list_list pfn value of the guest ker‐
257 nel is passed on the command line along with its NAMELIST and
258 the dumpfile.
259 --xen_phys_start physical-address
261 Supply the base physical address of the Xen hypervisor's text
262 and static data for older xendump dumpfiles that did not pass
263 that information in the dumpfile header.
264 --zero_excluded
266 If the makedumpfile(8) facility has filtered a compressed kdump
267 dumpfile to exclude various types of non-essential pages, or has
268 marked a compressed or ELF kdump dumpfile as incomplete due to
269 an ENOSPC or other error during its creation, any attempt to
270 read missing pages will fail. With this flag, reads from any of
271 those pages will return zero-filled memory.
272 --no_panic
274 Do not attempt to find the task that was running when the kernel
275 crashed. Set the initial context to that of the "swapper" task
276 on cpu 0.
277 --more Use /bin/more as the command output scroller, overriding the
279 default of /usr/bin/less and any settings in either ./.crashrc
280 or $HOME/.crashrc.
281 --less Use /usr/bin/less as the command output scroller, overriding any
283 settings in either ./.crashrc or $HOME/.crashrc.
284 --hex Set the default command output radix to 16, overriding the
286 default radix of 10, and any radix settings in either ./.crashrc
287 or $HOME/.crashrc.
288 --dec Set the default command output radix to 10, overriding any radix
290 settings in either ./.crashrc or $HOME/.crashrc. This is the
291 default radix setting.
292 --CRASHPAGER
294 Use the output paging command defined in the CRASHPAGER shell
295 environment variable, overriding any settings in either
296 ./.crashrc or $HOME/.crashrc.
297 --no_scroll
299 Do not pass run-time command output to any scrolling command.
300 --no_strip
302 Do not strip cloned kernel text symbol names.
303 --no_crashrc
305 Do not execute the commands in either $HOME/.crashrc or
306 ./.crashrc.
307 --mod directory
309 When loading the debuginfo data of kernel modules with the mod
310 -S command, search for their object files in directory instead
311 of in the standard location.
312 --kaslr offset|auto
314 If an x86_64 kernel was configured with CONFIG_RANDOMIZE_BASE,
315 the offset value is equal to the difference between the symbol
316 values compiled into the vmlinux file and their relocated KASLR
317 values. If set to auto, the KASLR offset value will be automat‐
318 ically calculated.
319 --reloc size
321 When analyzing live x86 kernels that were configured with a CON‐
322 FIG_PHYSICAL_START value that is larger than its CONFIG_PHYSI‐
323 CAL_ALIGN value, then it will be necessary to enter a relocation
324 size equal to the difference between the two values.
325 --hash count
327 Set the number of internal hash queue heads used for list gath‐
328 ering and verification. The default count is 32768.
329 --minimal
331 Bring up a session that is restricted to the log, dis, rd, sym,
332 eval, set and exit commands. This option may provide a way to
333 extract some minimal/quick information from a corrupted or trun‐
334 cated dumpfile, or in situations where one of the several kernel
335 subsystem initialization routines would abort the crash session.
336 --kvmhost [32|64]
338 When examining an x86 KVM guest dumpfile, this option specifies
339 that the KVM host that created the dumpfile was an x86 (32-bit)
340 or an x86_64 (64-bit) machine, overriding the automatically
341 determined value.
342 --kvmio <size>
344 override the automatically-calculated KVM guest I/O hole size.
345 --offline [show|hide]
347 Show or hide command output that is related to offline cpus.
348 The default setting is show.
349
351 Each crash command generally falls into one of the following cate‐
352 gories:
353 Symbolic display
355 Displays of kernel text/data, which take full advantage of the
356 power of gdb to format and display data structures symbolically.
357 System state
359 The majority of crash commands consist of a set of "kernel-
360 aware" commands, which delve into various kernel subsystems on a
361 system-wide or per-task basis.
362 Utility functions
364 A set of useful helper commands serving various purposes, some
365 simple, others quite powerful.
366 Session control
368 Commands that control the crash session itself.
369 The following alphabetical list consists of a very simple overview of
371 each crash command. However, since individual commands often have sev‐
372 eral options resulting in significantly different output, it is sug‐
373 gested that the full description of each command be viewed by executing
374 crash -h <command>, or during a crash session by simply entering help
375 command.
376 * "pointer to" is shorthand for either the struct or union com‐
378 mands. It displays the contents of a kernel structure or union.
379 alias creates a single-word alias for a command.
381 ascii displays an ascii chart or translates a numeric value into its
383 ascii components.
384 bt displays a task's kernel-stack backtrace. If it is given the -a
386 option, it displays the stack traces of the active tasks on all
387 CPUs. It is often used with the foreach command to display the
388 backtraces of all tasks with one command.
389 btop translates a byte value (physical offset) to its page number.
391 dev displays data concerning the character and block device assign‐
393 ments, I/O port usage, I/O memory usage, and PCI device data.
394 dis disassembles memory, either entire kernel functions, from a
396 location for a specified number of instructions, or from the
397 start of a function up to a specified memory location.
398 eval evaluates an expression or numeric type and displays the result
400 in hexadecimal, decimal, octal and binary.
401 exit causes crash to exit.
403 extend dynamically loads or unloads crash shared object extension mod‐
405 ules.
406 files displays information about open files in a context.
408 foreach
410 repeats a specified command for the specified (or all) tasks in
411 the system.
412 fuser displays the tasks using the specified file or socket.
414 gdb passes its argument to the embedded gdb module. It is useful
416 for executing gdb commands that have the same name as crash com‐
417 mands.
418 help alone displays the command menu; if followed by a command name,
420 a full description of a command, its options, and examples are
421 displayed. Its output is far more complete and useful than this
422 man page.
423 ipcs displays data about the System V IPC facilities.
425 irq displays data concerning interrupt request numbers and bottom-
427 half interrupt handling.
428 kmem displays information about the use of kernel memory.
430 list displays the contents of a linked list.
432 log displays the kernel log_buf contents in chronological order.
434 mach displays data specific to the machine type.
436 mod displays information about the currently installed kernel mod‐
438 ules, or adds or deletes symbolic or debugging information about
439 specified kernel modules.
440 mount displays information about the currently-mounted filesystems.
442 net display various network related data.
444 p passes its arguments to the gdb "print" command for evaluation
446 and display.
447 ps displays process status for specified, or all, processes in the
449 system.
450 pte translates the hexadecimal contents of a PTE into its physical
452 page address and page bit settings.
453 ptob translates a page frame number to its byte value.
455 ptov translates a hexadecimal physical address into a kernel virtual
457 address.
458 q is an alias for the "exit" command.
460 rd displays the contents of memory, with the output formatted in
462 several different manners.
463 repeat repeats a command indefinitely, optionally delaying a given num‐
465 ber of seconds between each command execution.
466 runq displays the tasks on the run queue.
468 search searches a range of user or kernel memory space for given value.
470 set either sets a new context, or gets the current context for dis‐
472 play.
473 sig displays signal-handling data of one or more tasks.
475 struct displays either a structure definition or the contents of a ker‐
477 nel structure at a specified address.
478 swap displays information about each configured swap device.
480 sym translates a symbol to its virtual address, or a static kernel
482 virtual address to its symbol -- or to a symbol-plus-offset
483 value, if appropriate.
484 sys displays system-specific data.
486 task displays the contents of a task_struct.
488 tree displays the contents of a red-black tree or a radix tree.
490 timer displays the timer queue entries, both old- and new-style, in
492 chronological order.
493 union is similar to the struct command, except that it works on kernel
495 unions.
496 vm displays basic virtual memory information of a context.
498 vtop translates a user or kernel virtual address to its physical
500 address.
501 waitq walks the wait queue list displaying the tasks which are blocked
503 on the specified wait queue.
504 whatis displays the definition of structures, unions, typedefs or
506 text/data symbols.
507 wr modifies the contents of memory on a live system. It can only
509 be used if /dev/mem is the device file being used to access sys‐
510 tem RAM, and should obviously be used with great care.
511 When crash is invoked with a Xen hypervisor binary as the NAMELIST, the
513 command set is slightly modified. The *, alias, ascii, bt, dis, eval,
514 exit, extend, gdb, help, list, log, p, pte, rd, repeat, search, set,
515 struct, sym, sys, union, whatis, wr and q commands are the same as
516 above. The following commands are specific to the Xen hypervisor:
517 domain displays the contents of the domain structure for selected, or
519 all, domains.
520 doms displays domain status for selected, or all, domains.
522 dumpinfo
524 displays Xen dump information for selected, or all, cpus.
525 pcpus displays physical cpu information for selected, or all, cpus.
527 vcpus displays vcpu status for selected, or all, vcpus.
529
531 .crashrc
532 Initialization commands. The file can be located in the user's
533 HOME directory and/or the current directory. Commands found in
534 the .crashrc file in the HOME directory are executed before
535 those in the current directory's .crashrc file.
536
538 EDITOR Command input is read using readline(3). If EDITOR is set to
539 emacs or vi then suitable keybindings are used. If EDITOR is
540 not set, then vi is used. This can be overridden by set vi or
541 set emacs commands located in a .crashrc file, or by entering -e
542 emacs on the crash command line.
543 CRASHPAGER
545 If CRASHPAGER is set, its value is used as the name of the pro‐
546 gram to which command output will be sent. If not, then command
547 output is sent to /usr/bin/less -E -X by default.
548 CRASH_MODULE_PATH
550 Specifies an alternative directory tree to search for kernel
551 module object files.
552 CRASH_EXTENSIONS
554 Specifies a directory containing extension modules that will be
555 loaded automatically if the -x command line option is used.
556
558 If crash does not work, look for a newer version: kernel evolution fre‐
559 quently makes crash updates necessary.
560 The command set scroll off will cause output to be sent directly to the
562 terminal rather than through a paging program. This is useful, for
563 example, if you are running crash in a window of emacs.
564
566 Dave Anderson <anderson@redhat.com> wrote crash.
567 Jay Fenlason <fenlason@redhat.com> and Dave Anderson <anderson@red‐
569 hat.com> wrote this man page.
570
572 The help command within crash provides more complete and accurate docu‐
573 mentation than this man page.
574 http://people.redhat.com/anderson - the home page of the crash utility.
576 netdump(8), gdb(1), makedumpfile(8)
578 CRASH(8)