1CRASH(8) System Manager's Manual CRASH(8)
2
6 crash - Analyze Linux crash dump data or a live system
7
9 crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS] (dumpfile form)
10 crash [OPTION]... [NAMELIST] (live system form)
11
13 Crash is a tool for interactively analyzing the state of the Linux sys‐
14 tem while it is running, or after a kernel crash has occurred and a
15 core dump has been created by the netdump, diskdump, LKCD, kdump, xen‐
16 dump kvmdump or VMware facilities. It is loosely based on the SVR4
17 UNIX crash command, but has been significantly enhanced by completely
18 merging it with the gdb(1) debugger. The marriage of the two effec‐
19 tively combines the kernel-specific nature of the traditional UNIX
20 crash utility with the source code level debugging capabilities of
21 gdb(1).
22 In the dumpfile form, both a NAMELIST and a MEMORY-IMAGE argument must
24 be entered. In the live system form, the NAMELIST argument must be
25 entered if the kernel's vmlinux file is not located in a known loca‐
26 tion, such as the /usr/lib/debug/lib/modules/<kernel-version> direc‐
27 tory.
28 The crash utility has also been extended to support the analysis of
30 dumpfiles generated by a crash of the Xen hypervisor. In that case,
31 the NAMELIST argument must be that of the xen-syms binary. Live system
32 analysis is not supported for the Xen hypervisor.
33 The crash utility command set consists of common kernel core analysis
35 tools such as kernel stack back traces of all processes, source code
36 disassembly, formatted kernel structure and variable displays, virtual
37 memory data, dumps of linked-lists, etc., along with several commands
38 that delve deeper into specific kernel subsystems. Appropriate gdb
39 commands may also be entered, which in turn are passed on to the gdb
40 module for execution. If desired, commands may be placed in either a
41 $HOME/.crashrc file and/or in a .crashrc file in the current directory.
42 During initialization, the commands in $HOME/.crashrc are executed
43 first, followed by those in the ./.crashrc file.
44 The crash utility is designed to be independent of Linux version depen‐
46 dencies. When new kernel source code impacts the correct functionality
47 of crash and its command set, the utility will be updated to recognize
48 new kernel code changes, while maintaining backwards compatibility with
49 earlier releases.
50
52 NAMELIST
53 This is a pathname to an uncompressed kernel image (a vmlinux
54 file), or a Xen hypervisor image (a xen-syms file) which has
55 been compiled with the "-g" option. If using the dumpfile form,
56 a vmlinux file may be compressed in either gzip or bzip2 for‐
57 mats.
58 MEMORY-IMAGE[@ADDRESS]
60 A kernel core dump file created by the netdump, diskdump, LKCD
61 kdump, xendump kvmdump or VMware facilities.
62 If a MEMORY-IMAGE argument is not entered, the session will be
64 invoked on the live system, which typically requires root privi‐
65 leges because of the device file used to access system RAM. By
66 default, /dev/crash will be used if it exists. If it does not
67 exist, then /dev/mem will be used; but if the kernel has been
68 configured with CONFIG_STRICT_DEVMEM, then /proc/kcore will be
69 used. It is permissible to explicitly enter /dev/crash,
70 /dev/mem or /proc/kcore.
71 An @ADDRESS value must be appended to the MEMORY-IMAGE if the
73 dumpfile is a raw RAM dumpfile that has no header information
74 describing the file contents. Multiple MEMORY-IMAGE@ADDRESS
75 ordered pairs may be entered, with each dumpfile containing a
76 contiguous block of RAM, where the ADDRESS value is the physical
77 start address of the block expressed in hexadecimal. The physi‐
78 cal address value(s) will be used to create a temporary ELF
79 header in /var/tmp, which will only exist during the crash ses‐
80 sion. If a raw RAM dumpile represents a live memory source,
81 such as that specified by the QEMU mem-path argument of a mem‐
82 ory-backend-file object, then "live:" must be prepended to the
83 MEMORY-IMAGE name.
84 As VMware facility, the crash utility is able to process VMware
86 VM memory dump generated by VM suspend or guest core dump. In
87 that case, .vmss or .guest file should be used as a MEMORY-IMAGE
88 and .vmem file must be located in the same folder.
89 mapfile
91 If the NAMELIST file is not the same kernel that is running
92 (live system form), or the kernel that was running when the sys‐
93 tem crashed (dumpfile form), then the System.map file of the
94 original kernel should be entered on the command line.
95 -h [option]
97 --help [option]
98 Without an option argument, display a crash usage help message.
99 If the option argument is a crash command name, the help page
100 for that command is displayed. If it is the string "input", a
101 page describing the various crash command line input options is
102 displayed. If it is the string "output", a page describing com‐
103 mand line output options is displayed. If it is the string
104 "all", then all of the possible help messages are displayed.
105 After the help message is displayed, crash exits.
106 -s Silently proceed directly to the "crash>" prompt without dis‐
108 playing any version, GPL, or crash initialization data during
109 startup, and by default, runtime command output is not passed to
110 any scrolling command.
111 -i file
113 Execute the command(s) contained in file prior to displaying the
114 "crash>" prompt for interactive user input.
115 -d num Set the internal debug level. The higher the number, the more
117 debugging data will be printed when crash initializes and runs.
118 -S Use /boot/System.map as the mapfile.
120 -e vi | emacs
122 Set the readline(3) command line editing mode to "vi" or
123 "emacs". The default editing mode is "vi".
124 -f Force the usage of a compressed vmlinux file if its original
126 name does not start with "vmlinux".
127 -k Indicate that the NAMELIST file is an LKCD "Kerntypes" debuginfo
129 file.
130 -g [namelist]
132 Determine if a vmlinux or xen-syms namelist file contains debug‐
133 ging data.
134 -t Display the system-crash timestamp and exit.
136 -L Attempt to lock all of its virtual address space into memory by
138 calling mlockall(MCL_CURRENT|MCL_FUTURE) during initialization.
139 If the system call fails, an error message will be displayed,
140 but the session continues.
141 -c tty-device
143 Open the tty-device as the console used for debug messages.
144 -p page-size
146 If a processor's page size cannot be determined by the dumpfile,
147 and the processor default cannot be used, use page-size.
148 -o filename
150 Only used with the MEMORY-IMAGE@ADDRESS format for raw RAM dump‐
151 files, specifies a filename of a new ELF vmcore that will be
152 created and used as the dumpfile. It will be saved to allow
153 future use as a standalone vmcore, replacing the original raw
154 RAM dumpfile.
155 -m option=value
157 --machdep option=value
158 Pass an option and value pair to machine-dependent code. These
159 architecture-specific option/pairs should only be required in
160 very rare circumstances:
161 X86_64:
163 phys_base=<physical-address>
164 irq_eframe_link=<value>
165 irq_stack_gap=<value>
166 max_physmem_bits=<value>
167 kernel_image_size=<value>
168 vm=orig (pre-2.6.11 virtual memory address ranges)
169 vm=2.6.11 (2.6.11 and later virtual memory address ranges)
170 vm=xen (Xen kernel virtual memory address ranges)
171 vm=xen-rhel4 (RHEL4 Xen kernel virtual address ranges)
172 vm=5level (5-level page tables)
173 page_offset=<PAGE_OFFSET-value>
174 PPC64:
175 vm=orig
176 vm=2.6.14 (4-level page tables)
177 IA64:
178 phys_start=<physical-address>
179 init_stack_size=<size>
180 vm=4l (4-level page tables)
181 ARM:
182 phys_base=<physical-address>
183 ARM64:
184 phys_offset=<physical-address>
185 kimage_voffset=<kimage_voffset-value>
186 max_physmem_bits=<value>
187 vabits_actual=<value>
188 X86:
189 page_offset=<CONFIG_PAGE_OFFSET-value>
190 -x Automatically load extension modules from a particular direc‐
192 tory. If a directory is specified in the CRASH_EXTENSIONS shell
193 environment variable, then that directory will be used. Other‐
194 wise /usr/lib64/crash/extensions (64-bit architectures) or
195 /usr/lib/crash/extensions (32-bit architectures) will be used;
196 if they do not exist, then the ./extensions directory will be
197 used.
198 --active
200 Track only the active task on each cpu.
201 --buildinfo
203 Display the crash binary's build date, the user ID of the
204 builder, the hostname of the machine where the build was done,
205 the target architecture, the version number, and the compiler
206 version.
207 --memory_module modname
209 Use the modname as an alternative kernel module to the crash.ko
210 module that creates the /dev/crash device.
211 --memory_device device
213 Use device as an alternative device to the /dev/crash, /dev/mem
214 or /proc/kcore devices.
215 --log dumpfile
217 Dump the contents of the kernel log buffer. A kernel namelist
218 argument is not necessary, but the dumpfile must contain the
219 VMCOREINFO data taken from the original /proc/vmcore ELF header.
220 --no_kallsyms
222 Do not use kallsyms-generated symbol information contained
223 within kernel module object files.
224 --no_modules
226 Do not access or display any kernel module related information.
227 --no_ikconf
229 Do not attempt to read configuration data that was built into
230 kernels configured with CONFIG_IKCONFIG.
231 --no_data_debug
233 Do not verify the validity of all structure member offsets and
234 structure sizes that it uses.
235 --no_kmem_cache
237 Do not initialize the kernel's slab cache infrastructure, and
238 commands that use kmem_cache-related data will not work.
239 --no_elf_notes
241 Do not use the registers from the ELF NT_PRSTATUS notes saved in
242 a compressed kdump header for backtraces.
243 --kmem_cache_delay
245 Delay the initialization of the kernel's slab cache infrastruc‐
246 ture until it is required by a run-time command.
247 --readnow
249 Pass this flag to the embedded gdb module, which will override
250 its two-stage strategy that it uses for reading symbol tables
251 from the NAMELIST.
252 --smp Specify that the system being analyzed is an SMP kernel.
254 -v
256 --version
257 Display the version of the crash utility, the version of the
258 embedded gdb module, GPL information, and copyright notices.
259 --cpus number
261 Specify the number of cpus in the SMP system being analyzed.
262 --osrelease dumpfile
264 Display the OSRELEASE vmcoreinfo string from a kdump dumpfile
265 header.
266 --hyper
268 Force the session to be that of a Xen hypervisor.
269 --p2m_mfn pfn
271 When a Xen Hypervisor or its dom0 kernel crashes, the dumpfile
272 is typically analyzed with either the Xen hypervisor or the dom0
273 kernel. It is also possible to analyze any of the guest domU
274 kernels if the pfn_to_mfn_list_list pfn value of the guest ker‐
275 nel is passed on the command line along with its NAMELIST and
276 the dumpfile.
277 --xen_phys_start physical-address
279 Supply the base physical address of the Xen hypervisor's text
280 and static data for older xendump dumpfiles that did not pass
281 that information in the dumpfile header.
282 --zero_excluded
284 If the makedumpfile(8) facility has filtered a compressed kdump
285 dumpfile to exclude various types of non-essential pages, or has
286 marked a compressed or ELF kdump dumpfile as incomplete due to
287 an ENOSPC or other error during its creation, any attempt to
288 read missing pages will fail. With this flag, reads from any of
289 those pages will return zero-filled memory.
290 --no_panic
292 Do not attempt to find the task that was running when the kernel
293 crashed. Set the initial context to that of the "swapper" task
294 on cpu 0.
295 --more Use /bin/more as the command output scroller, overriding the
297 default of /usr/bin/less and any settings in either ./.crashrc
298 or $HOME/.crashrc.
299 --less Use /usr/bin/less as the command output scroller, overriding any
301 settings in either ./.crashrc or $HOME/.crashrc.
302 --hex Set the default command output radix to 16, overriding the
304 default radix of 10, and any radix settings in either ./.crashrc
305 or $HOME/.crashrc.
306 --dec Set the default command output radix to 10, overriding any radix
308 settings in either ./.crashrc or $HOME/.crashrc. This is the
309 default radix setting.
310 --CRASHPAGER
312 Use the output paging command defined in the CRASHPAGER shell
313 environment variable, overriding any settings in either
314 ./.crashrc or $HOME/.crashrc.
315 --no_scroll
317 Do not pass run-time command output to any scrolling command.
318 --no_strip
320 Do not strip cloned kernel text symbol names.
321 --no_crashrc
323 Do not execute the commands in either $HOME/.crashrc or
324 ./.crashrc.
325 --mod directory
327 When loading the debuginfo data of kernel modules with the mod
328 -S command, search for their object files in directory instead
329 of in the standard location.
330 --kaslr offset|auto
332 If an x86_64 kernel was configured with CONFIG_RANDOMIZE_BASE,
333 the offset value is equal to the difference between the symbol
334 values compiled into the vmlinux file and their relocated KASLR
335 values. If set to auto, the KASLR offset value will be automat‐
336 ically calculated.
337 --reloc size
339 When analyzing live x86 kernels that were configured with a CON‐
340 FIG_PHYSICAL_START value that is larger than its CONFIG_PHYSI‐
341 CAL_ALIGN value, then it will be necessary to enter a relocation
342 size equal to the difference between the two values.
343 --hash count
345 Set the number of internal hash queue heads used for list gath‐
346 ering and verification. The default count is 32768.
347 --minimal
349 Bring up a session that is restricted to the log, dis, rd, sym,
350 eval, set and exit commands. This option may provide a way to
351 extract some minimal/quick information from a corrupted or trun‐
352 cated dumpfile, or in situations where one of the several kernel
353 subsystem initialization routines would abort the crash session.
354 --kvmhost [32|64]
356 When examining an x86 KVM guest dumpfile, this option specifies
357 that the KVM host that created the dumpfile was an x86 (32-bit)
358 or an x86_64 (64-bit) machine, overriding the automatically
359 determined value.
360 --kvmio <size>
362 override the automatically-calculated KVM guest I/O hole size.
363 --offline [show|hide]
365 Show or hide command output that is related to offline cpus.
366 The default setting is show.
367
369 Each crash command generally falls into one of the following cate‐
370 gories:
371 Symbolic display
373 Displays of kernel text/data, which take full advantage of the
374 power of gdb to format and display data structures symbolically.
375 System state
377 The majority of crash commands consist of a set of "kernel-
378 aware" commands, which delve into various kernel subsystems on a
379 system-wide or per-task basis.
380 Utility functions
382 A set of useful helper commands serving various purposes, some
383 simple, others quite powerful.
384 Session control
386 Commands that control the crash session itself.
387 The following alphabetical list consists of a very simple overview of
389 each crash command. However, since individual commands often have sev‐
390 eral options resulting in significantly different output, it is sug‐
391 gested that the full description of each command be viewed by executing
392 crash -h <command>, or during a crash session by simply entering help
393 command.
394 * "pointer to" is shorthand for either the struct or union com‐
396 mands. It displays the contents of a kernel structure or union.
397 alias creates a single-word alias for a command.
399 ascii displays an ascii chart or translates a numeric value into its
401 ascii components.
402 bt displays a task's kernel-stack backtrace. If it is given the -a
404 option, it displays the stack traces of the active tasks on all
405 CPUs. It is often used with the foreach command to display the
406 backtraces of all tasks with one command.
407 btop translates a byte value (physical offset) to its page number.
409 dev displays data concerning the character and block device assign‐
411 ments, I/O port usage, I/O memory usage, and PCI device data.
412 dis disassembles memory, either entire kernel functions, from a
414 location for a specified number of instructions, or from the
415 start of a function up to a specified memory location.
416 eval evaluates an expression or numeric type and displays the result
418 in hexadecimal, decimal, octal and binary.
419 exit causes crash to exit.
421 extend dynamically loads or unloads crash shared object extension mod‐
423 ules.
424 files displays information about open files in a context.
426 foreach
428 repeats a specified command for the specified (or all) tasks in
429 the system.
430 fuser displays the tasks using the specified file or socket.
432 gdb passes its argument to the embedded gdb module. It is useful
434 for executing gdb commands that have the same name as crash com‐
435 mands.
436 help alone displays the command menu; if followed by a command name,
438 a full description of a command, its options, and examples are
439 displayed. Its output is far more complete and useful than this
440 man page.
441 ipcs displays data about the System V IPC facilities.
443 irq displays data concerning interrupt request numbers and bottom-
445 half interrupt handling.
446 kmem displays information about the use of kernel memory.
448 list displays the contents of a linked list.
450 log displays the kernel log_buf contents in chronological order.
452 mach displays data specific to the machine type.
454 mod displays information about the currently installed kernel mod‐
456 ules, or adds or deletes symbolic or debugging information about
457 specified kernel modules.
458 mount displays information about the currently-mounted filesystems.
460 net display various network related data.
462 p passes its arguments to the gdb "print" command for evaluation
464 and display.
465 ps displays process status for specified, or all, processes in the
467 system.
468 pte translates the hexadecimal contents of a PTE into its physical
470 page address and page bit settings.
471 ptob translates a page frame number to its byte value.
473 ptov translates a hexadecimal physical address into a kernel virtual
475 address.
476 q is an alias for the "exit" command.
478 rd displays the contents of memory, with the output formatted in
480 several different manners.
481 repeat repeats a command indefinitely, optionally delaying a given num‐
483 ber of seconds between each command execution.
484 runq displays the tasks on the run queue.
486 search searches a range of user or kernel memory space for given value.
488 set either sets a new context, or gets the current context for dis‐
490 play.
491 sig displays signal-handling data of one or more tasks.
493 struct displays either a structure definition or the contents of a ker‐
495 nel structure at a specified address.
496 swap displays information about each configured swap device.
498 sym translates a symbol to its virtual address, or a static kernel
500 virtual address to its symbol -- or to a symbol-plus-offset
501 value, if appropriate.
502 sys displays system-specific data.
504 task displays the contents of a task_struct.
506 tree displays the contents of a red-black tree or a radix tree.
508 timer displays the timer queue entries, both old- and new-style, in
510 chronological order.
511 union is similar to the struct command, except that it works on kernel
513 unions.
514 vm displays basic virtual memory information of a context.
516 vtop translates a user or kernel virtual address to its physical
518 address.
519 waitq walks the wait queue list displaying the tasks which are blocked
521 on the specified wait queue.
522 whatis displays the definition of structures, unions, typedefs or
524 text/data symbols.
525 wr modifies the contents of memory on a live system. It can only
527 be used if /dev/mem is the device file being used to access sys‐
528 tem RAM, and should obviously be used with great care.
529 When crash is invoked with a Xen hypervisor binary as the NAMELIST, the
531 command set is slightly modified. The *, alias, ascii, bt, dis, eval,
532 exit, extend, gdb, help, list, log, p, pte, rd, repeat, search, set,
533 struct, sym, sys, union, whatis, wr and q commands are the same as
534 above. The following commands are specific to the Xen hypervisor:
535 domain displays the contents of the domain structure for selected, or
537 all, domains.
538 doms displays domain status for selected, or all, domains.
540 dumpinfo
542 displays Xen dump information for selected, or all, cpus.
543 pcpus displays physical cpu information for selected, or all, cpus.
545 vcpus displays vcpu status for selected, or all, vcpus.
547
549 .crashrc
550 Initialization commands. The file can be located in the user's
551 HOME directory and/or the current directory. Commands found in
552 the .crashrc file in the HOME directory are executed before
553 those in the current directory's .crashrc file.
554
556 EDITOR Command input is read using readline(3). If EDITOR is set to
557 emacs or vi then suitable keybindings are used. If EDITOR is
558 not set, then vi is used. This can be overridden by set vi or
559 set emacs commands located in a .crashrc file, or by entering -e
560 emacs on the crash command line.
561 CRASHPAGER
563 If CRASHPAGER is set, its value is used as the name of the pro‐
564 gram to which command output will be sent. If not, then command
565 output is sent to /usr/bin/less -E -X by default.
566 CRASH_MODULE_PATH
568 Specifies an alternative directory tree to search for kernel
569 module object files.
570 CRASH_EXTENSIONS
572 Specifies a directory containing extension modules that will be
573 loaded automatically if the -x command line option is used.
574
576 If crash does not work, look for a newer version: kernel evolution fre‐
577 quently makes crash updates necessary.
578 The command set scroll off will cause output to be sent directly to the
580 terminal rather than through a paging program. This is useful, for
581 example, if you are running crash in a window of emacs.
582
584 Dave Anderson <anderson@redhat.com> wrote crash.
585 Jay Fenlason <fenlason@redhat.com> and Dave Anderson <anderson@red‐
587 hat.com> wrote this man page.
588
590 The help command within crash provides more complete and accurate docu‐
591 mentation than this man page.
592 https://github.com/crash-utility - the home page of the crash utility.
594 netdump(8), gdb(1), makedumpfile(8)
596 CRASH(8)