1CRASH(8)                    System Manager's Manual                   CRASH(8)
2
3
4

NAME

6       crash - Analyze Linux crash dump data or a live system
7

SYNOPSIS

9       crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS]    (dumpfile form)
10       crash [OPTION]... [NAMELIST]                         (live system form)
11

DESCRIPTION

13       Crash is a tool for interactively analyzing the state of the Linux sys‐
14       tem while it is running, or after a kernel crash  has  occurred  and  a
15       core  dump has been created by the netdump, diskdump, LKCD, kdump, xen‐
16       dump kvmdump or VMware facilities.  It is loosely  based  on  the  SVR4
17       UNIX  crash  command, but has been significantly enhanced by completely
18       merging it with the gdb(1) debugger. The marriage  of  the  two  effec‐
19       tively  combines  the  kernel-specific  nature  of the traditional UNIX
20       crash utility with the source  code  level  debugging  capabilities  of
21       gdb(1).
22
23       In  the dumpfile form, both a NAMELIST and a MEMORY-IMAGE argument must
24       be entered.  In the live system form, the NAMELIST argument must be en‐
25       tered  if the kernel's vmlinux file is not located in a known location,
26       such as the /usr/lib/debug/lib/modules/<kernel-version> directory.
27
28       The crash utility has also been extended to  support  the  analysis  of
29       dumpfiles  generated  by  a crash of the Xen hypervisor.  In that case,
30       the NAMELIST argument must be that of the xen-syms binary.  Live system
31       analysis is not supported for the Xen hypervisor.
32
33       The  crash  utility command set consists of common kernel core analysis
34       tools such as kernel stack back traces of all  processes,  source  code
35       disassembly,  formatted kernel structure and variable displays, virtual
36       memory data, dumps of linked-lists, etc., along with  several  commands
37       that  delve  deeper  into  specific kernel subsystems.  Appropriate gdb
38       commands may also be entered, which in turn are passed on  to  the  gdb
39       module  for  execution.  If desired, commands may be placed in either a
40       $HOME/.crashrc file and/or in a .crashrc file in the current directory.
41       During  initialization,  the  commands  in  $HOME/.crashrc are executed
42       first, followed by those in the ./.crashrc file.
43
44       The crash utility is designed to be independent of Linux version depen‐
45       dencies.  When new kernel source code impacts the correct functionality
46       of crash and its command set, the utility will be updated to  recognize
47       new kernel code changes, while maintaining backwards compatibility with
48       earlier releases.
49

OPTIONS

51       NAMELIST
52              This is a pathname to an uncompressed kernel  image  (a  vmlinux
53              file),  or  a  Xen  hypervisor image (a xen-syms file) which has
54              been compiled with the "-g" option.  If using the dumpfile form,
55              a  vmlinux  file  may be compressed in either gzip or bzip2 for‐
56              mats.
57
58       MEMORY-IMAGE[@ADDRESS]
59              A kernel core dump file created by the netdump,  diskdump,  LKCD
60              kdump, xendump kvmdump or VMware facilities.
61
62              If  a  MEMORY-IMAGE argument is not entered, the session will be
63              invoked on the live system, which typically requires root privi‐
64              leges  because of the device file used to access system RAM.  By
65              default, /dev/crash will be used if it exists.  If it  does  not
66              exist,  then  /dev/mem  will be used; but if the kernel has been
67              configured with CONFIG_STRICT_DEVMEM, then /proc/kcore  will  be
68              used.    It  is  permissible  to  explicitly  enter  /dev/crash,
69              /dev/mem or /proc/kcore.
70
71              An @ADDRESS value must be appended to the  MEMORY-IMAGE  if  the
72              dumpfile  is  a  raw RAM dumpfile that has no header information
73              describing the file contents.  Multiple MEMORY-IMAGE@ADDRESS or‐
74              dered pairs may be entered, with each dumpfile containing a con‐
75              tiguous block of RAM, where the ADDRESS value  is  the  physical
76              start address of the block expressed in hexadecimal.  The physi‐
77              cal address value(s) will be used  to  create  a  temporary  ELF
78              header  in /var/tmp, which will only exist during the crash ses‐
79              sion.  If a raw RAM dumpile represents  a  live  memory  source,
80              such  as  that specified by the QEMU mem-path argument of a mem‐
81              ory-backend-file object, then "live:" must be prepended  to  the
82              MEMORY-IMAGE name.
83
84              As  VMware facility, the crash utility is able to process VMware
85              VM memory dump generated by VM suspend or guest  core  dump.  In
86              that case, .vmss or .guest file should be used as a MEMORY-IMAGE
87              and .vmem file must be located in the same folder.
88
89       mapfile
90              If the NAMELIST file is not the  same  kernel  that  is  running
91              (live system form), or the kernel that was running when the sys‐
92              tem crashed (dumpfile form), then the  System.map  file  of  the
93              original kernel should be entered on the command line.
94
95       -h [option]
96       --help [option]
97              Without  an option argument, display a crash usage help message.
98              If the option argument is a crash command name,  the  help  page
99              for  that  command is displayed.  If it is the string "input", a
100              page describing the various crash command line input options  is
101              displayed.  If it is the string "output", a page describing com‐
102              mand line output options is displayed.   If  it  is  the  string
103              "all",  then  all  of  the possible help messages are displayed.
104              After the help message is displayed, crash exits.
105
106       -s     Silently proceed directly to the "crash>"  prompt  without  dis‐
107              playing  any  version,  GPL, or crash initialization data during
108              startup, and by default, runtime command output is not passed to
109              any scrolling command.
110
111       -i file
112              Execute the command(s) contained in file prior to displaying the
113              "crash>" prompt for interactive user input.
114
115       -d num Set the internal debug level.  The higher the number,  the  more
116              debugging data will be printed when crash initializes and runs.
117
118       -S     Use /boot/System.map as the mapfile.
119
120       -e vi | emacs
121              Set  the  readline(3)  command  line  editing  mode  to  "vi" or
122              "emacs".  The default editing mode is "vi".
123
124       -f     Force the usage of a compressed vmlinux  file  if  its  original
125              name does not start with "vmlinux".
126
127       -k     Indicate that the NAMELIST file is an LKCD "Kerntypes" debuginfo
128              file.
129
130       -g [namelist]
131              Determine if a vmlinux or xen-syms namelist file contains debug‐
132              ging data.
133
134       -t     Display the system-crash timestamp and exit.
135
136       -L     Attempt  to lock all of its virtual address space into memory by
137              calling mlockall(MCL_CURRENT|MCL_FUTURE) during  initialization.
138              If  the  system  call fails, an error message will be displayed,
139              but the session continues.
140
141       -c tty-device
142              Open the tty-device as the console used for debug messages.
143
144       -p page-size
145              If a processor's page size cannot be determined by the dumpfile,
146              and the processor default cannot be used, use page-size.
147
148       -o filename
149              Only used with the MEMORY-IMAGE@ADDRESS format for raw RAM dump‐
150              files, specifies a filename of a new ELF  vmcore  that  will  be
151              created and used as the dumpfile.  It will be saved to allow fu‐
152              ture use as a standalone vmcore, replacing the original raw  RAM
153              dumpfile.
154
155       -m option=value
156       --machdep option=value
157              Pass  an option and value pair to machine-dependent code.  These
158              architecture-specific option/pairs should only  be  required  in
159              very rare circumstances:
160
161              X86_64:
162                phys_base=<physical-address>
163                irq_eframe_link=<value>
164                irq_stack_gap=<value>
165                max_physmem_bits=<value>
166                kernel_image_size=<value>
167                vm=orig       (pre-2.6.11 virtual memory address ranges)
168                vm=2.6.11     (2.6.11 and later virtual memory address ranges)
169                vm=xen        (Xen kernel virtual memory address ranges)
170                vm=xen-rhel4  (RHEL4 Xen kernel virtual address ranges)
171                vm=5level     (5-level page tables)
172                page_offset=<PAGE_OFFSET-value>
173              PPC64:
174                vm=orig
175                vm=2.6.14     (4-level page tables)
176              IA64:
177                phys_start=<physical-address>
178                init_stack_size=<size>
179                vm=4l         (4-level page tables)
180              ARM:
181                phys_base=<physical-address>
182              ARM64:
183                phys_offset=<physical-address>
184                kimage_voffset=<kimage_voffset-value>
185                max_physmem_bits=<value>
186                vabits_actual=<value>
187              X86:
188                page_offset=<CONFIG_PAGE_OFFSET-value>
189
190       -x     Automatically  load  extension  modules from a particular direc‐
191              tory.  If a directory is specified in the CRASH_EXTENSIONS shell
192              environment  variable, then that directory will be used.  Other‐
193              wise  /usr/lib64/crash/extensions  (64-bit   architectures)   or
194              /usr/lib/crash/extensions  (32-bit  architectures) will be used;
195              if they do not exist, then the ./extensions  directory  will  be
196              used.
197
198       --active
199              Track only the active task on each cpu.
200
201       --buildinfo
202              Display  the  crash  binary's  build  date,  the  user ID of the
203              builder, the hostname of the machine where the build  was  done,
204              the  target  architecture,  the version number, and the compiler
205              version.
206
207       --memory_module modname
208              Use the modname as an alternative kernel module to the  crash.ko
209              module that creates the /dev/crash device.
210
211       --memory_device device
212              Use  device as an alternative device to the /dev/crash, /dev/mem
213              or /proc/kcore devices.
214
215       --log dumpfile
216              Dump the contents of the kernel log buffer.  A  kernel  namelist
217              argument is not necessary, but the dumpfile must contain the VM‐
218              COREINFO data taken from the original /proc/vmcore ELF header.
219
220       --no_kallsyms
221              Do  not  use  kallsyms-generated  symbol  information  contained
222              within kernel module object files.
223
224       --no_modules
225              Do not access or display any kernel module related information.
226
227       --no_ikconf
228              Do  not  attempt  to read configuration data that was built into
229              kernels configured with CONFIG_IKCONFIG.
230
231       --no_data_debug
232              Do not verify the validity of all structure member  offsets  and
233              structure sizes that it uses.
234
235       --no_kmem_cache
236              Do  not  initialize  the kernel's slab cache infrastructure, and
237              commands that use kmem_cache-related data will not work.
238
239       --no_elf_notes
240              Do not use the registers from the ELF NT_PRSTATUS notes saved in
241              a compressed kdump header for backtraces.
242
243       --kmem_cache_delay
244              Delay  the initialization of the kernel's slab cache infrastruc‐
245              ture until it is required by a run-time command.
246
247       --readnow
248              Pass this flag to the embedded gdb module, which  will  override
249              its  two-stage  strategy  that it uses for reading symbol tables
250              from the NAMELIST.
251
252       --smp  Specify that the system being analyzed is an SMP kernel.
253
254       -v
255       --version
256              Display the version of the crash utility, the version of the em‐
257              bedded gdb module, GPL information, and copyright notices.
258
259       --cpus number
260              Specify the number of cpus in the SMP system being analyzed.
261
262       --osrelease dumpfile
263              Display  the  OSRELEASE  vmcoreinfo string from a kdump dumpfile
264              header.
265
266       --hyper
267              Force the session to be that of a Xen hypervisor.
268
269       --p2m_mfn pfn
270              When a Xen Hypervisor or its dom0 kernel crashes,  the  dumpfile
271              is typically analyzed with either the Xen hypervisor or the dom0
272              kernel.  It is also possible to analyze any of  the  guest  domU
273              kernels  if the pfn_to_mfn_list_list pfn value of the guest ker‐
274              nel is passed on the command line along with  its  NAMELIST  and
275              the dumpfile.
276
277       --xen_phys_start physical-address
278              Supply  the  base  physical address of the Xen hypervisor's text
279              and static data for older xendump dumpfiles that  did  not  pass
280              that information in the dumpfile header.
281
282       --zero_excluded
283              If  the makedumpfile(8) facility has filtered a compressed kdump
284              dumpfile to exclude various types of non-essential pages, or has
285              marked  a  compressed or ELF kdump dumpfile as incomplete due to
286              an ENOSPC or other error during its  creation,  any  attempt  to
287              read missing pages will fail.  With this flag, reads from any of
288              those pages will return zero-filled memory.
289
290       --no_panic
291              Do not attempt to find the task that was running when the kernel
292              crashed.   Set the initial context to that of the "swapper" task
293              on cpu 0.
294
295       --more Use /bin/more as the command output scroller, overriding the de‐
296              fault  of /usr/bin/less and any settings in either ./.crashrc or
297              $HOME/.crashrc.
298
299       --less Use /usr/bin/less as the command output scroller, overriding any
300              settings in either ./.crashrc or $HOME/.crashrc.
301
302       --hex  Set  the  default command output radix to 16, overriding the de‐
303              fault radix of 10, and any radix settings in  either  ./.crashrc
304              or $HOME/.crashrc.
305
306       --dec  Set the default command output radix to 10, overriding any radix
307              settings in either ./.crashrc or $HOME/.crashrc. This is the de‐
308              fault radix setting.
309
310       --CRASHPAGER
311              Use  the  output  paging command defined in the CRASHPAGER shell
312              environment  variable,  overriding  any   settings   in   either
313              ./.crashrc or $HOME/.crashrc.
314
315       --no_scroll
316              Do not pass run-time command output to any scrolling command.
317
318       --no_strip
319              Do not strip cloned kernel text symbol names.
320
321       --no_crashrc
322              Do   not  execute  the  commands  in  either  $HOME/.crashrc  or
323              ./.crashrc.
324
325       --mod directory
326              When loading the debuginfo data of kernel modules with  the  mod
327              -S  command,  search for their object files in directory instead
328              of in the standard location.
329
330       --src directory
331              Search for the kernel source code in directory instead of in the
332              standard location that is compiled into the debuginfo data.
333
334       --kaslr offset|auto
335              If  an  x86_64 kernel was configured with CONFIG_RANDOMIZE_BASE,
336              the offset value is equal to the difference between  the  symbol
337              values  compiled into the vmlinux file and their relocated KASLR
338              values.  If set to auto, the KASLR offset value will be automat‐
339              ically calculated.
340
341       --reloc size
342              When analyzing live x86 kernels that were configured with a CON‐
343              FIG_PHYSICAL_START value that is larger than  its  CONFIG_PHYSI‐
344              CAL_ALIGN value, then it will be necessary to enter a relocation
345              size equal to the difference between the two values.
346
347       --hash count
348              Set the number of internal hash queue heads used for list  gath‐
349              ering and verification.  The default count is 32768.
350
351       --minimal
352              Bring  up a session that is restricted to the log, dis, rd, sym,
353              eval, set and exit commands.  This option may provide a  way  to
354              extract some minimal/quick information from a corrupted or trun‐
355              cated dumpfile, or in situations where one of the several kernel
356              subsystem initialization routines would abort the crash session.
357
358       --kvmhost [32|64]
359              When  examining an x86 KVM guest dumpfile, this option specifies
360              that the KVM host that created the dumpfile was an x86  (32-bit)
361              or  an x86_64 (64-bit) machine, overriding the automatically de‐
362              termined value.
363
364       --kvmio <size>
365              override the automatically-calculated KVM guest I/O hole size.
366
367       --offline [show|hide]
368              Show or hide command output that is  related  to  offline  cpus.
369              The default setting is show.
370

COMMANDS

372       Each  crash  command  generally  falls  into one of the following cate‐
373       gories:
374
375       Symbolic display
376              Displays of kernel text/data, which take full advantage  of  the
377              power of gdb to format and display data structures symbolically.
378
379       System state
380              The  majority  of  crash  commands  consist of a set of "kernel-
381              aware" commands, which delve into various kernel subsystems on a
382              system-wide or per-task basis.
383
384       Utility functions
385              A  set  of useful helper commands serving various purposes, some
386              simple, others quite powerful.
387
388       Session control
389              Commands that control the crash session itself.
390
391       The following alphabetical list consists of a very simple  overview  of
392       each crash command.  However, since individual commands often have sev‐
393       eral options resulting in significantly different output,  it  is  sug‐
394       gested that the full description of each command be viewed by executing
395       crash -h <command>, or during a crash session by simply  entering  help
396       command.
397
398       *      "pointer  to"  is  shorthand for either the struct or union com‐
399              mands.  It displays the contents of a kernel structure or union.
400
401       alias  creates a single-word alias for a command.
402
403       ascii  displays an ascii chart or translates a numeric value  into  its
404              ascii components.
405
406       bt     displays a task's kernel-stack backtrace.  If it is given the -a
407              option, it displays the stack traces of the active tasks on  all
408              CPUs.   It is often used with the foreach command to display the
409              backtraces of all tasks with one command.
410
411       btop   translates a byte value (physical offset) to its page number.
412
413       dev    displays data concerning the character and block device  assign‐
414              ments, I/O port usage, I/O memory usage, and PCI device data.
415
416       dis    disassembles  memory, either entire kernel functions, from a lo‐
417              cation for a specified number of instructions, or from the start
418              of a function up to a specified memory location.
419
420       eval   evaluates  an expression or numeric type and displays the result
421              in hexadecimal, decimal, octal and binary.
422
423       exit   causes crash to exit.
424
425       extend dynamically loads or unloads crash shared object extension  mod‐
426              ules.
427
428       files  displays information about open files in a context.
429
430       foreach
431              repeats  a specified command for the specified (or all) tasks in
432              the system.
433
434       fuser  displays the tasks using the specified file or socket.
435
436       gdb    passes its argument to the embedded gdb module.   It  is  useful
437              for executing gdb commands that have the same name as crash com‐
438              mands.
439
440       help   alone displays the command menu; if followed by a command  name,
441              a  full  description of a command, its options, and examples are
442              displayed.  Its output is far more complete and useful than this
443              man page.
444
445       ipcs   displays data about the System V IPC facilities.
446
447       irq    displays  data  concerning interrupt request numbers and bottom-
448              half interrupt handling.
449
450       kmem   displays information about the use of kernel memory.
451
452       list   displays the contents of a linked list.
453
454       log    displays the kernel log_buf contents in chronological order.
455
456       mach   displays data specific to the machine type.
457
458       mod    displays information about the currently installed  kernel  mod‐
459              ules, or adds or deletes symbolic or debugging information about
460              specified kernel modules.
461
462       mount  displays information about the currently-mounted filesystems.
463
464       net    display various network related data.
465
466       p      passes its arguments to the gdb "print" command  for  evaluation
467              and display.
468
469       ps     displays  process status for specified, or all, processes in the
470              system.
471
472       pte    translates the hexadecimal contents of a PTE into  its  physical
473              page address and page bit settings.
474
475       ptob   translates a page frame number to its byte value.
476
477       ptov   translates  a hexadecimal physical address into a kernel virtual
478              address.
479
480       q      is an alias for the "exit" command.
481
482       rd     displays the contents of memory, with the  output  formatted  in
483              several different manners.
484
485       repeat repeats a command indefinitely, optionally delaying a given num‐
486              ber of seconds between each command execution.
487
488       runq   displays the tasks on the run queue.
489
490       search searches a range of user or kernel memory space for given value.
491
492       set    either sets a new context, or gets the current context for  dis‐
493              play.
494
495       sig    displays signal-handling data of one or more tasks.
496
497       struct displays either a structure definition or the contents of a ker‐
498              nel structure at a specified address.
499
500       swap   displays information about each configured swap device.
501
502       sym    translates a symbol to its virtual address, or a  static  kernel
503              virtual  address  to  its  symbol  -- or to a symbol-plus-offset
504              value, if appropriate.
505
506       sys    displays system-specific data.
507
508       task   displays the contents of a task_struct.
509
510       tree   displays the contents of a red-black tree or a radix tree.
511
512       timer  displays the timer queue entries, both old-  and  new-style,  in
513              chronological order.
514
515       union  is similar to the struct command, except that it works on kernel
516              unions.
517
518       vm     displays basic virtual memory information of a context.
519
520       vtop   translates a user or kernel virtual address to its physical  ad‐
521              dress.
522
523       waitq  walks the wait queue list displaying the tasks which are blocked
524              on the specified wait queue.
525
526       whatis displays the  definition  of  structures,  unions,  typedefs  or
527              text/data symbols.
528
529       wr     modifies  the  contents of memory on a live system.  It can only
530              be used if /dev/mem is the device file being used to access sys‐
531              tem RAM, and should obviously be used with great care.
532
533       When crash is invoked with a Xen hypervisor binary as the NAMELIST, the
534       command set is slightly modified.  The *, alias, ascii, bt, dis,  eval,
535       exit,  extend,  gdb,  help, list, log, p, pte, rd, repeat, search, set,
536       struct, sym, sys, union, whatis, wr and q  commands  are  the  same  as
537       above.  The following commands are specific to the Xen hypervisor:
538
539       domain displays  the  contents of the domain structure for selected, or
540              all, domains.
541
542       doms   displays domain status for selected, or all, domains.
543
544       dumpinfo
545              displays Xen dump information for selected, or all, cpus.
546
547       pcpus  displays physical cpu information for selected, or all, cpus.
548
549       vcpus  displays vcpu status for selected, or all, vcpus.
550

FILES

552       .crashrc
553              Initialization commands.  The file can be located in the  user's
554              HOME  directory and/or the current directory.  Commands found in
555              the .crashrc file in the  HOME  directory  are  executed  before
556              those in the current directory's .crashrc file.
557

ENVIRONMENT

559       EDITOR Command  input  is  read using readline(3).  If EDITOR is set to
560              emacs or vi then suitable keybindings are used.   If  EDITOR  is
561              not  set,  then vi is used.  This can be overridden by set vi or
562              set emacs commands located in a .crashrc file, or by entering -e
563              emacs on the crash command line.
564
565       CRASHPAGER
566              If  CRASHPAGER is set, its value is used as the name of the pro‐
567              gram to which command output will be sent.  If not, then command
568              output is sent to /usr/bin/less -E -X by default.
569
570       CRASH_MODULE_PATH
571              Specifies  an  alternative  directory  tree to search for kernel
572              module object files.
573
574       CRASH_EXTENSIONS
575              Specifies a directory containing extension modules that will  be
576              loaded automatically if the -x command line option is used.
577

NOTES

579       If crash does not work, look for a newer version: kernel evolution fre‐
580       quently makes crash updates necessary.
581
582       The command set scroll off will cause output to be sent directly to the
583       terminal rather than through a paging program.  This is useful, for ex‐
584       ample, if you are running crash in a window of emacs.
585

AUTHOR

587       Dave Anderson <anderson@redhat.com> wrote crash.
588
589       Jay Fenlason <fenlason@redhat.com>  and  Dave  Anderson  <anderson@red‐
590       hat.com> wrote this man page.
591

SEE ALSO

593       The help command within crash provides more complete and accurate docu‐
594       mentation than this man page.
595
596       https://github.com/crash-utility - the home page of the crash utility.
597
598       netdump(8), gdb(1), makedumpfile(8)
599
600
601
602                                                                      CRASH(8)
Impressum