1CRASH(8)                    System Manager's Manual                   CRASH(8)
2
3
4

NAME

6       crash - Analyze Linux crash dump data or a live system
7

SYNOPSIS

9       crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS]    (dumpfile form)
10       crash [OPTION]... [NAMELIST]                         (live system form)
11

DESCRIPTION

13       Crash is a tool for interactively analyzing the state of the Linux sys‐
14       tem while it is running, or after a kernel crash  has  occurred  and  a
15       core  dump has been created by the netdump, diskdump, LKCD, kdump, xen‐
16       dump or kvmdump facilities.  It is loosely based on the SVR4 UNIX crash
17       command,  but  has been significantly enhanced by completely merging it
18       with the gdb(1) debugger. The marriage of the two effectively  combines
19       the  kernel-specific  nature of the traditional UNIX crash utility with
20       the source code level debugging capabilities of gdb(1).
21
22       In the dumpfile form, both a NAMELIST and a MEMORY-IMAGE argument  must
23       be  entered.   In  the  live system form, the NAMELIST argument must be
24       entered if the kernel's vmlinux file is not located in  a  known  loca‐
25       tion,  such  as  the /usr/lib/debug/lib/modules/<kernel-version> direc‐
26       tory.
27
28       The crash utility has also been extended to  support  the  analysis  of
29       dumpfiles  generated  by  a crash of the Xen hypervisor.  In that case,
30       the NAMELIST argument must be that of the xen-syms binary.  Live system
31       analysis is not supported for the Xen hypervisor.
32
33       The  crash  utility command set consists of common kernel core analysis
34       tools such as kernel stack back traces of all  processes,  source  code
35       disassembly,  formatted kernel structure and variable displays, virtual
36       memory data, dumps of linked-lists, etc., along with  several  commands
37       that  delve  deeper  into  specific kernel subsystems.  Appropriate gdb
38       commands may also be entered, which in turn are passed on  to  the  gdb
39       module  for  execution.  If desired, commands may be placed in either a
40       $HOME/.crashrc file and/or in a .crashrc file in the current directory.
41       During  initialization,  the  commands  in  $HOME/.crashrc are executed
42       first, followed by those in the ./.crashrc file.
43
44       The crash utility is designed to be independent of Linux version depen‐
45       dencies.  When new kernel source code impacts the correct functionality
46       of crash and its command set, the utility will be updated to  recognize
47       new kernel code changes, while maintaining backwards compatibility with
48       earlier releases.
49

OPTIONS

51       NAMELIST
52              This is a pathname to an uncompressed kernel  image  (a  vmlinux
53              file),  or  a  Xen  hypervisor image (a xen-syms file) which has
54              been compiled with the "-g" option.  If using the dumpfile form,
55              a  vmlinux  file  may be compressed in either gzip or bzip2 for‐
56              mats.
57
58       MEMORY-IMAGE[@ADDRESS]
59              A kernel core dump file created by the netdump,  diskdump,  LKCD
60              kdump, xendump or kvmdump facilities.
61
62              If  a  MEMORY-IMAGE argument is not entered, the session will be
63              invoked on the live system, which typically requires root privi‐
64              leges  because of the device file used to access system RAM.  By
65              default, /dev/crash will be used if it exists.  If it  does  not
66              exist,  then  /dev/mem  will be used; but if the kernel has been
67              configured with CONFIG_STRICT_DEVMEM, then /proc/kcore  will  be
68              used.    It  is  permissible  to  explicitly  enter  /dev/crash,
69              /dev/mem or /proc/kcore.
70
71              An @ADDRESS value must be appended to the  MEMORY-IMAGE  if  the
72              dumpfile  is  a  raw RAM dumpfile that has no header information
73              describing the  file  contents.   Multiple  MEMORY-IMAGE@ADDRESS
74              ordered  pairs  may  be entered, with each dumpfile containing a
75              contiguous block of RAM, where the ADDRESS value is the physical
76              start address of the block expressed in hexadecimal.  The physi‐
77              cal address value(s) will be used  to  create  a  temporary  ELF
78              header  in /var/tmp, which will only exist during the crash ses‐
79              sion.  If a raw RAM dumpile represents  a  live  memory  source,
80              such  as  that specified by the QEMU mem-path argument of a mem‐
81              ory-backend-file object, then "live:" must be prepended  to  the
82              MEMORY-IMAGE name.
83
84       mapfile
85              If  the  NAMELIST  file  is  not the same kernel that is running
86              (live system form), or the kernel that was running when the sys‐
87              tem  crashed  (dumpfile  form),  then the System.map file of the
88              original kernel should be entered on the command line.
89
90       -h [option]
91       --help [option]
92              Without an option argument, display a crash usage help  message.
93              If  the  option  argument is a crash command name, the help page
94              for that command is displayed.  If it is the string  "input",  a
95              page  describing the various crash command line input options is
96              displayed.  If it is the string "output", a page describing com‐
97              mand  line  output  options  is  displayed.  If it is the string
98              "all", then all of the possible  help  messages  are  displayed.
99              After the help message is displayed, crash exits.
100
101       -s     Silently  proceed  directly  to the "crash>" prompt without dis‐
102              playing any version, GPL, or crash  initialization  data  during
103              startup, and by default, runtime command output is not passed to
104              any scrolling command.
105
106       -i file
107              Execute the command(s) contained in file prior to displaying the
108              "crash>" prompt for interactive user input.
109
110       -d num Set  the  internal debug level.  The higher the number, the more
111              debugging data will be printed when crash initializes and runs.
112
113       -S     Use /boot/System.map as the mapfile.
114
115       -e vi | emacs
116              Set the  readline(3)  command  line  editing  mode  to  "vi"  or
117              "emacs".  The default editing mode is "vi".
118
119       -f     Force  the  usage  of  a compressed vmlinux file if its original
120              name does not start with "vmlinux".
121
122       -k     Indicate that the NAMELIST file is an LKCD "Kerntypes" debuginfo
123              file.
124
125       -g [namelist]
126              Determine if a vmlinux or xen-syms namelist file contains debug‐
127              ging data.
128
129       -t     Display the system-crash timestamp and exit.
130
131       -L     Attempt to lock all of its virtual address space into memory  by
132              calling  mlockall(MCL_CURRENT|MCL_FUTURE) during initialization.
133              If the system call fails, an error message  will  be  displayed,
134              but the session continues.
135
136       -c tty-device
137              Open the tty-device as the console used for debug messages.
138
139       -p page-size
140              If a processor's page size cannot be determined by the dumpfile,
141              and the processor default cannot be used, use page-size.
142
143       -o filename
144              Only used with the MEMORY-IMAGE@ADDRESS format for raw RAM dump‐
145              files,  specifies  a  filename  of a new ELF vmcore that will be
146              created and used as the dumpfile.  It will  be  saved  to  allow
147              future  use  as  a standalone vmcore, replacing the original raw
148              RAM dumpfile.
149
150       -m option=value
151       --machdep option=value
152              Pass an option and value pair to machine-dependent code.   These
153              architecture-specific  option/pairs  should  only be required in
154              very rare circumstances:
155
156              X86_64:
157                phys_base=<physical-address>
158                irq_eframe_link=<value>
159                irq_stack_gap=<value>
160                max_physmem_bits=<value>
161                kernel_image_size=<value>
162                vm=orig       (pre-2.6.11 virtual memory address ranges)
163                vm=2.6.11     (2.6.11 and later virtual memory address ranges)
164                vm=xen        (Xen kernel virtual memory address ranges)
165                vm=xen-rhel4  (RHEL4 Xen kernel virtual address ranges)
166                vm=5level     (5-level page tables)
167              PPC64:
168                vm=orig
169                vm=2.6.14     (4-level page tables)
170              IA64:
171                phys_start=<physical-address>
172                init_stack_size=<size>
173                vm=4l         (4-level page tables)
174              ARM:
175                phys_base=<physical-address>
176              ARM64:
177                phys_offset=<physical-address>
178                kimage_voffset=<kimage_voffset-value>
179              X86:
180                page_offset=<CONFIG_PAGE_OFFSET-value>
181
182       -x     Automatically load extension modules from  a  particular  direc‐
183              tory.  If a directory is specified in the CRASH_EXTENSIONS shell
184              environment variable, then that directory will be used.   Other‐
185              wise   /usr/lib64/crash/extensions   (64-bit  architectures)  or
186              /usr/lib/crash/extensions (32-bit architectures) will  be  used;
187              if  they  do  not exist, then the ./extensions directory will be
188              used.
189
190       --active
191              Track only the active task on each cpu.
192
193       --buildinfo
194              Display the crash binary's  build  date,  the  user  ID  of  the
195              builder,  the  hostname of the machine where the build was done,
196              the target architecture, the version number,  and  the  compiler
197              version.
198
199       --memory_module modname
200              Use  the modname as an alternative kernel module to the crash.ko
201              module that creates the /dev/crash device.
202
203       --memory_device device
204              Use device as an alternative device to the /dev/crash,  /dev/mem
205              or /proc/kcore devices.
206
207       --log dumpfile
208              Dump  the  contents of the kernel log buffer.  A kernel namelist
209              argument is not necessary, but the  dumpfile  must  contain  the
210              VMCOREINFO data taken from the original /proc/vmcore ELF header.
211
212       --no_kallsyms
213              Do  not  use  kallsyms-generated  symbol  information  contained
214              within kernel module object files.
215
216       --no_modules
217              Do not access or display any kernel module related information.
218
219       --no_ikconf
220              Do not attempt to read configuration data that  was  built  into
221              kernels configured with CONFIG_IKCONFIG.
222
223       --no_data_debug
224              Do  not  verify the validity of all structure member offsets and
225              structure sizes that it uses.
226
227       --no_kmem_cache
228              Do not initialize the kernel's slab  cache  infrastructure,  and
229              commands that use kmem_cache-related data will not work.
230
231       --no_elf_notes
232              Do not use the registers from the ELF NT_PRSTATUS notes saved in
233              a compressed kdump header for backtraces.
234
235       --kmem_cache_delay
236              Delay the initialization of the kernel's slab cache  infrastruc‐
237              ture until it is required by a run-time command.
238
239       --readnow
240              Pass  this  flag to the embedded gdb module, which will override
241              its two-stage strategy that it uses for  reading  symbol  tables
242              from the NAMELIST.
243
244       --smp  Specify that the system being analyzed is an SMP kernel.
245
246       -v
247       --version
248              Display  the  version  of  the crash utility, the version of the
249              embedded gdb module, GPL information, and copyright notices.
250
251       --cpus number
252              Specify the number of cpus in the SMP system being analyzed.
253
254       --osrelease dumpfile
255              Display the OSRELEASE vmcoreinfo string from  a  kdump  dumpfile
256              header.
257
258       --hyper
259              Force the session to be that of a Xen hypervisor.
260
261       --p2m_mfn pfn
262              When  a  Xen Hypervisor or its dom0 kernel crashes, the dumpfile
263              is typically analyzed with either the Xen hypervisor or the dom0
264              kernel.   It  is  also possible to analyze any of the guest domU
265              kernels if the pfn_to_mfn_list_list pfn value of the guest  ker‐
266              nel  is  passed  on the command line along with its NAMELIST and
267              the dumpfile.
268
269       --xen_phys_start physical-address
270              Supply the base physical address of the  Xen  hypervisor's  text
271              and  static  data  for older xendump dumpfiles that did not pass
272              that information in the dumpfile header.
273
274       --zero_excluded
275              If the makedumpfile(8) facility has filtered a compressed  kdump
276              dumpfile to exclude various types of non-essential pages, or has
277              marked a compressed or ELF kdump dumpfile as incomplete  due  to
278              an  ENOSPC  or  other  error during its creation, any attempt to
279              read missing pages will fail.  With this flag, reads from any of
280              those pages will return zero-filled memory.
281
282       --no_panic
283              Do not attempt to find the task that was running when the kernel
284              crashed.  Set the initial context to that of the "swapper"  task
285              on cpu 0.
286
287       --more Use  /bin/more  as  the  command output scroller, overriding the
288              default of /usr/bin/less and any settings in  either  ./.crashrc
289              or $HOME/.crashrc.
290
291       --less Use /usr/bin/less as the command output scroller, overriding any
292              settings in either ./.crashrc or $HOME/.crashrc.
293
294       --hex  Set the default command  output  radix  to  16,  overriding  the
295              default radix of 10, and any radix settings in either ./.crashrc
296              or $HOME/.crashrc.
297
298       --dec  Set the default command output radix to 10, overriding any radix
299              settings  in  either  ./.crashrc  or $HOME/.crashrc. This is the
300              default radix setting.
301
302       --CRASHPAGER
303              Use the output paging command defined in  the  CRASHPAGER  shell
304              environment   variable,   overriding   any  settings  in  either
305              ./.crashrc or $HOME/.crashrc.
306
307       --no_scroll
308              Do not pass run-time command output to any scrolling command.
309
310       --no_strip
311              Do not strip cloned kernel text symbol names.
312
313       --no_crashrc
314              Do  not  execute  the  commands  in  either  $HOME/.crashrc   or
315              ./.crashrc.
316
317       --mod directory
318              When  loading  the debuginfo data of kernel modules with the mod
319              -S command, search for their object files in  directory  instead
320              of in the standard location.
321
322       --kaslr offset|auto
323              If  an  x86_64 kernel was configured with CONFIG_RANDOMIZE_BASE,
324              the offset value is equal to the difference between  the  symbol
325              values  compiled into the vmlinux file and their relocated KASLR
326              values.  If set to auto, the KASLR offset value will be automat‐
327              ically calculated.
328
329       --reloc size
330              When analyzing live x86 kernels that were configured with a CON‐
331              FIG_PHYSICAL_START value that is larger than  its  CONFIG_PHYSI‐
332              CAL_ALIGN value, then it will be necessary to enter a relocation
333              size equal to the difference between the two values.
334
335       --hash count
336              Set the number of internal hash queue heads used for list  gath‐
337              ering and verification.  The default count is 32768.
338
339       --minimal
340              Bring  up a session that is restricted to the log, dis, rd, sym,
341              eval, set and exit commands.  This option may provide a  way  to
342              extract some minimal/quick information from a corrupted or trun‐
343              cated dumpfile, or in situations where one of the several kernel
344              subsystem initialization routines would abort the crash session.
345
346       --kvmhost [32|64]
347              When  examining an x86 KVM guest dumpfile, this option specifies
348              that the KVM host that created the dumpfile was an x86  (32-bit)
349              or  an  x86_64  (64-bit)  machine,  overriding the automatically
350              determined value.
351
352       --kvmio <size>
353              override the automatically-calculated KVM guest I/O hole size.
354
355       --offline [show|hide]
356              Show or hide command output that is  related  to  offline  cpus.
357              The default setting is show.
358

COMMANDS

360       Each  crash  command  generally  falls  into one of the following cate‐
361       gories:
362
363       Symbolic display
364              Displays of kernel text/data, which take full advantage  of  the
365              power of gdb to format and display data structures symbolically.
366
367       System state
368              The  majority  of  crash  commands  consist of a set of "kernel-
369              aware" commands, which delve into various kernel subsystems on a
370              system-wide or per-task basis.
371
372       Utility functions
373              A  set  of useful helper commands serving various purposes, some
374              simple, others quite powerful.
375
376       Session control
377              Commands that control the crash session itself.
378
379       The following alphabetical list consists of a very simple  overview  of
380       each crash command.  However, since individual commands often have sev‐
381       eral options resulting in significantly different output,  it  is  sug‐
382       gested that the full description of each command be viewed by executing
383       crash -h <command>, or during a crash session by simply  entering  help
384       command.
385
386       *      "pointer  to"  is  shorthand for either the struct or union com‐
387              mands.  It displays the contents of a kernel structure or union.
388
389       alias  creates a single-word alias for a command.
390
391       ascii  displays an ascii chart or translates a numeric value  into  its
392              ascii components.
393
394       bt     displays a task's kernel-stack backtrace.  If it is given the -a
395              option, it displays the stack traces of the active tasks on  all
396              CPUs.   It is often used with the foreach command to display the
397              backtraces of all tasks with one command.
398
399       btop   translates a byte value (physical offset) to its page number.
400
401       dev    displays data concerning the character and block device  assign‐
402              ments, I/O port usage, I/O memory usage, and PCI device data.
403
404       dis    disassembles  memory,  either  entire  kernel  functions, from a
405              location for a specified number of  instructions,  or  from  the
406              start of a function up to a specified memory location.
407
408       eval   evaluates  an expression or numeric type and displays the result
409              in hexadecimal, decimal, octal and binary.
410
411       exit   causes crash to exit.
412
413       extend dynamically loads or unloads crash shared object extension  mod‐
414              ules.
415
416       files  displays information about open files in a context.
417
418       foreach
419              repeats  a specified command for the specified (or all) tasks in
420              the system.
421
422       fuser  displays the tasks using the specified file or socket.
423
424       gdb    passes its argument to the embedded gdb module.   It  is  useful
425              for executing gdb commands that have the same name as crash com‐
426              mands.
427
428       help   alone displays the command menu; if followed by a command  name,
429              a  full  description of a command, its options, and examples are
430              displayed.  Its output is far more complete and useful than this
431              man page.
432
433       ipcs   displays data about the System V IPC facilities.
434
435       irq    displays  data  concerning interrupt request numbers and bottom-
436              half interrupt handling.
437
438       kmem   displays information about the use of kernel memory.
439
440       list   displays the contents of a linked list.
441
442       log    displays the kernel log_buf contents in chronological order.
443
444       mach   displays data specific to the machine type.
445
446       mod    displays information about the currently installed  kernel  mod‐
447              ules, or adds or deletes symbolic or debugging information about
448              specified kernel modules.
449
450       mount  displays information about the currently-mounted filesystems.
451
452       net    display various network related data.
453
454       p      passes its arguments to the gdb "print" command  for  evaluation
455              and display.
456
457       ps     displays  process status for specified, or all, processes in the
458              system.
459
460       pte    translates the hexadecimal contents of a PTE into  its  physical
461              page address and page bit settings.
462
463       ptob   translates a page frame number to its byte value.
464
465       ptov   translates  a hexadecimal physical address into a kernel virtual
466              address.
467
468       q      is an alias for the "exit" command.
469
470       rd     displays the contents of memory, with the  output  formatted  in
471              several different manners.
472
473       repeat repeats a command indefinitely, optionally delaying a given num‐
474              ber of seconds between each command execution.
475
476       runq   displays the tasks on the run queue.
477
478       search searches a range of user or kernel memory space for given value.
479
480       set    either sets a new context, or gets the current context for  dis‐
481              play.
482
483       sig    displays signal-handling data of one or more tasks.
484
485       struct displays either a structure definition or the contents of a ker‐
486              nel structure at a specified address.
487
488       swap   displays information about each configured swap device.
489
490       sym    translates a symbol to its virtual address, or a  static  kernel
491              virtual  address  to  its  symbol  -- or to a symbol-plus-offset
492              value, if appropriate.
493
494       sys    displays system-specific data.
495
496       task   displays the contents of a task_struct.
497
498       tree   displays the contents of a red-black tree or a radix tree.
499
500       timer  displays the timer queue entries, both old-  and  new-style,  in
501              chronological order.
502
503       union  is similar to the struct command, except that it works on kernel
504              unions.
505
506       vm     displays basic virtual memory information of a context.
507
508       vtop   translates a user or kernel  virtual  address  to  its  physical
509              address.
510
511       waitq  walks the wait queue list displaying the tasks which are blocked
512              on the specified wait queue.
513
514       whatis displays the  definition  of  structures,  unions,  typedefs  or
515              text/data symbols.
516
517       wr     modifies  the  contents of memory on a live system.  It can only
518              be used if /dev/mem is the device file being used to access sys‐
519              tem RAM, and should obviously be used with great care.
520
521       When crash is invoked with a Xen hypervisor binary as the NAMELIST, the
522       command set is slightly modified.  The *, alias, ascii, bt, dis,  eval,
523       exit,  extend,  gdb,  help, list, log, p, pte, rd, repeat, search, set,
524       struct, sym, sys, union, whatis, wr and q  commands  are  the  same  as
525       above.  The following commands are specific to the Xen hypervisor:
526
527       domain displays  the  contents of the domain structure for selected, or
528              all, domains.
529
530       doms   displays domain status for selected, or all, domains.
531
532       dumpinfo
533              displays Xen dump information for selected, or all, cpus.
534
535       pcpus  displays physical cpu information for selected, or all, cpus.
536
537       vcpus  displays vcpu status for selected, or all, vcpus.
538

FILES

540       .crashrc
541              Initialization commands.  The file can be located in the  user's
542              HOME  directory and/or the current directory.  Commands found in
543              the .crashrc file in the  HOME  directory  are  executed  before
544              those in the current directory's .crashrc file.
545

ENVIRONMENT

547       EDITOR Command  input  is  read using readline(3).  If EDITOR is set to
548              emacs or vi then suitable keybindings are used.   If  EDITOR  is
549              not  set,  then vi is used.  This can be overridden by set vi or
550              set emacs commands located in a .crashrc file, or by entering -e
551              emacs on the crash command line.
552
553       CRASHPAGER
554              If  CRASHPAGER is set, its value is used as the name of the pro‐
555              gram to which command output will be sent.  If not, then command
556              output is sent to /usr/bin/less -E -X by default.
557
558       CRASH_MODULE_PATH
559              Specifies  an  alternative  directory  tree to search for kernel
560              module object files.
561
562       CRASH_EXTENSIONS
563              Specifies a directory containing extension modules that will  be
564              loaded automatically if the -x command line option is used.
565

NOTES

567       If crash does not work, look for a newer version: kernel evolution fre‐
568       quently makes crash updates necessary.
569
570       The command set scroll off will cause output to be sent directly to the
571       terminal  rather  than  through  a paging program.  This is useful, for
572       example, if you are running crash in a window of emacs.
573

AUTHOR

575       Dave Anderson <anderson@redhat.com> wrote crash.
576
577       Jay Fenlason <fenlason@redhat.com>  and  Dave  Anderson  <anderson@red‐
578       hat.com> wrote this man page.
579

SEE ALSO

581       The help command within crash provides more complete and accurate docu‐
582       mentation than this man page.
583
584       http://people.redhat.com/anderson - the home page of the crash utility.
585
586       netdump(8), gdb(1), makedumpfile(8)
587
588
589
590                                                                      CRASH(8)
Impressum