1CRASH(8)                    System Manager's Manual                   CRASH(8)
2
3
4

NAME

6       crash - Analyze Linux crash dump data or a live system
7

SYNOPSIS

9       crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS]    (dumpfile form)
10       crash [OPTION]... [NAMELIST]                         (live system form)
11

DESCRIPTION

13       Crash is a tool for interactively analyzing the state of the Linux sys‐
14       tem while it is running, or after a kernel crash  has  occurred  and  a
15       core  dump has been created by the netdump, diskdump, LKCD, kdump, xen‐
16       dump or kvmdump facilities.  It is loosely based on the SVR4 UNIX crash
17       command,  but  has been significantly enhanced by completely merging it
18       with the gdb(1) debugger. The marriage of the two effectively  combines
19       the  kernel-specific  nature of the traditional UNIX crash utility with
20       the source code level debugging capabilities of gdb(1).
21
22       In the dumpfile form, both a NAMELIST and a MEMORY-IMAGE argument  must
23       be  entered.   In  the  live system form, the NAMELIST argument must be
24       entered if the kernel's vmlinux file is not located in  a  known  loca‐
25       tion,  such  as  the /usr/lib/debug/lib/modules/<kernel-version> direc‐
26       tory.
27
28       The crash utility has also been extended to  support  the  analysis  of
29       dumpfiles  generated  by  a crash of the Xen hypervisor.  In that case,
30       the NAMELIST argument must be that of the xen-syms binary.  Live system
31       analysis is not supported for the Xen hypervisor.
32
33       The  crash  utility command set consists of common kernel core analysis
34       tools such as kernel stack back traces of all  processes,  source  code
35       disassembly,  formatted kernel structure and variable displays, virtual
36       memory data, dumps of linked-lists, etc., along with  several  commands
37       that  delve  deeper  into  specific kernel subsystems.  Appropriate gdb
38       commands may also be entered, which in turn are passed on  to  the  gdb
39       module  for  execution.  If desired, commands may be placed in either a
40       $HOME/.crashrc file and/or in a .crashrc file in the current directory.
41       During  initialization,  the  commands  in  $HOME/.crashrc are executed
42       first, followed by those in the ./.crashrc file.
43
44       The crash utility is designed to be independent of Linux version depen‐
45       dencies.  When new kernel source code impacts the correct functionality
46       of crash and its command set, the utility will be updated to  recognize
47       new kernel code changes, while maintaining backwards compatibility with
48       earlier releases.
49

OPTIONS

51       NAMELIST
52              This is a pathname to an uncompressed kernel  image  (a  vmlinux
53              file),  or  a  Xen  hypervisor image (a xen-syms file) which has
54              been compiled with the "-g" option.  If using the dumpfile form,
55              a  vmlinux  file  may be compressed in either gzip or bzip2 for‐
56              mats.
57
58       MEMORY-IMAGE[@ADDRESS]
59              A kernel core dump file created by the netdump,  diskdump,  LKCD
60              kdump, xendump or kvmdump facilities.
61
62              If  a  MEMORY-IMAGE argument is not entered, the session will be
63              invoked on the live system, which typically requires root privi‐
64              leges  because of the device file used to access system RAM.  By
65              default, /dev/crash will be used if it exists.  If it  does  not
66              exist,  then  /dev/mem  will be used; but if the kernel has been
67              configured with CONFIG_STRICT_DEVMEM, then /proc/kcore  will  be
68              used.    It  is  permissible  to  explicitly  enter  /dev/crash,
69              /dev/mem or /proc/kcore.
70
71              An @ADDRESS value must be appended to the  MEMORY-IMAGE  if  the
72              dumpfile  is  a  raw RAM dumpfile that has no header information
73              describing the  file  contents.   Multiple  MEMORY-IMAGE@ADDRESS
74              ordered  pairs  may  be entered, with each dumpfile containing a
75              contiguous block of RAM, where the ADDRESS value is the physical
76              start address of the block expressed in hexadecimal.  The physi‐
77              cal address value(s) will be used  to  create  a  temporary  ELF
78              header  in /var/tmp, which will only exist during the crash ses‐
79              sion.  If a raw RAM dumpile represents  a  live  memory  source,
80              such  as  that specified by the QEMU mem-path argument of a mem‐
81              ory-backend-file object, then "live:" must be prepended  to  the
82              MEMORY-IMAGE name.
83
84       mapfile
85              If  the  NAMELIST  file  is  not the same kernel that is running
86              (live system form), or the kernel that was running when the sys‐
87              tem  crashed  (dumpfile  form),  then the System.map file of the
88              original kernel should be entered on the command line.
89
90       -h [option]
91       --help [option]
92              Without an option argument, display a crash usage help  message.
93              If  the  option  argument is a crash command name, the help page
94              for that command is displayed.  If it is the string  "input",  a
95              page  describing the various crash command line input options is
96              displayed.  If it is the string "output", a page describing com‐
97              mand  line  output  options  is  displayed.  If it is the string
98              "all", then all of the possible  help  messages  are  displayed.
99              After the help message is displayed, crash exits.
100
101       -s     Silently  proceed  directly  to the "crash>" prompt without dis‐
102              playing any version, GPL, or crash  initialization  data  during
103              startup, and by default, runtime command output is not passed to
104              any scrolling command.
105
106       -i file
107              Execute the command(s) contained in file prior to displaying the
108              "crash>" prompt for interactive user input.
109
110       -d num Set  the  internal debug level.  The higher the number, the more
111              debugging data will be printed when crash initializes and runs.
112
113       -S     Use /boot/System.map as the mapfile.
114
115       -e vi | emacs
116              Set the  readline(3)  command  line  editing  mode  to  "vi"  or
117              "emacs".  The default editing mode is "vi".
118
119       -f     Force  the  usage  of  a compressed vmlinux file if its original
120              name does not start with "vmlinux".
121
122       -k     Indicate that the NAMELIST file is an LKCD "Kerntypes" debuginfo
123              file.
124
125       -g [namelist]
126              Determine if a vmlinux or xen-syms namelist file contains debug‐
127              ging data.
128
129       -t     Display the system-crash timestamp and exit.
130
131       -L     Attempt to lock all of its virtual address space into memory  by
132              calling  mlockall(MCL_CURRENT|MCL_FUTURE) during initialization.
133              If the system call fails, an error message  will  be  displayed,
134              but the session continues.
135
136       -c tty-device
137              Open the tty-device as the console used for debug messages.
138
139       -p page-size
140              If a processor's page size cannot be determined by the dumpfile,
141              and the processor default cannot be used, use page-size.
142
143       -o filename
144              Only used with the MEMORY-IMAGE@ADDRESS format for raw RAM dump‐
145              files,  specifies  a  filename  of a new ELF vmcore that will be
146              created and used as the dumpfile.  It will  be  saved  to  allow
147              future  use  as  a standalone vmcore, replacing the original raw
148              RAM dumpfile.
149
150       -m option=value
151       --machdep option=value
152              Pass an option and value pair to machine-dependent code.   These
153              architecture-specific  option/pairs  should  only be required in
154              very rare circumstances:
155
156              X86_64:
157                phys_base=<physical-address>
158                irq_eframe_link=<value>
159                irq_stack_gap=<value>
160                max_physmem_bits=<value>
161                kernel_image_size=<value>
162                vm=orig       (pre-2.6.11 virtual memory address ranges)
163                vm=2.6.11     (2.6.11 and later virtual memory address ranges)
164                vm=xen        (Xen kernel virtual memory address ranges)
165                vm=xen-rhel4  (RHEL4 Xen kernel virtual address ranges)
166                vm=5level     (5-level page tables)
167              PPC64:
168                vm=orig
169                vm=2.6.14     (4-level page tables)
170              IA64:
171                phys_start=<physical-address>
172                init_stack_size=<size>
173                vm=4l         (4-level page tables)
174              ARM:
175                phys_base=<physical-address>
176              ARM64:
177                phys_offset=<physical-address>
178                kimage_voffset=<kimage_voffset-value>
179                max_physmem_bits=<value>
180              X86:
181                page_offset=<CONFIG_PAGE_OFFSET-value>
182
183       -x     Automatically load extension modules from  a  particular  direc‐
184              tory.  If a directory is specified in the CRASH_EXTENSIONS shell
185              environment variable, then that directory will be used.   Other‐
186              wise   /usr/lib64/crash/extensions   (64-bit  architectures)  or
187              /usr/lib/crash/extensions (32-bit architectures) will  be  used;
188              if  they  do  not exist, then the ./extensions directory will be
189              used.
190
191       --active
192              Track only the active task on each cpu.
193
194       --buildinfo
195              Display the crash binary's  build  date,  the  user  ID  of  the
196              builder,  the  hostname of the machine where the build was done,
197              the target architecture, the version number,  and  the  compiler
198              version.
199
200       --memory_module modname
201              Use  the modname as an alternative kernel module to the crash.ko
202              module that creates the /dev/crash device.
203
204       --memory_device device
205              Use device as an alternative device to the /dev/crash,  /dev/mem
206              or /proc/kcore devices.
207
208       --log dumpfile
209              Dump  the  contents of the kernel log buffer.  A kernel namelist
210              argument is not necessary, but the  dumpfile  must  contain  the
211              VMCOREINFO data taken from the original /proc/vmcore ELF header.
212
213       --no_kallsyms
214              Do  not  use  kallsyms-generated  symbol  information  contained
215              within kernel module object files.
216
217       --no_modules
218              Do not access or display any kernel module related information.
219
220       --no_ikconf
221              Do not attempt to read configuration data that  was  built  into
222              kernels configured with CONFIG_IKCONFIG.
223
224       --no_data_debug
225              Do  not  verify the validity of all structure member offsets and
226              structure sizes that it uses.
227
228       --no_kmem_cache
229              Do not initialize the kernel's slab  cache  infrastructure,  and
230              commands that use kmem_cache-related data will not work.
231
232       --no_elf_notes
233              Do not use the registers from the ELF NT_PRSTATUS notes saved in
234              a compressed kdump header for backtraces.
235
236       --kmem_cache_delay
237              Delay the initialization of the kernel's slab cache  infrastruc‐
238              ture until it is required by a run-time command.
239
240       --readnow
241              Pass  this  flag to the embedded gdb module, which will override
242              its two-stage strategy that it uses for  reading  symbol  tables
243              from the NAMELIST.
244
245       --smp  Specify that the system being analyzed is an SMP kernel.
246
247       -v
248       --version
249              Display  the  version  of  the crash utility, the version of the
250              embedded gdb module, GPL information, and copyright notices.
251
252       --cpus number
253              Specify the number of cpus in the SMP system being analyzed.
254
255       --osrelease dumpfile
256              Display the OSRELEASE vmcoreinfo string from  a  kdump  dumpfile
257              header.
258
259       --hyper
260              Force the session to be that of a Xen hypervisor.
261
262       --p2m_mfn pfn
263              When  a  Xen Hypervisor or its dom0 kernel crashes, the dumpfile
264              is typically analyzed with either the Xen hypervisor or the dom0
265              kernel.   It  is  also possible to analyze any of the guest domU
266              kernels if the pfn_to_mfn_list_list pfn value of the guest  ker‐
267              nel  is  passed  on the command line along with its NAMELIST and
268              the dumpfile.
269
270       --xen_phys_start physical-address
271              Supply the base physical address of the  Xen  hypervisor's  text
272              and  static  data  for older xendump dumpfiles that did not pass
273              that information in the dumpfile header.
274
275       --zero_excluded
276              If the makedumpfile(8) facility has filtered a compressed  kdump
277              dumpfile to exclude various types of non-essential pages, or has
278              marked a compressed or ELF kdump dumpfile as incomplete  due  to
279              an  ENOSPC  or  other  error during its creation, any attempt to
280              read missing pages will fail.  With this flag, reads from any of
281              those pages will return zero-filled memory.
282
283       --no_panic
284              Do not attempt to find the task that was running when the kernel
285              crashed.  Set the initial context to that of the "swapper"  task
286              on cpu 0.
287
288       --more Use  /bin/more  as  the  command output scroller, overriding the
289              default of /usr/bin/less and any settings in  either  ./.crashrc
290              or $HOME/.crashrc.
291
292       --less Use /usr/bin/less as the command output scroller, overriding any
293              settings in either ./.crashrc or $HOME/.crashrc.
294
295       --hex  Set the default command  output  radix  to  16,  overriding  the
296              default radix of 10, and any radix settings in either ./.crashrc
297              or $HOME/.crashrc.
298
299       --dec  Set the default command output radix to 10, overriding any radix
300              settings  in  either  ./.crashrc  or $HOME/.crashrc. This is the
301              default radix setting.
302
303       --CRASHPAGER
304              Use the output paging command defined in  the  CRASHPAGER  shell
305              environment   variable,   overriding   any  settings  in  either
306              ./.crashrc or $HOME/.crashrc.
307
308       --no_scroll
309              Do not pass run-time command output to any scrolling command.
310
311       --no_strip
312              Do not strip cloned kernel text symbol names.
313
314       --no_crashrc
315              Do  not  execute  the  commands  in  either  $HOME/.crashrc   or
316              ./.crashrc.
317
318       --mod directory
319              When  loading  the debuginfo data of kernel modules with the mod
320              -S command, search for their object files in  directory  instead
321              of in the standard location.
322
323       --kaslr offset|auto
324              If  an  x86_64 kernel was configured with CONFIG_RANDOMIZE_BASE,
325              the offset value is equal to the difference between  the  symbol
326              values  compiled into the vmlinux file and their relocated KASLR
327              values.  If set to auto, the KASLR offset value will be automat‐
328              ically calculated.
329
330       --reloc size
331              When analyzing live x86 kernels that were configured with a CON‐
332              FIG_PHYSICAL_START value that is larger than  its  CONFIG_PHYSI‐
333              CAL_ALIGN value, then it will be necessary to enter a relocation
334              size equal to the difference between the two values.
335
336       --hash count
337              Set the number of internal hash queue heads used for list  gath‐
338              ering and verification.  The default count is 32768.
339
340       --minimal
341              Bring  up a session that is restricted to the log, dis, rd, sym,
342              eval, set and exit commands.  This option may provide a  way  to
343              extract some minimal/quick information from a corrupted or trun‐
344              cated dumpfile, or in situations where one of the several kernel
345              subsystem initialization routines would abort the crash session.
346
347       --kvmhost [32|64]
348              When  examining an x86 KVM guest dumpfile, this option specifies
349              that the KVM host that created the dumpfile was an x86  (32-bit)
350              or  an  x86_64  (64-bit)  machine,  overriding the automatically
351              determined value.
352
353       --kvmio <size>
354              override the automatically-calculated KVM guest I/O hole size.
355
356       --offline [show|hide]
357              Show or hide command output that is  related  to  offline  cpus.
358              The default setting is show.
359

COMMANDS

361       Each  crash  command  generally  falls  into one of the following cate‐
362       gories:
363
364       Symbolic display
365              Displays of kernel text/data, which take full advantage  of  the
366              power of gdb to format and display data structures symbolically.
367
368       System state
369              The  majority  of  crash  commands  consist of a set of "kernel-
370              aware" commands, which delve into various kernel subsystems on a
371              system-wide or per-task basis.
372
373       Utility functions
374              A  set  of useful helper commands serving various purposes, some
375              simple, others quite powerful.
376
377       Session control
378              Commands that control the crash session itself.
379
380       The following alphabetical list consists of a very simple  overview  of
381       each crash command.  However, since individual commands often have sev‐
382       eral options resulting in significantly different output,  it  is  sug‐
383       gested that the full description of each command be viewed by executing
384       crash -h <command>, or during a crash session by simply  entering  help
385       command.
386
387       *      "pointer  to"  is  shorthand for either the struct or union com‐
388              mands.  It displays the contents of a kernel structure or union.
389
390       alias  creates a single-word alias for a command.
391
392       ascii  displays an ascii chart or translates a numeric value  into  its
393              ascii components.
394
395       bt     displays a task's kernel-stack backtrace.  If it is given the -a
396              option, it displays the stack traces of the active tasks on  all
397              CPUs.   It is often used with the foreach command to display the
398              backtraces of all tasks with one command.
399
400       btop   translates a byte value (physical offset) to its page number.
401
402       dev    displays data concerning the character and block device  assign‐
403              ments, I/O port usage, I/O memory usage, and PCI device data.
404
405       dis    disassembles  memory,  either  entire  kernel  functions, from a
406              location for a specified number of  instructions,  or  from  the
407              start of a function up to a specified memory location.
408
409       eval   evaluates  an expression or numeric type and displays the result
410              in hexadecimal, decimal, octal and binary.
411
412       exit   causes crash to exit.
413
414       extend dynamically loads or unloads crash shared object extension  mod‐
415              ules.
416
417       files  displays information about open files in a context.
418
419       foreach
420              repeats  a specified command for the specified (or all) tasks in
421              the system.
422
423       fuser  displays the tasks using the specified file or socket.
424
425       gdb    passes its argument to the embedded gdb module.   It  is  useful
426              for executing gdb commands that have the same name as crash com‐
427              mands.
428
429       help   alone displays the command menu; if followed by a command  name,
430              a  full  description of a command, its options, and examples are
431              displayed.  Its output is far more complete and useful than this
432              man page.
433
434       ipcs   displays data about the System V IPC facilities.
435
436       irq    displays  data  concerning interrupt request numbers and bottom-
437              half interrupt handling.
438
439       kmem   displays information about the use of kernel memory.
440
441       list   displays the contents of a linked list.
442
443       log    displays the kernel log_buf contents in chronological order.
444
445       mach   displays data specific to the machine type.
446
447       mod    displays information about the currently installed  kernel  mod‐
448              ules, or adds or deletes symbolic or debugging information about
449              specified kernel modules.
450
451       mount  displays information about the currently-mounted filesystems.
452
453       net    display various network related data.
454
455       p      passes its arguments to the gdb "print" command  for  evaluation
456              and display.
457
458       ps     displays  process status for specified, or all, processes in the
459              system.
460
461       pte    translates the hexadecimal contents of a PTE into  its  physical
462              page address and page bit settings.
463
464       ptob   translates a page frame number to its byte value.
465
466       ptov   translates  a hexadecimal physical address into a kernel virtual
467              address.
468
469       q      is an alias for the "exit" command.
470
471       rd     displays the contents of memory, with the  output  formatted  in
472              several different manners.
473
474       repeat repeats a command indefinitely, optionally delaying a given num‐
475              ber of seconds between each command execution.
476
477       runq   displays the tasks on the run queue.
478
479       search searches a range of user or kernel memory space for given value.
480
481       set    either sets a new context, or gets the current context for  dis‐
482              play.
483
484       sig    displays signal-handling data of one or more tasks.
485
486       struct displays either a structure definition or the contents of a ker‐
487              nel structure at a specified address.
488
489       swap   displays information about each configured swap device.
490
491       sym    translates a symbol to its virtual address, or a  static  kernel
492              virtual  address  to  its  symbol  -- or to a symbol-plus-offset
493              value, if appropriate.
494
495       sys    displays system-specific data.
496
497       task   displays the contents of a task_struct.
498
499       tree   displays the contents of a red-black tree or a radix tree.
500
501       timer  displays the timer queue entries, both old-  and  new-style,  in
502              chronological order.
503
504       union  is similar to the struct command, except that it works on kernel
505              unions.
506
507       vm     displays basic virtual memory information of a context.
508
509       vtop   translates a user or kernel  virtual  address  to  its  physical
510              address.
511
512       waitq  walks the wait queue list displaying the tasks which are blocked
513              on the specified wait queue.
514
515       whatis displays the  definition  of  structures,  unions,  typedefs  or
516              text/data symbols.
517
518       wr     modifies  the  contents of memory on a live system.  It can only
519              be used if /dev/mem is the device file being used to access sys‐
520              tem RAM, and should obviously be used with great care.
521
522       When crash is invoked with a Xen hypervisor binary as the NAMELIST, the
523       command set is slightly modified.  The *, alias, ascii, bt, dis,  eval,
524       exit,  extend,  gdb,  help, list, log, p, pte, rd, repeat, search, set,
525       struct, sym, sys, union, whatis, wr and q  commands  are  the  same  as
526       above.  The following commands are specific to the Xen hypervisor:
527
528       domain displays  the  contents of the domain structure for selected, or
529              all, domains.
530
531       doms   displays domain status for selected, or all, domains.
532
533       dumpinfo
534              displays Xen dump information for selected, or all, cpus.
535
536       pcpus  displays physical cpu information for selected, or all, cpus.
537
538       vcpus  displays vcpu status for selected, or all, vcpus.
539

FILES

541       .crashrc
542              Initialization commands.  The file can be located in the  user's
543              HOME  directory and/or the current directory.  Commands found in
544              the .crashrc file in the  HOME  directory  are  executed  before
545              those in the current directory's .crashrc file.
546

ENVIRONMENT

548       EDITOR Command  input  is  read using readline(3).  If EDITOR is set to
549              emacs or vi then suitable keybindings are used.   If  EDITOR  is
550              not  set,  then vi is used.  This can be overridden by set vi or
551              set emacs commands located in a .crashrc file, or by entering -e
552              emacs on the crash command line.
553
554       CRASHPAGER
555              If  CRASHPAGER is set, its value is used as the name of the pro‐
556              gram to which command output will be sent.  If not, then command
557              output is sent to /usr/bin/less -E -X by default.
558
559       CRASH_MODULE_PATH
560              Specifies  an  alternative  directory  tree to search for kernel
561              module object files.
562
563       CRASH_EXTENSIONS
564              Specifies a directory containing extension modules that will  be
565              loaded automatically if the -x command line option is used.
566

NOTES

568       If crash does not work, look for a newer version: kernel evolution fre‐
569       quently makes crash updates necessary.
570
571       The command set scroll off will cause output to be sent directly to the
572       terminal  rather  than  through  a paging program.  This is useful, for
573       example, if you are running crash in a window of emacs.
574

AUTHOR

576       Dave Anderson <anderson@redhat.com> wrote crash.
577
578       Jay Fenlason <fenlason@redhat.com>  and  Dave  Anderson  <anderson@red‐
579       hat.com> wrote this man page.
580

SEE ALSO

582       The help command within crash provides more complete and accurate docu‐
583       mentation than this man page.
584
585       http://people.redhat.com/anderson - the home page of the crash utility.
586
587       netdump(8), gdb(1), makedumpfile(8)
588
589
590
591                                                                      CRASH(8)
Impressum