1PCP-ATOP(1)                 General Commands Manual                PCP-ATOP(1)
2
3
4

NAME

6       pcp-atop - Advanced System and Process Monitor
7

SYNOPSIS

9       Interactive Usage:
10
11       pcp      [pcp options]      atop     [-g|-m|-d|-n|-u|-p|-s|-c|-v|-o|-y]
12       [-C|-M|-D|-N|-A] [-afFG1xR] [-L linelen] [-Plabel[,label]...] [interval
13       [samples]]
14
15       Writing and reading raw logfiles:
16
17       pcp atop -w rawfile [-a] [-S] [interval [samples]]
18       pcp atop    -r    [    rawfile    ]   [-b   hh:mm   ]   [-e   hh:mm   ]
19       [-g|-m|-d|-n|-u|-p|-s|-c|-v|-o|-y] [-C|-M|-D|-N|-A] [-fFG1xR] [-L line‐
20       len] [-Plabel[,label]...]
21

DESCRIPTION

23       The  program pcp-atop is an interactive monitor to view various aspects
24       of load on a system.  It shows the  occupation  of  the  most  critical
25       hardware  resources (from a performance point of view) on system level,
26       i.e. cpu, memory, disk and network.
27       It also shows which processes are responsible for  the  indicated  load
28       with  respect  to  cpu  and memory load on process level.  Disk load is
29       shown per process if "storage accounting" is active in the kernel.
30
31       Every interval (default: 10 seconds) information  is  shown  about  the
32       resource  occupation  on  system  level (cpu, memory, disks and network
33       layers), followed by a list of processes which have been active  during
34       the  last  interval (note that all processes that were unchanged during
35       the last interval are not shown, unless the key 'a' has been pressed or
36       unless  sorting  on  memory occupation is done).  If the list of active
37       processes does not entirely fit on the screen, only the top of the list
38       is shown (sorted in order of activity).
39       The  intervals  are  repeated  till the number of samples (specified as
40       command argument) is reached, or till the key 'q' is pressed in  inter‐
41       active mode.
42
43       When invoked via the pcp(1) command, the PCPIntro(1) options -h/--host,
44       -a/--archive, -O/--origin, -s/--samples,  -t/--interval,  -Z/--timezone
45       and  several  other  pcp options become indirectly available.  The long
46       option form of these is directly available.  Additionally,  the  --hot‐
47       proc  option can be used to request the per-process PCP metrics be used
48       instead of the default proc metrics from pmdaproc(1).
49
50       When pcp-atop is started, it checks whether the standard output channel
51       is  connected to a screen, or to a file/pipe. In the first case it pro‐
52       duces screen control codes (via the ncurses library) and behaves inter‐
53       actively; in the second case it produces flat ASCII-output.
54
55       In  interactive  mode, the output of pcp-atop scales dynamically to the
56       current dimensions of the screen/window.
57       If the window is resized horizontally, columns will be added or removed
58       automatically.  For this purpose, every column has a particular weight.
59       The columns with the highest weights that fit within the current  width
60       will be shown.
61       If  the  window is resized vertically, lines of the process/thread list
62       will be added or removed automatically.
63
64       Furthermore in interactive mode the output  of  pcp-atop  can  be  con‐
65       trolled  by  pressing  particular keys.  However it is also possible to
66       specify such key as flag on the command line.  In  that  case  pcp-atop
67       switches to the indicated mode on beforehand; this mode can be modified
68       again interactively. Specifying such key as flag is  especially  useful
69       when  running  pcp-atop  with  output  to  a pipe or file (non-interac‐
70       tively).  These flags are the same as the keys that can be  pressed  in
71       interactive mode (see section INTERACTIVE COMMANDS).
72       Additional  flags  are available to support storage of pcp-atop data in
73       PCP archive format (see section PCP DATA STORAGE).
74

COLORS

76       For the resource consumption on system level, pcp-atop uses  colors  to
77       indicate  that  a  critical  occupation  percentage  has  been (almost)
78       reached.  A critical occupation percentage means that  is  likely  that
79       this load causes a noticeable negative performance influence for appli‐
80       cations using this resource. The critical  percentage  depends  on  the
81       type  of resource: e.g. the performance influence of a disk with a busy
82       percentage of 80% might be more noticeable for applications/user than a
83       CPU with a busy percentage of 90%.
84       Currently  pcp-atop  uses  the  following default values to calculate a
85       weighted percentage per resource:
86
87        Processor
88            A busy percentage of 90% or higher is considered `critical'.
89
90        Disk
91            A busy percentage of 70% or higher is considered `critical'.
92
93        Network
94            A busy percentage of 90% or higher for the load of an interface is
95            considered `critical'.
96
97        Memory
98            An  occupation percentage of 90% is considered `critical'.  Notice
99            that this occupation percentage is the accumulated memory consump‐
100            tion  of the kernel (including slab) and all processes; the memory
101            for the page cache (`cache' and `buff' in the  MEM-line)  and  the
102            reclaimable part of the slab (`slrec`) is not implied!
103            If  the  number  of pages swapped out (`swout' in the PAG-line) is
104            larger than 10 per  second,  the  memory  resource  is  considered
105            `critical'.   A  value  of  at  least  1  per second is considered
106            `almost critical'.
107            If the committed virtual memory exceeds  the  limit  (`vmcom'  and
108            `vmlim'  in the SWP-line), the SWP-line is colored due to overcom‐
109            mitting the system.
110
111        Swap
112            An occupation percentage of 80% is considered  `critical'  because
113            swap space might be completely exhausted in the near future; it is
114            not critical from a performance point-of-view.
115
116       These default values can be modified in  the  configuration  file  (see
117       separate man-page of pcp-atoprc).
118
119       When  a  resource  exceeds its critical occupation percentage, the con‐
120       cerning values in the screen line are colored red by default.
121       When a resource exceeded (default) 80% of its critical  percentage  (so
122       it  is  almost  critical), the concerning values in the screen line are
123       colored cyan by default. This `almost critical percentage'  (one  value
124       for all resources) can be modified in the configuration file (see sepa‐
125       rate man-page of pcp-atoprc).
126       The default colors red and cyan can be modified  in  the  configuration
127       file as well (see separate man-page of pcp-atoprc).
128
129       With the key 'x' (or flag -x), the use of colors can be suppressed.
130

GPU STATISTICS GATHERING

132       GPU  statistics  can  be  gathered by pmdanvidia(1) which is a separate
133       data collection daemon  process.   It  gathers  cumulative  utilization
134       counters  of  every  Nvidia  GPU  in the system, as well as utilization
135       counters of every process that uses a GPU.  When atop notices that  the
136       daemon  is  active,  it reads these GPU utilization counters with every
137       interval.
138
139       Find a description about the utilization counters in the section OUTPUT
140       DESCRIPTION.
141

INTERACTIVE COMMANDS

143       When  running  pcp-atop interactively (no output redirection), keys can
144       be pressed to control the output. In general, lower case  keys  can  be
145       used  to show other information for the active processes and upper case
146       keys  can  be  used  to  influence  the  sort  order  of   the   active
147       process/thread list.
148
149       g    Show generic output (default).
150
151            Per  process  the  following fields are shown in case of a window-
152            width of 80 positions: process-id, cpu consumption during the last
153            interval  in system and user mode, the virtual and resident memory
154            growth of the process.
155
156            The subsequent columns depend on the used kernel:
157            When the kernel supports "storage  accounting"  (>=  2.6.20),  the
158            data transfer for read/write on disk, the status and exit code are
159            shown for each process.  When the kernel does not support "storage
160            accounting",  the username, number of threads in the thread group,
161            the status and exit code are shown.
162            The last columns contain the state, the occupation percentage  for
163            the chosen resource (default: cpu) and the process name.
164
165            When  more  than  80 positions are available, other information is
166            added.
167
168       m    Show memory related output.
169
170            Per process the following fields are shown in case  of  a  window-
171            width  of 80 positions: process-id, minor and major memory faults,
172            size of virtual shared text, total  virtual  process  size,  total
173            resident  process  size,  virtual  and resident growth during last
174            interval, memory occupation percentage and process name.
175
176            When more than 80 positions are available,  other  information  is
177            added.
178
179            For  memory  consumption, always all processes are shown (also the
180            processes that were not active during the interval).
181
182       d    Show disk-related output.
183
184            When "storage accounting" is active in the kernel,  the  following
185            fields  are  shown:  process-id,  amount  of  data read from disk,
186            amount of data written to disk, amount of data  that  was  written
187            but  has been withdrawn again (WCANCL), disk occupation percentage
188            and process name.
189
190       s    Show scheduling characteristics.
191
192            Per process the following fields are shown in case  of  a  window-
193            width  of  80  positions:  process-id,  number of threads in state
194            'running' (R), number of threads in state 'interruptible sleeping'
195            (S),  number  of  threads in state 'uninterruptible sleeping' (D),
196            scheduling policy (normal timesharing, realtime round-robin, real‐
197            time  fifo), nice value, priority, realtime priority, current pro‐
198            cessor, status, exit code, state, the  occupation  percentage  for
199            the chosen resource and the process name.
200
201            When  more  than  80 positions are available, other information is
202            added.
203
204       v    Show various process characteristics.
205
206            Per process the following fields are shown in case  of  a  window-
207            width of 80 positions: process-id, user name and group, start date
208            and time, status (e.g. exit code if  the  process  has  finished),
209            state,  the  occupation percentage for the chosen resource and the
210            process name.
211
212            When more than 80 positions are available,  other  information  is
213            added.
214
215       c    Show the command line of the process.
216
217            Per  process the following fields are shown: process-id, the occu‐
218            pation percentage for the chosen resource  and  the  command  line
219            including arguments.
220
221       e    Show GPU utilization.
222
223            Per  process  at least the following fields are shown: process-id,
224            range of GPU numbers on which the process currently runs, GPU busy
225            percentage  on  all  GPUs,  memory  busy percentage (i.e. read and
226            write accesses on memory) on all GPUs, memory  occupation  at  the
227            moment of the sample, average memory occupation during the sample,
228            and GPU percentage.
229
230            When the pmdanvidia daemon does not run with root privileges,  the
231            GPU  busy percentage and the memory busy percentage are not avail‐
232            able on process level.   In  that  case,  the  GPU  percentage  on
233            process  level  reflects  the GPU memory occupation instead of the
234            GPU busy percentage (which is preferred).
235
236       o    Show the user-defined line of the process.
237
238            In the configuration file the keyword ownprocline can be specified
239            with the description of a user-defined output-line.
240            Refer to the man-page of pcp-atoprc for a detailed description.
241
242       y    Show the individual threads within a process (toggle).
243
244            Single-threaded processes are still shown as one line.
245            For  multi-threaded  processes,  one  line  represents the process
246            while additional lines show the activity per individual thread (in
247            a  different  color).  Depending  on the option 'a' (all or active
248            toggle), all threads are shown  or  only  the  threads  that  were
249            active during the last interval.
250            Whether this key is active or not can be seen in the header line.
251
252       u    Show the process activity accumulated per user.
253
254            Per  user  the  following  fields  are  shown: number of processes
255            active or terminated during last interval (or in total if combined
256            with  command `a'), accumulated cpu consumption during last inter‐
257            val in system and user mode, the current virtual and resident mem‐
258            ory  space  consumed  by active processes (or all processes of the
259            user if combined with command `a').
260            When "storage accounting" is active in the kernel, the accumulated
261            read  and write throughput on disk is shown.  When the kernel mod‐
262            ule `netatop' has been installed, the number of received and  sent
263            network packets are shown.
264            The last columns contain the accumulated occupation percentage for
265            the chosen resource (default: cpu) and the user name.
266
267       p    Show the process activity accumulated per  program  (i.e.  process
268            name).
269
270            Per  program  the  following fields are shown: number of processes
271            active or terminated during last interval (or in total if combined
272            with  command `a'), accumulated cpu consumption during last inter‐
273            val in system and user mode, the current virtual and resident mem‐
274            ory  space  consumed  by active processes (or all processes of the
275            user if combined with command `a').
276            When "storage accounting" is active in the kernel, the accumulated
277            read  and write throughput on disk is shown.  When the kernel mod‐
278            ule `netatop' has been installed, the number of received and  sent
279            network packets are shown.
280            The last columns contain the accumulated occupation percentage for
281            the chosen resource (default: cpu) and the program name.
282
283       j    Show the process activity accumulated per Docker container.
284
285            Per container the following fields are shown: number of  processes
286            active or terminated during last interval (or in total if combined
287            with command `a'), accumulated cpu consumption during last  inter‐
288            val in system and user mode, the current virtual and resident mem‐
289            ory space consumed by active processes (or all  processes  of  the
290            user if combined with command `a').
291            When "storage accounting" is active in the kernel, the accumulated
292            read and write throughput on disk is shown.  When the kernel  mod‐
293            ule  `netatop' has been installed, the number of received and sent
294            network packets are shown.
295            The last columns contain the accumulated occupation percentage for
296            the  chosen  resource  (default:  cpu) and the Docker container id
297            (CID).
298
299       C    Sort the current list in the order of cpu  consumption  (default).
300            The one-but-last column changes to ``CPU''.
301
302       E    Sort  the current list in the order of GPU utilization (preferred,
303            but only applicable when the pmdanvidia  daemon  runs  under  root
304            privileges)  or the order of GPU memory occupation).  The one-but-
305            last column changes to ``GPU''.
306
307       M    Sort the current list in the order of resident memory consumption.
308            The  one-but-last column changes to ``MEM''. In case of sorting on
309            memory, the full process list will be shown (not only  the  active
310            processes).
311
312       D    Sort  the  current list in the order of disk accesses issued.  The
313            one-but-last column changes to ``DSK''.
314
315       N    Sort the current list in the order of network bandwidth  (received
316            and transmitted).  The one-but-last column changes to ``NET''.
317
318       A    Sort  the current list automatically in the order of the most busy
319            system resource during this  interval.   The  one-but-last  column
320            shows either ``ACPU'', ``AMEM'', ``ADSK'' or ``ANET'' (the preced‐
321            ing  'A'  indicates  automatic  sorting-order).   The  most   busy
322            resource  is determined by comparing the weighted busy-percentages
323            of the system resources, as described earlier in the section  COL‐
324            ORS.
325            This  option  remains valid until another sorting-order is explic‐
326            itly selected again.
327            A sorting-order for disk is only possible when  "storage  account‐
328            ing" is active.  A sorting-order for network is only possible when
329            the kernel module `netatop' is loaded.
330
331       Miscellaneous interactive commands:
332
333       ?    Request for help information (also the key 'h' can be pressed).
334
335       V    Request for version information (version number and date).
336
337       R    Gather and calculate the proportional set size of processes  (tog‐
338            gle).   Gathering  of  all values that are needed to calculate the
339            PSIZE of a process is a relatively time-consuming  task,  so  this
340            key  should only be active when analyzing the resident memory con‐
341            sumption of processes.
342
343       x    Suppress colors to highlight critical resources (toggle).
344            Whether this key is active or not can be seen in the header line.
345
346       z    The pause key can be used to freeze the current situation in order
347            to investigate the output on the screen. While pcp-atop is paused,
348            the keys described above can be pressed to show other  information
349            about  the  current  list of processes.  Whenever the pause key is
350            pressed again, pcp-atop will continue with a next sample.
351
352       i    Modify the interval timer (default: 10 seconds).  If  an  interval
353            timer of 0 is entered, the interval timer is switched off. In that
354            case a new sample can only be triggered manually by  pressing  the
355            key 't'.
356
357       t    Trigger a new sample manually. This key can be pressed if the cur‐
358            rent sample should be finished before the timer has  exceeded,  or
359            if  no  timer  is set at all (interval timer defined as 0). In the
360            latter case pcp-atop can be used as a  stopwatch  to  measure  the
361            load being caused by a particular application transaction, without
362            knowing on beforehand how many seconds this transaction will last.
363
364            When viewing the contents of a raw file, this key can be  used  to
365            show the next sample from the file.
366
367       T    When  viewing  the contents of a raw file, this key can be used to
368            show the previous sample from the file.
369
370       b    When viewing the contents of a raw file, this key can be  used  to
371            branch  to  a certain timestamp within the file (either forward or
372            backward).
373
374       r    Reset all counters to zero to see the system and process  activity
375            since boot again.
376
377            When  viewing  the contents of a raw file, this key can be used to
378            rewind to the beginning of the file again.
379
380       U    Specify a search string for  specific  user  names  as  a  regular
381            expression.   From  now  on, only (active) processes will be shown
382            from a user which matches the regular expression.  The system sta‐
383            tistics  are still system wide.  If the Enter-key is pressed with‐
384            out specifying a name, (active) processes of  all  users  will  be
385            shown again.
386            Whether this key is active or not can be seen in the header line.
387
388       I    Specify a list with one or more PIDs to be selected.  From now on,
389            only processes will be shown with a PID which matches one  of  the
390            given  list.  The system statistics are still system wide.  If the
391            Enter-key is pressed without specifying a PID, all  (active)  pro‐
392            cesses will be shown again.
393            Whether this key is active or not can be seen in the header line.
394
395       P    Specify  a  search  string for specific process names as a regular
396            expression.  From now on, only processes will be shown with a name
397            which  matches  the regular expression.  The system statistics are
398            still system wide.  If the Enter-key is pressed without specifying
399            a name, all (active) processes will be shown again.
400            Whether this key is active or not can be seen in the header line.
401
402       /    Specify a specific command line search string as a regular expres‐
403            sion.  From now on, only processes will be shown  with  a  command
404            line  which matches the regular expression.  The system statistics
405            are still system wide.  If the Enter-key is pressed without speci‐
406            fying a string, all (active) processes will be shown again.
407            Whether this key is active or not can be seen in the header line.
408
409       J    Specify  a  Docker  container  id  of 12 (hexadecimal) characters.
410            From now on, only processes will be shown that run  in  that  spe‐
411            cific  Docker  container  (CID).   The system statistics are still
412            system wide.  If the Enter-key is  pressed  without  specifying  a
413            container id, all (active) processes will be shown again.
414            Whether this key is active or not can be seen in the header line.
415
416       S    Specify search strings for specific logical volume names, specific
417            disk names  and  specific  network  interface  names.  All  search
418            strings  are  interpreted  as a regular expressions.  From now on,
419            only those system resources are shown that  match  the  concerning
420            regular  expression.  If the Enter-key is pressed without specify‐
421            ing a search string, all (active) system resources  of  that  type
422            will be shown again.
423            Whether this key is active or not can be seen in the header line.
424
425       a    The  `all/active'  key  can  be  used to toggle between only show‐
426            ing/accumulating the processes that were active  during  the  last
427            interval (default) or showing/accumulating all processes.
428            Whether this key is active or not can be seen in the header line.
429
430       G    By  default,  pcp-atop  shows/accumulates  the  processes that are
431            alive and the processes that are exited during the last  interval.
432            With  this  key  (toggle), showing/accumulating the processes that
433            are exited can be suppressed.
434            Whether this key is active or not can be seen in the header line.
435
436       f    Show a fixed (maximum) number of header lines for system resources
437            (toggle).   By  default  only  the  lines  are  shown about system
438            resources (CPUs, paging, logical volumes,  disks,  network  inter‐
439            faces)  that  really  have  been  active during the last interval.
440            With this key you can force pcp-atop to  show  lines  of  inactive
441            resources as well.
442            Whether this key is active or not can be seen in the header line.
443
444       F    Suppress  sorting of system resources (toggle).  By default system
445            resources (CPUs, logical volumes, disks, network  interfaces)  are
446            sorted on utilization.
447            Whether this key is active or not can be seen in the header line.
448
449       1    Show  relevant  counters  as  an average per second (in the format
450            `..../s') instead of as a total during the interval (toggle).
451            Whether this key is active or not can be seen in the header line.
452
453       l    Limit the number of system level lines for the  counters  per-cpu,
454            the active disks and the network interfaces.  By default lines are
455            shown of all CPUs, disks and network interfaces  which  have  been
456            active during the last interval.  Limiting these lines can be use‐
457            ful on systems with huge number CPUs, disks or interfaces in order
458            to  be  able  to run pcp-atop on a screen/window with e.g. only 24
459            lines.
460            For all mentioned resources the maximum number  of  lines  can  be
461            specified interactively. When using the flag -l the maximum number
462            of per-cpu lines is set to 0, the maximum number of disk lines  to
463            5  and  the  maximum number of interface lines to 3.  These values
464            can be modified again in interactive mode.
465
466       k    Send a signal to an active process (a.k.a. kill a process).
467
468       q    Quit the program.
469
470       PgDn Show the next page of the process/thread list.
471            With the arrow-down key the list can be  scrolled  downwards  with
472            single lines.
473
474       ^F   Show the next page of the process/thread list (forward).
475            With  the  arrow-down  key the list can be scrolled downwards with
476            single lines.
477
478       PgUp Show the previous page of the process/thread list.
479            With the arrow-up key the list can be scrolled upwards with single
480            lines.
481
482       ^B   Show the previous page of the process/thread list (backward).
483            With the arrow-up key the list can be scrolled upwards with single
484            lines.
485
486       ^L   Redraw the screen.
487

PCP DATA STORAGE

489       In order to store system and process  level  statistics  for  long-term
490       analysis  (e.g.  to check the system load and the active processes run‐
491       ning yesterday between 3:00 and 4:00 PM), pcp-atop can store the system
492       and  process  level statistics in the PCP archive format, as an archive
493       folio (see mkaf(1)).
494       All information about processes and threads is stored in the archive.
495       The interval (default: 10 seconds)  and  number  of  samples  (default:
496       infinite)  can  be  passed  as last arguments. Instead of the number of
497       samples, the flag -S can be used to indicate that pcp-atop should  fin‐
498       ish anyhow before midnight.
499
500       A  PCP archive can be read and visualized again with the flag -r .  The
501       argument is a comma-separated list of names, each of which may  be  the
502       base  name  of  an archive or the name of a directory containing one or
503       more   archives.    If   no   argument   is   specified,    the    file
504       $PCP_LOG_DIR/pmlogger/HOST/YYYYMMDD is opened for input (where YYYYMMDD
505       are digits representing the current date, and HOST is the  hostname  of
506       the  machine  being  logged).  If a filename is specified in the format
507       YYYYMMDD (representing any valid date),  the  file  $PCP_LOG_DIR/pmlog‐
508       ger/HOST/YYYYMMDD is opened.  If a filename with the symbolic name y is
509       specified, yesterday's daily logfile is opened (this can be repeated so
510       'yyyy' indicates the logfile of four days ago).
511       The  samples from the file can be viewed interactively by using the key
512       't' to show the next sample, the key 'T' to show the  previous  sample,
513       the  key 'b' to branch to a particular time or the key 'r' to rewind to
514       the begin of the file.
515       When output is redirected to a file or pipe, pcp-atop prints  all  sam‐
516       ples  in  plain ASCII. The default line length is 80 characters in that
517       case; with the flag -L followed by an alternate line length,  more  (or
518       less) columns will be shown.
519       With  the  flag -b (begin time) and/or -e (end time) followed by a time
520       argument of the form HH:MM, a certain time period within the  raw  file
521       can be selected.
522

OUTPUT DESCRIPTION

524       The  first  sample  shows  the  system  level  activity since boot (the
525       elapsed time in the header shows the time since boot).  Note that  par‐
526       ticular counters could have reached their maximum value (several times)
527       and started by zero again, so do not rely on these figures.
528
529       For every sample pcp-atop first shows the lines related to system level
530       activity.  If a particular system resource has not been used during the
531       interval, the entire line related to this resource  is  suppressed.  So
532       the number of system level lines may vary for each sample.
533       After  that  a list is shown of processes which have been active during
534       the last interval. This list is by default sorted on  cpu  consumption,
535       but  this  order  can  be  changed  by  the  keys  which are previously
536       described.
537
538       If values have to be shown by pcp-atop which do not fit in  the  column
539       width, another format is used. If e.g. a cpu-consumption of 233216 mil‐
540       liseconds should be shown in a column width of 4 positions, it is shown
541       as `233s' (in seconds).  For large memory figures, another unit is cho‐
542       sen if the value does not fit (Mb instead of Kb, Gb instead of  Mb,  Tb
543       instead  of Gb, ...).  For other values, a kind of exponent notation is
544       used (value 123456789 shown in a column of 5 positions gives 123e6).
545

OUTPUT DESCRIPTION - SYSTEM LEVEL

547       The system level information consists of the following output lines:
548
549       PRC  Process and thread level totals.
550            This line contains the total cpu  time  consumed  in  system  mode
551            (`sys')  and  in user mode (`user'), the total number of processes
552            present at this moment (`#proc'),  the  total  number  of  threads
553            present  at  this  moment  in state `running' (`#trun'), `sleeping
554            interruptible'   (`#tslpi')   and    `sleeping    uninterruptible'
555            (`#tslpu'), the number of zombie processes (`#zombie'), the number
556            of clone system calls (`clones'), and the number of processes that
557            ended  during  the  interval  (`#exit') when process accounting is
558            used. Instead of `#exit` the last column may indicate that process
559            accounting could not be activated (`no procacct`).
560            If  the  screen-width does not allow all of these counters, only a
561            relevant subset is shown.
562
563       CPU  CPU utilization.
564            At least one line is shown for the total occupation  of  all  CPUs
565            together.
566            In  case  of a multi-processor system, an additional line is shown
567            for every individual processor (with `cpu' in lower case),  sorted
568            on  activity.  Inactive  CPUs  will  not be shown by default.  The
569            lines showing the per-cpu occupation contain the cpu number in the
570            field combined with the wait percentage.
571
572            Every  line  contains  the  percentage of cpu time spent in kernel
573            mode by all active processes (`sys'), the percentage of  cpu  time
574            consumed in user mode (`user') for all active processes (including
575            processes running with a nice value larger than  zero),  the  per‐
576            centage of cpu time spent for interrupt handling (`irq') including
577            softirq, the percentage of unused cpu time while no processes were
578            waiting  for  disk  I/O (`idle'), and the percentage of unused cpu
579            time while at least one process was waiting for disk I/O (`wait').
580            In case of per-cpu occupation, the cpu number and  the  wait  per‐
581            centage  (`w') for that cpu.  The number of lines showing the per-
582            cpu occupation can be limited.
583
584            For virtual machines, the  steal-percentage  (`steal')  shows  the
585            percentage of cpu time stolen by other virtual machines running on
586            the same hardware.
587            For physical machines hosting one or more  virtual  machines,  the
588            guest-percentage  (`guest')  shows the percentage of cpu time used
589            by the virtual machines. Notice that this percentage overlaps  the
590            user-percentage!
591
592            When  PMC performance monitoring counters are supported by the CPU
593            and the kernel (and pcp-atop runs with root privileges), the  num‐
594            ber  of  instructions  per  CPU cycle (`ipc') is shown.  The first
595            sample always shows the value 'initial', because the counters  are
596            just activated at the moment that pcp-atop is started.
597            When the CPU busy percentage is high and the IPC is less than 1.0,
598            it is likely that the CPU is frequently waiting for memory  access
599            during  instruction  execution (larger CPU caches or faster memory
600            might be helpful to improve performance).  When the CPU busy  per‐
601            centage is high and the IPC is greater than 1.0, it is likely that
602            the CPU is instruction-bound (more/faster cores might  be  helpful
603            to improve performance).
604            Furthermore,  per  CPU  the effective number of cycles (`cycl') is
605            shown.  This value can reach the current CPU frequency if such CPU
606            is 100% busy.  When an idle CPU is halted, the number of effective
607            cycles can be (considerably) lower than the current frequency.
608            Notice that the average  instructions  per  cycle  and  number  of
609            cycles is shown in the CPU line for all CPUs.
610            See also: http://www.brendangregg.com/blog/2017-05-09/cpu-utiliza
611            tion-is-wrong.html
612
613
614            In case of frequency scaling, all previously  mentioned  CPU  per‐
615            centages  are  relative  to the used scaling of the CPU during the
616            interval.  If a CPU has been active for e.g. 50% in user mode dur‐
617            ing  the interval while the frequency scaling of that CPU was 40%,
618            only 20% of the full capacity of the CPU has  been  used  in  user
619            mode.
620
621            If  the  screen-width does not allow all of these counters, only a
622            relevant subset is shown.
623
624       CPL  CPU load information.
625            This line contains the load average figures reflecting the  number
626            of  threads  that  are available to run on a CPU (i.e. part of the
627            runqueue) or that are waiting for  disk  I/O.  These  figures  are
628            averaged over 1 (`avg1'), 5 (`avg5') and 15 (`avg15') minutes.
629            Furthermore  the number of context switches (`csw'), the number of
630            serviced interrupts (`intr') and the number of available CPUs  are
631            shown.
632
633            If  the  screen-width does not allow all of these counters, only a
634            relevant subset is shown.
635
636       GPU  GPU utilization (Nvidia).
637            Read the section GPU STATISTICS GATHERING in this document to find
638            the details about the activation of the pmdanvidia daemon.
639
640            In  the  first column of every line, the bus-id (last nine charac‐
641            ters) and the GPU number are shown.  The subsequent  columns  show
642            the  percentage of time that one or more kernels were executing on
643            the GPU (`gpubusy'), the percentage of time that  global  (device)
644            memory  was being read or written (`membusy'), the occupation per‐
645            centage of memory (`memocc'), the total memory (`total'), the mem‐
646            ory being in use at the moment of the sample (`used'), the average
647            memory being in use during the sample time (`usavg'),  the  number
648            of  processes  being active on the GPU at the moment of the sample
649            (`#proc'), and the type of GPU.
650
651            If the screen-width does not allow all of these counters,  only  a
652            relevant subset is shown.
653            The number of lines showing the GPUs can be limited.
654
655       MEM  Memory occupation.
656            This  line  contains  the total amount of physical memory (`tot'),
657            the amount of memory which is currently free (`free'), the  amount
658            of memory in use as page cache including the total resident shared
659            memory (`cache'), the amount of memory within the page cache  that
660            has to be flushed to disk (`dirty'), the amount of memory used for
661            filesystem meta data (`buff'), the amount of memory being used for
662            kernel  mallocs  (`slab'),  the  amount  of  slab  memory  that is
663            reclaimable (`slrec'), the resident size of shared memory  includ‐
664            ing  tmpfs (`shmem`), the resident size of shared memory (`shrss`)
665            the amount of shared memory that is currently  swapped  (`shswp`),
666            the amount of memory that is currently claimed by vmware's balloon
667            driver (`vmbal`), the amount of memory that is  claimed  for  huge
668            pages (`hptot`), and the amount of huge page memory that is really
669            in use (`hpuse`).
670
671            If the screen-width does not allow all of these counters,  only  a
672            relevant subset is shown.
673
674       SWP  Swap occupation and overcommit info.
675            This  line contains the total amount of swap space on disk (`tot')
676            and the amount of free swap space (`free').
677            Furthermore the committed virtual memory space (`vmcom')  and  the
678            maximum limit of the committed space (`vmlim', which is by default
679            swap size plus 50% of memory size) is shown.  The committed  space
680            is  the reserved virtual space for all allocations of private mem‐
681            ory space for processes. The kernel only verifies whether the com‐
682            mitted  space  exceeds  the limit if strict overcommit handling is
683            configured (vm.overcommit_memory is 2).
684
685       PAG  Paging frequency.
686            This line contains the number of scanned pages (`scan') due to the
687            fact  that  free memory drops below a particular threshold and the
688            number times that the kernel tries to  reclaim  pages  due  to  an
689            urgent need (`stall').
690            Also  the  number  of memory pages the system read from swap space
691            (`swin') and the number of memory pages the system wrote  to  swap
692            space (`swout') are shown.
693
694       PSI  Pressure Stall Information.
695            This  line  contains three percentages per category: average pres‐
696            sure percentage over the last 10, 60 and 300 seconds (separated by
697            slashes).
698            The  categories  are:  CPU  for  'some'  (`cs'), memory for 'some'
699            (`ms'), memory for 'full' (`mf'), I/O for 'some' (`is'),  and  I/O
700            for 'full' (`if').
701
702       LVM/MDD/DSK
703            Logical volume/multiple device/disk utilization.
704            Per  active  unit  one  line is produced, sorted on unit activity.
705            Such line shows the name (e.g. VolGroup00-lvtmp for a logical vol‐
706            ume  or sda for a hard disk), the busy percentage i.e. the portion
707            of time that the unit was busy  handling  requests  (`busy'),  the
708            number  of  read  requests  issued  (`read'),  the number of write
709            requests  issued  (`write'),  the  number  of  KiBytes  per   read
710            (`KiB/r'),  the  number of KiBytes per write (`KiB/w'), the number
711            of MiBytes per second throughput for reads (`MBr/s'),  the  number
712            of MiBytes per second throughput for writes (`MBw/s'), the average
713            queue depth (`avq') and the average number of milliseconds  needed
714            by a request (`avio') for seek, latency and data transfer.
715            If  the  screen-width does not allow all of these counters, only a
716            relevant subset is shown.
717
718            The number of lines showing the units can  be  limited  per  class
719            (LVM,  MDD  or  DSK)  with the 'l' key or statically (see separate
720            man-page of pcp-atoprc(5)).  By specifying the value 0 for a  par‐
721            ticular class, no lines will be shown any more for that class.
722
723       NFM  Network Filesystem (NFS) mount at the client side.
724            For each NFS-mounted filesystem, a line is shown that contains the
725            mounted server directory, the name  of  the  server  (`srv'),  the
726            total number of bytes physically read from the server (`read') and
727            the total  number  of  bytes  physically  written  to  the  server
728            (`write').   Data  transfer  is  subdivided in the number of bytes
729            read via normal read() system calls (`nread'), the number of bytes
730            written  via  normal  read() system calls (`nwrit'), the number of
731            bytes read via direct I/O (`dread'), the number of  bytes  written
732            via  direct  I/O  (`dwrit'),  the  number of bytes read via memory
733            mapped I/O pages (`mread'), and the number of  bytes  written  via
734            memory mapped I/O pages (`mwrit').
735
736       NFC  Network Filesystem (NFS) client side counters.
737            This  line  contains  the number of RPC calls issues by local pro‐
738            cesses (`rpc'), the number of read RPC calls  (`read`)  and  write
739            RPC  calls (`rpwrite') issued to the NFS server, the number of RPC
740            calls being retransmitted (`retxmit') and the number of authoriza‐
741            tion refreshes (`autref').
742
743       NFS  Network Filesystem (NFS) server side counters.
744            This  line  contains  the  number  of  RPC calls received from NFS
745            clients (`rpc'), the number of read RPC calls received  (`cread`),
746            the  number  of  write RPC calls received (`cwrit'), the number of
747            Megabytes/second returned to read requests by clients  (`MBcr/s`),
748            the number of Megabytes/second passed in write requests by clients
749            (`MBcw/s`),  the  number  of  network  requests  handled  via  TCP
750            (`nettcp'),  the  number  of  network  requests  handled  via  UDP
751            (`netudp'), the number of reply cache hits (`rchits'), the  number
752            of  reply  cache  misses  (`rcmiss')  and  the  number of uncached
753            requests (`rcnoca').  Furthermore some error  counters  indicating
754            the  number  of  requests  with  a  bad format (`badfmt') or a bad
755            authorization (`badaut'), and a counter indicating the  number  of
756            bad clients (`badcln').
757
758       NET  Network utilization (TCP/IP).
759            One  line  is  shown  for activity of the transport layer (TCP and
760            UDP), one line for the IP layer and one line per active interface.
761            For the transport layer, counters are shown concerning the  number
762            of  received  TCP  segments  including  those  received  in  error
763            (`tcpi'), the number of transmitted TCP segments  excluding  those
764            containing  only  retransmitted octets (`tcpo'), the number of UDP
765            datagrams received (`udpi'), the number of UDP datagrams transmit‐
766            ted (`udpo'), the number of active TCP opens (`tcpao'), the number
767            of passive TCP opens (`tcppo'), the number of TCP output  retrans‐
768            missions  (`tcprs'), the number of TCP input errors (`tcpie'), the
769            number of TCP output resets (`tcpor'), the number of UDP no  ports
770            (`udpnp'), and the number of UDP input errors (`udpie').
771            If  the  screen-width does not allow all of these counters, only a
772            relevant subset is shown.
773            These counters are related to IPv4 and IPv6 combined.
774
775            For the IP layer, counters are shown concerning the number  of  IP
776            datagrams  received  from  interfaces, including those received in
777            error (`ipi'), the number of IP datagrams that local  higher-layer
778            protocols offered for transmission (`ipo'), the number of received
779            IP datagrams which were forwarded to other  interfaces  (`ipfrw'),
780            the  number  of IP datagrams which were delivered to local higher-
781            layer protocols (`deliv'), the number of received  ICMP  datagrams
782            (`icmpi'), and the number of transmitted ICMP datagrams (`icmpo').
783            If  the  screen-width does not allow all of these counters, only a
784            relevant subset is shown.
785            These counters are related to IPv4 and IPv6 combined.
786
787            For every active network interface one line is  shown,  sorted  on
788            the interface activity.  Such line shows the name of the interface
789            and its busy percentage in the first column.  The busy  percentage
790            for  half  duplex  is  determined by comparing the interface speed
791            with the number of bits transmitted and received per  second;  for
792            full  duplex  the  interface speed is compared with the highest of
793            either the transmitted or the received bits.  When  the  interface
794            speed  can  not  be  determined (e.g. for the loopback interface),
795            `---' is shown instead of the percentage.
796            Furthermore the number of received packets (`pcki'), the number of
797            transmitted  packets  (`pcko'),  the  line  speed of the interface
798            (`sp'), the effective amount of bits received per  second  (`si'),
799            the  effective  amount  of bits transmitted per second (`so'), the
800            number of collisions (`coll'), the number  of  received  multicast
801            packets  (`mlti'),  the  number of errors while receiving a packet
802            (`erri'),  the  number  of  errors  while  transmitting  a  packet
803            (`erro'), the number of received packets dropped (`drpi'), and the
804            number of transmitted packets dropped (`drpo').
805            If the screen-width does not allow all of these counters,  only  a
806            relevant subset is shown.
807            The number of lines showing the network interfaces can be limited.
808
809       IFB  Infiniband utilization.
810            For  every  active  Infiniband  port  one line is shown, sorted on
811            activity.  Such line shows the name of the port and its busy  per‐
812            centage in the first column.  The busy percentage is determined by
813            taking the highest of either the transmitted or the received  bits
814            during the interval, multiplying that value by the number of lanes
815            and comparing it against the maximum port speed.
816            Furthermore the number of received packets divided by  the  number
817            of  lanes  (`pcki'),  the number of transmitted packets divided by
818            the number of lanes (`pcko'), the maximum line speed  (`sp'),  the
819            effective amount of bits received per second (`si'), the effective
820            amount of bits transmitted per second (`so'), and  the  number  of
821            lanes (`lanes').
822            If  the  screen-width does not allow all of these counters, only a
823            relevant subset is shown.
824            The number of lines showing the Infiniband ports can be limited.
825

OUTPUT DESCRIPTION - PROCESS LEVEL

827       Following the system level information, the processes  are  shown  from
828       which  the  resource  utilization has changed during the last interval.
829       These processes might have used cpu time  or  issued  disk  or  network
830       requests.  However a process is also shown if part of it has been paged
831       out due to lack of memory  (while  the  process  itself  was  in  sleep
832       state).
833
834       Per  process the following fields may be shown (in alphabetical order),
835       depending on the current output mode as described in the section INTER‐
836       ACTIVE COMMANDS and depending on the current width of your window:
837
838       AVGRSZ   The average size of one read-action on disk.
839
840       AVGWSZ   The average size of one write-action on disk.
841
842       CID      Container  ID  (Docker) of 12 hexadecimal digits, referring to
843                the container in which the process/thread is  running.   If  a
844                process  has  been started and finished during the last inter‐
845                val, a `?' is shown because the container ID is  not  part  of
846                the standard process accounting record.
847
848       CMD      The  name  of  the  process.   This  name can be surrounded by
849                "less/greater than" signs  (`<name>')  which  means  that  the
850                process has finished during the last interval.
851                Behind  the abbreviation `CMD' in the header line, the current
852                page  number  and  the  total   number   of   pages   of   the
853                process/thread list are shown.
854
855       COMMAND-LINE
856                The full command line of the process (including arguments). If
857                the length of the command  line  exceeds  the  length  of  the
858                screen line, the arrow keys -> and <- can be used for horizon‐
859                tal scroll.
860                Behind the verb `COMMAND-LINE' in the header line, the current
861                page   number   and   the   total   number  of  pages  of  the
862                process/thread list are shown.
863
864       CPU      The occupation percentage  of  this  process  related  to  the
865                available capacity for this resource on system level.
866
867       CPUNR    The  identification of the CPU the (main) thread is running on
868                or has recently been running on.
869
870       CTID     Container ID (OpenVZ).  If a process has been started and fin‐
871                ished  during  the  last  interval, a `?' is shown because the
872                container ID is not part of the  standard  process  accounting
873                record.
874
875       DSK      The occupation percentage of this process related to the total
876                load that is  produced  by  all  processes  (i.e.  total  disk
877                accesses by all processes during the last interval).
878                This  information  is shown when per process "storage account‐
879                ing" is active in the kernel.
880
881       EGID     Effective group-id under which this process executes.
882
883       ENDATE   Date that the process has been finished.  If  the  process  is
884                still running, this field shows `active'.
885
886       ENTIME   Time  that  the  process  has been finished. If the process is
887                still running, this field shows `active'.
888
889       ENVID    Virtual environment identified (OpenVZ only).
890
891       EUID     Effective user-id under which this process executes.
892
893       EXC      The exit code of a terminated process (second position of col‐
894                umn  `ST' is E) or the fatal signal number (second position of
895                column `ST' is S or C).
896
897       FSGID    Filesystem group-id under which this process executes.
898
899       FSUID    Filesystem user-id under which this process executes.
900
901       GPU      When the pmdanvidia daemon does not run with root  privileges,
902                the GPU percentage reflects the GPU memory occupation percent‐
903                age (memory of all GPUs is 100%).
904                When the pmdanvidia daemon runs with root privileges, the  GPU
905                percentage reflects the GPU busy percentage.
906
907       GPUBUSY  Busy percentage on all GPUs (one GPU is 100%).
908                When  the pmdanvidia daemon does not run with root privileges,
909                this value is not available.
910
911       GPUNUMS  Comma-separated list of GPUs used by the  process  during  the
912                interval.  When  the comma-separated list exceeds the width of
913                the column, a hexadecimal value is shown.
914
915       MAJFLT   The number of page faults issued by  this  process  that  have
916                been solved by creating/loading the requested memory page.
917
918       MEM      The  occupation  percentage  of  this  process  related to the
919                available capacity for this resource on system level.
920
921       MEMAVG   Average memory occupation during  the  interval  on  all  used
922                GPUs.
923
924       MEMBUSY  Busy  percentage of memory on all GPUs (one GPU is 100%), i.e.
925                the time needed for read and write accesses on memory.
926                When the pmdanvidia daemon does not run with root  privileges,
927                this value is not available.
928
929       MEMNOW   Memory  occupation  at  the  moment  of the sample on all used
930                GPUs.
931
932       MINFLT   The number of page faults issued by  this  process  that  have
933                been  solved  by reclaiming the requested memory page from the
934                free list of pages.
935
936       NET      The occupation percentage of this process related to the total
937                load  that is produced by all processes (i.e. consumed network
938                bandwidth of all processes during the last interval).
939                This  information  will  only  be  shown  when  kernel  module
940                `netatop' is loaded.
941
942       NICE     The  more  or  less  static  priority  that  can be given to a
943                process on a scale from -20 (high priority) to +19 (low prior‐
944                ity).
945
946       NPROCS   The  number of active and terminated processes accumulated for
947                this user or program.
948
949       PID      Process-id.
950
951       POLI     The policies 'norm' (normal,  which  is  SCHED_OTHER),  'btch'
952                (batch)  and 'idle' refer to timesharing processes.  The poli‐
953                cies 'fifo' (SCHED_FIFO)  and  'rr'  (round  robin,  which  is
954                SCHED_RR) refer to realtime processes.
955
956       PPID     Parent process-id.
957
958       PRI      The  process' priority ranges from 0 (highest priority) to 139
959                (lowest priority). Priority 0 to 99 are used for realtime pro‐
960                cesses (fixed priority independent of their behavior) and pri‐
961                ority 100 to 139 for timesharing processes (variable  priority
962                depending on their recent CPU consumption and the nice value).
963
964       PSIZE    The proportional memory size of this process (or user).
965                Every  process  shares  resident  memory with other processes.
966                E.g. when a particular program is started several  times,  the
967                code pages (text) are only loaded once in memory and shared by
968                all incarnations. Also the code of shared libraries is  shared
969                by  all processes using that shared library, as well as shared
970                memory and memory-mapped files.  For the PSIZE calculation  of
971                a  process,  the  resident  memory of a process that is shared
972                with other processes is divided  by  the  number  of  sharers.
973                This means, that every process is accounted for a proportional
974                part of that memory. Accumulating the PSIZE values of all pro‐
975                cesses  in the system gives a reliable impression of the total
976                resident memory consumed by all processes.
977                Since gathering of all values that are needed to calculate the
978                PSIZE  is  a  relatively  time-consuming task, the 'R' key (or
979                '-R' flag) should  be  active.  Gathering  these  values  also
980                requires  superuser privileges (otherwise '?K' is shown in the
981                output).
982
983       RDDSK    When the kernel maintains standard io statistics (>= 2.6.20):
984                The read data transfer issued physically on disk  (so  reading
985                from the disk cache is not accounted for).
986                Unfortunately,  the  kernel  aggregates  the data tranfer of a
987                process to the data transfer of its parent process when termi‐
988                nating, so you might see transfers for (parent) processes like
989                cron, bash or init, that are not really issued by them.
990
991       RGID     The real group-id under which the process executes.
992
993       RGROW    The amount of resident memory that the process has grown  dur‐
994                ing  the  last  interval.  A  resident growth can be caused by
995                touching memory pages which were not physically created/loaded
996                before (load-on-demand).  Note that a resident growth can also
997                be negative e.g. when part of the process is paged out due  to
998                lack of memory or when the process frees dynamically allocated
999                memory.  For a process which started during the last interval,
1000                the  resident  growth  reflects the total resident size of the
1001                process at that moment.
1002
1003       RSIZE    The total resident memory usage consumed by this  process  (or
1004                user).   Notice that the RSIZE of a process includes all resi‐
1005                dent memory used by that process, even if certain memory parts
1006                are  shared  with other processes (see also the explanation of
1007                PSIZE).
1008
1009       RTPR     Realtime priority according the POSIX standard.  Value can  be
1010                0  for a timesharing process (policy 'norm', 'btch' or 'idle')
1011                or ranges from 1 (lowest) till 99  (highest)  for  a  realtime
1012                process (policy 'rr' or 'fifo').
1013
1014       RUID     The real user-id under which the process executes.
1015
1016       S        The  current state of the (main) thread: `R' for running (cur‐
1017                rently processing or in the runqueue), `S' for sleeping inter‐
1018                ruptible  (wait  for an event to occur), `D' for sleeping non-
1019                interruptible, `Z' for zombie (waiting to be synchronized with
1020                its  parent  process),  `T' for stopped (suspended or traced),
1021                `W' for swapping, and `E' (exit) for processes which have fin‐
1022                ished during the last interval.
1023
1024       SGID     The saved group-id of the process.
1025
1026       ST       The status of a process.
1027                The  first  position indicates if the process has been started
1028                during the last interval (the value N means 'new process').
1029
1030                The second position indicates if the process has been finished
1031                during the last interval.
1032                The  value  E means 'exit' on the process' own initiative; the
1033                exit code is displayed in the column `EXC'.
1034                The value S means that the process has been terminated  unvol‐
1035                untarily by a signal; the signal number is displayed in the in
1036                the column `EXC'.
1037                The value C means that the process has been terminated  unvol‐
1038                untarily  by  a  signal,  producing a core dump in its current
1039                directory; the signal number is displayed in the column `EXC'.
1040
1041       STDATE   The start date of the process.
1042
1043       STTIME   The start time of the process.
1044
1045       SUID     The saved user-id of the process.
1046
1047       SWAPSZ   The swap space consumed by this process (or user).
1048
1049       SYSCPU   CPU time consumption of this process in  system  mode  (kernel
1050                mode), usually due to system call handling.
1051
1052       THR      Total  number  of  threads  within  this process.  All related
1053                threads are contained in a thread group, represented  by  pcp-
1054                atop as one line or as a separate line when the 'y' key (or -y
1055                flag) is active.
1056
1057                On Linux 2.4 systems it is hardly possible to determine  which
1058                threads (i.e. processes) are related to the same thread group.
1059                Every thread is represented by pcp-atop as a separate line.
1060
1061       TID      Thread-id.  All threads within a process run with the same PID
1062                but  with  a different TID. This value is shown for individual
1063                threads in multi-threaded processes (when using the key 'y').
1064
1065       TRUN     Number of threads within this process that are  in  the  state
1066                'running' (R).
1067
1068       TSLPI    Number  of  threads  within this process that are in the state
1069                'interruptible sleeping' (S).
1070
1071       TSLPU    Number of threads within this process that are  in  the  state
1072                'uninterruptible sleeping' (D).
1073
1074       USRCPU   CPU time consumption of this process in user mode, due to pro‐
1075                cessing the own program text.
1076
1077       VDATA    The virtual memory size of  the  private  data  used  by  this
1078                process (including heap and shared library data).
1079
1080       VGROW    The amount of virtual memory that the process has grown during
1081                the last interval. A virtual growth  can  be  caused  by  e.g.
1082                issueing a malloc() or attaching a shared memory segment. Note
1083                that a virtual growth can also be negative by e.g. issueing  a
1084                free()  or  detaching  a shared memory segment.  For a process
1085                which started during the last  interval,  the  virtual  growth
1086                reflects the total virtual size of the process at that moment.
1087
1088       VPID     Virtual process-id (within an OpenVZ container).  If a process
1089                has been started and finished during the last interval, a  `?'
1090                is  shown  because  the  virtual process-id is not part of the
1091                standard process accounting record.
1092
1093       VSIZE    The total virtual memory usage consumed by  this  process  (or
1094                user).
1095
1096       VSLIBS   The  virtual  memory  size  of the (shared) text of all shared
1097                libraries used by this process.
1098
1099       VSTACK   The virtual memory size of the (private) stack  used  by  this
1100                process
1101
1102       VSTEXT   The virtual memory size of the (shared) text of the executable
1103                program.
1104
1105       WRDSK    When the kernel maintains standard io statistics (>= 2.6.20):
1106                The write data transfer issued physically on disk (so  writing
1107                to  the  disk  cache  is  not accounted for).  This counter is
1108                maintained for the application process that writes its data to
1109                the  cache  (assuming that this data is physically transferred
1110                to disk later on). Notice that disk I/O needed for swapping is
1111                not taken into account.
1112                Unfortunately,  the  kernel  aggregates  the data tranfer of a
1113                process to the data transfer of its parent process when termi‐
1114                nating, so you might see transfers for (parent) processes like
1115                cron, bash or init, that are not really issued by them.
1116
1117       WCANCL   When the kernel maintains standard io statistics (>= 2.6.20):
1118                The write data transfer previously accounted for this  process
1119                or  another  process  that has been cancelled.  Suppose that a
1120                process writes new data to a file and  that  data  is  removed
1121                again  before  the  cache  buffers  have been flushed to disk.
1122                Then the original process shows the  written  data  as  WRDSK,
1123                while  the  process  that removes/truncates the file shows the
1124                unflushed removed data as WCANCL.
1125

PARSEABLE OUTPUT

1127       With the flag -P followed by a list of one or more labels  (comma-sepa‐
1128       rated),  parseable output is produced for each sample.  The labels that
1129       can be specified for system-level statistics correspond to  the  labels
1130       (first  verb of each line) that can be found in the interactive output:
1131       "CPU", "cpu", "CPL", "GPU", "MEM", "SWP", "PAG", "PSI",  "LVM",  "MDD",
1132       "DSK", "NFM", "NFC", "NFS", "NET" and "IFB".
1133       For process-level statistics special labels are introduced: "PRG" (gen‐
1134       eral), "PRC" (cpu), "PRE" (GPU), "PRM" (memory), "PRD" (disk,  only  if
1135       "storage accounting" is active).
1136       With  the  label  "ALL",  all  system  and process level statistics are
1137       shown.
1138
1139       For every interval all requested lines are  shown  whereafter  pcp-atop
1140       shows  a line just containing the label "SEP" as a separator before the
1141       lines for the next sample are generated.
1142       When a sample contains the values since boot,  pcp-atop  shows  a  line
1143       just  containing the label "RESET" before the lines for this sample are
1144       generated.
1145
1146       The first part of  each  output-line  consists  of  the  following  six
1147       fields: label (the name of the label), host (the name of this machine),
1148       epoch (the time of this interval as number of seconds since  1-1-1970),
1149       date  (date  of this interval in format YYYY/MM/DD), time (time of this
1150       interval in format HH:MM:SS), and interval (number of  seconds  elapsed
1151       for this interval).
1152
1153       The subsequent fields of each output-line depend on the label:
1154
1155       CPU      Subsequent  fields: total number of clock-ticks per second for
1156                this machine, number of processors, consumption for  all  CPUs
1157                in system mode (clock-ticks), consumption for all CPUs in user
1158                mode (clock-ticks), consumption for all CPUs in user mode  for
1159                niced  processes  (clock-ticks),  consumption  for all CPUs in
1160                idle mode (clock-ticks), consumption for all CPUs in wait mode
1161                (clock-ticks),  consumption  for  all CPUs in irq mode (clock-
1162                ticks), consumption for  all  CPUs  in  softirq  mode  (clock-
1163                ticks),  consumption for all CPUs in steal mode (clock-ticks),
1164                consumption for all CPUs in guest mode (clock-ticks)  overlap‐
1165                ping user mode, frequency of all CPUs and frequency percentage
1166                of all CPUs.
1167
1168       cpu      Subsequent fields: total number of clock-ticks per second  for
1169                this  machine,  processor-number,  consumption for this CPU in
1170                system mode (clock-ticks), consumption for this  CPU  in  user
1171                mode  (clock-ticks), consumption for this CPU in user mode for
1172                niced processes (clock-ticks), consumption  for  this  CPU  in
1173                idle mode (clock-ticks), consumption for this CPU in wait mode
1174                (clock-ticks), consumption for this CPU in  irq  mode  (clock-
1175                ticks),  consumption  for  this  CPU  in  softirq mode (clock-
1176                ticks), consumption for this CPU in steal mode  (clock-ticks),
1177                consumption  for this CPU in guest mode (clock-ticks) overlap‐
1178                ping user mode, frequency of all CPUs, frequency percentage of
1179                all CPUs, instructions executed by all CPUs and cycles for all
1180                CPUs.
1181
1182       CPL      Subsequent fields: number of processors, load average for last
1183                minute,  load  average for last five minutes, load average for
1184                last fifteen minutes, number of context-switches,  and  number
1185                of device interrupts.
1186
1187       GPU      Subsequent  fields:  GPU  number,  bus-id  string, type of GPU
1188                string, GPU busy percentage during  last  second  (-1  if  not
1189                available),  memory  busy percentage during last second (-1 if
1190                not available), total memory size (KiB), used memory (KiB)  at
1191                this  moment, number of samples taken during interval, cumula‐
1192                tive GPU busy percentage during the interval (to be divided by
1193                the  number  of samples for the average busy percentage, -1 if
1194                not available), cumulative memory busy percentage  during  the
1195                interval (to be divided by the number of samples for the aver‐
1196                age busy percentage, -1 if not available), and cumulative mem‐
1197                ory  occupation during the interval (to be divided by the num‐
1198                ber of samples for the average occupation).
1199
1200       MEM      Subsequent fields: page size for this machine (in bytes), size
1201                of  physical memory (pages), size of free memory (pages), size
1202                of page cache (pages), size of buffer cache (pages),  size  of
1203                slab  (pages),  dirty pages in cache (pages), reclaimable part
1204                of slab (pages), total size of vmware's balloon pages (pages),
1205                total  size  of shared memory (pages), size of resident shared
1206                memory (pages), size of swapped shared  memory  (pages),  huge
1207                page  size  (in bytes), total size of huge pages (huge pages),
1208                and size of free huge pages (huge pages).
1209
1210       SWP      Subsequent fields: page size for this machine (in bytes), size
1211                of  swap  (pages),  size of free swap (pages), 0 (future use),
1212                size of committed space (pages), and limit for committed space
1213                (pages).
1214
1215       PAG      Subsequent fields: page size for this machine (in bytes), num‐
1216                ber of page scans, number of allocstalls, 0 (future use), num‐
1217                ber of swapins, and number of swapouts.
1218
1219       PSI      Subsequent fields: PSI statistics present on this system (n or
1220                y), CPU some avg10, CPU some avg60, CPU some avg300, CPU  some
1221                accumulated  microseconds  during interval, memory some avg10,
1222                memory some avg60, memory some avg300, memory some accumulated
1223                microseconds  during  interval, memory full avg10, memory full
1224                avg60, memory full avg300, memory full  accumulated  microsec‐
1225                onds during interval, I/O some avg10, I/O some avg60, I/O some
1226                avg300, I/O some accumulated microseconds during interval, I/O
1227                full  avg10,  I/O  full  avg60,  I/O full avg300, and I/O full
1228                accumulated microseconds during interval.
1229
1230       LVM/MDD/DSK
1231                For every logical volume/multiple device/hard disk one line is
1232                shown.
1233                Subsequent fields: name, number of milliseconds spent for I/O,
1234                number of reads issued,  number  of  sectors  transferred  for
1235                reads,  number  of writes issued, and number of sectors trans‐
1236                ferred for write.
1237
1238       NFM      Subsequent fields: mounted NFS  filesystem,  total  number  of
1239                bytes  read,  total  number  of bytes written, number of bytes
1240                read by normal system calls, number of bytes written by normal
1241                system  calls,  number  of bytes read by direct I/O, number of
1242                bytes written by direct I/O, number of pages read  by  memory-
1243                mapped I/O, and number of pages written by memory-mapped I/O.
1244
1245       NFC      Subsequent  fields:  number  of  transmitted  RPCs,  number of
1246                transmitted read RPCs, number of transmitted write RPCs,  num‐
1247                ber  of  RPC  retransmissions,  and  number  of  authorization
1248                refreshes.
1249
1250       NFS      Subsequent fields: number of handled RPCs, number of  received
1251                read RPCs, number of received write RPCs, number of bytes read
1252                by clients, number of bytes written by clients, number of RPCs
1253                with bad format, number of RPCs with bad authorization, number
1254                of RPCs from bad  client,  total  number  of  handled  network
1255                requests,  number  of handled network requests via TCP, number
1256                of handled network requests via UDP,  number  of  handled  TCP
1257                connections,  number  of hits on reply cache, number of misses
1258                on reply cache, and number of uncached requests.
1259
1260       NET      First one line is produced for the upper layers of the  TCP/IP
1261                stack.
1262                Subsequent   fields:  the  verb  "upper",  number  of  packets
1263                received by TCP, number of packets transmitted by TCP,  number
1264                of  packets  received by UDP, number of packets transmitted by
1265                UDP, number of packets  received  by  IP,  number  of  packets
1266                transmitted  by IP, number of packets delivered to higher lay‐
1267                ers by IP, and number of packets forwarded by IP.
1268
1269                Next one line is shown for every interface.
1270                Subsequent fields: name of the interface,  number  of  packets
1271                received  by  the  interface,  number of bytes received by the
1272                interface, number of packets  transmitted  by  the  interface,
1273                number of bytes transmitted by the interface, interface speed,
1274                and duplex mode (0=half, 1=full).
1275
1276       IFB      Subsequent fields: name of the InfiniBand interface, port num‐
1277                ber,  number  of  lanes,  maximum rate (Mbps), number of bytes
1278                received, number  of  bytes  transmitted,  number  of  packets
1279                received, and number of packets transmitted.
1280
1281       PRG      For every process one line is shown.
1282                Subsequent  fields:  PID  (unique  ID  of task), name (between
1283                brackets), state, real uid, real gid, TGID  (group  number  of
1284                related tasks/threads), total number of threads, exit code (in
1285                case of  fatal  signal:  signal  number  +  256),  start  time
1286                (epoch), full command line (between brackets), PPID, number of
1287                threads in state 'running' (R), number  of  threads  in  state
1288                'interruptible  sleeping'  (S),  number  of  threads  in state
1289                'uninterruptible sleeping' (D), effective uid, effective  gid,
1290                saved  uid, saved gid, filesystem uid, filesystem gid, elapsed
1291                time (hertz), is_process (y/n), OpenVZ   virtual  pid  (VPID),
1292                OpenVZ container id (CTID) and Docker container id (CID).
1293
1294       PRC      For every process one line is shown.
1295                Subsequent  fields: PID, name (between brackets), state, total
1296                number of clock-ticks per second for  this  machine,  CPU-con‐
1297                sumption  in user mode (clockticks), CPU-consumption in system
1298                mode (clockticks), nice value,  priority,  realtime  priority,
1299                scheduling  policy,  current  CPU,  sleep average, TGID (group
1300                number of related tasks/threads) and is_process (y/n).
1301
1302       PRE      For every process one line is shown.
1303                Subsequent  fields:  PID,  name  (between  brackets),  process
1304                state,  GPU  state  (A  for active, E for exited, N for no GPU
1305                user), number of GPUs used by this process, bitlist reflecting
1306                used  GPUs,  GPU  busy percentage during interval, memory busy
1307                percentage during interval, memory occupation  (KiB)  at  this
1308                moment cumulative memory occupation (KiB) during interval, and
1309                number of samples taken during interval.
1310
1311       PRM      For every process one line is shown.
1312                Subsequent fields: PID, name (between brackets),  state,  page
1313                size   for  this  machine  (in  bytes),  virtual  memory  size
1314                (Kbytes), resident memory size (Kbytes),  shared  text  memory
1315                size (Kbytes), virtual memory growth (Kbytes), resident memory
1316                growth (Kbytes), number of minor page faults, number of  major
1317                page  faults, virtual library exec size (Kbytes), virtual data
1318                size (Kbytes), virtual stack size (Kbytes),  swap  space  used
1319                (Kbytes),   TGID  (group  number  of  related  tasks/threads),
1320                is_process (y/n) and proportional set size (Kbytes) if in  'R'
1321                option is specified.
1322
1323       PRD      For every process one line is shown.
1324                Subsequent  fields: PID, name (between brackets), state, obso‐
1325                leted kernel patch installed  ('n'),  standard  io  statistics
1326                used  ('y' or 'n'), number of reads on disk, cumulative number
1327                of sectors read, number of writes on disk,  cumulative  number
1328                of  sectors written, cancelled number of written sectors, TGID
1329                (group number of related tasks/threads) and is_process (y/n).
1330                If the standard I/O statistics (>= 2.6.20) are not  used,  the
1331                disk  I/O counters per process are not relevant.  The counters
1332                'number of reads on disk' and 'number of writes on  disk'  are
1333                obsoleted anyhow.
1334
1335       PRN      For every process one line is shown.
1336                Subsequent fields: PID, name (between brackets), state, kernel
1337                module 'netatop' loaded ('y' or 'n'),  number  of  TCP-packets
1338                transmitted,  cumulative size of TCP-packets transmitted, num‐
1339                ber of TCP-packets received, cumulative  size  of  TCP-packets
1340                received,  number  of UDP-packets transmitted, cumulative size
1341                of UDP-packets transmitted, number  of  UDP-packets  received,
1342                cumulative  size  of  UDP-packets  transmitted,  number of raw
1343                packets transmitted (obsolete, always 0), number of raw  pack‐
1344                ets  received  (obsolete,  always  0),  TGID  (group number of
1345                related tasks/threads) and is_process (y/n).
1346

SIGNALS

1348       By sending the SIGUSR1 signal to pcp-atop a new sample will be  forced,
1349       even  if  the current timer interval has not exceeded yet. The behavior
1350       is similar to pressing the `t` key in an interactive session.
1351
1352       By sending the SIGUSR2 signal to pcp-atop a final sample will be forced
1353       after which pcp-atop will terminate.
1354

EXAMPLES

1356       To  monitor the current system load interactively with an interval of 5
1357       seconds:
1358
1359         pcp atop 5
1360
1361       To monitor the system load and write it to a file (in plain ASCII) with
1362       an  interval  of  one  minute during half an hour with active processes
1363       sorted on memory consumption:
1364
1365         pcp atop -M 60 30 > /log/pcp-atop.mem
1366
1367       Store information about the system and process activity in  a  PCP  ar‐
1368       chive folio with an interval of ten minutes during an hour:
1369
1370         pcp atop -w /tmp/pcp-atop 600 6
1371
1372       View the contents of this file interactively:
1373
1374         pcp atop -r /tmp/pcp-atop
1375
1376       View  the processor and disk utilization of this file in parseable for‐
1377       mat:
1378
1379         pcp atop -PCPU,DSK -r /tmp/pcp-atop.raw
1380
1381       View the contents of today's standard logfile interactively:
1382
1383         pcp atop -r
1384
1385       View the contents of the standard logfile of the day  before  yesterday
1386       interactively:
1387
1388         pcp atop -r yy
1389
1390       View the contents of the standard logfile of 2014, June 7 from 02:00 PM
1391       onwards interactively:
1392
1393         pcp atop -r 20140607 -b 14:00
1394

FILES

1396       /etc/atoprc
1397            Configuration file containing  system-wide  default  values.   See
1398            related man-page.
1399
1400       ~/.atoprc
1401            Configuration   file  containing  personal  default  values.   See
1402            related man-page.
1403

NOTES

1405       pcp-atop is based on the  source  code  of  the  atop(1)  command  from
1406       https://atoptool.nl,      maintained      by      Gerlof      Langeveld
1407       (gerlof.langeveld@atoptool.nl), and aims to be command line and  output
1408       compatible  with  it  as  much as possible.  Some features of that atop
1409       command are not available in pcp-atop.
1410
1411       Some features of pcp-atop (such as reporting on the Apache HTTP daemon,
1412       Infiniband,  NFS  client  mounts, hardware event counts and GPU statis‐
1413       tics) are only activated if the corresonding PCP metrics are available.
1414       Refer to the documentation for pmdaapache(1), pmdainfiniband(1), pmdan‐
1415       fsclient(1), pmdanvidia(1) and pmdaperfevent(1) for further details  on
1416       activating these metrics.
1417

SEE ALSO

1419       pcp(1),   pcp-atopsar(1),   pmdaapache(1),  pmdainfiniband(1),  pmdanf‐
1420       sclient(1), pmdanvidia(1), pmdaproc(1),  mkaf(1),  pmlogger(1),  pmlog‐
1421       ger_daily(1), PCPIntro(1) and pcp-atoprc(5).
1422
1423
1424
1425Performance Co-Pilot                  PCP                          PCP-ATOP(1)
Impressum