1PCP-ATOP(1)                 General Commands Manual                PCP-ATOP(1)
2
3
4

NAME

6       pcp-atop - Advanced System and Process Monitor
7

SYNOPSIS

9       Interactive Usage:
10
11       pcp  [pcp options]  atop [-aAcCdDfFgGmMnNopRsuvxy1] [-L linelen] [-Pla‐
12       bel[,label]...] [interval [samples]]
13
14       Writing and reading PCP archive folios:
15
16       pcp atop -w folio [-a] [-S] [interval [samples]]
17       pcp atop -r [folio] [-AcCdDfFgGmMnNopRsuvxy1] [-b hh:mm] [-e hh:mm] [-L
18       linelen] [-Plabel[,label]...]
19

DESCRIPTION

21       The  program pcp-atop is an interactive monitor to view various aspects
22       of load on a system.  It shows the  occupation  of  the  most  critical
23       hardware  resources (from a performance point of view) on system level,
24       i.e. cpu, memory, disk and network.
25       It also shows which processes are responsible for  the  indicated  load
26       with  respect  to  cpu  and memory load on process level.  Disk load is
27       shown per process if "storage accounting" is active in the kernel.
28
29       Every interval (default: 10 seconds) information  is  shown  about  the
30       resource  occupation  on  system  level (cpu, memory, disks and network
31       layers), followed by a list of processes which have been active  during
32       the  last  interval (note that all processes that were unchanged during
33       the last interval are not shown, unless the key 'a' has been pressed or
34       unless  sorting  on  memory occupation is done).  If the list of active
35       processes does not entirely fit on the screen, only the top of the list
36       is shown (sorted in order of activity).
37       The  intervals  are  repeated  till the number of samples (specified as
38       command argument) is reached, or till the key 'q' is pressed in  inter‐
39       active mode.
40
41       When invoked via the pcp(1) command, the PCPIntro(1) options -h/--host,
42       -a/--archive, -O/--origin, -s/--samples,  -t/--interval,  -Z/--timezone
43       and  several  other  pcp options become indirectly available.  The long
44       option form of these is directly available.  Additionally,  the  --hot‐
45       proc  option can be used to request the per-process PCP metrics be used
46       instead of the default proc metrics from pmdaproc(1).
47
48       When pcp-atop is started, it checks whether the standard output channel
49       is connected to a screen, or to a file/pipe.  In the first case it pro‐
50       duces screen control codes (via the ncurses library) and behaves inter‐
51       actively; in the second case it produces flat ASCII-output.
52
53       In  interactive  mode, the output of pcp-atop scales dynamically to the
54       current dimensions of the screen/window.
55       If the window is resized horizontally, columns will be added or removed
56       automatically.  For this purpose, every column has a particular weight.
57       The columns with the highest weights that fit within the current  width
58       will be shown.
59       If  the  window is resized vertically, lines of the process/thread list
60       will be added or removed automatically.
61
62       Furthermore in interactive mode the output  of  pcp-atop  can  be  con‐
63       trolled  by  pressing  particular keys.  However it is also possible to
64       specify such key as flag on the command line.  In  that  case  pcp-atop
65       switches to the indicated mode on beforehand; this mode can be modified
66       again interactively.  Specifying such key as flag is especially  useful
67       when  running  pcp-atop  with  output  to  a pipe or file (non-interac‐
68       tively).  These flags are the same as the keys that can be  pressed  in
69       interactive mode (see section INTERACTIVE COMMANDS).
70       Additional  flags  are available to support storage of pcp-atop data in
71       PCP archive format (see section PCP DATA STORAGE).
72

COLORS

74       For the resource consumption on system level, pcp-atop uses  colors  to
75       indicate  that  a  critical  occupation  percentage  has  been (almost)
76       reached.  A critical occupation percentage means that  is  likely  that
77       this load causes a noticeable negative performance influence for appli‐
78       cations using this resource.  The critical percentage  depends  on  the
79       type  of resource: e.g. the performance influence of a disk with a busy
80       percentage of 80% might be more noticeable for applications/user than a
81       CPU with a busy percentage of 90%.
82       Currently  pcp-atop  uses  the  following default values to calculate a
83       weighted percentage per resource:
84
85        Processor
86            A busy percentage of 90% or higher is considered `critical'.
87
88        Disk
89            A busy percentage of 70% or higher is considered `critical'.
90
91        Network
92            A busy percentage of 90% or higher for the load of an interface is
93            considered `critical'.
94
95        Memory
96            An  occupation percentage of 90% is considered `critical'.  Notice
97            that this occupation percentage is the accumulated memory consump‐
98            tion  of the kernel (including slab) and all processes; the memory
99            for the page cache (`cache' and `buff' in the  MEM-line)  and  the
100            reclaimable part of the slab (`slrec`) is not implied!
101            If  the  number  of pages swapped out (`swout' in the PAG-line) is
102            larger than 10 per  second,  the  memory  resource  is  considered
103            `critical'.   A  value  of  at  least  1  per second is considered
104            `almost critical'.
105            If the committed virtual memory exceeds  the  limit  (`vmcom'  and
106            `vmlim'  in the SWP-line), the SWP-line is colored due to overcom‐
107            mitting the system.
108
109        Swap
110            An occupation percentage of 80% is considered  `critical'  because
111            swap space might be completely exhausted in the near future; it is
112            not critical from a performance point-of-view.
113
114       These default values can be modified in  the  configuration  file  (see
115       separate man-page of pcp-atoprc(5)).
116
117       When  a  resource  exceeds its critical occupation percentage, the con‐
118       cerning values in the screen line are colored red by default.
119       When a resource exceeded (default) 80% of its critical  percentage  (so
120       it  is  almost  critical), the concerning values in the screen line are
121       colored cyan by default.  This `almost critical percentage' (one  value
122       for all resources) can be modified in the configuration file (see sepa‐
123       rate man-page of pcp-atoprc(5)).
124       The default colors red and cyan can be modified  in  the  configuration
125       file as well (see separate man-page of pcp-atoprc(5)).
126
127       With the key 'x' (or flag -x), the use of colors can be suppressed.
128

GPU STATISTICS GATHERING

130       GPU  statistics  can  be  gathered by pmdanvidia(1) which is a separate
131       data collection daemon  process.   It  gathers  cumulative  utilization
132       counters  of  every  Nvidia  GPU  in the system, as well as utilization
133       counters of every process that uses a GPU.  When pcp-atop notices  that
134       the  daemon  is  active,  it  reads these GPU utilization counters with
135       every interval.
136
137       Find a description about the utilization counters in the section OUTPUT
138       DESCRIPTION.
139

INTERACTIVE COMMANDS

141       When  running  pcp-atop interactively (no output redirection), keys can
142       be pressed to control the output.  In general, lower case keys  can  be
143       used  to show other information for the active processes and upper case
144       keys  can  be  used  to  influence  the  sort  order  of   the   active
145       process/thread list.
146
147       g    Show generic output (default).
148
149            Per  process  the  following fields are shown in case of a window-
150            width of 80 positions: process-id, cpu consumption during the last
151            interval  in system and user mode, the virtual and resident memory
152            growth of the process.
153
154            The subsequent columns depend on the used kernel:
155            When the kernel supports "storage  accounting"  (>=  2.6.20),  the
156            data transfer for read/write on disk, the status and exit code are
157            shown for each process.  When the kernel does not support "storage
158            accounting",  the username, number of threads in the thread group,
159            the status and exit code are shown.
160            The last columns contain the state, the occupation percentage  for
161            the chosen resource (default: cpu) and the process name.
162
163            When  more  than  80 positions are available, other information is
164            added.
165
166       m    Show memory related output.
167
168            Per process the following fields are shown in case  of  a  window-
169            width  of 80 positions: process-id, minor and major memory faults,
170            size of virtual shared text, total  virtual  process  size,  total
171            resident  process  size,  virtual  and resident growth during last
172            interval, memory occupation percentage and process name.
173
174            When more than 80 positions are available,  other  information  is
175            added.
176
177            For  memory  consumption, always all processes are shown (also the
178            processes that were not active during the interval).
179
180       d    Show disk-related output.
181
182            When "storage accounting" is active in the kernel,  the  following
183            fields  are  shown:  process-id,  amount  of  data read from disk,
184            amount of data written to disk, amount of data  that  was  written
185            but  has been withdrawn again (WCANCL), disk occupation percentage
186            and process name.
187
188       s    Show scheduling characteristics.
189
190            Per process the following fields are shown in case  of  a  window-
191            width  of  80  positions:  process-id,  number of threads in state
192            'running' (R), number of threads in state 'interruptible sleeping'
193            (S),  number  of  threads in state 'uninterruptible sleeping' (D),
194            scheduling policy (normal timesharing, realtime round-robin, real‐
195            time  fifo), nice value, priority, realtime priority, current pro‐
196            cessor, status, exit code, state, the  occupation  percentage  for
197            the chosen resource and the process name.
198
199            When  more  than  80 positions are available, other information is
200            added.
201
202       v    Show various process characteristics.
203
204            Per process the following fields are shown in case  of  a  window-
205            width of 80 positions: process-id, user name and group, start date
206            and time, status (e.g. exit code if  the  process  has  finished),
207            state,  the  occupation percentage for the chosen resource and the
208            process name.
209
210            When more than 80 positions are available,  other  information  is
211            added.
212
213       c    Show the command line of the process.
214
215            Per  process the following fields are shown: process-id, the occu‐
216            pation percentage for the chosen resource  and  the  command  line
217            including arguments.
218
219       e    Show GPU utilization.
220
221            Per  process  at least the following fields are shown: process-id,
222            range of GPU numbers on which the process currently runs, GPU busy
223            percentage  on  all  GPUs,  memory  busy percentage (i.e. read and
224            write accesses on memory) on all GPUs, memory  occupation  at  the
225            moment of the sample, average memory occupation during the sample,
226            and GPU percentage.
227
228            When the pmdanvidia daemon does not run with root privileges,  the
229            GPU  busy percentage and the memory busy percentage are not avail‐
230            able on process level.   In  that  case,  the  GPU  percentage  on
231            process  level  reflects  the GPU memory occupation instead of the
232            GPU busy percentage (which is preferred).
233
234       o    Show the user-defined line of the process.
235
236            In the configuration file the keyword ownprocline can be specified
237            with the description of a user-defined output-line.
238            Refer to the man-page of pcp-atoprc(5) for a detailed description.
239
240       y    Show the individual threads within a process (toggle).
241
242            Single-threaded processes are still shown as one line.
243            For  multi-threaded  processes,  one  line  represents the process
244            while additional lines show the activity per individual thread (in
245            a  different  color).   Depending on the option 'a' (all or active
246            toggle), all threads are shown  or  only  the  threads  that  were
247            active during the last interval.
248            Whether this key is active or not can be seen in the header line.
249
250       u    Show the process activity accumulated per user.
251
252            Per  user  the  following  fields  are  shown: number of processes
253            active or terminated during last interval (or in total if combined
254            with  command `a'), accumulated cpu consumption during last inter‐
255            val in system and user mode, the current virtual and resident mem‐
256            ory  space  consumed  by active processes (or all processes of the
257            user if combined with command `a').
258            When "storage accounting" is active in the kernel, the accumulated
259            read  and write throughput on disk is shown.  When the kernel mod‐
260            ule `netatop' has been installed, the number of received and  sent
261            network packets are shown.
262            The last columns contain the accumulated occupation percentage for
263            the chosen resource (default: cpu) and the user name.
264
265       p    Show the process activity accumulated per  program  (i.e.  process
266            name).
267
268            Per  program  the  following fields are shown: number of processes
269            active or terminated during last interval (or in total if combined
270            with  command `a'), accumulated cpu consumption during last inter‐
271            val in system and user mode, the current virtual and resident mem‐
272            ory  space  consumed  by active processes (or all processes of the
273            user if combined with command `a').
274            When "storage accounting" is active in the kernel, the accumulated
275            read and write throughput on disk is shown.
276            The last columns contain the accumulated occupation percentage for
277            the chosen resource (default: cpu) and the program name.
278
279       j    Show the process activity accumulated per Docker container.
280
281            Per container the following fields are shown: number of  processes
282            active or terminated during last interval (or in total if combined
283            with command `a'), accumulated cpu consumption during last  inter‐
284            val in system and user mode, the current virtual and resident mem‐
285            ory space consumed by active processes (or all  processes  of  the
286            user if combined with command `a').
287            When "storage accounting" is active in the kernel, the accumulated
288            read and write throughput on disk is shown.
289            The last columns contain the accumulated occupation percentage for
290            the  chosen  resource  (default:  cpu) and the Docker container id
291            (CID).
292
293       C    Sort the current list in the order of cpu  consumption  (default).
294            The one-but-last column changes to ``CPU''.
295
296       E    Sort  the current list in the order of GPU utilization (preferred,
297            but only applicable when the pmdanvidia  daemon  runs  under  root
298            privileges)  or the order of GPU memory occupation).  The one-but-
299            last column changes to ``GPU''.
300
301       M    Sort the current list in the order of resident memory consumption.
302            The one-but-last column changes to ``MEM''.  In case of sorting on
303            memory, the full process list will be shown (not only  the  active
304            processes).
305
306       D    Sort  the  current list in the order of disk accesses issued.  The
307            one-but-last column changes to ``DSK''.
308
309       N    Sort the current list in the order of network bandwidth  (received
310            and transmitted).  The one-but-last column changes to ``NET''.
311
312       A    Sort  the current list automatically in the order of the most busy
313            system resource during this  interval.   The  one-but-last  column
314            shows either ``ACPU'', ``AMEM'', ``ADSK'' or ``ANET'' (the preced‐
315            ing  'A'  indicates  automatic  sorting-order).   The  most   busy
316            resource  is determined by comparing the weighted busy-percentages
317            of the system resources, as described earlier in the section  COL‐
318            ORS.
319            This  option  remains valid until another sorting-order is explic‐
320            itly selected again.
321            A sorting-order for disk is only possible when  "storage  account‐
322            ing" is active.
323
324       Miscellaneous interactive commands:
325
326       ?    Request for help information (also the key 'h' can be pressed).
327
328       V    Request for version information (version number and date).
329
330       R    Gather  and calculate the proportional set size of processes (tog‐
331            gle).  Gathering of all values that are needed  to  calculate  the
332            PSIZE  of  a  process is a relatively time-consuming task, so this
333            key should only be active when analyzing the resident memory  con‐
334            sumption of processes.
335
336       x    Suppress colors to highlight critical resources (toggle).
337            Whether this key is active or not can be seen in the header line.
338
339       z    The pause key can be used to freeze the current situation in order
340            to investigate the  output  on  the  screen.   While  pcp-atop  is
341            paused,  the  keys  described  above  can be pressed to show other
342            information about the current list  of  processes.   Whenever  the
343            pause  key  is pressed again, pcp-atop will continue with the next
344            sample.
345
346       i    Modify the interval timer (default: 10 seconds).  If  an  interval
347            timer  of  0  is  entered, the interval timer is switched off.  In
348            that case a new sample can only be triggered manually by  pressing
349            the key 't'.
350
351       t    Trigger  a  new  sample  manually.  This key can be pressed if the
352            current sample should be finished before the timer  has  exceeded,
353            or  if  no  timer is set at all (interval timer defined as 0).  In
354            the latter case pcp-atop can be used as a stopwatch to measure the
355            load being caused by a particular application transaction, without
356            knowing on beforehand how many seconds this transaction will last.
357
358            When viewing the contents of an archive folio,  this  key  can  be
359            used to show the next sample from the folio.
360
361       T    When  viewing  the  contents  of an archive folio, this key can be
362            used to show the previous sample from the folio.
363
364       b    When viewing the contents of an archive folio,  this  key  can  be
365            used  to  move to a certain timestamp within the file (either for‐
366            ward or backward).
367
368       r    Reset all counters to zero to see the system and process  activity
369            since boot again.
370
371            When  viewing  the contents of an archive, this key can be used to
372            rewind to the beginning of the file again.
373
374       U    Specify a search string for  specific  user  names  as  a  regular
375            expression.   From  now  on, only (active) processes will be shown
376            from a user which matches the regular expression.  The system sta‐
377            tistics  are still system wide.  If the Enter-key is pressed with‐
378            out specifying a name, (active) processes of  all  users  will  be
379            shown again.
380            Whether this key is active or not can be seen in the header line.
381
382       I    Specify a list with one or more PIDs to be selected.  From now on,
383            only processes will be shown with a PID which matches one  of  the
384            given  list.  The system statistics are still system wide.  If the
385            Enter-key is pressed without specifying a PID, all  (active)  pro‐
386            cesses will be shown again.
387            Whether this key is active or not can be seen in the header line.
388
389       P    Specify  a  search  string for specific process names as a regular
390            expression.  From now on, only processes will be shown with a name
391            which  matches  the regular expression.  The system statistics are
392            still system wide.  If the Enter-key is pressed without specifying
393            a name, all (active) processes will be shown again.
394            Whether this key is active or not can be seen in the header line.
395
396       /    Specify a specific command line search string as a regular expres‐
397            sion.  From now on, only processes will be shown  with  a  command
398            line  which matches the regular expression.  The system statistics
399            are still system wide.  If the Enter-key is pressed without speci‐
400            fying a string, all (active) processes will be shown again.
401            Whether this key is active or not can be seen in the header line.
402
403       J    Specify  a  Docker  container  id  of 12 (hexadecimal) characters.
404            From now on, only processes will be shown that run  in  that  spe‐
405            cific  Docker  container  (CID).   The system statistics are still
406            system wide.  If the Enter-key is  pressed  without  specifying  a
407            container id, all (active) processes will be shown again.
408            Whether this key is active or not can be seen in the header line.
409
410       S    Specify search strings for specific logical volume names, specific
411            disk names and  specific  network  interface  names.   All  search
412            strings  are  interpreted  as a regular expressions.  From now on,
413            only those system resources are shown that  match  the  concerning
414            regular  expression.  If the Enter-key is pressed without specify‐
415            ing a search string, all (active) system resources  of  that  type
416            will be shown again.
417            Whether this key is active or not can be seen in the header line.
418
419       a    The  `all/active'  key  can  be  used to toggle between only show‐
420            ing/accumulating the processes that were active  during  the  last
421            interval (default) or showing/accumulating all processes.
422            Whether this key is active or not can be seen in the header line.
423
424       G    By  default,  pcp-atop  shows/accumulates  the  processes that are
425            alive and the processes that are exited during the last  interval.
426            With  this  key  (toggle), showing/accumulating the processes that
427            are exited can be suppressed.
428            Whether this key is active or not can be seen in the header line.
429
430       f    Show a fixed (maximum) number of header lines for system resources
431            (toggle).   By  default  only  the  lines  are  shown about system
432            resources (CPUs, paging, logical volumes,  disks,  network  inter‐
433            faces)  that  really  have  been  active during the last interval.
434            With this key you can force pcp-atop to  show  lines  of  inactive
435            resources as well.
436            Whether this key is active or not can be seen in the header line.
437
438       F    Suppress  sorting of system resources (toggle).  By default system
439            resources (CPUs, logical volumes, disks, network  interfaces)  are
440            sorted on utilization.
441            Whether this key is active or not can be seen in the header line.
442
443       1    Show  relevant  counters  as  an average per second (in the format
444            `..../s') instead of as a total during the interval (toggle).
445            Whether this key is active or not can be seen in the header line.
446
447       l    Limit the number of system level lines for the  counters  per-cpu,
448            the active disks and the network interfaces.  By default lines are
449            shown of all CPUs, disks and network interfaces  which  have  been
450            active during the last interval.  Limiting these lines can be use‐
451            ful on systems with huge number CPUs, disks or interfaces in order
452            to  be  able  to run pcp-atop on a screen/window with e.g. only 24
453            lines.
454            For all mentioned resources the maximum number  of  lines  can  be
455            specified interactively. When using the flag -l the maximum number
456            of per-cpu lines is set to 0, the maximum number of disk lines  to
457            5  and  the  maximum number of interface lines to 3.  These values
458            can be modified again in interactive mode.
459
460       k    Send a signal to an active process (a.k.a. kill a process).
461
462       q    Quit the program.
463
464       PgDn Show the next page of the process/thread list.
465            With the arrow-down key the list can be  scrolled  downwards  with
466            single lines.
467
468       ^F   Show the next page of the process/thread list (forward).
469            With  the  arrow-down  key the list can be scrolled downwards with
470            single lines.
471
472       PgUp Show the previous page of the process/thread list.
473            With the arrow-up key the list can be scrolled upwards with single
474            lines.
475
476       ^B   Show the previous page of the process/thread list (backward).
477            With the arrow-up key the list can be scrolled upwards with single
478            lines.
479
480       ^L   Redraw the screen.
481

PCP DATA STORAGE

483       In order to store system and process  level  statistics  for  long-term
484       analysis  (e.g.  to check the system load and the active processes run‐
485       ning yesterday between 3:00 and 4:00 PM), pcp-atop can store the system
486       and  process  level statistics in the PCP archive format, as an archive
487       folio (see mkaf(1)).
488       All information about processes and threads is stored in the archive.
489       The interval (default: 10 seconds)  and  number  of  samples  (default:
490       infinite)  can  be  passed as last arguments.  Instead of the number of
491       samples, the flag -S can be used to indicate that pcp-atop should  fin‐
492       ish anyhow before midnight.
493
494       A PCP archive can be read and visualized again with the -r option.  The
495       argument is a comma-separated list of names, each of which may  be  the
496       base  name  of  an archive or the name of a directory containing one or
497       more   archives.    If   no   argument   is   specified,    the    file
498       $PCP_LOG_DIR/pmlogger/HOST/YYYYMMDD is opened for input (where YYYYMMDD
499       are digits representing the current date, and HOST is the  hostname  of
500       the  machine  being  logged).  If a filename is specified in the format
501       YYYYMMDD (representing any valid date),  the  file  $PCP_LOG_DIR/pmlog‐
502       ger/HOST/YYYYMMDD is opened.  If a filename with the symbolic name y is
503       specified, yesterday's daily logfile is opened (this can be repeated so
504       'yyyy' indicates the logfile of four days ago).
505       The  samples from the file can be viewed interactively by using the key
506       't' to show the next sample, the key 'T' to show the  previous  sample,
507       the  key 'b' to branch to a particular time or the key 'r' to rewind to
508       the begin of the file.
509       When output is redirected to a file or pipe, pcp-atop prints  all  sam‐
510       ples  in plain ASCII.  The default line length is 80 characters in that
511       case; with the flag -L followed by an alternate line length,  more  (or
512       less) columns will be shown.
513       With  the  flag -b (begin time) and/or -e (end time) followed by a time
514       argument of the form HH:MM, a certain time period  within  the  archive
515       can be selected.
516

OUTPUT DESCRIPTION

518       The  first  sample  shows  the  system  level  activity since boot (the
519       elapsed time in the header shows the time since boot).  Note that  par‐
520       ticular counters could have reached their maximum value (several times)
521       and started by zero again, so do not rely on these figures.
522
523       For every sample pcp-atop first shows the lines related to system level
524       activity.  If a particular system resource has not been used during the
525       interval, the entire line related to this resource is  suppressed.   So
526       the number of system level lines may vary for each sample.
527       After  that  a list is shown of processes which have been active during
528       the last interval.  This list is by default sorted on cpu  consumption,
529       but  this  order  can  be  changed  by  the  keys  which are previously
530       described.
531
532       If values have to be shown by pcp-atop which do not fit in  the  column
533       width, another format is used. If e.g. a cpu-consumption of 233216 mil‐
534       liseconds should be shown in a column width of 4 positions, it is shown
535       as `233s' (in seconds).  For large memory figures, another unit is cho‐
536       sen if the value does not fit (Mb instead of Kb, Gb instead of  Mb,  Tb
537       instead  of Gb, ...).  For other values, a kind of exponent notation is
538       used (value 123456789 shown in a column of 5 positions gives 123e6).
539

OUTPUT DESCRIPTION - SYSTEM LEVEL

541       The system level information consists of the following output lines:
542
543       PRC  Process and thread level totals.
544            This line contains the total cpu  time  consumed  in  system  mode
545            (`sys')  and  in user mode (`user'), the total number of processes
546            present at this moment (`#proc'),  the  total  number  of  threads
547            present  at  this  moment  in state `running' (`#trun'), `sleeping
548            interruptible'   (`#tslpi')   and    `sleeping    uninterruptible'
549            (`#tslpu'), the number of zombie processes (`#zombie'), the number
550            of clone system calls (`clones'), and the number of processes that
551            ended  during  the  interval  (`#exit') when process accounting is
552            used. Instead of `#exit` the last column may indicate that process
553            accounting could not be activated (`no procacct`).
554            If  the  screen-width does not allow all of these counters, only a
555            relevant subset is shown.
556
557       CPU  CPU utilization.
558            At least one line is shown for the total occupation  of  all  CPUs
559            together.
560            In  case  of a multi-processor system, an additional line is shown
561            for every individual processor (with `cpu' in lower case),  sorted
562            on  activity.   Inactive  CPUs  will not be shown by default.  The
563            lines showing the per-cpu occupation contain the cpu number in the
564            field combined with the wait percentage.
565
566            Every  line  contains  the  percentage of cpu time spent in kernel
567            mode by all active processes (`sys'), the percentage of  cpu  time
568            consumed in user mode (`user') for all active processes (including
569            processes running with a nice value larger than  zero),  the  per‐
570            centage of cpu time spent for interrupt handling (`irq') including
571            softirq, the percentage of unused cpu time while no processes were
572            waiting  for  disk  I/O (`idle'), and the percentage of unused cpu
573            time while at least one process was waiting for disk I/O (`wait').
574            In case of per-cpu occupation, the cpu number and  the  wait  per‐
575            centage  (`w') for that cpu.  The number of lines showing the per-
576            cpu occupation can be limited.
577
578            For virtual machines, the  steal-percentage  (`steal')  shows  the
579            percentage of cpu time stolen by other virtual machines running on
580            the same hardware.
581            For physical machines hosting one or more  virtual  machines,  the
582            guest-percentage  (`guest')  shows the percentage of cpu time used
583            by the virtual machines.  Notice that this percentage overlaps the
584            user-percentage!
585
586            When  PMC performance monitoring counters are supported by the CPU
587            and the kernel (and pmdaperfevent(1) runs with  root  privileges),
588            the  number  of  instructions per CPU cycle (`ipc') is shown.  The
589            first sample always shows the value 'initial', because  the  coun‐
590            ters are just activated at the moment that pcp-atop is started.
591            When the CPU busy percentage is high and the IPC is less than 1.0,
592            it is likely that the CPU is frequently waiting for memory  access
593            during  instruction  execution (larger CPU caches or faster memory
594            might be helpful to improve performance).  When the CPU busy  per‐
595            centage is high and the IPC is greater than 1.0, it is likely that
596            the CPU is instruction-bound (more/faster cores might  be  helpful
597            to improve performance).
598            Furthermore,  per  CPU  the effective number of cycles (`cycl') is
599            shown.  This value can reach the current CPU frequency if such CPU
600            is 100% busy.  When an idle CPU is halted, the number of effective
601            cycles can be (considerably) lower than the current frequency.
602            Notice that the average  instructions  per  cycle  and  number  of
603            cycles is shown in the CPU line for all CPUs.
604            See also: http://www.brendangregg.com/blog/2017-05-09/cpu-utiliza
605            tion-is-wrong.html
606
607            In case of frequency scaling, all previously  mentioned  CPU  per‐
608            centages  are  relative  to the used scaling of the CPU during the
609            interval.  If a CPU has been active for e.g. 50% in user mode dur‐
610            ing  the interval while the frequency scaling of that CPU was 40%,
611            only 20% of the full capacity of the CPU has  been  used  in  user
612            mode.
613
614            If  the  screen-width does not allow all of these counters, only a
615            relevant subset is shown.
616
617       CPL  CPU load information.
618            This line contains the load average figures reflecting the  number
619            of  threads  that  are available to run on a CPU (i.e. part of the
620            runqueue) or that are waiting for  disk  I/O.  These  figures  are
621            averaged over 1 (`avg1'), 5 (`avg5') and 15 (`avg15') minutes.
622            Furthermore  the number of context switches (`csw'), the number of
623            serviced interrupts (`intr') and the number of available CPUs  are
624            shown.
625
626            If  the  screen-width does not allow all of these counters, only a
627            relevant subset is shown.
628
629       GPU  GPU utilization (Nvidia).
630            Read the section GPU STATISTICS GATHERING in this document to find
631            the details about the activation of the pmdanvidia daemon.
632
633            In  the  first column of every line, the bus-id (last nine charac‐
634            ters) and the GPU number are shown.  The subsequent  columns  show
635            the  percentage of time that one or more kernels were executing on
636            the GPU (`gpubusy'), the percentage of time that  global  (device)
637            memory  was being read or written (`membusy'), the occupation per‐
638            centage of memory (`memocc'), the total memory (`total'), the mem‐
639            ory being in use at the moment of the sample (`used'), the average
640            memory being in use during the sample time (`usavg'),  the  number
641            of  processes  being active on the GPU at the moment of the sample
642            (`#proc'), and the type of GPU.
643
644            If the screen-width does not allow all of these counters,  only  a
645            relevant subset is shown.
646            The number of lines showing the GPUs can be limited.
647
648       MEM  Memory occupation.
649            This  line  contains  the total amount of physical memory (`tot'),
650            the amount of memory which is currently free (`free'), the  amount
651            of memory in use as page cache including the total resident shared
652            memory (`cache'), the amount of memory within the page cache  that
653            has to be flushed to disk (`dirty'), the amount of memory used for
654            filesystem meta data (`buff'), the amount of memory being used for
655            kernel  mallocs  (`slab'),  the  amount  of  slab  memory  that is
656            reclaimable (`slrec'), the resident size of shared memory  includ‐
657            ing  tmpfs (`shmem`), the resident size of shared memory (`shrss`)
658            the amount of shared memory that is currently  swapped  (`shswp`),
659            the amount of memory that is currently claimed by vmware's balloon
660            driver (`vmbal`), the amount of memory that is  claimed  for  huge
661            pages (`hptot`), and the amount of huge page memory that is really
662            in use (`hpuse`).
663
664            If the screen-width does not allow all of these counters,  only  a
665            relevant subset is shown.
666
667       SWP  Swap occupation and overcommit info.
668            This  line contains the total amount of swap space on disk (`tot')
669            and the amount of free swap space (`free').
670            Furthermore the committed virtual memory space (`vmcom')  and  the
671            maximum limit of the committed space (`vmlim', which is by default
672            swap size plus 50% of memory size) is shown.  The committed  space
673            is  the reserved virtual space for all allocations of private mem‐
674            ory space for processes.  The kernel  only  verifies  whether  the
675            committed space exceeds the limit if strict overcommit handling is
676            configured (vm.overcommit_memory is 2).
677
678       PAG  Paging frequency.
679            This line contains the number of scanned pages (`scan') due to the
680            fact  that  free memory drops below a particular threshold and the
681            number times that the kernel tries to  reclaim  pages  due  to  an
682            urgent need (`stall').
683            Also  the  number  of memory pages the system read from swap space
684            (`swin') and the number of memory pages the system wrote  to  swap
685            space (`swout') are shown.
686
687       PSI  Pressure Stall Information.
688            This  line  contains three percentages per category: average pres‐
689            sure percentage over the last 10, 60 and 300 seconds (separated by
690            slashes).
691            The  categories  are:  CPU  for  'some'  (`cs'), memory for 'some'
692            (`ms'), memory for 'full' (`mf'), I/O for 'some' (`is'),  and  I/O
693            for 'full' (`if').
694
695       LVM/MDD/DSK
696            Logical volume/multiple device/disk utilization.
697            Per  active  unit  one  line is produced, sorted on unit activity.
698            Such line shows the name (e.g. VolGroup00-lvtmp for a logical vol‐
699            ume  or sda for a hard disk), the busy percentage i.e. the portion
700            of time that the unit was busy  handling  requests  (`busy'),  the
701            number  of  read  requests  issued  (`read'),  the number of write
702            requests  issued  (`write'),  the  number  of  KiBytes  per   read
703            (`KiB/r'),  the  number of KiBytes per write (`KiB/w'), the number
704            of MiBytes per second throughput for reads (`MBr/s'),  the  number
705            of MiBytes per second throughput for writes (`MBw/s'), the average
706            queue depth (`avq') and the average number of milliseconds  needed
707            by a request (`avio') for seek, latency and data transfer.
708            If  the  screen-width does not allow all of these counters, only a
709            relevant subset is shown.
710
711            The number of lines showing the units can  be  limited  per  class
712            (LVM,  MDD  or  DSK)  with the 'l' key or statically (see separate
713            man-page of pcp-atoprc(5)).  By specifying the value 0 for a  par‐
714            ticular class, no lines will be shown any more for that class.
715
716       NFM  Network Filesystem (NFS) mount at the client side.
717            For each NFS-mounted filesystem, a line is shown that contains the
718            mounted server directory, the name  of  the  server  (`srv'),  the
719            total number of bytes physically read from the server (`read') and
720            the total  number  of  bytes  physically  written  to  the  server
721            (`write').   Data  transfer  is  subdivided in the number of bytes
722            read via normal read() system calls (`nread'), the number of bytes
723            written  via  normal  read() system calls (`nwrit'), the number of
724            bytes read via direct I/O (`dread'), the number of  bytes  written
725            via  direct  I/O  (`dwrit'),  the  number of bytes read via memory
726            mapped I/O pages (`mread'), and the number of  bytes  written  via
727            memory mapped I/O pages (`mwrit').
728
729       NFC  Network Filesystem (NFS) client side counters.
730            This  line  contains  the number of RPC calls issues by local pro‐
731            cesses (`rpc'), the number of read RPC calls  (`read`)  and  write
732            RPC  calls (`rpwrite') issued to the NFS server, the number of RPC
733            calls being retransmitted (`retxmit') and the number of authoriza‐
734            tion refreshes (`autref').
735
736       NFS  Network Filesystem (NFS) server side counters.
737            This  line  contains  the  number  of  RPC calls received from NFS
738            clients (`rpc'), the number of read RPC calls received  (`cread`),
739            the  number  of  write RPC calls received (`cwrit'), the number of
740            Megabytes/second returned to read requests by clients  (`MBcr/s`),
741            the number of Megabytes/second passed in write requests by clients
742            (`MBcw/s`),  the  number  of  network  requests  handled  via  TCP
743            (`nettcp'),  the  number  of  network  requests  handled  via  UDP
744            (`netudp'), the number of reply cache hits (`rchits'), the  number
745            of  reply  cache  misses  (`rcmiss')  and  the  number of uncached
746            requests (`rcnoca').  Furthermore some error  counters  indicating
747            the  number  of  requests  with  a  bad format (`badfmt') or a bad
748            authorization (`badaut'), and a counter indicating the  number  of
749            bad clients (`badcln').
750
751       NET  Network utilization (TCP/IP).
752            One  line  is  shown  for activity of the transport layer (TCP and
753            UDP), one line for the IP layer and one line per active interface.
754            For the transport layer, counters are shown concerning the  number
755            of  received  TCP  segments  including  those  received  in  error
756            (`tcpi'), the number of transmitted TCP segments  excluding  those
757            containing  only  retransmitted octets (`tcpo'), the number of UDP
758            datagrams received (`udpi'), the number of UDP datagrams transmit‐
759            ted (`udpo'), the number of active TCP opens (`tcpao'), the number
760            of passive TCP opens (`tcppo'), the number of TCP output  retrans‐
761            missions  (`tcprs'), the number of TCP input errors (`tcpie'), the
762            number of TCP output resets (`tcpor'), the number of UDP no  ports
763            (`udpnp'), and the number of UDP input errors (`udpie').
764            If  the  screen-width does not allow all of these counters, only a
765            relevant subset is shown.
766            These counters are related to IPv4 and IPv6 combined.
767
768            For the IP layer, counters are shown concerning the number  of  IP
769            datagrams  received  from  interfaces, including those received in
770            error (`ipi'), the number of IP datagrams that local  higher-layer
771            protocols offered for transmission (`ipo'), the number of received
772            IP datagrams which were forwarded to other  interfaces  (`ipfrw'),
773            the  number  of IP datagrams which were delivered to local higher-
774            layer protocols (`deliv'), the number of received  ICMP  datagrams
775            (`icmpi'), and the number of transmitted ICMP datagrams (`icmpo').
776            If  the  screen-width does not allow all of these counters, only a
777            relevant subset is shown.
778            These counters are related to IPv4 and IPv6 combined.
779
780            For every active network interface one line is  shown,  sorted  on
781            the interface activity.  Such line shows the name of the interface
782            and its busy percentage in the first column.  The busy  percentage
783            for  half  duplex  is  determined by comparing the interface speed
784            with the number of bits transmitted and received per  second;  for
785            full  duplex  the  interface speed is compared with the highest of
786            either the transmitted or the received bits.  When  the  interface
787            speed  can  not  be  determined (e.g. for the loopback interface),
788            `---' is shown instead of the percentage.
789            Furthermore the number of received packets (`pcki'), the number of
790            transmitted  packets  (`pcko'),  the  line  speed of the interface
791            (`sp'), the effective amount of bits received per  second  (`si'),
792            the  effective  amount  of bits transmitted per second (`so'), the
793            number of collisions (`coll'), the number  of  received  multicast
794            packets  (`mlti'),  the  number of errors while receiving a packet
795            (`erri'),  the  number  of  errors  while  transmitting  a  packet
796            (`erro'), the number of received packets dropped (`drpi'), and the
797            number of transmitted packets dropped (`drpo').
798            If the screen-width does not allow all of these counters,  only  a
799            relevant subset is shown.
800            The number of lines showing the network interfaces can be limited.
801
802       IFB  Infiniband utilization.
803            For  every  active  Infiniband  port  one line is shown, sorted on
804            activity.  Such line shows the name of the port and its busy  per‐
805            centage in the first column.  The busy percentage is determined by
806            taking the highest of either the transmitted or the received  bits
807            during the interval, multiplying that value by the number of lanes
808            and comparing it against the maximum port speed.
809            Furthermore the number of received packets divided by  the  number
810            of  lanes  (`pcki'),  the number of transmitted packets divided by
811            the number of lanes (`pcko'), the maximum line speed  (`sp'),  the
812            effective amount of bits received per second (`si'), the effective
813            amount of bits transmitted per second (`so'), and  the  number  of
814            lanes (`lanes').
815            If  the  screen-width does not allow all of these counters, only a
816            relevant subset is shown.
817            The number of lines showing the Infiniband ports can be limited.
818

OUTPUT DESCRIPTION - PROCESS LEVEL

820       Following the system level information, the processes  are  shown  from
821       which  the  resource  utilization has changed during the last interval.
822       These processes might have used cpu time  or  issued  disk  or  network
823       requests.  However a process is also shown if part of it has been paged
824       out due to lack of memory  (while  the  process  itself  was  in  sleep
825       state).
826
827       Per  process the following fields may be shown (in alphabetical order),
828       depending on the current output mode as described in the section INTER‐
829       ACTIVE COMMANDS and depending on the current width of your window:
830
831       AVGRSZ   The average size of one read-action on disk.
832
833       AVGWSZ   The average size of one write-action on disk.
834
835       CID      Container  ID  (Docker) of 12 hexadecimal digits, referring to
836                the container in which the process/thread is  running.   If  a
837                process  has  been started and finished during the last inter‐
838                val, a `?' is shown because the container ID is  not  part  of
839                the standard process accounting record.
840
841       CMD      The  name  of  the  process.   This  name can be surrounded by
842                "less/greater than" signs  (`<name>')  which  means  that  the
843                process has finished during the last interval.
844                Behind  the abbreviation `CMD' in the header line, the current
845                page  number  and  the  total   number   of   pages   of   the
846                process/thread list are shown.
847
848       COMMAND-LINE
849                The full command line of the process (including arguments). If
850                the length of the command  line  exceeds  the  length  of  the
851                screen line, the arrow keys -> and <- can be used for horizon‐
852                tal scroll.
853                Behind the verb `COMMAND-LINE' in the header line, the current
854                page   number   and   the   total   number  of  pages  of  the
855                process/thread list are shown.
856
857       CPU      The occupation percentage  of  this  process  related  to  the
858                available capacity for this resource on system level.
859
860       CPUNR    The  identification of the CPU the (main) thread is running on
861                or has recently been running on.
862
863       CTID     Container ID (OpenVZ).  If a process has been started and fin‐
864                ished  during  the  last  interval, a `?' is shown because the
865                container ID is not part of the  standard  process  accounting
866                record.
867
868       DSK      The occupation percentage of this process related to the total
869                load that is  produced  by  all  processes  (i.e.  total  disk
870                accesses by all processes during the last interval).
871                This  information  is shown when per process "storage account‐
872                ing" is active in the kernel.
873
874       EGID     Effective group-id under which this process executes.
875
876       ENDATE   Date that the process has been finished.  If  the  process  is
877                still running, this field shows `active'.
878
879       ENTIME   Time  that  the  process has been finished.  If the process is
880                still running, this field shows `active'.
881
882       ENVID    Virtual environment identified (OpenVZ only).
883
884       EUID     Effective user-id under which this process executes.
885
886       EXC      The exit code of a terminated process (second position of col‐
887                umn  `ST' is E) or the fatal signal number (second position of
888                column `ST' is S or C).
889
890       FSGID    Filesystem group-id under which this process executes.
891
892       FSUID    Filesystem user-id under which this process executes.
893
894       GPU      When the pmdanvidia daemon does not run with root  privileges,
895                the GPU percentage reflects the GPU memory occupation percent‐
896                age (memory of all GPUs is 100%).
897                When the pmdanvidia daemon runs with root privileges, the  GPU
898                percentage reflects the GPU busy percentage.
899
900       GPUBUSY  Busy percentage on all GPUs (one GPU is 100%).
901                When  the pmdanvidia daemon does not run with root privileges,
902                this value is not available.
903
904       GPUNUMS  Comma-separated list of GPUs used by the  process  during  the
905                interval.   When the comma-separated list exceeds the width of
906                the column, a hexadecimal value is shown.
907
908       MAJFLT   The number of page faults issued by  this  process  that  have
909                been solved by creating/loading the requested memory page.
910
911       MEM      The  occupation  percentage  of  this  process  related to the
912                available capacity for this resource on system level.
913
914       MEMAVG   Average memory occupation during  the  interval  on  all  used
915                GPUs.
916
917       MEMBUSY  Busy  percentage of memory on all GPUs (one GPU is 100%), i.e.
918                the time needed for read and write accesses on memory.
919                When the pmdanvidia daemon does not run with root  privileges,
920                this value is not available.
921
922       MEMNOW   Memory  occupation  at  the  moment  of the sample on all used
923                GPUs.
924
925       MINFLT   The number of page faults issued by  this  process  that  have
926                been  solved  by reclaiming the requested memory page from the
927                free list of pages.
928
929       NICE     The more or less static  priority  that  can  be  given  to  a
930                process on a scale from -20 (high priority) to +19 (low prior‐
931                ity).
932
933       NPROCS   The number of active and terminated processes accumulated  for
934                this user or program.
935
936       PID      Process-id.
937
938       POLI     The  policies  'norm'  (normal,  which is SCHED_OTHER), 'btch'
939                (batch) and 'idle' refer to timesharing processes.  The  poli‐
940                cies  'fifo'  (SCHED_FIFO)  and  'rr'  (round  robin, which is
941                SCHED_RR) refer to realtime processes.
942
943       PPID     Parent process-id.
944
945       PRI      The process' priority ranges from 0 (highest priority) to  139
946                (lowest  priority).   Priority  0  to 99 are used for realtime
947                processes (fixed priority independent of their  behavior)  and
948                priority 100 to 139 for timesharing processes (variable prior‐
949                ity depending on their recent CPU  consumption  and  the  nice
950                value).
951
952       PSIZE    The proportional memory size of this process (or user).
953                Every  process  shares  resident  memory with other processes.
954                E.g. when a particular program is started several  times,  the
955                code pages (text) are only loaded once in memory and shared by
956                all incarnations.  Also the code of shared libraries is shared
957                by  all processes using that shared library, as well as shared
958                memory and memory-mapped files.  For the PSIZE calculation  of
959                a  process,  the  resident  memory of a process that is shared
960                with other processes is divided  by  the  number  of  sharers.
961                This means, that every process is accounted for a proportional
962                part of that memory.  Accumulating the  PSIZE  values  of  all
963                processes  in  the  system  gives a reliable impression of the
964                total resident memory consumed by all processes.
965                Since gathering of all values that are needed to calculate the
966                PSIZE  is  a  relatively  time-consuming task, the 'R' key (or
967                '-R' flag) should be  active.   Gathering  these  values  also
968                requires  superuser privileges (otherwise '?K' is shown in the
969                output).
970
971       RDDSK    When the kernel maintains standard io statistics (>= 2.6.20):
972                The read data transfer issued physically on disk  (so  reading
973                from the disk cache is not accounted for).
974                Unfortunately,  the  kernel  aggregates  the data tranfer of a
975                process to the data transfer of its parent process when termi‐
976                nating, so you might see transfers for (parent) processes like
977                cron, bash or init, that are not really issued by them.
978
979       RGID     The real group-id under which the process executes.
980
981       RGROW    The amount of resident memory that the process has grown  dur‐
982                ing  the  last  interval.   A resident growth can be caused by
983                touching memory pages which were not physically created/loaded
984                before (load-on-demand).  Note that a resident growth can also
985                be negative e.g. when part of the process is paged out due  to
986                lack of memory or when the process frees dynamically allocated
987                memory.  For a process which started during the last interval,
988                the  resident  growth  reflects the total resident size of the
989                process at that moment.
990
991       RSIZE    The total resident memory usage consumed by this  process  (or
992                user).   Notice that the RSIZE of a process includes all resi‐
993                dent memory used by that process, even if certain memory parts
994                are  shared  with other processes (see also the explanation of
995                PSIZE).
996
997       RTPR     Realtime priority according the POSIX standard.  Value can  be
998                0  for a timesharing process (policy 'norm', 'btch' or 'idle')
999                or ranges from 1 (lowest) till 99  (highest)  for  a  realtime
1000                process (policy 'rr' or 'fifo').
1001
1002       RUID     The real user-id under which the process executes.
1003
1004       S        The  current state of the (main) thread: `R' for running (cur‐
1005                rently processing or in the runqueue), `S' for sleeping inter‐
1006                ruptible  (wait  for an event to occur), `D' for sleeping non-
1007                interruptible, `Z' for zombie (waiting to be synchronized with
1008                its  parent  process),  `T' for stopped (suspended or traced),
1009                `W' for swapping, and `E' (exit) for processes which have fin‐
1010                ished during the last interval.
1011
1012       SGID     The saved group-id of the process.
1013
1014       ST       The status of a process.
1015                The  first  position indicates if the process has been started
1016                during the last interval (the value N means 'new process').
1017
1018                The second position indicates if the process has been finished
1019                during the last interval.
1020                The  value  E means 'exit' on the process' own initiative; the
1021                exit code is displayed in the column `EXC'.
1022                The value S means that the process has been terminated  unvol‐
1023                untarily by a signal; the signal number is displayed in the in
1024                the column `EXC'.
1025                The value C means that the process has been terminated  unvol‐
1026                untarily  by  a  signal,  producing a core dump in its current
1027                directory; the signal number is displayed in the column `EXC'.
1028
1029       STDATE   The start date of the process.
1030
1031       STTIME   The start time of the process.
1032
1033       SUID     The saved user-id of the process.
1034
1035       SWAPSZ   The swap space consumed by this process (or user).
1036
1037       SYSCPU   CPU time consumption of this process in  system  mode  (kernel
1038                mode), usually due to system call handling.
1039
1040       THR      Total  number  of  threads  within  this process.  All related
1041                threads are contained in a thread group, represented  by  pcp-
1042                atop as one line or as a separate line when the 'y' key (or -y
1043                flag) is active.
1044
1045       TID      Thread-id.  All threads within a process run with the same PID
1046                but  with a different TID.  This value is shown for individual
1047                threads in multi-threaded processes (when using the key 'y').
1048
1049       TRUN     Number of threads within this process that are  in  the  state
1050                'running' (R).
1051
1052       TSLPI    Number  of  threads  within this process that are in the state
1053                'interruptible sleeping' (S).
1054
1055       TSLPU    Number of threads within this process that are  in  the  state
1056                'uninterruptible sleeping' (D).
1057
1058       USRCPU   CPU time consumption of this process in user mode, due to pro‐
1059                cessing the own program text.
1060
1061       VDATA    The virtual memory size of  the  private  data  used  by  this
1062                process (including heap and shared library data).
1063
1064       VGROW    The amount of virtual memory that the process has grown during
1065                the last interval.  A virtual growth can  be  caused  by  e.g.
1066                issueing  a  malloc()  or  attaching  a shared memory segment.
1067                Note that a virtual growth can also be negative by e.g. issue‐
1068                ing  a  free()  or  detaching  a shared memory segment.  For a
1069                process which started during the last  interval,  the  virtual
1070                growth  reflects the total virtual size of the process at that
1071                moment.
1072
1073       VPID     Virtual process-id (within an OpenVZ container).  If a process
1074                has  been started and finished during the last interval, a `?'
1075                is shown because the virtual process-id is  not  part  of  the
1076                standard process accounting record.
1077
1078       VSIZE    The  total  virtual  memory usage consumed by this process (or
1079                user).
1080
1081       VSLIBS   The virtual memory size of the (shared)  text  of  all  shared
1082                libraries used by this process.
1083
1084       VSTACK   The  virtual  memory  size of the (private) stack used by this
1085                process
1086
1087       VSTEXT   The virtual memory size of the (shared) text of the executable
1088                program.
1089
1090       WRDSK    When the kernel maintains standard io statistics (>= 2.6.20):
1091                The  write data transfer issued physically on disk (so writing
1092                to the disk cache is not  accounted  for).   This  counter  is
1093                maintained for the application process that writes its data to
1094                the cache (assuming that this data is  physically  transferred
1095                to  disk  later on).  Notice that disk I/O needed for swapping
1096                is not taken into account.
1097                Unfortunately, the kernel aggregates the  data  tranfer  of  a
1098                process to the data transfer of its parent process when termi‐
1099                nating, so you might see transfers for (parent) processes like
1100                cron, bash or init, that are not really issued by them.
1101
1102       WCANCL   When the kernel maintains standard io statistics (>= 2.6.20):
1103                The  write data transfer previously accounted for this process
1104                or another process that has been cancelled.   Suppose  that  a
1105                process  writes  new  data  to a file and that data is removed
1106                again before the cache buffers  have  been  flushed  to  disk.
1107                Then  the  original  process  shows the written data as WRDSK,
1108                while the process that removes/truncates the  file  shows  the
1109                unflushed removed data as WCANCL.
1110

PARSEABLE OUTPUT

1112       With  the flag -P followed by a list of one or more labels (comma-sepa‐
1113       rated), parseable output is produced for each sample.  The labels  that
1114       can  be  specified for system-level statistics correspond to the labels
1115       (first verb of each line) that can be found in the interactive  output:
1116       "CPU",  "cpu",  "CPL", "GPU", "MEM", "SWP", "PAG", "PSI", "LVM", "MDD",
1117       "DSK", "NFM", "NFC", "NFS", "NET" and "IFB".
1118       For process-level statistics special labels are introduced: "PRG" (gen‐
1119       eral),  "PRC"  (cpu), "PRE" (GPU), "PRM" (memory), "PRD" (disk, only if
1120       "storage accounting" is active).
1121       With the label "ALL", all  system  and  process  level  statistics  are
1122       shown.
1123
1124       For  every  interval  all requested lines are shown whereafter pcp-atop
1125       shows a line just containing the label "SEP" as a separator before  the
1126       lines for the next sample are generated.
1127       When  a  sample  contains  the values since boot, pcp-atop shows a line
1128       just containing the label "RESET" before the lines for this sample  are
1129       generated.
1130
1131       The  first  part  of  each  output-line  consists  of the following six
1132       fields: label (the name of the label), host (the name of this machine),
1133       epoch  (the time of this interval as number of seconds since 1-1-1970),
1134       date (date of this interval in format YYYY/MM/DD), time (time  of  this
1135       interval  in  format HH:MM:SS), and interval (number of seconds elapsed
1136       for this interval).
1137
1138       The subsequent fields of each output-line depend on the label:
1139
1140       CPU      Subsequent fields: total number of clock-ticks per second  for
1141                this  machine,  number of processors, consumption for all CPUs
1142                in system mode (clock-ticks), consumption for all CPUs in user
1143                mode  (clock-ticks), consumption for all CPUs in user mode for
1144                niced processes (clock-ticks), consumption  for  all  CPUs  in
1145                idle mode (clock-ticks), consumption for all CPUs in wait mode
1146                (clock-ticks), consumption for all CPUs in  irq  mode  (clock-
1147                ticks),  consumption  for  all  CPUs  in  softirq mode (clock-
1148                ticks), consumption for all CPUs in steal mode  (clock-ticks),
1149                consumption  for all CPUs in guest mode (clock-ticks) overlap‐
1150                ping user mode, frequency of all CPUs and frequency percentage
1151                of all CPUs.
1152
1153       cpu      Subsequent  fields: total number of clock-ticks per second for
1154                this machine, processor-number, consumption for  this  CPU  in
1155                system  mode  (clock-ticks),  consumption for this CPU in user
1156                mode (clock-ticks), consumption for this CPU in user mode  for
1157                niced  processes  (clock-ticks),  consumption  for this CPU in
1158                idle mode (clock-ticks), consumption for this CPU in wait mode
1159                (clock-ticks),  consumption  for  this CPU in irq mode (clock-
1160                ticks), consumption for  this  CPU  in  softirq  mode  (clock-
1161                ticks),  consumption for this CPU in steal mode (clock-ticks),
1162                consumption for this CPU in guest mode (clock-ticks)  overlap‐
1163                ping user mode, frequency of all CPUs, frequency percentage of
1164                all CPUs, instructions executed by all CPUs and cycles for all
1165                CPUs.
1166
1167       CPL      Subsequent fields: number of processors, load average for last
1168                minute, load average for last five minutes, load  average  for
1169                last  fifteen  minutes, number of context-switches, and number
1170                of device interrupts.
1171
1172       GPU      Subsequent fields: GPU number,  bus-id  string,  type  of  GPU
1173                string,  GPU  busy  percentage  during  last second (-1 if not
1174                available), memory busy percentage during last second  (-1  if
1175                not  available), total memory size (KiB), used memory (KiB) at
1176                this moment, number of samples taken during interval,  cumula‐
1177                tive GPU busy percentage during the interval (to be divided by
1178                the number of samples for the average busy percentage,  -1  if
1179                not  available),  cumulative memory busy percentage during the
1180                interval (to be divided by the number of samples for the aver‐
1181                age busy percentage, -1 if not available), and cumulative mem‐
1182                ory occupation during the interval (to be divided by the  num‐
1183                ber of samples for the average occupation).
1184
1185       MEM      Subsequent fields: page size for this machine (in bytes), size
1186                of physical memory (pages), size of free memory (pages),  size
1187                of  page  cache (pages), size of buffer cache (pages), size of
1188                slab (pages), dirty pages in cache (pages),  reclaimable  part
1189                of slab (pages), total size of vmware's balloon pages (pages),
1190                total size of shared memory (pages), size of  resident  shared
1191                memory  (pages),  size  of swapped shared memory (pages), huge
1192                page size (in bytes), total size of huge pages  (huge  pages),
1193                and size of free huge pages (huge pages).
1194
1195       SWP      Subsequent fields: page size for this machine (in bytes), size
1196                of swap (pages), size of free swap (pages),  0  (future  use),
1197                size of committed space (pages), and limit for committed space
1198                (pages).
1199
1200       PAG      Subsequent fields: page size for this machine (in bytes), num‐
1201                ber of page scans, number of allocstalls, 0 (future use), num‐
1202                ber of swapins, and number of swapouts.
1203
1204       PSI      Subsequent fields: PSI statistics present on this system (n or
1205                y),  CPU some avg10, CPU some avg60, CPU some avg300, CPU some
1206                accumulated microseconds during interval, memory  some  avg10,
1207                memory some avg60, memory some avg300, memory some accumulated
1208                microseconds during interval, memory full avg10,  memory  full
1209                avg60,  memory  full avg300, memory full accumulated microsec‐
1210                onds during interval, I/O some avg10, I/O some avg60, I/O some
1211                avg300, I/O some accumulated microseconds during interval, I/O
1212                full avg10, I/O full avg60, I/O  full  avg300,  and  I/O  full
1213                accumulated microseconds during interval.
1214
1215       LVM/MDD/DSK
1216                For every logical volume/multiple device/hard disk one line is
1217                shown.
1218                Subsequent fields: name, number of milliseconds spent for I/O,
1219                number  of  reads  issued,  number  of sectors transferred for
1220                reads, number of writes issued, and number of  sectors  trans‐
1221                ferred for write.
1222
1223       NFM      Subsequent  fields:  mounted  NFS  filesystem, total number of
1224                bytes read, total number of bytes  written,  number  of  bytes
1225                read by normal system calls, number of bytes written by normal
1226                system calls, number of bytes read by direct  I/O,  number  of
1227                bytes  written  by direct I/O, number of pages read by memory-
1228                mapped I/O, and number of pages written by memory-mapped I/O.
1229
1230       NFC      Subsequent fields:  number  of  transmitted  RPCs,  number  of
1231                transmitted  read RPCs, number of transmitted write RPCs, num‐
1232                ber  of  RPC  retransmissions,  and  number  of  authorization
1233                refreshes.
1234
1235       NFS      Subsequent  fields: number of handled RPCs, number of received
1236                read RPCs, number of received write RPCs, number of bytes read
1237                by clients, number of bytes written by clients, number of RPCs
1238                with bad format, number of RPCs with bad authorization, number
1239                of  RPCs  from  bad  client,  total  number of handled network
1240                requests, number of handled network requests via  TCP,  number
1241                of  handled  network  requests  via UDP, number of handled TCP
1242                connections, number of hits on reply cache, number  of  misses
1243                on reply cache, and number of uncached requests.
1244
1245       NET      First  one line is produced for the upper layers of the TCP/IP
1246                stack.
1247                Subsequent  fields:  the  verb  "upper",  number  of   packets
1248                received  by TCP, number of packets transmitted by TCP, number
1249                of packets received by UDP, number of packets  transmitted  by
1250                UDP,  number  of  packets  received  by  IP, number of packets
1251                transmitted by IP, number of packets delivered to higher  lay‐
1252                ers by IP, and number of packets forwarded by IP.
1253
1254                Next one line is shown for every interface.
1255                Subsequent  fields:  name  of the interface, number of packets
1256                received by the interface, number of  bytes  received  by  the
1257                interface,  number  of  packets  transmitted by the interface,
1258                number of bytes transmitted by the interface, interface speed,
1259                and duplex mode (0=half, 1=full).
1260
1261       IFB      Subsequent fields: name of the InfiniBand interface, port num‐
1262                ber, number of lanes, maximum rate  (Mbps),  number  of  bytes
1263                received,  number  of  bytes  transmitted,  number  of packets
1264                received, and number of packets transmitted.
1265
1266       PRG      For every process one line is shown.
1267                Subsequent fields: PID (unique  ID  of  task),  name  (between
1268                brackets),  state,  real  uid, real gid, TGID (group number of
1269                related tasks/threads), total number of threads, exit code (in
1270                case  of  fatal  signal:  signal  number  +  256),  start time
1271                (epoch), full command line (between brackets), PPID, number of
1272                threads  in  state  'running'  (R), number of threads in state
1273                'interruptible sleeping'  (S),  number  of  threads  in  state
1274                'uninterruptible  sleeping' (D), effective uid, effective gid,
1275                saved uid, saved gid, filesystem uid, filesystem gid,  elapsed
1276                time  (hertz),  is_process  (y/n), OpenVZ  virtual pid (VPID),
1277                OpenVZ container id (CTID) and Docker container id (CID).
1278
1279       PRC      For every process one line is shown.
1280                Subsequent fields: PID, name (between brackets), state,  total
1281                number  of  clock-ticks  per second for this machine, CPU-con‐
1282                sumption in user mode (clockticks), CPU-consumption in  system
1283                mode  (clockticks),  nice  value, priority, realtime priority,
1284                scheduling policy, current CPU,  sleep  average,  TGID  (group
1285                number of related tasks/threads) and is_process (y/n).
1286
1287       PRE      For every process one line is shown.
1288                Subsequent  fields:  PID,  name  (between  brackets),  process
1289                state, GPU state (A for active, E for exited,  N  for  no  GPU
1290                user), number of GPUs used by this process, bitlist reflecting
1291                used GPUs, GPU busy percentage during  interval,  memory  busy
1292                percentage  during  interval,  memory occupation (KiB) at this
1293                moment cumulative memory occupation (KiB) during interval, and
1294                number of samples taken during interval.
1295
1296       PRM      For every process one line is shown.
1297                Subsequent  fields:  PID, name (between brackets), state, page
1298                size  for  this  machine  (in  bytes),  virtual  memory   size
1299                (Kbytes),  resident  memory  size (Kbytes), shared text memory
1300                size (Kbytes), virtual memory growth (Kbytes), resident memory
1301                growth  (Kbytes), number of minor page faults, number of major
1302                page faults, virtual library exec size (Kbytes), virtual  data
1303                size  (Kbytes),  virtual  stack size (Kbytes), swap space used
1304                (Kbytes),  TGID  (group  number  of  related   tasks/threads),
1305                is_process  (y/n) and proportional set size (Kbytes) if in 'R'
1306                option is specified.
1307
1308       PRD      For every process one line is shown.
1309                Subsequent fields: PID, name (between brackets), state,  obso‐
1310                leted  kernel  patch  installed  ('n'), standard io statistics
1311                used ('y' or 'n'), number of reads on disk, cumulative  number
1312                of  sectors  read, number of writes on disk, cumulative number
1313                of sectors written, cancelled number of written sectors,  TGID
1314                (group number of related tasks/threads) and is_process (y/n).
1315                If  the  standard I/O statistics (>= 2.6.20) are not used, the
1316                disk I/O counters per process are not relevant.  The  counters
1317                'number  of  reads on disk' and 'number of writes on disk' are
1318                obsoleted anyhow.
1319
1320       PRN      For every process one line is shown.
1321                Subsequent fields: PID, name (between brackets), state, kernel
1322                module  'netatop'  loaded  ('y' or 'n'), number of TCP-packets
1323                transmitted, cumulative size of TCP-packets transmitted,  num‐
1324                ber  of  TCP-packets  received, cumulative size of TCP-packets
1325                received, number of UDP-packets transmitted,  cumulative  size
1326                of  UDP-packets  transmitted,  number of UDP-packets received,
1327                cumulative size of  UDP-packets  transmitted,  number  of  raw
1328                packets  transmitted (obsolete, always 0), number of raw pack‐
1329                ets received (obsolete,  always  0),  TGID  (group  number  of
1330                related tasks/threads) and is_process (y/n).
1331

SIGNALS

1333       By  sending the SIGUSR1 signal to pcp-atop a new sample will be forced,
1334       even if the current timer interval has not exceeded yet.  The  behavior
1335       is similar to pressing the `t` key in an interactive session.
1336
1337       By sending the SIGUSR2 signal to pcp-atop a final sample will be forced
1338       after which pcp-atop will terminate.
1339

EXAMPLES

1341       To monitor the current system load interactively with an interval of  5
1342       seconds:
1343
1344         pcp atop 5
1345
1346       To monitor the system load and write it to a file (in plain ASCII) with
1347       an interval of one minute during half an  hour  with  active  processes
1348       sorted on memory consumption:
1349
1350         pcp atop -M 60 30 > /log/pcp-atop.mem
1351
1352       Store  information  about  the system and process activity in a PCP ar‐
1353       chive folio with an interval of ten minutes during an hour:
1354
1355         pcp atop -w /tmp/pcp-atop 600 6
1356
1357       View the contents of this file interactively:
1358
1359         pcp atop -r /tmp/pcp-atop
1360
1361       View the processor and disk utilization of this file in parseable  for‐
1362       mat:
1363
1364         pcp atop -PCPU,DSK -r /tmp/pcp-atop.folio
1365
1366       View the contents of today's standard logfile interactively:
1367
1368         pcp atop -r
1369
1370       View  the  contents of the standard logfile of the day before yesterday
1371       interactively:
1372
1373         pcp atop -r yy
1374
1375       View the contents of the standard logfile of 2014, June 7 from 02:00 PM
1376       onwards interactively:
1377
1378         pcp atop -r 20140607 -b 14:00
1379

NOTES

1381       pcp-atop  is  based  on  the  source  code  of the atop(1) command from
1382       https://atoptool.nl,      maintained      by      Gerlof      Langeveld
1383       (gerlof.langeveld@atoptool.nl),  and aims to be command line and output
1384       compatible with it as much as possible.  Some  features  of  that  atop
1385       command are not available in pcp-atop.
1386
1387       Some features of pcp-atop (such as reporting on the Apache HTTP daemon,
1388       Infiniband, NFS client mounts, hardware event counts  and  GPU  statis‐
1389       tics) are only activated if the corresonding PCP metrics are available.
1390       Refer to the documentation for pmdaapache(1), pmdainfiniband(1), pmdan‐
1391       fsclient(1),  pmdanvidia(1) and pmdaperfevent(1) for further details on
1392       activating these metrics.
1393

FILES

1395       /etc/atoprc
1396            Configuration file containing  system-wide  default  values.   See
1397            related man-page.
1398
1399       ~/.atoprc
1400            Configuration   file  containing  personal  default  values.   See
1401            related man-page.
1402

PCP ENVIRONMENT

1404       Environment variables with the prefix PCP_ are used to parameterize the
1405       file  and  directory names used by PCP.  On each installation, the file
1406       /etc/pcp.conf contains the  local  values  for  these  variables.   The
1407       $PCP_CONF  variable may be used to specify an alternative configuration
1408       file, as described in pcp.conf(5).
1409
1410       For environment variables affecting PCP tools, see pmGetOptions(3).
1411

SEE ALSO

1413       pcp(1),  pcp-atopsar(1),  pmdaapache(1),   pmdainfiniband(1),   pmdanf‐
1414       sclient(1),  pmdanvidia(1),  pmdaproc(1),  mkaf(1), pmlogger(1), pmlog‐
1415       ger_daily(1), PCPIntro(1) and pcp-atoprc(5).
1416
1417
1418
1419Performance Co-Pilot                  PCP                          PCP-ATOP(1)
Impressum