1klockstat(8)                System Manager's Manual               klockstat(8)
2
3
4

NAME

6       klockstat  -  Traces kernel mutex lock events and display locks statisā€
7       tics. Uses Linux eBPF/bcc.
8

SYNOPSIS

10       klockstat [-h] [-i] [-n] [-s] [-c] [-S FIELDS] [-p] [-t] [-d DURATION]
11

DESCRIPTION

13       klockstat traces kernel mutex lock events and display locks  statistics
14       and displays following data:
15
16                                         Caller    Avg  Spin  Count   Max spin
17       Total spin
18                             psi_avgs_work+0x2e       3675       5        5468
19       18379
20                            flush_to_ldisc+0x22        2833       2       4210
21       5667
22                              n_tty_write+0x30c       3914       1        3914
23       3914
24                                      isig+0x5d        2390       1       2390
25       2390
26                          tty_buffer_flush+0x2a       1604       1        1604
27       1604
28                             commit_echoes+0x22        1400       1       1400
29       1400
30                 n_tty_receive_buf_common+0x3b9       1399       1        1399
31       1399
32
33                                         Caller    Avg  Hold  Count   Max hold
34       Total hold
35                            flush_to_ldisc+0x22      42558       2       76135
36       85116
37                             psi_avgs_work+0x2e       14821       5      20446
38       74106
39                 n_tty_receive_buf_common+0x3b9      12300       1       12300
40       12300
41                              n_tty_write+0x30c       10712       1      10712
42       10712
43                                      isig+0x5d       3362       1        3362
44       3362
45                          tty_buffer_flush+0x2a        3078       1       3078
46       3078
47                             commit_echoes+0x22       3017       1        3017
48       3017
49
50
51       Every caller to using kernel's mutex is displayed on every line.
52
53       First portion of lines show the lock acquiring data, showing the amount
54       of time it took to acquired given lock.
55
56         'Caller'       - symbol acquiring the mutex
57         'Average Spin' - average time to acquire the mutex
58         'Count'        - number of times mutex was acquired
59         'Max spin'     - maximum time to acquire the mutex
60         'Total spin'   - total time spent in acquiring the mutex
61
62       Second portion of lines show the lock holding data, showing the  amount
63       of time it took to hold given lock.
64
65         'Caller'       - symbol holding the mutex
66         'Average Hold' - average time mutex was held
67         'Count'        - number of times mutex was held
68         'Max hold'     - maximum time mutex was held
69         'Total hold'   - total time spent in holding the mutex
70
71       This  works  by  tracing  mutex_lock/unlock  kprobes, updating the lock
72       stats in maps and processing them in the python part.
73
74       Since this uses BPF, only the root user can use this tool.
75

REQUIREMENTS

77       CONFIG_BPF and bcc.
78

OPTIONS

80       -h     Print usage message.
81
82       -i SEC print summary at this interval (seconds)
83
84       -c CALLER
85              print locks taken by given caller
86
87       -S FIELDS
88              sort data on specific  columns  defined  by  FIELDS  string  (by
89              default the data is sorted by Max columns)
90
91              FIELDS  string contains 1 or 2 fields describing columns to sort
92              on for both acquiring and holding  data.  Following  fields  are
93              available:
94
95                acq_max      for 'Max spin' column
96                acq_total    for 'Total spin' column
97                acq_count    for 'Count' column
98
99                hld_max      for 'Max hold' column
100                hld_total    for 'Total hold' column
101                hld_count    for 'Count' column
102
103              See EXAMPLES.
104
105
106       -n COUNT
107              print COUNT number of locks
108
109       -s COUNT
110              print COUNT number of stack entries
111
112       -p PID Trace this process ID only (filtered in-kernel).
113
114       -t TID Trace this thread ID only (filtered in-kernel).
115
116       -d DURATION
117              Total duration of trace in seconds.
118
119       --stack-storage-size STACK_STORAGE_SIZE
120              Change  the number of unique stack traces that can be stored and
121              displayed.
122

EXAMPLES

124       Sort lock acquired results on acquired count:
125              # klockstat -S acq_count
126
127
128       Sort lock held results on total held time:
129              # klockstat -S hld_total
130
131
132       Combination of above:
133              # klockstat -S acq_count,hld_total
134
135
136       Trace system wide:
137              # klockstat
138
139
140       Trace for 5 seconds only:
141              # klockstat -d 5
142
143
144       Display stats every 5 seconds
145              # klockstat -i 5
146
147
148       Trace locks for PID 123:
149              # klockstat -p 123
150
151
152       Trace locks for PID 321:
153              # klockstat -t 321
154
155
156       Display stats only for lock callers with 'pipe_' substring
157              # klockstat -c pipe_
158
159
160       Display 3 locks:
161              # klockstat -n 3
162
163
164       Display 10 levels of stack for the most expensive lock:
165              # klockstat -n 1 -s 10
166
167              Tracing lock events... Hit Ctrl-C to end.  ^C
168                                                Caller   Avg Spin  Count   Max
169              spin Total spin
170                                        pipe_wait+0xa9           670    397691
171              17273  266775437
172                                        pipe_wait+0xa9
173                                       pipe_read+0x206
174                                   new_sync_read+0x12a
175                                         vfs_read+0x9d
176                                        ksys_read+0x5f
177                                    do_syscall_64+0x5b
178                   entry_SYSCALL_64_after_hwframe+0x44
179
180                                                Caller   Avg Hold  Count   Max
181              hold Total hold
182                                   flush_to_ldisc+0x22         28381         3
183              65484      85144
184                                   flush_to_ldisc+0x22
185                                process_one_work+0x1b0
186                                    worker_thread+0x50
187                                          kthread+0xfb
188                                    ret_from_fork+0x35
189
190

SOURCE

192       This is from bcc.
193
194              https://github.com/iovisor/bcc
195
196       Also look in the bcc distribution for a  companion  _examples.txt  file
197       containing example usage, output, and commentary for this tool.
198

OS

200       Linux
201

STABILITY

203       Unstable - in development.
204

CREDITS

206       This  tool  is based on work of David Valin <dvalin@redhat.com> and his
207       script.
208

AUTHOR

210       Jiri Olsa
211
212
213
214USER COMMANDS                     2019-10-22                      klockstat(8)
Impressum