1klockstat(8) System Manager's Manual klockstat(8)
2
6 klockstat - Traces kernel mutex lock events and display locks statis‐
7 tics. Uses Linux eBPF/bcc.
8
10 klockstat [-h] [-i] [-n] [-s] [-c] [-S FIELDS] [-p] [-t] [-d DURATION]
11
13 klockstat traces kernel mutex lock events and display locks statistics
14 and displays following data:
15 Caller Avg Spin Count Max spin
17 Total spin
18 psi_avgs_work+0x2e 3675 5 5468
19 18379
20 flush_to_ldisc+0x22 2833 2 4210
21 5667
22 n_tty_write+0x30c 3914 1 3914
23 3914
24 isig+0x5d 2390 1 2390
25 2390
26 tty_buffer_flush+0x2a 1604 1 1604
27 1604
28 commit_echoes+0x22 1400 1 1400
29 1400
30 n_tty_receive_buf_common+0x3b9 1399 1 1399
31 1399
32 Caller Avg Hold Count Max hold
34 Total hold
35 flush_to_ldisc+0x22 42558 2 76135
36 85116
37 psi_avgs_work+0x2e 14821 5 20446
38 74106
39 n_tty_receive_buf_common+0x3b9 12300 1 12300
40 12300
41 n_tty_write+0x30c 10712 1 10712
42 10712
43 isig+0x5d 3362 1 3362
44 3362
45 tty_buffer_flush+0x2a 3078 1 3078
46 3078
47 commit_echoes+0x22 3017 1 3017
48 3017
49 Every caller to using kernel's mutex is displayed on every line.
52 First portion of lines show the lock acquiring data, showing the amount
54 of time it took to acquired given lock.
55 'Caller' - symbol acquiring the mutex
57 'Average Spin' - average time to acquire the mutex
58 'Count' - number of times mutex was acquired
59 'Max spin' - maximum time to acquire the mutex
60 'Total spin' - total time spent in acquiring the mutex
61 Second portion of lines show the lock holding data, showing the amount
63 of time it took to hold given lock.
64 'Caller' - symbol holding the mutex
66 'Average Hold' - average time mutex was held
67 'Count' - number of times mutex was held
68 'Max hold' - maximum time mutex was held
69 'Total hold' - total time spent in holding the mutex
70 This works by tracing mutex_lock/unlock kprobes, updating the lock
72 stats in maps and processing them in the python part.
73 Since this uses BPF, only the root user can use this tool.
75
77 CONFIG_BPF and bcc.
78
80 -h Print usage message.
81 -i SEC print summary at this interval (seconds)
83 -c CALLER
85 print locks taken by given caller
86 -S FIELDS
88 sort data on specific columns defined by FIELDS string (by
89 default the data is sorted by Max columns)
90 FIELDS string contains 1 or 2 fields describing columns to sort
92 on for both acquiring and holding data. Following fields are
93 available:
94 acq_max for 'Max spin' column
96 acq_total for 'Total spin' column
97 acq_count for 'Count' column
98 hld_max for 'Max hold' column
100 hld_total for 'Total hold' column
101 hld_count for 'Count' column
102 See EXAMPLES.
104 -n COUNT
107 print COUNT number of locks
108 -s COUNT
110 print COUNT number of stack entries
111 -p PID Trace this process ID only (filtered in-kernel).
113 -t TID Trace this thread ID only (filtered in-kernel).
115 -d DURATION
117 Total duration of trace in seconds.
118 --stack-storage-size STACK_STORAGE_SIZE
120 Change the number of unique stack traces that can be stored and
121 displayed.
122
124 Sort lock acquired results on acquired count:
125 # klockstat -S acq_count
126 Sort lock held results on total held time:
129 # klockstat -S hld_total
130 Combination of above:
133 # klockstat -S acq_count,hld_total
134 Trace system wide:
137 # klockstat
138 Trace for 5 seconds only:
141 # klockstat -d 5
142 Display stats every 5 seconds
145 # klockstat -i 5
146 Trace locks for PID 123:
149 # klockstat -p 123
150 Trace locks for PID 321:
153 # klockstat -t 321
154 Display stats only for lock callers with 'pipe_' substring
157 # klockstat -c pipe_
158 Display 3 locks:
161 # klockstat -n 3
162 Display 10 levels of stack for the most expensive lock:
165 # klockstat -n 1 -s 10
166 Tracing lock events... Hit Ctrl-C to end. ^C
168 Caller Avg Spin Count Max
169 spin Total spin
170 pipe_wait+0xa9 670 397691
171 17273 266775437
172 pipe_wait+0xa9
173 pipe_read+0x206
174 new_sync_read+0x12a
175 vfs_read+0x9d
176 ksys_read+0x5f
177 do_syscall_64+0x5b
178 entry_SYSCALL_64_after_hwframe+0x44
179 Caller Avg Hold Count Max
181 hold Total hold
182 flush_to_ldisc+0x22 28381 3
183 65484 85144
184 flush_to_ldisc+0x22
185 process_one_work+0x1b0
186 worker_thread+0x50
187 kthread+0xfb
188 ret_from_fork+0x35
189
192 This is from bcc.
193 https://github.com/iovisor/bcc
195 Also look in the bcc distribution for a companion _examples.txt file
197 containing example usage, output, and commentary for this tool.
198
200 Linux
201
203 Unstable - in development.
204
206 This tool is based on work of David Valin <dvalin@redhat.com> and his
207 script.
208
210 Jiri Olsa
211USER COMMANDS 2019-10-22 klockstat(8)