1glocktop(8) System Manager's Manual glocktop(8)
2
6 glocktop - Display or print active GFS2 locks.
7
10 glocktop [OPTIONS]
11
14 The glocktop tool is used to display active GFS2 inter-node locks, also
15 known as glocks. Simply put, it's a tool to filter and interpret the
16 contents of the glocks debugfs file. The glocks debugfs file shows all
17 glocks known to GFS2, their holders, and technical data such as flags.
18 The glocktop tool will only show the glocks that are important: glocks
19 that are being held or for which there are waiters. It also interprets
20 the debugfs file of DLM (Distributed Lock Manager).
21
24 -d <delay>
25 Specify a time delay (in seconds) between reports. (Default is
26 30 seconds)
27 -h Print help information.
29 -i Interactive mode. In this mode, glocktop acts more like the top
31 command. It shows the pertinent glocks on the terminal session
32 (as many as it can fit). The advantage is that it uses different
33 colors to draw attention to what's important. The disadvantage
34 is that it's limited by the size of your display, so you may not
35 see all the glocks.
36 -D Omit DLM status. This may be used to reduce the amount of output
38 for interactive mode.
39 -n <iterations>
41 End the program after the specified number of iterations
42 (reports). The default is to keep running until interrupted.
43 -r Show resource group reservation information. Normally, glocktop
45 omits resource group reservation information to condense the
46 output. This information is only important when debugging infor‐
47 mation related to the GFS2 block allocator and file system frag‐
48 mentation.
49 -s <freq>
51 Print glock summary information every <freq> reports. The glock
52 summary information is bulky and often not needed, so it's only
53 printed once every 10 reports. You can eliminate it entirely
54 from the output by specifying a value of 0. If you want the sta‐
55 tistics to print after every report, specify freq as 1.
56 -t Trace directory path. A lot of GFS2 glock performance problems
58 are caused by an application's contention for one or two direc‐
59 tories. These show up as regular inodes in the output, but
60 there's no good way to tell from the output which directory is
61 contended. Ordinarily, glocktop won't try to look up the full
62 pathname of a contended directory because it's slow, especially
63 if there are millions of glocks. This option instructs glocktop
64 to try to determine the full directory path names when it can,
65 so you can tell the full path (within the mount point) of con‐
66 tended directories.
67 -H Don't show Held glocks, unless there are also waiters for the
69 lock. Ordinarily, glocktop will show glocks that are held (but
70 not iopen glocks which are almost always held by the thousands)
71 as well as glocks for which there are waiters. If it only showed
72 glocks with waiters, you could see, for example, that a glock is
73 being blocked on one node, but you couldn't see the information
74 for a different node currently holding the lock and thus, block‐
75 ing the waiter. This option forces glocktop to stop printing
76 information for glocks with no waiters (on that node). The
77 advantage is that the output is smaller and easier to look at.
78 The disadvantage is that you can't see information from the node
79 that's blocking the waiter, unless both waiter and holder are on
80 the same node.
81
83 @ name This is the GFS2 file system name for which the information is
84 printed. It also gives the time stamp of the report, and the
85 cluster node name.
86 G: This line represents a glock (internode GFS2 lock).
88 G: s:UN n:2/609b4 f:lIqob t:EX d:EX/0 a:0 v:0 r:3 m:200
89 (inode)
90 D: This line gives you glocktop's interpretation of the glock's
92 state as far as DLM (distributed lock manager) is concerned.
93 D: Granted PR on node 2 to pid 17511 [python]
94 H: This line represents a glock holder: a process that's either
96 holding the glock, or is waiting to hold it. The value after S:
97 represents how this holder needs the lock: EX (Exclusive), SH
98 (Shared), PR (Protected Read), or UN (Unlocked). The value after
99 F: indicates the holder flags: a W indicates the holder is Wait‐
100 ing for the lock to be granted. An H indicates the holder is
101 currently holding the lock.
102 H: s:EX f:W e:0 p:17511 [python] gfs2_unlink+0x7e/0x250 [gfs2]
103 U: These lines represent glocktop's user interpretation of the
105 data, both glock and holder. Lines that begin with (N/A:...) can
106 probably be ignored because they ought to be unimportant: system
107 files such as journals, etc.
108 U: W inode 183f5 Is:Shared, Want:Exclusive [Demote
109 pending, Reply pending, Queued, Blocking]
110 U: W ---> waiting pid 17511 [python] (Granted PR on node 2
111 to pid 17511 [python])
112 C: These lines give you the call trace (call stack) of the process
114 that's either holding or waiting to hold the glock.
115 S These lines give you the summary of all glocks for this file
117 system: How many of each category are unlocked, locked, how many
118 are held in EX, SH, and DF, and how many are waiting. G Waiting
119 is how many glocks have waiters. P Waiting is how many processes
120 are waiting. Thus, you could have one glock that's got ten pro‐
121 cesses waiting, or ten glocks that have ten processes waiting.
122
124 # glocktop
125 @ nate_bob1 Wed Jan 27 07:24:14 2016 @host-050
127 G: s:EX n:9/1 f:Iqb t:EX d:EX/0 a:0 v:0 r:2 m:200 (journal)
128 D: Granted EX on node 2 to pid 17468 [ended]
129 H: s:EX f:eH e:0 p:17468 [(ended)] gfs2_glock_nq_num+0x5b/0xa0 [gfs2]
130 U: (N/A:Journl) H journal 1 Held:Exclusive [Queued, Blocking]
131 U: (N/A:Journl) H ---> held by pid 17468 [(ended)] (Granted EX on node 2 to pid 17468 [ended])
132 G: s:SH n:1/1 f:Iqb t:SH d:EX/0 a:0 v:0 r:2 m:200 (non-disk)
133 D: Granted PR on node 2 to pid 17468 [ended]
134 H: s:SH f:eEH e:0 p:17468 [(ended)] gfs2_glock_nq_num+0x5b/0xa0 [gfs2]
135 U: (N/A:Not EX) H non-disk 1 Held:Shared [Queued, Blocking]
136 U: (N/A:Not EX) H ---> held by pid 17468 [(ended)] (Granted PR on node 2 to pid 17468 [ended])
137 G: s:EX n:2/181ec f:yIqob t:EX d:EX/0 a:1 v:0 r:3 m:200 (inode)
138 D: Granted EX on this node to pid 17468 [ended]
139 H: s:EX f:H e:0 p:17468 [(ended)] init_per_node+0x17d/0x280 [gfs2]
140 I: n:12/98796 t:8 f:0x00 d:0x00000201 s:24
141 U: (N/A:System) H inode 181ec Held:Exclusive [Dirty, Queued, Blocking]
142 U: (N/A:System) H ---> held by pid 17468 [(ended)] (Granted EX on this node to pid 17468 [ended])
143 G: s:EX n:2/181ed f:Iqob t:EX d:EX/0 a:0 v:0 r:3 m:200 (inode)
144 D: Granted EX on this node to pid 17468 [ended]
145 H: s:EX f:H e:0 p:17468 [(ended)] init_per_node+0x1b0/0x280 [gfs2]
146 I: n:13/98797 t:8 f:0x00 d:0x00000200 s:1048576
147 U: (N/A:System) H inode 181ed Held:Exclusive [Queued, Blocking]
148 U: (N/A:System) H ---> held by pid 17468 [(ended)] (Granted EX on this node to pid 17468 [ended])
149 G: s:SH n:2/183f5 f:ldrIqob t:EX d:UN/0 a:0 v:0 r:5 m:10 (inode)
150 D: Granted PR on node 2 to pid 17511 [python]
151 H: s:EX f:W e:0 p:17511 [python] gfs2_unlink+0x7e/0x250 [gfs2]
152 I: n:1/99317 t:4 f:0x00 d:0x00000003 s:2048
153 U: W inode 183f5 Is:Shared, Want:Exclusive [Demote pending, Reply pending, Queued, Blocking]
154 U: W ---> waiting pid 17511 [python] (Granted PR on node 2 to pid 17511 [python])
155 C: gfs2_unlink+0xdc/0x250 [gfs2]
156 C: vfs_unlink+0xa0/0xf0
157 C: do_unlinkat+0x163/0x260
158 C: sys_unlink+0x16/0x20
159 G: s:SH n:2/805b f:Iqob t:SH d:EX/0 a:0 v:0 r:3 m:200 (inode)
160 D: Granted PR on node 2 to pid 17468 [ended]
161 H: s:SH f:eEcH e:0 p:17468 [(ended)] init_journal+0x185/0x500 [gfs2]
162 I: n:5/32859 t:8 f:0x01 d:0x00000200 s:134217728
163 U: (N/A:Not EX) H inode 805b Held:Shared [Queued, Blocking]
164 U: (N/A:Not EX) H ---> held by pid 17468 [(ended)] (Granted PR on node 2 to pid 17468 [ended])
165 S glocks nondisk inode rgrp iopen flock quota jrnl Total
166 S --------- ------- -------- ------- ------- ------- ----- ---- --------
167 S Unlocked: 1 5 4 0 0 0 0 10
168 S Locked: 2 245 6 58 0 0 1 313
169 S Total: 3 250 10 58 0 0 1 323
170 S
171 S Held EX: 0 2 0 0 0 0 1 3
172 S Held SH: 1 1 0 57 0 0 0 59
173 S Held DF: 0 0 0 0 0 0 0 0
174 S G Waiting: 0 1 0 0 0 0 0 1
175 S P Waiting: 0 1 0 0 0 0 0 1
176 S DLM wait: 0
177 @ nate_bob0 Wed Jan 27 07:24:14 2016 @host-050
179 G: s:EX n:2/180e9 f:yIqob t:EX d:EX/0 a:1 v:0 r:3 m:200 (inode)
180 D: Granted EX on this node to pid 17465 [ended]
181 H: s:EX f:H e:0 p:17465 [(ended)] init_per_node+0x17d/0x280 [gfs2]
182 I: n:9/98537 t:8 f:0x00 d:0x00000201 s:24
183 U: (N/A:System) H inode 180e9 Held:Exclusive [Dirty, Queued, Blocking]
184 U: (N/A:System) H ---> held by pid 17465 [(ended)] (Granted EX on this node to pid 17465 [ended])
185 G: s:UN n:2/609b4 f:lIqob t:EX d:EX/0 a:0 v:0 r:3 m:200 (inode)
186 D: Granted EX on node 2 to pid 14367 [ended]
187 H: s:EX f:W e:0 p:16297 [delete_workqueu] gfs2_delete_inode+0x9d/0x450 [gfs2]
188 U: W inode 609b4 Is:Unlocked, Want:Exclusive [Queued, Blocking]
189 U: W ---> waiting pid 16297 [delete_workqueu] (Granted EX on node 2 to pid 14367 [ended])
190 C: gfs2_delete_inode+0xa5/0x450 [gfs2]
191 C: generic_delete_inode+0xde/0x1d0
192 C: generic_drop_inode+0x65/0x80
193 C: gfs2_drop_inode+0x37/0x40 [gfs2]
194 G: s:SH n:2/19 f:Iqob t:SH d:EX/0 a:0 v:0 r:3 m:200 (inode)
195 D: Granted PR on this node to pid 17465 [ended]
196 H: s:SH f:eEcH e:0 p:17465 [(ended)] init_journal+0x185/0x500 [gfs2]
197 I: n:4/25 t:8 f:0x01 d:0x00000200 s:134217728
198 U: (N/A:Not EX) H inode 19 Held:Shared [Queued, Blocking]
199 U: (N/A:Not EX) H ---> held by pid 17465 [(ended)] (Granted PR on this node to pid 17465 [ended])
200 G: s:EX n:2/180ea f:Iqob t:EX d:EX/0 a:0 v:0 r:3 m:200 (inode)
201 D: Granted EX on this node to pid 17465 [ended]
202 H: s:EX f:H e:0 p:17465 [(ended)] init_per_node+0x1b0/0x280 [gfs2]
203 I: n:10/98538 t:8 f:0x00 d:0x00000200 s:1048576
204 U: (N/A:System) H inode 180ea Held:Exclusive [Queued, Blocking]
205 U: (N/A:System) H ---> held by pid 17465 [(ended)] (Granted EX on this node to pid 17465 [ended])
206 G: s:EX n:9/0 f:Iqb t:EX d:EX/0 a:0 v:0 r:2 m:200 (journal)
207 D: Granted EX on this node to pid 17465 [ended]
208 H: s:EX f:eH e:0 p:17465 [(ended)] gfs2_glock_nq_num+0x5b/0xa0 [gfs2]
209 U: (N/A:Journl) H journal 0 Held:Exclusive [Queued, Blocking]
210 U: (N/A:Journl) H ---> held by pid 17465 [(ended)] (Granted EX on this node to pid 17465 [ended])
211 G: s:UN n:2/4fe12 f:ldIqob t:EX d:UN/0 a:0 v:0 r:4 m:10 (inode)
212 H: s:EX f:W e:0 p:17523 [python] gfs2_rename+0x344/0x8b0 [gfs2]
213 H: s:SH f:AW e:0 p:17527 [python] gfs2_permission+0x176/0x210 [gfs2]
214 U: W inode 4fe12 Is:Unlocked, Want:Exclusive [Demote pending, Queued, Blocking]
215 U: W ---> waiting pid 17523 [python]
216 C: gfs2_permission+0x17f/0x210 [gfs2]
217 C: __link_path_walk+0xb3/0x1000
218 C: path_walk+0x6a/0xe0
219 C: filename_lookup+0x6b/0xc0
220 U: W ---> waiting pid 17527 [python]
221 C: do_unlinkat+0x107/0x260
222 C: sys_unlink+0x16/0x20
223 C: system_call_fastpath+0x16/0x1b
224 C: 0xffffffffffffffff
225 G: s:SH n:1/1 f:Iqb t:SH d:EX/0 a:0 v:0 r:2 m:200 (non-disk)
226 D: Granted PR on node 2 to pid 14285 [ended]
227 D: Granted PR on this node to pid 17465 [ended]
228 H: s:SH f:eEH e:0 p:17465 [(ended)] gfs2_glock_nq_num+0x5b/0xa0 [gfs2]
229 U: (N/A:Not EX) H non-disk 1 Held:Shared [Queued, Blocking]
230 U: (N/A:Not EX) H ---> held by pid 17465 [(ended)] (Granted PR on node 2 to pid 14285 [ended]) (Granted PR on this node to pid 17465 [ended])
231 S glocks nondisk inode rgrp iopen flock quota jrnl Total
232 S --------- ------- -------- ------- ------- ------- ----- ---- --------
233 S Unlocked: 1 8 7 0 0 0 0 16
234 S Locked: 2 208 3 41 0 0 1 256
235 S Total: 3 216 10 41 0 0 1 272
236 S
237 S Held EX: 0 2 0 0 0 0 1 3
238 S Held SH: 1 1 0 41 0 0 0 43
239 S Held DF: 0 0 0 0 0 0 0 0
240 S G Waiting: 0 2 0 0 0 0 0 2
241 S P Waiting: 0 3 0 0 0 0 0 3
242 S DLM wait: 0
243 From this example output, we can see there are two GFS2 file systems
245 mounted on system host-050: nate_bob1 and nate_bob0. In nate_bob1, we
246 can see six glocks, but we can ignore all of them marked (N/A:...)
247 because they are system files or held in SHared mode, and therefore
248 other nodes should be able to hold the lock in SHared as well.
249 There is one glock, for inode 183f5, which is has a process waiting to
251 hold it. The lock is currently in SHared mode (s:SH on the G: line) but
252 process 17511 (python) wants to hold the lock in EXclusive mode (S:EX
253 on the H: line). That process has a call stack that indicates it is
254 trying to hold the glock from gfs2_unlink. The DLM says the lock is
255 currently granted on node 2 in PR (Protected Read) mode.
256 For file system nate_bob0, there are 7 glocks listed. All but two are
258 uninteresting. Locks 2/609b4 and 2/4fe12 have processes waiting to hold
259 them.
260 In the summary data for nate_bob0, you can see there are 3 processes
262 waiting for 2 inode glocks (so one of those glocks has multiple pro‐
263 cesses waiting).
264 Since DLM wait is 0 in the summary data for both GFS2 mount points,
266 nobody is waiting for DLM to grant the lock.
267
270 Since the GFS2 debugfs files are completely separate from the DLM
271 debugfs files, and locks can change status in a few nanoseconds time,
272 there will always be a lag between the GFS2 view of a lock and the DLM
273 view of a lock. If there is some kind of long-term hang, they are more
274 likely to match. However, under ordinary conditions, by the time
275 glocktop gets around to fetching the DLM status of a lock, the informa‐
276 tion has changed. Therefore, don't be surprised if the DLM's view of a
277 lock is at odds with its glock.
278 Since iopen glocks are held by the thousands, glocktop skips most of
280 the information related to them unless there's a waiter. For that rea‐
281 son, iopen lock problems may be difficult to debug with glocktop.
282 It doesn't handle very large numbers (millions) of glocks.
284 glocktop(8)