1glocktop(8)                 System Manager's Manual                glocktop(8)
2
3
4

NAME

6       glocktop - Display or print active GFS2 locks.
7
8

SYNOPSIS

10       glocktop [OPTIONS]
11
12

DESCRIPTION

14       The glocktop tool is used to display active GFS2 inter-node locks, also
15       known as glocks. Simply put, it's a tool to filter  and  interpret  the
16       contents  of the glocks debugfs file. The glocks debugfs file shows all
17       glocks known to GFS2, their holders, and technical data such as  flags.
18       The  glocktop tool will only show the glocks that are important: glocks
19       that are being held or for which there are waiters. It also  interprets
20       the debugfs file of DLM (Distributed Lock Manager).
21
22

OPTIONS

24       -d <delay>
25              Specify  a  time delay (in seconds) between reports. (Default is
26              30 seconds)
27
28       -h     Print help information.
29
30       -i     Interactive mode. In this mode, glocktop acts more like the  top
31              command.   It shows the pertinent glocks on the terminal session
32              (as many as it can fit). The advantage is that it uses different
33              colors  to  draw attention to what's important. The disadvantage
34              is that it's limited by the size of your display, so you may not
35              see all the glocks.
36
37       -D     Omit DLM status. This may be used to reduce the amount of output
38              for interactive mode.
39
40       -n <iterations>
41              End  the  program  after  the  specified  number  of  iterations
42              (reports). The default is to keep running until interrupted.
43
44       -r     Show  resource group reservation information. Normally, glocktop
45              omits resource group reservation  information  to  condense  the
46              output. This information is only important when debugging infor‐
47              mation related to the GFS2 block allocator and file system frag‐
48              mentation.
49
50       -s <freq>
51              Print glock summary information every <freq> reports.  The glock
52              summary information is bulky and often not needed, so it's  only
53              printed  once  every  10  reports. You can eliminate it entirely
54              from the output by specifying a value of 0. If you want the sta‐
55              tistics to print after every report, specify freq as 1.
56
57       -t     Trace  directory  path. A lot of GFS2 glock performance problems
58              are caused by an application's contention for one or two  direc‐
59              tories.  These  show  up  as  regular  inodes in the output, but
60              there's no good way to tell from the output which  directory  is
61              contended.  Ordinarily,  glocktop  won't try to look up the full
62              pathname of a contended directory because it's slow,  especially
63              if  there are millions of glocks. This option instructs glocktop
64              to try to determine the full directory path names when  it  can,
65              so  you  can tell the full path (within the mount point) of con‐
66              tended directories.
67
68       -H     Don't show Held glocks, unless there are also  waiters  for  the
69              lock.   Ordinarily, glocktop will show glocks that are held (but
70              not iopen glocks which are almost always held by the  thousands)
71              as well as glocks for which there are waiters. If it only showed
72              glocks with waiters, you could see, for example, that a glock is
73              being  blocked on one node, but you couldn't see the information
74              for a different node currently holding the lock and thus, block‐
75              ing  the  waiter.  This  option forces glocktop to stop printing
76              information for glocks with no  waiters  (on  that  node).   The
77              advantage  is  that the output is smaller and easier to look at.
78              The disadvantage is that you can't see information from the node
79              that's blocking the waiter, unless both waiter and holder are on
80              the same node.
81

OUTPUT LINES

83       @ name This is the GFS2 file system name for which the  information  is
84              printed.  It  also  gives  the time stamp of the report, and the
85              cluster node name.
86
87       G:     This line represents a glock (internode GFS2 lock).
88               G:  s:UN n:2/609b4  f:lIqob  t:EX  d:EX/0  a:0  v:0  r:3  m:200
89              (inode)
90
91       D:     This  line  gives  you  glocktop's interpretation of the glock's
92              state as far as DLM (distributed lock manager) is concerned.
93                D: Granted PR on node 2 to pid 17511 [python]
94
95       H:     This line represents a glock holder:  a  process  that's  either
96              holding  the glock, or is waiting to hold it. The value after S:
97              represents how this holder needs the lock:  EX  (Exclusive),  SH
98              (Shared), PR (Protected Read), or UN (Unlocked). The value after
99              F: indicates the holder flags: a W indicates the holder is Wait‐
100              ing  for  the  lock  to be granted. An H indicates the holder is
101              currently holding the lock.
102                H: s:EX f:W e:0 p:17511 [python] gfs2_unlink+0x7e/0x250 [gfs2]
103
104       U:     These lines represent  glocktop's  user  interpretation  of  the
105              data, both glock and holder. Lines that begin with (N/A:...) can
106              probably be ignored because they ought to be unimportant: system
107              files such as journals, etc.
108                U:  W inode      183f5     Is:Shared, Want:Exclusive   [Demote
109              pending, Reply pending, Queued, Blocking]
110                U:  W  --->  waiting pid 17511 [python]  (Granted PR on node 2
111              to pid 17511 [python])
112
113       C:     These  lines give you the call trace (call stack) of the process
114              that's either holding or waiting to hold the glock.
115
116       S      These lines give you the summary of all  glocks  for  this  file
117              system: How many of each category are unlocked, locked, how many
118              are held in EX, SH, and DF, and how many are waiting. G  Waiting
119              is how many glocks have waiters. P Waiting is how many processes
120              are waiting. Thus, you could have one glock that's got ten  pro‐
121              cesses waiting, or ten glocks that have ten processes waiting.
122

EXAMPLE OUTPUT

124              # glocktop
125
126              @ nate_bob1       Wed Jan 27 07:24:14 2016  @host-050
127               G:  s:EX n:9/1 f:Iqb t:EX d:EX/0 a:0 v:0 r:2 m:200 (journal)
128                D: Granted EX on node 2 to pid 17468 [ended]
129                H: s:EX f:eH e:0 p:17468 [(ended)] gfs2_glock_nq_num+0x5b/0xa0 [gfs2]
130                U: (N/A:Journl)  H journal    1         Held:Exclusive   [Queued, Blocking]
131                U: (N/A:Journl)  H  --->  held by pid 17468 [(ended)]  (Granted EX on node 2 to pid 17468 [ended])
132               G:  s:SH n:1/1 f:Iqb t:SH d:EX/0 a:0 v:0 r:2 m:200 (non-disk)
133                D: Granted PR on node 2 to pid 17468 [ended]
134                H: s:SH f:eEH e:0 p:17468 [(ended)] gfs2_glock_nq_num+0x5b/0xa0 [gfs2]
135                U: (N/A:Not EX)  H non-disk   1         Held:Shared   [Queued, Blocking]
136                U: (N/A:Not EX)  H  --->  held by pid 17468 [(ended)]  (Granted PR on node 2 to pid 17468 [ended])
137               G:  s:EX n:2/181ec f:yIqob t:EX d:EX/0 a:1 v:0 r:3 m:200 (inode)
138                D: Granted EX on this node to pid 17468 [ended]
139                H: s:EX f:H e:0 p:17468 [(ended)] init_per_node+0x17d/0x280 [gfs2]
140                I: n:12/98796 t:8 f:0x00 d:0x00000201 s:24
141                U: (N/A:System)  H inode      181ec     Held:Exclusive   [Dirty, Queued, Blocking]
142                U: (N/A:System)  H  --->  held by pid 17468 [(ended)]  (Granted EX on this node to pid 17468 [ended])
143               G:  s:EX n:2/181ed f:Iqob t:EX d:EX/0 a:0 v:0 r:3 m:200 (inode)
144                D: Granted EX on this node to pid 17468 [ended]
145                H: s:EX f:H e:0 p:17468 [(ended)] init_per_node+0x1b0/0x280 [gfs2]
146                I: n:13/98797 t:8 f:0x00 d:0x00000200 s:1048576
147                U: (N/A:System)  H inode      181ed     Held:Exclusive   [Queued, Blocking]
148                U: (N/A:System)  H  --->  held by pid 17468 [(ended)]  (Granted EX on this node to pid 17468 [ended])
149               G:  s:SH n:2/183f5 f:ldrIqob t:EX d:UN/0 a:0 v:0 r:5 m:10 (inode)
150                D: Granted PR on node 2 to pid 17511 [python]
151                H: s:EX f:W e:0 p:17511 [python] gfs2_unlink+0x7e/0x250 [gfs2]
152                I: n:1/99317 t:4 f:0x00 d:0x00000003 s:2048
153                U:  W inode      183f5     Is:Shared, Want:Exclusive   [Demote pending, Reply pending, Queued, Blocking]
154                U:  W  --->  waiting pid 17511 [python]  (Granted PR on node 2 to pid 17511 [python])
155                C:              gfs2_unlink+0xdc/0x250 [gfs2]
156                C:              vfs_unlink+0xa0/0xf0
157                C:              do_unlinkat+0x163/0x260
158                C:              sys_unlink+0x16/0x20
159               G:  s:SH n:2/805b f:Iqob t:SH d:EX/0 a:0 v:0 r:3 m:200 (inode)
160                D: Granted PR on node 2 to pid 17468 [ended]
161                H: s:SH f:eEcH e:0 p:17468 [(ended)] init_journal+0x185/0x500 [gfs2]
162                I: n:5/32859 t:8 f:0x01 d:0x00000200 s:134217728
163                U: (N/A:Not EX)  H inode      805b      Held:Shared   [Queued, Blocking]
164                U: (N/A:Not EX)  H  --->  held by pid 17468 [(ended)]  (Granted PR on node 2 to pid 17468 [ended])
165              S    glocks  nondisk    inode    rgrp   iopen   flock quota jrnl    Total
166              S  --------- ------- -------- ------- ------- ------- ----- ---- --------
167              S  Unlocked:       1        5       4       0       0     0    0       10
168              S    Locked:       2      245       6      58       0     0    1      313
169              S     Total:       3      250      10      58       0     0    1      323
170              S
171              S   Held EX:       0        2       0       0       0     0    1        3
172              S   Held SH:       1        1       0      57       0     0    0       59
173              S   Held DF:       0        0       0       0       0     0    0        0
174              S G Waiting:       0        1       0       0       0     0    0        1
175              S P Waiting:       0        1       0       0       0     0    0        1
176              S  DLM wait:       0
177
178              @ nate_bob0       Wed Jan 27 07:24:14 2016  @host-050
179               G:  s:EX n:2/180e9 f:yIqob t:EX d:EX/0 a:1 v:0 r:3 m:200 (inode)
180                D: Granted EX on this node to pid 17465 [ended]
181                H: s:EX f:H e:0 p:17465 [(ended)] init_per_node+0x17d/0x280 [gfs2]
182                I: n:9/98537 t:8 f:0x00 d:0x00000201 s:24
183                U: (N/A:System)  H inode      180e9     Held:Exclusive   [Dirty, Queued, Blocking]
184                U: (N/A:System)  H  --->  held by pid 17465 [(ended)]  (Granted EX on this node to pid 17465 [ended])
185               G:  s:UN n:2/609b4 f:lIqob t:EX d:EX/0 a:0 v:0 r:3 m:200 (inode)
186                D: Granted EX on node 2 to pid 14367 [ended]
187                H: s:EX f:W e:0 p:16297 [delete_workqueu] gfs2_delete_inode+0x9d/0x450 [gfs2]
188                U:  W inode      609b4     Is:Unlocked, Want:Exclusive   [Queued, Blocking]
189                U:  W  --->  waiting pid 16297 [delete_workqueu]  (Granted EX on node 2 to pid 14367 [ended])
190                C:              gfs2_delete_inode+0xa5/0x450 [gfs2]
191                C:              generic_delete_inode+0xde/0x1d0
192                C:              generic_drop_inode+0x65/0x80
193                C:              gfs2_drop_inode+0x37/0x40 [gfs2]
194               G:  s:SH n:2/19 f:Iqob t:SH d:EX/0 a:0 v:0 r:3 m:200 (inode)
195                D: Granted PR on this node to pid 17465 [ended]
196                H: s:SH f:eEcH e:0 p:17465 [(ended)] init_journal+0x185/0x500 [gfs2]
197                I: n:4/25 t:8 f:0x01 d:0x00000200 s:134217728
198                U: (N/A:Not EX)  H inode      19        Held:Shared   [Queued, Blocking]
199                U: (N/A:Not EX)  H  --->  held by pid 17465 [(ended)]  (Granted PR on this node to pid 17465 [ended])
200               G:  s:EX n:2/180ea f:Iqob t:EX d:EX/0 a:0 v:0 r:3 m:200 (inode)
201                D: Granted EX on this node to pid 17465 [ended]
202                H: s:EX f:H e:0 p:17465 [(ended)] init_per_node+0x1b0/0x280 [gfs2]
203                I: n:10/98538 t:8 f:0x00 d:0x00000200 s:1048576
204                U: (N/A:System)  H inode      180ea     Held:Exclusive   [Queued, Blocking]
205                U: (N/A:System)  H  --->  held by pid 17465 [(ended)]  (Granted EX on this node to pid 17465 [ended])
206               G:  s:EX n:9/0 f:Iqb t:EX d:EX/0 a:0 v:0 r:2 m:200 (journal)
207                D: Granted EX on this node to pid 17465 [ended]
208                H: s:EX f:eH e:0 p:17465 [(ended)] gfs2_glock_nq_num+0x5b/0xa0 [gfs2]
209                U: (N/A:Journl)  H journal    0         Held:Exclusive   [Queued, Blocking]
210                U: (N/A:Journl)  H  --->  held by pid 17465 [(ended)]  (Granted EX on this node to pid 17465 [ended])
211               G:  s:UN n:2/4fe12 f:ldIqob t:EX d:UN/0 a:0 v:0 r:4 m:10 (inode)
212                H: s:EX f:W e:0 p:17523 [python] gfs2_rename+0x344/0x8b0 [gfs2]
213                H: s:SH f:AW e:0 p:17527 [python] gfs2_permission+0x176/0x210 [gfs2]
214                U:  W inode      4fe12     Is:Unlocked, Want:Exclusive   [Demote pending, Queued, Blocking]
215                U:  W  --->  waiting pid 17523 [python]
216                C:              gfs2_permission+0x17f/0x210 [gfs2]
217                C:              __link_path_walk+0xb3/0x1000
218                C:              path_walk+0x6a/0xe0
219                C:              filename_lookup+0x6b/0xc0
220                U:  W  --->  waiting pid 17527 [python]
221                C:              do_unlinkat+0x107/0x260
222                C:              sys_unlink+0x16/0x20
223                C:              system_call_fastpath+0x16/0x1b
224                C:              0xffffffffffffffff
225               G:  s:SH n:1/1 f:Iqb t:SH d:EX/0 a:0 v:0 r:2 m:200 (non-disk)
226                D: Granted PR on node 2 to pid 14285 [ended]
227                D: Granted PR on this node to pid 17465 [ended]
228                H: s:SH f:eEH e:0 p:17465 [(ended)] gfs2_glock_nq_num+0x5b/0xa0 [gfs2]
229                U: (N/A:Not EX)  H non-disk   1         Held:Shared   [Queued, Blocking]
230                U: (N/A:Not EX)  H  --->  held by pid 17465 [(ended)]  (Granted PR on node 2 to pid 14285 [ended]) (Granted PR on this node to pid 17465 [ended])
231              S    glocks  nondisk    inode    rgrp   iopen   flock quota jrnl    Total
232              S  --------- ------- -------- ------- ------- ------- ----- ---- --------
233              S  Unlocked:       1        8       7       0       0     0    0       16
234              S    Locked:       2      208       3      41       0     0    1      256
235              S     Total:       3      216      10      41       0     0    1      272
236              S
237              S   Held EX:       0        2       0       0       0     0    1        3
238              S   Held SH:       1        1       0      41       0     0    0       43
239              S   Held DF:       0        0       0       0       0     0    0        0
240              S G Waiting:       0        2       0       0       0     0    0        2
241              S P Waiting:       0        3       0       0       0     0    0        3
242              S  DLM wait:       0
243
244       From  this  example  output, we can see there are two GFS2 file systems
245       mounted on system host-050: nate_bob1 and nate_bob0. In  nate_bob1,  we
246       can  see  six  glocks,  but  we can ignore all of them marked (N/A:...)
247       because they are system files or held in  SHared  mode,  and  therefore
248       other nodes should be able to hold the lock in SHared as well.
249
250       There  is one glock, for inode 183f5, which is has a process waiting to
251       hold it. The lock is currently in SHared mode (s:SH on the G: line) but
252       process  17511  (python) wants to hold the lock in EXclusive mode (S:EX
253       on the H: line). That process has a call stack  that  indicates  it  is
254       trying  to  hold  the  glock from gfs2_unlink. The DLM says the lock is
255       currently granted on node 2 in PR (Protected Read) mode.
256
257       For file system nate_bob0, there are 7 glocks listed. All but  two  are
258       uninteresting. Locks 2/609b4 and 2/4fe12 have processes waiting to hold
259       them.
260
261       In the summary data for nate_bob0, you can see there  are  3  processes
262       waiting  for  2  inode glocks (so one of those glocks has multiple pro‐
263       cesses waiting).
264
265       Since DLM wait is 0 in the summary data for  both  GFS2  mount  points,
266       nobody is waiting for DLM to grant the lock.
267
268

KNOWN BUGS AND LIMITATIONS

270       Since  the  GFS2  debugfs  files  are  completely separate from the DLM
271       debugfs files, and locks can change status in a few  nanoseconds  time,
272       there  will always be a lag between the GFS2 view of a lock and the DLM
273       view of a lock.  If there is some kind of long-term hang, they are more
274       likely  to  match.   However,  under  ordinary  conditions, by the time
275       glocktop gets around to fetching the DLM status of a lock, the informa‐
276       tion  has changed. Therefore, don't be surprised if the DLM's view of a
277       lock is at odds with its glock.
278
279       Since iopen glocks are held by the thousands, glocktop  skips  most  of
280       the  information related to them unless there's a waiter. For that rea‐
281       son, iopen lock problems may be difficult to debug with glocktop.
282
283       It doesn't handle very large numbers (millions) of glocks.
284
285
286
287
288                                                                   glocktop(8)
Impressum