1deadlock(8) System Manager's Manual deadlock(8)
2
6 deadlock - Find potential deadlocks (lock order inversions) in a run‐
7 ning program.
8
10 deadlock [-h] [--binary BINARY] [--dump-graph DUMP_GRAPH] [--verbose]
11 [--lock-symbols LOCK_SYMBOLS] [--unlock-symbols UNLOCK_SYMBOLS] pid
12
14 deadlock finds potential deadlocks in a running process. The program
15 attaches uprobes on `pthread_mutex_lock` and `pthread_mutex_unlock` by
16 default to build a mutex wait directed graph, and then looks for a cy‐
17 cle in this graph. This graph has the following properties:
18 - Nodes in the graph represent mutexes.
20 - Edge (A, B) exists if there exists some thread T where lock(A) was
22 called and lock(B) was called before unlock(A) was called.
23 If there is a cycle in this graph, this indicates that there is a lock
25 order inversion (potential deadlock). If the program finds a lock order
26 inversion, the program will dump the cycle of mutexes, dump the stack
27 traces where each mutex was acquired, and then exit.
28 This program can only find potential deadlocks that occur while the
30 program is tracing the process. It cannot find deadlocks that may have
31 occurred before the program was attached to the process.
32 This tool does not work for shared mutexes or recursive mutexes.
34 Since this uses BPF, only the root user can use this tool.
36
38 CONFIG_BPF and bcc
39
41 -h, --help
42 show this help message and exit
43 --binary BINARY
45 If set, trace the mutexes from the binary at this path. For
46 statically-linked binaries, this argument is not required. For
47 dynamically-linked binaries, this argument is required and
48 should be the path of the pthread library the binary is using.
49 Example: /lib/x86_64-linux-gnu/libpthread.so.0
50 --dump-graph DUMP_GRAPH
52 If set, this will dump the mutex graph to the specified file.
53 --verbose
55 Print statistics about the mutex wait graph.
56 --lock-symbols LOCK_SYMBOLS
58 Comma-separated list of lock symbols to trace. Default is
59 pthread_mutex_lock. These symbols cannot be inlined in the bi‐
60 nary.
61 --unlock-symbols UNLOCK_SYMBOLS
63 Comma-separated list of unlock symbols to trace. Default is
64 pthread_mutex_unlock. These symbols cannot be inlined in the bi‐
65 nary.
66 -t THREADS, --threads THREADS
68 Specifies the maximum number of threads to trace. default 65536.
69 Note. 40 bytes per thread.
70 -e EDGES, --edges EDGES
72 Specifies the maximum number of edge cases that can be recorded.
73 default 65536. Note. 88 bytes per edge case.
74 pid Pid to trace
76
78 Find potential deadlocks in PID 181. The --binary argument is not
79 needed for statically-linked binaries.
80 # deadlock 181
81 Find potential deadlocks in PID 181. If the process was created from a
83 dynamically-linked executable, the --binary argument is required and
84 must be the path of the pthread library:
85 # deadlock 181 --binary /lib/x86_64-linux-gnu/libpthread.so.0
86 Find potential deadlocks in PID 181. If the process was created from a
88 statically-linked executable, optionally pass the location of the bi‐
89 nary. On older kernels without https://lkml.org/lkml/2017/1/13/585, bi‐
90 naries that contain `:` in the path cannot be attached with uprobes. As
91 a workaround, we can create a symlink to the binary, and provide the
92 symlink name instead with the `--binary` option:
93 # deadlock 181 --binary /usr/local/bin/lockinversion
94 Find potential deadlocks in PID 181 and dump the mutex wait graph to a
96 file:
97 # deadlock 181 --dump-graph graph.json
98 Find potential deadlocks in PID 181 and print mutex wait graph statis‐
100 tics:
101 # deadlock 181 --verbose
102 Find potential deadlocks in PID 181 with custom mutexes:
104 # deadlock 181 --lock-symbols custom_mutex1_lock,custom_mu‐
105 tex2_lock --unlock_symbols custom_mutex1_unlock,custom_mu‐
106 tex2_unlock
107
109 This program does not output any fields. Rather, it will keep running
110 until it finds a potential deadlock, or the user hits Ctrl-C. If the
111 program finds a potential deadlock, it will output the stack traces and
112 lock order inversion in the following format and exit:
113 Potential Deadlock Detected!
115 Cycle in lock order graph: Mutex M0 => Mutex M1 => Mutex M0
117 Mutex M1 acquired here while holding Mutex M0 in Thread T:
119 [stack trace]
120 Mutex M0 previously acquired by the same Thread T here:
122 [stack trace]
123 Mutex M0 acquired here while holding Mutex M1 in Thread S:
125 [stack trace]
126 Mutex M1 previously acquired by the same Thread S here:
128 [stack trace]
129 Thread T created by Thread R here:
131 [stack trace]
132 Thread S created by Thread Q here:
134 [stack trace]
135
137 This traces all mutex lock and unlock events and all thread creation
138 events on the traced process. The overhead of this can be high if the
139 process has many threads and mutexes. You should only run this on a
140 process where the slowdown is acceptable.
141
143 This is from bcc.
144 https://github.com/iovisor/bcc
146 Also look in the bcc distribution for a companion _examples.txt file
148 containing example usage, output, and commentary for this tool.
149
151 Linux
152
154 Unstable - in development.
155
157 Kenny Yu
158USER COMMANDS 2017-02-01 deadlock(8)