1deadlock_detector(8) System Manager's Manual deadlock_detector(8)
2
6 deadlock_detector - Find potential deadlocks (lock order inversions) in
7 a running program.
8
10 deadlock_detector [-h] [--binary BINARY] [--dump-graph DUMP_GRAPH]
11 [--verbose] [--lock-symbols LOCK_SYMBOLS] [--unlock-symbols UNLOCK_SYM‐
12 BOLS] pid
13
15 deadlock_detector finds potential deadlocks in a running process. The
16 program attaches uprobes on `pthread_mutex_lock` and
17 `pthread_mutex_unlock` by default to build a mutex wait directed graph,
18 and then looks for a cycle in this graph. This graph has the following
19 properties:
20 - Nodes in the graph represent mutexes.
22 - Edge (A, B) exists if there exists some thread T where lock(A) was
24 called and lock(B) was called before unlock(A) was called.
25 If there is a cycle in this graph, this indicates that there is a lock
27 order inversion (potential deadlock). If the program finds a lock order
28 inversion, the program will dump the cycle of mutexes, dump the stack
29 traces where each mutex was acquired, and then exit.
30 This program can only find potential deadlocks that occur while the
32 program is tracing the process. It cannot find deadlocks that may have
33 occurred before the program was attached to the process.
34 This tool does not work for shared mutexes or recursive mutexes.
36 Since this uses BPF, only the root user can use this tool.
38
40 CONFIG_BPF and bcc
41
43 -h, --help
44 show this help message and exit
45 --binary BINARY
47 If set, trace the mutexes from the binary at this path. For
48 statically-linked binaries, this argument is not required. For
49 dynamically-linked binaries, this argument is required and
50 should be the path of the pthread library the binary is using.
51 Example: /lib/x86_64-linux-gnu/libpthread.so.0
52 --dump-graph DUMP_GRAPH
54 If set, this will dump the mutex graph to the specified file.
55 --verbose
57 Print statistics about the mutex wait graph.
58 --lock-symbols LOCK_SYMBOLS
60 Comma-separated list of lock symbols to trace. Default is
61 pthread_mutex_lock. These symbols cannot be inlined in the
62 binary.
63 --unlock-symbols UNLOCK_SYMBOLS
65 Comma-separated list of unlock symbols to trace. Default is
66 pthread_mutex_unlock. These symbols cannot be inlined in the
67 binary.
68 pid Pid to trace
70
72 Find potential deadlocks in PID 181. The --binary argument is not
73 needed for statically-linked binaries.
74 # deadlock_detector 181
75 Find potential deadlocks in PID 181. If the process was created from a
77 dynamically-linked executable, the --binary argument is required and
78 must be the path of the pthread library:
79 # deadlock_detector 181 --binary /lib/x86_64-linux-
80 gnu/libpthread.so.0
81 Find potential deadlocks in PID 181. If the process was created from a
83 statically-linked executable, optionally pass the location of the
84 binary. On older kernels without https://lkml.org/lkml/2017/1/13/585,
85 binaries that contain `:` in the path cannot be attached with uprobes.
86 As a workaround, we can create a symlink to the binary, and provide the
87 symlink name instead with the `--binary` option:
88 # deadlock_detector 181 --binary /usr/local/bin/lockinversion
89 Find potential deadlocks in PID 181 and dump the mutex wait graph to a
91 file:
92 # deadlock_detector 181 --dump-graph graph.json
93 Find potential deadlocks in PID 181 and print mutex wait graph statis‐
95 tics:
96 # deadlock_detector 181 --verbose
97 Find potential deadlocks in PID 181 with custom mutexes:
99 # deadlock_detector 181 --lock-symbols custom_mutex1_lock,cus‐
100 tom_mutex2_lock --unlock_symbols custom_mutex1_unlock,cus‐
101 tom_mutex2_unlock
102
104 This program does not output any fields. Rather, it will keep running
105 until it finds a potential deadlock, or the user hits Ctrl-C. If the
106 program finds a potential deadlock, it will output the stack traces and
107 lock order inversion in the following format and exit:
108 Potential Deadlock Detected!
110 Cycle in lock order graph: Mutex M0 => Mutex M1 => Mutex M0
112 Mutex M1 acquired here while holding Mutex M0 in Thread T:
114 [stack trace]
115 Mutex M0 previously acquired by the same Thread T here:
117 [stack trace]
118 Mutex M0 acquired here while holding Mutex M1 in Thread S:
120 [stack trace]
121 Mutex M1 previously acquired by the same Thread S here:
123 [stack trace]
124 Thread T created by Thread R here:
126 [stack trace]
127 Thread S created by Thread Q here:
129 [stack trace]
130
132 This traces all mutex lock and unlock events and all thread creation
133 events on the traced process. The overhead of this can be high if the
134 process has many threads and mutexes. You should only run this on a
135 process where the slowdown is acceptable.
136
138 This is from bcc.
139 https://github.com/iovisor/bcc
141 Also look in the bcc distribution for a companion _examples.txt file
143 containing example usage, output, and commentary for this tool.
144
146 Linux
147
149 Unstable - in development.
150
152 Kenny Yu
153USER COMMANDS 2017-02-01 deadlock_detector(8)