1deadlock(8)                 System Manager's Manual                deadlock(8)
2
3
4

NAME

6       deadlock  -  Find potential deadlocks (lock order inversions) in a run‐
7       ning program.
8

SYNOPSIS

10       deadlock [-h] [--binary BINARY] [--dump-graph  DUMP_GRAPH]  [--verbose]
11       [--lock-symbols LOCK_SYMBOLS] [--unlock-symbols UNLOCK_SYMBOLS] pid
12

DESCRIPTION

14       deadlock  finds  potential  deadlocks in a running process. The program
15       attaches uprobes on `pthread_mutex_lock` and `pthread_mutex_unlock`  by
16       default  to build a mutex wait directed graph, and then looks for a cy‐
17       cle in this graph.  This graph has the following properties:
18
19       - Nodes in the graph represent mutexes.
20
21       - Edge (A, B) exists if there exists some thread T  where  lock(A)  was
22       called and lock(B) was called before unlock(A) was called.
23
24       If  there is a cycle in this graph, this indicates that there is a lock
25       order inversion (potential deadlock). If the program finds a lock order
26       inversion,  the  program will dump the cycle of mutexes, dump the stack
27       traces where each mutex was acquired, and then exit.
28
29       This program can only find potential deadlocks  that  occur  while  the
30       program  is tracing the process. It cannot find deadlocks that may have
31       occurred before the program was attached to the process.
32
33       This tool does not work for shared mutexes or recursive mutexes.
34
35       Since this uses BPF, only the root user can use this tool.
36

REQUIREMENTS

38       CONFIG_BPF and bcc
39

OPTIONS

41       -h, --help
42              show this help message and exit
43
44       --binary BINARY
45              If set, trace the mutexes from the  binary  at  this  path.  For
46              statically-linked  binaries, this argument is not required.  For
47              dynamically-linked  binaries,  this  argument  is  required  and
48              should  be  the path of the pthread library the binary is using.
49              Example: /lib/x86_64-linux-gnu/libpthread.so.0
50
51       --dump-graph DUMP_GRAPH
52              If set, this will dump the mutex graph to the specified file.
53
54       --verbose
55              Print statistics about the mutex wait graph.
56
57       --lock-symbols LOCK_SYMBOLS
58              Comma-separated list  of  lock  symbols  to  trace.  Default  is
59              pthread_mutex_lock.   These symbols cannot be inlined in the bi‐
60              nary.
61
62       --unlock-symbols UNLOCK_SYMBOLS
63              Comma-separated list of unlock  symbols  to  trace.  Default  is
64              pthread_mutex_unlock. These symbols cannot be inlined in the bi‐
65              nary.
66
67       -t THREADS, --threads THREADS
68              Specifies the maximum number of threads to trace. default 65536.
69              Note. 40 bytes per thread.
70
71       -e EDGES, --edges EDGES
72              Specifies the maximum number of edge cases that can be recorded.
73              default 65536. Note. 88 bytes per edge case.
74
75       pid    Pid to trace
76

EXAMPLES

78       Find potential deadlocks in PID  181.  The  --binary  argument  is  not
79       needed for statically-linked binaries.
80              # deadlock 181
81
82       Find  potential deadlocks in PID 181. If the process was created from a
83       dynamically-linked executable, the --binary argument  is  required  and
84       must be the path of the pthread library:
85              # deadlock 181 --binary /lib/x86_64-linux-gnu/libpthread.so.0
86
87       Find  potential deadlocks in PID 181. If the process was created from a
88       statically-linked executable, optionally pass the location of  the  bi‐
89       nary. On older kernels without https://lkml.org/lkml/2017/1/13/585, bi‐
90       naries that contain `:` in the path cannot be attached with uprobes. As
91       a  workaround,  we  can create a symlink to the binary, and provide the
92       symlink name instead with the `--binary` option:
93              # deadlock 181 --binary /usr/local/bin/lockinversion
94
95       Find potential deadlocks in PID 181 and dump the mutex wait graph to  a
96       file:
97              # deadlock 181 --dump-graph graph.json
98
99       Find  potential deadlocks in PID 181 and print mutex wait graph statis‐
100       tics:
101              # deadlock 181 --verbose
102
103       Find potential deadlocks in PID 181 with custom mutexes:
104              #  deadlock  181  --lock-symbols   custom_mutex1_lock,custom_mu‐
105              tex2_lock    --unlock_symbols    custom_mutex1_unlock,custom_mu‐
106              tex2_unlock
107

OUTPUT

109       This program does not output any fields. Rather, it will  keep  running
110       until  it  finds  a potential deadlock, or the user hits Ctrl-C. If the
111       program finds a potential deadlock, it will output the stack traces and
112       lock order inversion in the following format and exit:
113
114       Potential Deadlock Detected!
115
116       Cycle in lock order graph: Mutex M0 => Mutex M1 => Mutex M0
117
118       Mutex M1 acquired here while holding Mutex M0 in Thread T:
119              [stack trace]
120
121       Mutex M0 previously acquired by the same Thread T here:
122              [stack trace]
123
124       Mutex M0 acquired here while holding Mutex M1 in Thread S:
125              [stack trace]
126
127       Mutex M1 previously acquired by the same Thread S here:
128              [stack trace]
129
130       Thread T created by Thread R here:
131              [stack trace]
132
133       Thread S created by Thread Q here:
134              [stack trace]
135

OVERHEAD

137       This  traces  all  mutex lock and unlock events and all thread creation
138       events on the traced process. The overhead of this can be high  if  the
139       process  has  many  threads  and mutexes. You should only run this on a
140       process where the slowdown is acceptable.
141

SOURCE

143       This is from bcc.
144
145              https://github.com/iovisor/bcc
146
147       Also look in the bcc distribution for a  companion  _examples.txt  file
148       containing example usage, output, and commentary for this tool.
149

OS

151       Linux
152

STABILITY

154       Unstable - in development.
155

AUTHOR

157       Kenny Yu
158
159
160
161USER COMMANDS                     2017-02-01                       deadlock(8)
Impressum