1deadlock_detector(8)        System Manager's Manual       deadlock_detector(8)
2
3
4

NAME

6       deadlock_detector - Find potential deadlocks (lock order inversions) in
7       a running program.
8

SYNOPSIS

10       deadlock_detector  [-h]  [--binary  BINARY]  [--dump-graph  DUMP_GRAPH]
11       [--verbose] [--lock-symbols LOCK_SYMBOLS] [--unlock-symbols UNLOCK_SYM‐
12       BOLS] pid
13

DESCRIPTION

15       deadlock_detector finds potential deadlocks in a running  process.  The
16       program     attaches     uprobes     on     `pthread_mutex_lock`    and
17       `pthread_mutex_unlock` by default to build a mutex wait directed graph,
18       and then looks for a cycle in this graph.  This graph has the following
19       properties:
20
21       - Nodes in the graph represent mutexes.
22
23       - Edge (A, B) exists if there exists some thread T  where  lock(A)  was
24       called and lock(B) was called before unlock(A) was called.
25
26       If  there is a cycle in this graph, this indicates that there is a lock
27       order inversion (potential deadlock). If the program finds a lock order
28       inversion,  the  program will dump the cycle of mutexes, dump the stack
29       traces where each mutex was acquired, and then exit.
30
31       This program can only find potential deadlocks  that  occur  while  the
32       program  is tracing the process. It cannot find deadlocks that may have
33       occurred before the program was attached to the process.
34
35       This tool does not work for shared mutexes or recursive mutexes.
36
37       Since this uses BPF, only the root user can use this tool.
38

REQUIREMENTS

40       CONFIG_BPF and bcc
41

OPTIONS

43       -h, --help
44              show this help message and exit
45
46       --binary BINARY
47              If set, trace the mutexes from the  binary  at  this  path.  For
48              statically-linked  binaries, this argument is not required.  For
49              dynamically-linked  binaries,  this  argument  is  required  and
50              should  be  the path of the pthread library the binary is using.
51              Example: /lib/x86_64-linux-gnu/libpthread.so.0
52
53       --dump-graph DUMP_GRAPH
54              If set, this will dump the mutex graph to the specified file.
55
56       --verbose
57              Print statistics about the mutex wait graph.
58
59       --lock-symbols LOCK_SYMBOLS
60              Comma-separated list  of  lock  symbols  to  trace.  Default  is
61              pthread_mutex_lock.   These  symbols  cannot  be  inlined in the
62              binary.
63
64       --unlock-symbols UNLOCK_SYMBOLS
65              Comma-separated list of unlock  symbols  to  trace.  Default  is
66              pthread_mutex_unlock.  These  symbols  cannot  be inlined in the
67              binary.
68
69       pid    Pid to trace
70

EXAMPLES

72       Find potential deadlocks in PID  181.  The  --binary  argument  is  not
73       needed for statically-linked binaries.
74              # deadlock_detector 181
75
76       Find  potential deadlocks in PID 181. If the process was created from a
77       dynamically-linked executable, the --binary argument  is  required  and
78       must be the path of the pthread library:
79              #     deadlock_detector    181    --binary    /lib/x86_64-linux-
80              gnu/libpthread.so.0
81
82       Find potential deadlocks in PID 181. If the process was created from  a
83       statically-linked  executable,  optionally  pass  the  location  of the
84       binary. On older kernels  without  https://lkml.org/lkml/2017/1/13/585,
85       binaries  that contain `:` in the path cannot be attached with uprobes.
86       As a workaround, we can create a symlink to the binary, and provide the
87       symlink name instead with the `--binary` option:
88              # deadlock_detector 181 --binary /usr/local/bin/lockinversion
89
90       Find  potential deadlocks in PID 181 and dump the mutex wait graph to a
91       file:
92              # deadlock_detector 181 --dump-graph graph.json
93
94       Find potential deadlocks in PID 181 and print mutex wait graph  statis‐
95       tics:
96              # deadlock_detector 181 --verbose
97
98       Find potential deadlocks in PID 181 with custom mutexes:
99              #  deadlock_detector  181 --lock-symbols custom_mutex1_lock,cus‐
100              tom_mutex2_lock    --unlock_symbols    custom_mutex1_unlock,cus‐
101              tom_mutex2_unlock
102

OUTPUT

104       This  program  does not output any fields. Rather, it will keep running
105       until it finds a potential deadlock, or the user hits  Ctrl-C.  If  the
106       program finds a potential deadlock, it will output the stack traces and
107       lock order inversion in the following format and exit:
108
109       Potential Deadlock Detected!
110
111       Cycle in lock order graph: Mutex M0 => Mutex M1 => Mutex M0
112
113       Mutex M1 acquired here while holding Mutex M0 in Thread T:
114              [stack trace]
115
116       Mutex M0 previously acquired by the same Thread T here:
117              [stack trace]
118
119       Mutex M0 acquired here while holding Mutex M1 in Thread S:
120              [stack trace]
121
122       Mutex M1 previously acquired by the same Thread S here:
123              [stack trace]
124
125       Thread T created by Thread R here:
126              [stack trace]
127
128       Thread S created by Thread Q here:
129              [stack trace]
130

OVERHEAD

132       This traces all mutex lock and unlock events and  all  thread  creation
133       events  on  the traced process. The overhead of this can be high if the
134       process has many threads and mutexes. You should only  run  this  on  a
135       process where the slowdown is acceptable.
136

SOURCE

138       This is from bcc.
139
140              https://github.com/iovisor/bcc
141
142       Also  look  in  the bcc distribution for a companion _examples.txt file
143       containing example usage, output, and commentary for this tool.
144

OS

146       Linux
147

STABILITY

149       Unstable - in development.
150

AUTHOR

152       Kenny Yu
153
154
155
156USER COMMANDS                     2017-02-01              deadlock_detector(8)
Impressum