bcc-fileslower(8)

1fileslower(8)               System Manager's Manual              fileslower(8)
2
3
4

NAME

6       fileslower - Trace slow synchronous file reads and writes.
7

SYNOPSIS

9       fileslower [-h] [-p PID] [-a] [min_ms]
10

DESCRIPTION

12       This script uses kernel dynamic tracing of synchronous reads and writes
13       at the VFS interface, to identify slow file reads and  writes  for  any
14       file system.
15
16       This  version  traces  __vfs_read()  and __vfs_write() and only showing
17       synchronous I/O (the path to new_sync_read() and new_sync_write()), and
18       I/O with filenames. This approach provides a view of just two file sys‐
19       tem request types: file reads and writes. There are typically many oth‐
20       ers:  asynchronous  I/O,  directory operations, file handle operations,
21       file open()s, fflush(), etc.
22
23       WARNING: See the OVERHEAD section.
24
25       By default, a minimum millisecond threshold of 10 is used.
26
27       Since this works by tracing various kernel  __vfs_*()  functions  using
28       dynamic  tracing,  it  will need updating to match any changes to these
29       functions. A future version  should  switch  to  using  FS  tracepoints
30       instead.
31
32       Since this uses BPF, only the root user can use this tool.
33

REQUIREMENTS

35       CONFIG_BPF and bcc.
36

OPTIONS

38       -p PID Trace this PID only.
39
40       -a     Include non-regular file types in output (sockets, FIFOs, etc).
41
42       min_ms Minimum  I/O  latency  (duration)  to  trace,  in  milliseconds.
43              Default is 10 ms.
44

EXAMPLES

46       Trace synchronous file reads and writes slower than 10 ms:
47              # fileslower
48
49       Trace slower than 1 ms:
50              # fileslower 1
51
52       Trace slower than 1 ms, for PID 181 only:
53              # fileslower -p 181 1
54

FIELDS

56       TIME(s)
57              Time of I/O completion since the first I/O seen, in seconds.
58
59       COMM   Process name.
60
61       PID    Process ID.
62
63       D      Direction of I/O. R == read, W == write.
64
65       BYTES  Size of I/O, in bytes.
66
67       LAT(ms)
68              Latency (duration) of I/O, measured from  when  the  application
69              issued it to VFS to when it completed. This time is inclusive of
70              block device I/O, file system CPU cycles, file system locks, run
71              queue  latency, etc. It's a more accurate measure of the latency
72              suffered by applications performing file  system  I/O,  than  to
73              measure this down at the block device interface.
74
75       FILENAME
76              A cached kernel file name (comes from dentry->d_name.name).
77

OVERHEAD

79       Depending  on  the  frequency of application reads and writes, overhead
80       can become severe, in the worst case slowing applications by 2x. In the
81       best  case,  the overhead is negligible. Hopefully for real world work‐
82       loads the overhead is often at the lower end of the  spectrum  --  test
83       before  use. The reason for high overhead is that this traces VFS reads
84       and writes, which includes FS cache reads and writes,  and  can  exceed
85       one  million  events  per second if the application is I/O heavy. While
86       the instrumentation is extremely lightweight, and uses  in-kernel  eBPF
87       maps for efficient timing and filtering, multiply that cost by one mil‐
88       lion events per second and that cost becomes a million times worse. You
89       can  get an idea of the possible cost by just counting the instrumented
90       events using the bcc funccount tool, eg:
91
92       # ./funccount.py -i 1 -r '^__vfs_(read|write)$'
93
94       This also costs overhead, but is somewhat less than fileslower.
95
96       If the overhead is prohibitive for your workload, I'd recommend  moving
97       down-stack a little from VFS into the file system functions (ext4, xfs,
98       etc).  Look for updates to bcc for specific file system tools  that  do
99       this.  The advantage of a per-file system approach is that we can trace
100       post-cache, greatly reducing events and overhead. The  disadvantage  is
101       needing  custom  tracing  approaches  for  each  different  file system
102       (whereas VFS is generic).
103

SOURCE

105       This is from bcc.
106
107              https://github.com/iovisor/bcc
108
109       Also look in the bcc distribution for a  companion  _examples.txt  file
110       containing example usage, output, and commentary for this tool.
111

OS

113       Linux
114

STABILITY

116       Unstable - in development.
117

AUTHOR

119       Brendan Gregg
120