bcc-nfsslower(8)

1nfsslower(8)                System Manager's Manual               nfsslower(8)
2
3
4

NAME

6       nfsslower - Trace slow NFS file operations, with per-event details.
7

SYNOPSIS

9       nfsslower [-h] [-j] [-p PID] [min_ms]
10

DESCRIPTION

12       This  tool  traces common NFSv3 & NFSv4 file operations: reads, writes,
13       opens, and getattrs. It measures the time spent  in  these  operations,
14       and prints details for each that exceeded a threshold.
15
16       WARNING: See the OVERHEAD section.
17
18       By default, a minimum millisecond threshold of 10 is used. If a thresh‐
19       old of 0 is used, all events are printed (warning: verbose).
20
21       Since this works by tracing  the  nfs_file_operations  interface  func‐
22       tions, it will need updating to match any changes to these functions.
23
24       This  tool  uses  kprobes  to  instrument the kernel for entry and exit
25       information, in the future a preferred way would be to use tracepoints.
26       Currently  there  aren't  any  tracepoints available for nfs_read_file,
27       nfs_write_file and nfs_open_file, nfs_getattr does have entry and  exit
28       tracepoints but we chose to use kprobes for consistency
29
30       Since this uses BPF, only the root user can use this tool.
31

REQUIREMENTS

33       CONFIG_BPF and bcc.
34

OPTIONS

36       -p PID Trace this PID only.
37
38       -j     Trace output in CSV format.
39
40       min_ms Minimum  I/O  latency  (duration)  to  trace,  in  milliseconds.
41              Default is 10 ms.
42

EXAMPLES

44       Trace synchronous file reads and writes slower than 10 ms:
45              # nfsslower
46
47       Trace slower than 1 ms:
48              # nfsslower 1
49
50       Trace slower than 1 ms, and output just the fields in  parsable  format
51       (CSV):
52              # nfsslower -j 1
53
54       Trace all file reads and writes (warning: the output will be verbose):
55              # nfsslower 0
56
57       Trace slower than 1 ms, for PID 181 only:
58              # nfsslower -p 181 1
59

FIELDS

61       TIME(s)
62              Time of I/O completion since the first I/O seen, in seconds.
63
64       COMM   Process name.
65
66       PID    Process ID.
67
68       T      Type  of  operation.  R  ==  read,  W  == write, O == open, G ==
69              getattr.
70
71       OFF_KB File offset for the I/O, in Kbytes.
72
73       BYTES  Size of I/O, in bytes.
74
75       LAT(ms)
76              Latency (duration) of I/O, measured from when it was  issued  by
77              VFS to the filesystem, to when it completed. This time is inclu‐
78              sive of RPC latency, network latency, cache lookup, remote file‐
79              server  processing latency, etc.  Its a more accurate measure of
80              the latency suffered by applications performing  NFS  read/write
81              calls to a fileserver.
82
83       FILENAME
84              A cached kernel file name (comes from dentry->d_iname).
85
86       ENDTIME_us
87              Completion timestamp, microseconds (-j only).
88
89       OFFSET_b
90              File offset, bytes (-j only).
91
92       LATENCY_us
93              Latency (duration) of the I/O, in microseconds (-j only).
94

OVERHEAD

96       This  adds  low-overhead  instrumentation  to NFS operations, including
97       reads and writes from the file system cache. Such read, writes and par‐
98       ticularly getattrs can be very frequent (depending on the workload; eg,
99       1M/sec), at which point the overhead of this tool (even if it prints no
100       "slower"  events) can begin to become significant. Measure and quantify
101       before use. If this continues to be a problem, consider switching to  a
102       tool  that  prints  in-kernel summaries only. This tool has been tested
103       with NFSv3 & NVSv4, but it might work with NFSv{1,2}, since it is trac‐
104       ing the generic functions from nfs_file_operations.
105
106       Note  that the overhead of this tool should be less than fileslower(8),
107       as this tool targets NFS functions only, and not  all  file  read/write
108       paths.
109

SOURCE

111       This is from bcc.
112
113              https://github.com/iovisor/bcc
114
115       Also  look  in  the  bcc  distribution  for a companion nfsslower_exam‐
116       ples.txt file containing example usage, output, and commentary for this
117       tool.
118

OS

120       Linux
121

STABILITY

123       Unstable - in development.
124

AUTHOR

126       Samuel Nair
127