1NFSTEST_RDMA(1) nfstest_rdma 1.1 NFSTEST_RDMA(1)
2
6 nfstest_rdma - NFS-over-RDMA functional tests
7
9 nfstest_rdma --server <server> [options]
10
12 Verify correct functionality of NFS-over-RDMA
13 Remote Direct Memory Access (RDMA) provides fast data transfers between
15 servers and storage. NFS over RDMA is best used when a large amount of
16 data needs to be transferred with higher performance than regular NFS.
17 NFS over RDMA is usually used over InfiniBand which provides higher
18 performance and a lower latency.
19 Although NFS over RDMA is mostly used over InfiniBand, Ethernet could
21 be used as the link protocol as well. RDMA over Converged Ethernet
22 (RoCE) which allows RDMA over Ethernet by encapsulating the InfiniBand
23 transport packet over Ethernet. RoCE provides a couple of variants: Ro‐
24 CEv1 and RoCEv2. One is RoCEv1 which is an Ethernet link layer proto‐
25 col and provides RDMA functionality between two hosts in the same Eth‐
26 ernet broadcast domain. While the second is RoCEv2 or RRoCE (Remote
27 RoCE) which is an internet layer protocol so these packets can be
28 routed. RoCEv2 runs over UDP/IPv4 or UDP/IPv6. There is also another
29 variant called iWARP which runs over the TCP protocol. Testing is cur‐
30 rently supported for all of these variants except for iWARP.
31 NFS over RDMA has a couple of extra layers in the packet: InfiniBand
33 layer and RPC-over-RDMA or RPCoRDMA layer. The InfiniBand layer con‐
34 tains the OpCode which specifies the type of RDMA operation to perform
35 and the PSN which is the packet sequence number. The RPCoRDMA layer
36 contains the XID and the RDMA chunk lists. The RDMA read chunk list is
37 used to transfer DDP (Direct Data Placement) data from the NFS client
38 to the server, e.g., NFS write call. On the other hand, the RDMA write
39 chunk list is used to transfer DDP data from the NFS server back to the
40 client, e.g., NFS read reply. Only certain NFS operations can be trans‐
41 ferred using DDP and only the opaque part of the operation is trans‐
42 ferred using either RDMA reads or writes while the rest of the NFS
43 packet is transferred via the receive buffer using the RDMA SEND opera‐
44 tion. Finally, the RDMA reply chunk is used to transfer replies having
45 a variable length reply which could be larger than the receive buffer
46 and could not be transferred using the write chunk list because it does
47 not contain a single large opaque item.
48 Tests are divided into three groups: basic, read and write. The basic
50 tests deal mostly with verifying NFS packets using the reply chunk and
51 some other basic RDMA functionality. The read tests deal with verifying
52 NFS read which in turn verify the RDMA write functionality. Finally,
53 the write tests deal with verifying NFS write which in turn verify the
54 RDMA read functionality. Also, if the NFS read or write is small
55 enough the client could not use the RDMA write or read functionality,
56 but instead could use the receive buffer and transfer the data using
57 the RDMA SEND operations.
58 Tests verify the RPCoRDMA layer is sent when necessary and that the
60 RDMA chunk lists are sent with the correct information which includes
61 the number of chunks, number of segments in each chunk and the correct
62 information for each segment. Tests verify each segment information is
63 correct and their corresponding RDMA read or write information which
64 includes correct handle, virtual offset, DMA length and the XDR posi‐
65 tion for the case of RDMA reads. In addition, the correct number of
66 RDMA I/O fragments is also verified and their corresponding lengths and
67 packet sequence numbers.
68
72 --version
73 show program's version number and exit
74 -h, --help
76 show this help message and exit
77 -f FILE, --file=FILE
79 File where options are specified besides the system wide file
80 /etc/nfstest, user wide file $HOME/.nfstest or in the current
81 directory .nfstest file
82 NFS specific options:
85 -s SERVER, --server=SERVER
86 Server name or IP address
87 -e EXPORT, --export=EXPORT
89 Exported file system to mount [default: '/']
90 --nfsversion=NFSVERSION
92 NFS version, e.g., 3, 4, 4.1, etc. [default: 4.1]
93 -m MTPOINT, --mtpoint=MTPOINT
95 Mount point [default: '/mnt/t']
96 -p PORT, --port=PORT
98 NFS server port [default: 20049]
99 --proto=PROTO
101 NFS protocol name [default: 'rdma']
102 --sec=SEC
104 Security flavor [default: 'sys']
105 --nconnect=NCONNECT
107 Multiple TCP connections option [default: '1']
108 -o MTOPTS, --mtopts=MTOPTS
110 Mount options [default: 'hard']
111 --datadir=DATADIR
113 Data directory where files are created, directory is created on
114 the mount point [default: '']
115 Logging options:
118 -v VERBOSE, --verbose=VERBOSE
119 Verbose level for debug messages [default:
120 'opts|info|dbg1|dbg2|dbg3']
121 --tverbose=TVERBOSE
123 Verbose level for test messages [default: '1']
124 --createlog
126 Create log file
127 --rexeclog
129 Create rexec log files
130 --warnings
132 Display warnings
133 --tag=TAG
135 Informational tag, it is displayed as an INFO message [default:
136 '']
137 --notty
139 Do not use terminal colors on output
140 --isatty
142 Use terminal colors on output -- useful when running with nohup
143 Packet trace options:
146 --createtraces
147 Create a packet trace for each test
148 --tbsize=TBSIZE
150 Capture buffer size for tcpdump [default: 192k]
151 --trcdelay=TRCDELAY
153 Seconds to delay before stopping packet trace [default: 2.0]
154 --keeptraces
156 Do not remove any trace files [default: remove trace files if no
157 errors]
158 --rmtraces
160 Remove trace files [default: remove trace files if no errors]
161 -i INTERFACE, --interface=INTERFACE
163 Device interface [default: automatically selected]
164 File options:
167 --nfiles=NFILES
168 Number of files to create [default: 0]
169 --filesize=FILESIZE
171 File size to use for test files [default: 64k]
172 --rsize=RSIZE
174 Read size to use when reading files [default: 4k]
175 --wsize=WSIZE
177 Write size to use when writing files [default: 4k]
178 --iodelay=IODELAY
180 Seconds to delay I/O operations [default: 0.1]
181 --offset-delta=OFFSET_DELTA
183 Read/Write offset delta [default: 4k]
184 Path options:
187 --sudo=SUDO
188 Full path of binary for sudo [default: '/usr/bin/sudo']
189 --kill=KILL
191 Full path of binary for kill [default: '/usr/bin/kill']
192 --nfsstat=NFSSTAT
194 Full path of binary for nfsstat [default: '/usr/sbin/nfsstat']
195 --tcpdump=TCPDUMP
197 Full path of binary for tcpdump [default: '/usr/sbin/tcpdump']
198 --iptables=IPTABLES
200 Full path of binary for iptables [default: '/usr/sbin/iptables']
201 --messages=MESSAGES
203 Full path of log messages file [default: '/var/log/messages']
204 --trcevents=TRCEVENTS
206 Full path of tracing events directory [default: '/sys/kernel/de‐
207 bug/tracing/events']
208 --trcpipe=TRCPIPE
210 Full path of trace pipe file [default: '/sys/kernel/debug/trac‐
211 ing/trace_pipe']
212 --tmpdir=TMPDIR
214 Temporary directory [default: '/tmp']
215 Debug options:
218 --nocleanup
219 Do not cleanup created files
220 --notimestamps
222 Do not display timestamps in debug messages
223 --bugmsgs=BUGMSGS
225 File containing test messages to mark as bugs if they failed
226 --nomount
228 Do not mount server and run the tests on local disk space
229 --basename=BASENAME
231 Base name for all files and logs [default: automatically gener‐
232 ated]
233 --nfsdebug=NFSDEBUG
235 Set NFS kernel debug flags and save log messages [default: '']
236 --rpcdebug=RPCDEBUG
238 Set RPC kernel debug flags and save log messages [default: '']
239 --tracepoints=TRACEPOINTS
241 List of trace points modules to enable [default: '']
242 --nfsstats
244 Get NFS stats [default: 'False']
245 --pktdisp
247 Display main packets related to the given test
248 --nfserrors
250 Fail every NFS error found in the packet trace
251 --client-ipaddr=CLIENT_IPADDR
253 IP address of localhost
254 Reporting options:
257 --xunit-report
258 Generate xUnit compatible test report
259 --xunit-report-file=XUNIT_REPORT_FILE
261 Path to xout report file
262 Test options:
265 --runtest=RUNTEST
266 Comma separated list of tests to run, if list starts with a '^'
267 then all tests are run except the ones listed [default: 'all']
268 --small-filesize=SMALL_FILESIZE
270 File size to use for small files [default: 4k]
271 --large-filesize=LARGE_FILESIZE
273 File size to use for large files [default: 1m]
274 --strict
276 Mark warnings for missing fragments as failures [default: False]
277
279 basic
280 Run all NFS-over-RDMA basic functionality tests: basic01, basic02,
281 basic03, basic04, basic05
282 basic01
284 Verify basic NFS-over-RDMA functionality
285 basic02
287 Verify NFS-over-RDMA reply chunk on EXCHANGE_ID/SETCLIENTID
288 basic03
290 Verify NFS-over-RDMA reply chunk on READDIR
291 basic04
293 Verify NFS-over-RDMA reply chunk on READLINK
294 basic05
296 Verify NFS-over-RDMA reply chunk on GETATTR(FATTR4_ACL)
297 read
299 Run all NFS-over-RDMA functionality tests where file is opened for
300 reading: read01, read02, read03, read04
301 read01
303 Verify NFS-over-RDMA functionality on a file opened for reading (very small file)
304 read02
306 Verify NFS-over-RDMA functionality on a file opened for reading (small file)
307 read03
309 Verify NFS-over-RDMA functionality on a file opened for reading (medium file)
310 read04
312 Verify NFS-over-RDMA functionality on a file opened for reading (large file)
313 write
315 Run all NFS-over-RDMA functionality tests where file is opened for
316 writing: write01, write02, write03, write04
317 write01
319 Verify NFS-over-RDMA functionality on a file opened for writing (very small file)
320 write02
322 Verify NFS-over-RDMA functionality on a file opened for writing (small file)
323 write03
325 Verify NFS-over-RDMA functionality on a file opened for writing (medium file)
326 write04
328 Verify NFS-over-RDMA functionality on a file opened for writing (large file)
329 all
331 Run all tests: basic01, basic02, basic03, basic04, basic05, read01,
332 read02, read03, read04, write01, write02, write03, write04
333
335 The only required option is --server
336 $ nfstest_rdma --server 192.168.0.11
337
339 The user id in the local host must have access to run commands as root
340 using the 'sudo' command without the need for a password.
341
343 formatstr(3), nfstest.test_util(3), nfstest_alloc(1), nfstest_cache(1),
344 nfstest_delegation(1), nfstest_dio(1), nfstest_fcmp(1), nf‐
345 stest_file(1), nfstest_interop(1), nfstest_io(1), nfstest_lock(1), nf‐
346 stest_pkt(1), nfstest_pnfs(1), nfstest_posix(1), nfstest_sparse(1), nf‐
347 stest_ssc(1), nfstest_xattr(1), nfstest_xid(1), packet.applica‐
348 tion.rpc_const(3), packet.application.rpcordma_const(3),
349 packet.nfs.nfs3_const(3), packet.nfs.nfs4_const(3), packet.trans‐
350 port.ib(3), packet.transport.rdmap(3)
351
354 No known bugs.
355
357 Jorge Mora (mora@netapp.com)
358NFStest 3.2 21 March 2023 NFSTEST_RDMA(1)