1fio(1) General Commands Manual fio(1)
2
6 fio - flexible I/O tester
7
9 fio [options] [jobfile]...
10
12 fio is a tool that will spawn a number of threads or processes doing a
13 particular type of I/O action as specified by the user. The typical
14 use of fio is to write a job file matching the I/O load one wants to
15 simulate.
16
18 --debug=type
19 Enable verbose tracing type of various fio actions. May be `all'
20 for all types or individual types separated by a comma (e.g.
21 `--debug=file,mem' will enable file and memory debugging).
22 `help' will list all available tracing options.
23 --parse-only
25 Parse options only, don't start any I/O.
26 --output=filename
28 Write output to filename.
29 --output-format=format
31 Set the reporting format to `normal', `terse', `json', or
32 `json+'. Multiple formats can be selected, separate by a comma.
33 `terse' is a CSV based format. `json+' is like `json', except it
34 adds a full dump of the latency buckets.
35 --bandwidth-log
37 Generate aggregate bandwidth logs.
38 --minimal
40 Print statistics in a terse, semicolon-delimited format.
41 --append-terse
43 Print statistics in selected mode AND terse, semicolon-delimited
44 format. Deprecated, use --output-format instead to select mul‐
45 tiple formats.
46 --terse-version=version
48 Set terse version output format (default `3', or `2', `4', `5').
49 --version
51 Print version information and exit.
52 --help Print a summary of the command line options and exit.
54 --cpuclock-test
56 Perform test and validation of internal CPU clock.
57 --crctest=[test]
59 Test the speed of the built-in checksumming functions. If no
60 argument is given, all of them are tested. Alternatively, a
61 comma separated list can be passed, in which case the given ones
62 are tested.
63 --cmdhelp=command
65 Print help information for command. May be `all' for all com‐
66 mands.
67 --enghelp=[ioengine[,command]]
69 List all commands defined by ioengine, or print help for command
70 defined by ioengine. If no ioengine is given, list all available
71 ioengines.
72 --showcmd=jobfile
74 Convert jobfile to a set of command-line options.
75 --readonly
77 Turn on safety read-only checks, preventing writes. The --read‐
78 only option is an extra safety guard to prevent users from acci‐
79 dentally starting a write workload when that is not desired. Fio
80 will only write if `rw=write/randwrite/rw/randrw' is given. This
81 extra safety net can be used as an extra precaution as --read‐
82 only will also enable a write check in the I/O engine core to
83 prevent writes due to unknown user space bug(s).
84 --eta=when
86 Specifies when real-time ETA estimate should be printed. when
87 may be `always', `never' or `auto'.
88 --eta-newline=time
90 Force a new line for every time period passed. When the unit is
91 omitted, the value is interpreted in seconds.
92 --status-interval=time
94 Force a full status dump of cumulative (from job start) values
95 at time intervals. This option does *not* provide per-period
96 measurements. So values such as bandwidth are running averages.
97 When the time unit is omitted, time is interpreted in seconds.
98 --section=name
100 Only run specified section name in job file. Multiple sections
101 can be specified. The --section option allows one to combine
102 related jobs into one file. E.g. one job file could define
103 light, moderate, and heavy sections. Tell fio to run only the
104 "heavy" section by giving `--section=heavy' command line option.
105 One can also specify the "write" operations in one section and
106 "verify" operation in another section. The --section option only
107 applies to job sections. The reserved *global* section is always
108 parsed and used.
109 --alloc-size=kb
111 Set the internal smalloc pool size to kb in KiB. The
112 --alloc-size switch allows one to use a larger pool size for
113 smalloc. If running large jobs with randommap enabled, fio can
114 run out of memory. Smalloc is an internal allocator for shared
115 structures from a fixed size memory pool and can grow to 16
116 pools. The pool size defaults to 16MiB. NOTE: While running
117 `.fio_smalloc.*' backing store files are visible in `/tmp'.
118 --warnings-fatal
120 All fio parser warnings are fatal, causing fio to exit with an
121 error.
122 --max-jobs=nr
124 Set the maximum number of threads/processes to support to nr.
125 --server=args
127 Start a backend server, with args specifying what to listen to.
128 See CLIENT/SERVER section.
129 --daemonize=pidfile
131 Background a fio server, writing the pid to the given pidfile
132 file.
133 --client=hostname
135 Instead of running the jobs locally, send and run them on the
136 given hostname or set of hostnames. See CLIENT/SERVER section.
137 --remote-config=file
139 Tell fio server to load this local file.
140 --idle-prof=option
142 Report CPU idleness. option is one of the following:
143 calibrate
145 Run unit work calibration only and exit.
146 system Show aggregate system idleness and unit work.
148 percpu As system but also show per CPU idleness.
150 --inflate-log=log
152 Inflate and output compressed log.
153 --trigger-file=file
155 Execute trigger command when file exists.
156 --trigger-timeout=time
158 Execute trigger at this time.
159 --trigger=command
161 Set this command as local trigger.
162 --trigger-remote=command
164 Set this command as remote trigger.
165 --aux-path=path
167 Use this path for fio state generated files.
168
170 Any parameters following the options will be assumed to be job files,
171 unless they match a job file parameter. Multiple job files can be
172 listed and each job file will be regarded as a separate group. Fio will
173 stonewall execution between each group.
174 Fio accepts one or more job files describing what it is supposed to do.
176 The job file format is the classic ini file, where the names enclosed
177 in [] brackets define the job name. You are free to use any ASCII name
178 you want, except *global* which has special meaning. Following the job
179 name is a sequence of zero or more parameters, one per line, that
180 define the behavior of the job. If the first character in a line is a
181 ';' or a '#', the entire line is discarded as a comment.
182 A *global* section sets defaults for the jobs described in that file. A
184 job may override a *global* section parameter, and a job file may even
185 have several *global* sections if so desired. A job is only affected by
186 a *global* section residing above it.
187 The --cmdhelp option also lists all options. If used with an command
189 argument, --cmdhelp will detail the given command.
190 See the `examples/' directory for inspiration on how to write job
192 files. Note the copyright and license requirements currently apply to
193 `examples/' files.
194
196 Some parameters take an option of a given type, such as an integer or a
197 string. Anywhere a numeric value is required, an arithmetic expression
198 may be used, provided it is surrounded by parentheses. Supported opera‐
199 tors are:
200 addition (+)
202 subtraction (-)
204 multiplication (*)
206 division (/)
208 modulus (%)
210 exponentiation (^)
212 For time values in expressions, units are microseconds by default. This
214 is different than for time values not in expressions (not enclosed in
215 parentheses).
216
218 The following parameter types are used.
219 str String. A sequence of alphanumeric characters.
221 time Integer with possible time suffix. Without a unit value is
223 interpreted as seconds unless otherwise specified. Accepts a
224 suffix of 'd' for days, 'h' for hours, 'm' for minutes, 's' for
225 seconds, 'ms' (or 'msec') for milliseconds and 'us' (or 'usec')
226 for microseconds. For example, use 10m for 10 minutes.
227 int Integer. A whole number value, which may contain an integer pre‐
229 fix and an integer suffix.
230 [*integer prefix*] **number** [*integer suffix*]
232 The optional *integer prefix* specifies the number's base. The
234 default is decimal. *0x* specifies hexadecimal.
235 The optional *integer suffix* specifies the number's units, and
237 includes an optional unit prefix and an optional unit. For quan‐
238 tities of data, the default unit is bytes. For quantities of
239 time, the default unit is seconds unless otherwise specified.
240 With `kb_base=1000', fio follows international standards for
242 unit prefixes. To specify power-of-10 decimal values defined in
243 the International System of Units (SI):
244 K means kilo (K) or 1000
246 M means mega (M) or 1000**2
247 G means giga (G) or 1000**3
248 T means tera (T) or 1000**4
249 P means peta (P) or 1000**5
250 To specify power-of-2 binary values defined in IEC 80000-13:
252 Ki means kibi (Ki) or 1024
254 Mi means mebi (Mi) or 1024**2
255 Gi means gibi (Gi) or 1024**3
256 Ti means tebi (Ti) or 1024**4
257 Pi means pebi (Pi) or 1024**5
258 With `kb_base=1024' (the default), the unit prefixes are oppo‐
260 site from those specified in the SI and IEC 80000-13 standards
261 to provide compatibility with old scripts. For example, 4k means
262 4096.
263 For quantities of data, an optional unit of 'B' may be included
265 (e.g., 'kB' is the same as 'k').
266 The *integer suffix* is not case sensitive (e.g., m/mi mean
268 mebi/mega, not milli). 'b' and 'B' both mean byte, not bit.
269 Examples with `kb_base=1000':
271 4 KiB: 4096, 4096b, 4096B, 4k, 4kb, 4kB, 4K, 4KB
273 1 MiB: 1048576, 1m, 1024k
274 1 MB: 1000000, 1mi, 1000ki
275 1 TiB: 1073741824, 1t, 1024m, 1048576k
276 1 TB: 1000000000, 1ti, 1000mi, 1000000ki
277 Examples with `kb_base=1024' (default):
279 4 KiB: 4096, 4096b, 4096B, 4k, 4kb, 4kB, 4K, 4KB
281 1 MiB: 1048576, 1m, 1024k
282 1 MB: 1000000, 1mi, 1000ki
283 1 TiB: 1073741824, 1t, 1024m, 1048576k
284 1 TB: 1000000000, 1ti, 1000mi, 1000000ki
285 To specify times (units are not case sensitive):
287 D means days
289 H means hours
290 M mean minutes
291 s or sec means seconds (default)
292 ms or msec means milliseconds
293 us or usec means microseconds
294 If the option accepts an upper and lower range, use a colon ':'
296 or minus '-' to separate such values. See irange parameter type.
297 If the lower value specified happens to be larger than the upper
298 value the two values are swapped.
299 bool Boolean. Usually parsed as an integer, however only defined for
301 true and false (1 and 0).
302 irange Integer range with suffix. Allows value range to be given, such
304 as 1024-4096. A colon may also be used as the separator, e.g.
305 1k:4k. If the option allows two sets of ranges, they can be
306 specified with a ',' or '/' delimiter: 1k-4k/8k-32k. Also see
307 int parameter type.
308 float_list
310 A list of floating point numbers, separated by a ':' character.
311
313 With the above in mind, here follows the complete list of fio job
314 parameters.
315 Units
317 kb_base=int
318 Select the interpretation of unit prefixes in input parameters.
319 1000 Inputs comply with IEC 80000-13 and the Interna‐
321 tional System of Units (SI). Use:
322 - power-of-2 values with IEC prefixes (e.g., KiB)
324 - power-of-10 values with SI prefixes (e.g., kB)
325 1024 Compatibility mode (default). To avoid breaking
327 old scripts:
328 - power-of-2 values with SI prefixes
330 - power-of-10 values with IEC prefixes
331 See bs for more details on input parameters.
333 Outputs always use correct prefixes. Most outputs include both
335 side-by-side, like:
336 bw=2383.3kB/s (2327.4KiB/s)
338 If only one value is reported, then kb_base selects the one to
340 use:
341 1000 -- SI prefixes
343 1024 -- IEC prefixes
344 unit_base=int
346 Base unit for reporting. Allowed values are:
347 0 Use auto-detection (default).
349 8 Byte based.
351 1 Bit based.
353 Job description
355 name=str
356 ASCII name of the job. This may be used to override the name
357 printed by fio for this job. Otherwise the job name is used. On
358 the command line this parameter has the special purpose of also
359 signaling the start of a new job.
360 description=str
362 Text description of the job. Doesn't do anything except dump
363 this text description when this job is run. It's not parsed.
364 loops=int
366 Run the specified number of iterations of this job. Used to
367 repeat the same workload a given number of times. Defaults to 1.
368 numjobs=int
370 Create the specified number of clones of this job. Each clone of
371 job is spawned as an independent thread or process. May be used
372 to setup a larger number of threads/processes doing the same
373 thing. Each thread is reported separately; to see statistics for
374 all clones as a whole, use group_reporting in conjunction with
375 new_group. See --max-jobs. Default: 1.
376 Time related parameters
378 runtime=time
379 Tell fio to terminate processing after the specified period of
380 time. It can be quite hard to determine for how long a specified
381 job will run, so this parameter is handy to cap the total run‐
382 time to a given time. When the unit is omitted, the value is
383 intepreted in seconds.
384 time_based
386 If set, fio will run for the duration of the runtime specified
387 even if the file(s) are completely read or written. It will sim‐
388 ply loop over the same workload as many times as the runtime
389 allows.
390 startdelay=irange(int)
392 Delay the start of job for the specified amount of time. Can be
393 a single value or a range. When given as a range, each thread
394 will choose a value randomly from within the range. Value is in
395 seconds if a unit is omitted.
396 ramp_time=time
398 If set, fio will run the specified workload for this amount of
399 time before logging any performance numbers. Useful for letting
400 performance settle before logging results, thus minimizing the
401 runtime required for stable results. Note that the ramp_time is
402 considered lead in time for a job, thus it will increase the
403 total runtime if a special timeout or runtime is specified. When
404 the unit is omitted, the value is given in seconds.
405 clocksource=str
407 Use the given clocksource as the base of timing. The supported
408 options are:
409 gettimeofday
411 gettimeofday(2)
412 clock_gettime
414 clock_gettime(2)
415 cpu Internal CPU clock source
417 cpu is the preferred clocksource if it is reliable, as it is
419 very fast (and fio is heavy on time calls). Fio will automati‐
420 cally use this clocksource if it's supported and considered
421 reliable on the system it is running on, unless another clock‐
422 source is specifically set. For x86/x86-64 CPUs, this means sup‐
423 porting TSC Invariant.
424 gtod_reduce=bool
426 Enable all of the gettimeofday(2) reducing options (dis‐
427 able_clat, disable_slat, disable_bw_measurement) plus reduce
428 precision of the timeout somewhat to really shrink the gettime‐
429 ofday(2) call count. With this option enabled, we only do about
430 0.4% of the gettimeofday(2) calls we would have done if all time
431 keeping was enabled.
432 gtod_cpu=int
434 Sometimes it's cheaper to dedicate a single thread of execution
435 to just getting the current time. Fio (and databases, for
436 instance) are very intensive on gettimeofday(2) calls. With this
437 option, you can set one CPU aside for doing nothing but logging
438 current time to a shared memory location. Then the other
439 threads/processes that run I/O workloads need only copy that
440 segment, instead of entering the kernel with a gettimeofday(2)
441 call. The CPU set aside for doing these time calls will be
442 excluded from other uses. Fio will manually clear it from the
443 CPU mask of other jobs.
444 Target file/device
446 directory=str
447 Prefix filenames with this directory. Used to place files in a
448 different location than `./'. You can specify a number of direc‐
449 tories by separating the names with a ':' character. These
450 directories will be assigned equally distributed to job clones
451 created by numjobs as long as they are using generated file‐
452 names. If specific filename(s) are set fio will use the first
453 listed directory, and thereby matching the filename semantic
454 which generates a file each clone if not specified, but let all
455 clones use the same if set.
456 See the filename option for information on how to escape ':' and
458 '´ characters within the directory path itself.
459 filename=str
461 Fio normally makes up a filename based on the job name, thread
462 number, and file number (see filename_format). If you want to
463 share files between threads in a job or several jobs with fixed
464 file paths, specify a filename for each of them to override the
465 default. If the ioengine is file based, you can specify a number
466 of files by separating the names with a ':' colon. So if you
467 wanted a job to open `/dev/sda' and `/dev/sdb' as the two work‐
468 ing files, you would use `filename=/dev/sda:/dev/sdb'. This also
469 means that whenever this option is specified, nrfiles is
470 ignored. The size of regular files specified by this option will
471 be size divided by number of files unless an explicit size is
472 specified by filesize.
473 Each colon and backslash in the wanted path must be escaped with
475 a '´ character. For instance, if the path is
476 `/dev/dsk/foo@3,0:c' then you would use `file‐
477 name=/dev/dsk/foo@3,0\:c' and if the path is `F:\\filename' then
478 you would use `filename=F\:\\filename'.
479 On Windows, disk devices are accessed as `\\\\.\\PhysicalDrive0'
481 for the first device, `\\\\.\\PhysicalDrive1' for the second
482 etc. Note: Windows and FreeBSD prevent write access to areas of
483 the disk containing in-use data (e.g. filesystems).
484 The filename `-' is a reserved name, meaning *stdin* or *std‐
486 out*. Which of the two depends on the read/write direction set.
487 filename_format=str
489 If sharing multiple files between jobs, it is usually necessary
490 to have fio generate the exact names that you want. By default,
491 fio will name a file based on the default file format specifica‐
492 tion of `jobname.jobnumber.filenumber'. With this option, that
493 can be customized. Fio will recognize and replace the following
494 keywords in this string:
495 $jobname
497 The name of the worker thread or process.
498 $jobnum
500 The incremental number of the worker thread or
501 process.
502 $filenum
504 The incremental number of the file for that worker
505 thread or process.
506 To have dependent jobs share a set of files, this option can be
508 set to have fio generate filenames that are shared between the
509 two. For instance, if `testfiles.$filenum' is specified, file
510 number 4 for any job will be named `testfiles.4'. The default of
511 `$jobname.$jobnum.$filenum' will be used if no other format
512 specifier is given.
513 unique_filename=bool
515 To avoid collisions between networked clients, fio defaults to
516 prefixing any generated filenames (with a directory specified)
517 with the source of the client connecting. To disable this behav‐
518 ior, set this option to 0.
519 opendir=str
521 Recursively open any files below directory str.
522 lockfile=str
524 Fio defaults to not locking any files before it does I/O to
525 them. If a file or file descriptor is shared, fio can serialize
526 I/O to that file to make the end result consistent. This is
527 usual for emulating real workloads that share files. The lock
528 modes are:
529 none No locking. The default.
531 exclusive
533 Only one thread or process may do I/O at a time,
534 excluding all others.
535 readwrite
537 Read-write locking on the file. Many readers may
538 access the file at the same time, but writes get
539 exclusive access.
540 nrfiles=int
542 Number of files to use for this job. Defaults to 1. The size of
543 files will be size divided by this unless explicit size is spec‐
544 ified by filesize. Files are created for each thread separately,
545 and each file will have a file number within its name by
546 default, as explained in filename section.
547 openfiles=int
549 Number of files to keep open at the same time. Defaults to the
550 same as nrfiles, can be set smaller to limit the number simulta‐
551 neous opens.
552 file_service_type=str
554 Defines how fio decides which file from a job to service next.
555 The following types are defined:
556 random Choose a file at random.
558 roundrobin
560 Round robin over opened files. This is the
561 default.
562 sequential
564 Finish one file before moving on to the next. Mul‐
565 tiple files can still be open depending on open‐
566 files.
567 zipf Use a Zipf distribution to decide what file to
569 access.
570 pareto Use a Pareto distribution to decide what file to
572 access.
573 normal Use a Gaussian (normal) distribution to decide
575 what file to access.
576 gauss Alias for normal.
578 For random, roundrobin, and sequential, a postfix can be
580 appended to tell fio how many I/Os to issue before switching to
581 a new file. For example, specifying `file_service_type=random:8'
582 would cause fio to issue 8 I/Os before selecting a new file at
583 random. For the non-uniform distributions, a floating point
584 postfix can be given to influence how the distribution is
585 skewed. See random_distribution for a description of how that
586 would work.
587 ioscheduler=str
589 Attempt to switch the device hosting the file to the specified
590 I/O scheduler before running.
591 create_serialize=bool
593 If true, serialize the file creation for the jobs. This may be
594 handy to avoid interleaving of data files, which may greatly
595 depend on the filesystem used and even the number of processors
596 in the system. Default: true.
597 create_fsync=bool
599 fsync(2) the data file after creation. This is the default.
600 create_on_open=bool
602 If true, don't pre-create files but allow the job's open() to
603 create a file when it's time to do I/O. Default: false --
604 pre-create all necessary files when the job starts.
605 create_only=bool
607 If true, fio will only run the setup phase of the job. If files
608 need to be laid out or updated on disk, only that will be done
609 -- the actual job contents are not executed. Default: false.
610 allow_file_create=bool
612 If true, fio is permitted to create files as part of its work‐
613 load. If this option is false, then fio will error out if the
614 files it needs to use don't already exist. Default: true.
615 allow_mounted_write=bool
617 If this isn't set, fio will abort jobs that are destructive
618 (e.g. that write) to what appears to be a mounted device or par‐
619 tition. This should help catch creating inadvertently destruc‐
620 tive tests, not realizing that the test will destroy data on the
621 mounted file system. Note that some platforms don't allow writ‐
622 ing against a mounted device regardless of this option. Default:
623 false.
624 pre_read=bool
626 If this is given, files will be pre-read into memory before
627 starting the given I/O operation. This will also clear the
628 invalidate flag, since it is pointless to pre-read and then drop
629 the cache. This will only work for I/O engines that are
630 seek-able, since they allow you to read the same data multiple
631 times. Thus it will not work on non-seekable I/O engines (e.g.
632 network, splice). Default: false.
633 unlink=bool
635 Unlink the job files when done. Not the default, as repeated
636 runs of that job would then waste time recreating the file set
637 again and again. Default: false.
638 unlink_each_loop=bool
640 Unlink job files after each iteration or loop. Default: false.
641 zonesize=int
643 Divide a file into zones of the specified size. See zoneskip.
644 zonerange=int
646 Give size of an I/O zone. See zoneskip.
647 zoneskip=int
649 Skip the specified number of bytes when zonesize data has been
650 read. The two zone options can be used to only do I/O on zones
651 of a file.
652 I/O type
654 direct=bool
655 If value is true, use non-buffered I/O. This is usually
656 O_DIRECT. Note that OpenBSD and ZFS on Solaris don't support
657 direct I/O. On Windows the synchronous ioengines don't support
658 direct I/O. Default: false.
659 atomic=bool
661 If value is true, attempt to use atomic direct I/O. Atomic
662 writes are guaranteed to be stable once acknowledged by the
663 operating system. Only Linux supports O_ATOMIC right now.
664 buffered=bool
666 If value is true, use buffered I/O. This is the opposite of the
667 direct option. Defaults to true.
668 readwrite=str, rw=str
670 Type of I/O pattern. Accepted values are:
671 read Sequential reads.
673 write Sequential writes.
675 trim Sequential trims (Linux block devices only).
677 randread
679 Random reads.
680 randwrite
682 Random writes.
683 randtrim
685 Random trims (Linux block devices only).
686 rw,readwrite
688 Sequential mixed reads and writes.
689 randrw Random mixed reads and writes.
691 trimwrite
693 Sequential trim+write sequences. Blocks will be
694 trimmed first, then the same blocks will be writ‐
695 ten to.
696 Fio defaults to read if the option is not specified. For the
698 mixed I/O types, the default is to split them 50/50. For certain
699 types of I/O the result may still be skewed a bit, since the
700 speed may be different.
701 It is possible to specify the number of I/Os to do before get‐
703 ting a new offset by appending `:<nr>' to the end of the string
704 given. For a random read, it would look like `rw=randread:8' for
705 passing in an offset modifier with a value of 8. If the suffix
706 is used with a sequential I/O pattern, then the `<nr>' value
707 specified will be added to the generated offset for each I/O
708 turning sequential I/O into sequential I/O with holes. For
709 instance, using `rw=write:4k' will skip 4k for every write. Also
710 see the rw_sequencer option.
711 rw_sequencer=str
713 If an offset modifier is given by appending a number to the
714 `rw=str' line, then this option controls how that number modi‐
715 fies the I/O offset being generated. Accepted values are:
716 sequential
718 Generate sequential offset.
719 identical
721 Generate the same offset.
722 sequential is only useful for random I/O, where fio would nor‐
724 mally generate a new random offset for every I/O. If you append
725 e.g. 8 to randread, you would get a new random offset for every
726 8 I/Os. The result would be a seek for only every 8 I/Os,
727 instead of for every I/O. Use `rw=randread:8' to specify that.
728 As sequential I/O is already sequential, setting sequential for
729 that would not result in any differences. identical behaves in a
730 similar fashion, except it sends the same offset 8 number of
731 times before generating a new offset.
732 unified_rw_reporting=bool
734 Fio normally reports statistics on a per data direction basis,
735 meaning that reads, writes, and trims are accounted and reported
736 separately. If this option is set fio sums the results and
737 report them as "mixed" instead.
738 randrepeat=bool
740 Seed the random number generator used for random I/O patterns in
741 a predictable way so the pattern is repeatable across runs.
742 Default: true.
743 allrandrepeat=bool
745 Seed all random number generators in a predictable way so
746 results are repeatable across runs. Default: false.
747 randseed=int
749 Seed the random number generators based on this seed value, to
750 be able to control what sequence of output is being generated.
751 If not set, the random sequence depends on the randrepeat set‐
752 ting.
753 fallocate=str
755 Whether pre-allocation is performed when laying down files.
756 Accepted values are:
757 none Do not pre-allocate space.
759 native Use a platform's native pre-allocation call but
761 fall back to none behavior if it fails/is not
762 implemented.
763 posix Pre-allocate via posix_fallocate(3).
765 keep Pre-allocate via fallocate(2) with FAL‐
767 LOC_FL_KEEP_SIZE set.
768 0 Backward-compatible alias for none.
770 1 Backward-compatible alias for posix.
772 May not be available on all supported platforms. keep is only
774 available on Linux. If using ZFS on Solaris this cannot be set
775 to posix because ZFS doesn't support pre-allocation. Default:
776 native if any pre-allocation methods are available, none if not.
777 fadvise_hint=str
779 Use posix_fadvise(2) to advise the kernel what I/O patterns are
780 likely to be issued. Accepted values are:
781 0 Backwards compatible hint for "no hint".
783 1 Backwards compatible hint for "advise with fio
785 workload type". This uses FADV_RANDOM for a random
786 workload, and FADV_SEQUENTIAL for a sequential
787 workload.
788 sequential
790 Advise using FADV_SEQUENTIAL.
791 random Advise using FADV_RANDOM.
793 write_hint=str
795 Use fcntl(2) to advise the kernel what life time to expect from
796 a write. Only supported on Linux, as of version 4.13. Accepted
797 values are:
798 none No particular life time associated with this file.
800 short Data written to this file has a short life time.
802 medium Data written to this file has a medium life time.
804 long Data written to this file has a long life time.
806 extreme
808 Data written to this file has a very long life
809 time.
810 The values are all relative to each other, and no absolute mean‐
812 ing should be associated with them.
813 offset=int
815 Start I/O at the provided offset in the file, given as either a
816 fixed size in bytes or a percentage. If a percentage is given,
817 the next blockalign-ed offset will be used. Data before the
818 given offset will not be touched. This effectively caps the file
819 size at `real_size - offset'. Can be combined with size to con‐
820 strain the start and end range of the I/O workload. A percent‐
821 age can be specified by a number between 1 and 100 followed by
822 '%', for example, `offset=20%' to specify 20%.
823 offset_increment=int
825 If this is provided, then the real offset becomes `offset + off‐
826 set_increment * thread_number', where the thread number is a
827 counter that starts at 0 and is incremented for each sub-job
828 (i.e. when numjobs option is specified). This option is useful
829 if there are several jobs which are intended to operate on a
830 file in parallel disjoint segments, with even spacing between
831 the starting points.
832 number_ios=int
834 Fio will normally perform I/Os until it has exhausted the size
835 of the region set by size, or if it exhaust the allocated time
836 (or hits an error condition). With this setting, the range/size
837 can be set independently of the number of I/Os to perform. When
838 fio reaches this number, it will exit normally and report sta‐
839 tus. Note that this does not extend the amount of I/O that will
840 be done, it will only stop fio if this condition is met before
841 other end-of-job criteria.
842 fsync=int
844 If writing to a file, issue an fsync(2) (or its equivalent) of
845 the dirty data for every number of blocks given. For example, if
846 you give 32 as a parameter, fio will sync the file after every
847 32 writes issued. If fio is using non-buffered I/O, we may not
848 sync the file. The exception is the sg I/O engine, which syn‐
849 chronizes the disk cache anyway. Defaults to 0, which means fio
850 does not periodically issue and wait for a sync to complete.
851 Also see end_fsync and fsync_on_close.
852 fdatasync=int
854 Like fsync but uses fdatasync(2) to only sync data and not meta‐
855 data blocks. In Windows, FreeBSD, and DragonFlyBSD there is no
856 fdatasync(2) so this falls back to using fsync(2). Defaults to
857 0, which means fio does not periodically issue and wait for a
858 data-only sync to complete.
859 write_barrier=int
861 Make every N-th write a barrier write.
862 sync_file_range=str:int
864 Use sync_file_range(2) for every int number of write operations.
865 Fio will track range of writes that have happened since the last
866 sync_file_range(2) call. str can currently be one or more of:
867 wait_before
869 SYNC_FILE_RANGE_WAIT_BEFORE
870 write SYNC_FILE_RANGE_WRITE
872 wait_after
874 SYNC_FILE_RANGE_WRITE_AFTER
875 So if you do `sync_file_range=wait_before,write:8', fio would
877 use `SYNC_FILE_RANGE_WAIT_BEFORE | SYNC_FILE_RANGE_WRITE' for
878 every 8 writes. Also see the sync_file_range(2) man page. This
879 option is Linux specific.
880 overwrite=bool
882 If true, writes to a file will always overwrite existing data.
883 If the file doesn't already exist, it will be created before the
884 write phase begins. If the file exists and is large enough for
885 the specified write phase, nothing will be done. Default: false.
886 end_fsync=bool
888 If true, fsync(2) file contents when a write stage has com‐
889 pleted. Default: false.
890 fsync_on_close=bool
892 If true, fio will fsync(2) a dirty file on close. This differs
893 from end_fsync in that it will happen on every file close, not
894 just at the end of the job. Default: false.
895 rwmixread=int
897 Percentage of a mixed workload that should be reads. Default:
898 50.
899 rwmixwrite=int
901 Percentage of a mixed workload that should be writes. If both
902 rwmixread and rwmixwrite is given and the values do not add up
903 to 100%, the latter of the two will be used to override the
904 first. This may interfere with a given rate setting, if fio is
905 asked to limit reads or writes to a certain rate. If that is the
906 case, then the distribution may be skewed. Default: 50.
907 random_distribution=str:float[,str:float][,str:float]
909 By default, fio will use a completely uniform random distribu‐
910 tion when asked to perform random I/O. Sometimes it is useful to
911 skew the distribution in specific ways, ensuring that some parts
912 of the data is more hot than others. fio includes the following
913 distribution models:
914 random Uniform random distribution
916 zipf Zipf distribution
918 pareto Pareto distribution
920 normal Normal (Gaussian) distribution
922 zoned Zoned random distribution
924 When using a zipf or pareto distribution, an input value is also
926 needed to define the access pattern. For zipf, this is the `Zipf
927 theta'. For pareto, it's the `Pareto power'. Fio includes a
928 test program, fio-genzipf, that can be used visualize what the
929 given input values will yield in terms of hit rates. If you
930 wanted to use zipf with a `theta' of 1.2, you would use `ran‐
931 dom_distribution=zipf:1.2' as the option. If a non-uniform model
932 is used, fio will disable use of the random map. For the normal
933 distribution, a normal (Gaussian) deviation is supplied as a
934 value between 0 and 100.
935 For a zoned distribution, fio supports specifying percentages of
937 I/O access that should fall within what range of the file or
938 device. For example, given a criteria of:
939 60% of accesses should be to the first 10%
941 30% of accesses should be to the next 20%
942 8% of accesses should be to the next 30%
943 2% of accesses should be to the next 40%
944 we can define that through zoning of the random accesses. For
946 the above example, the user would do:
947 random_distribution=zoned:60/10:30/20:8/30:2/40
949 similarly to how bssplit works for setting ranges and percent‐
951 ages of block sizes. Like bssplit, it's possible to specify sep‐
952 arate zones for reads, writes, and trims. If just one set is
953 given, it'll apply to all of them.
954 percentage_random=int[,int][,int]
956 For a random workload, set how big a percentage should be ran‐
957 dom. This defaults to 100%, in which case the workload is fully
958 random. It can be set from anywhere from 0 to 100. Setting it to
959 0 would make the workload fully sequential. Any setting in
960 between will result in a random mix of sequential and random
961 I/O, at the given percentages. Comma-separated values may be
962 specified for reads, writes, and trims as described in block‐
963 size.
964 norandommap
966 Normally fio will cover every block of the file when doing ran‐
967 dom I/O. If this option is given, fio will just get a new random
968 offset without looking at past I/O history. This means that some
969 blocks may not be read or written, and that some blocks may be
970 read/written more than once. If this option is used with verify
971 and multiple blocksizes (via bsrange), only intact blocks are
972 verified, i.e., partially-overwritten blocks are ignored.
973 softrandommap=bool
975 See norandommap. If fio runs with the random block map enabled
976 and it fails to allocate the map, if this option is set it will
977 continue without a random block map. As coverage will not be as
978 complete as with random maps, this option is disabled by
979 default.
980 random_generator=str
982 Fio supports the following engines for generating I/O offsets
983 for random I/O:
984 tausworthe
986 Strong 2^88 cycle random number generator.
987 lfsr Linear feedback shift register generator.
989 tausworthe64
991 Strong 64-bit 2^258 cycle random number generator.
992 tausworthe is a strong random number generator, but it requires
994 tracking on the side if we want to ensure that blocks are only
995 read or written once. lfsr guarantees that we never generate the
996 same offset twice, and it's also less computationally expensive.
997 It's not a true random generator, however, though for I/O pur‐
998 poses it's typically good enough. lfsr only works with single
999 block sizes, not with workloads that use multiple block sizes.
1000 If used with such a workload, fio may read or write some blocks
1001 multiple times. The default value is tausworthe, unless the
1002 required space exceeds 2^32 blocks. If it does, then taus‐
1003 worthe64 is selected automatically.
1004 Block size
1006 blocksize=int[,int][,int], bs=int[,int][,int]
1007 The block size in bytes used for I/O units. Default: 4096. A
1008 single value applies to reads, writes, and trims. Comma-sepa‐
1009 rated values may be specified for reads, writes, and trims. A
1010 value not terminated in a comma applies to subsequent types.
1011 Examples:
1012 bs=256k means 256k for reads, writes and trims.
1014 bs=8k,32k means 8k for reads, 32k for writes and
1015 trims.
1016 bs=8k,32k, means 8k for reads, 32k for writes, and
1017 default for trims.
1018 bs=,8k means default for reads, 8k for writes and
1019 trims.
1020 bs=,8k, means default for reads, 8k for writes,
1021 and default for trims.
1022 blocksize_range=irange[,irange][,irange],
1024 bsrange=irange[,irange][,irange]
1025 A range of block sizes in bytes for I/O units. The issued I/O
1026 unit will always be a multiple of the minimum size, unless
1027 blocksize_unaligned is set. Comma-separated ranges may be spec‐
1028 ified for reads, writes, and trims as described in blocksize.
1029 Example:
1030 bsrange=1k-4k,2k-8k
1032 bssplit=str[,str][,str]
1034 Sometimes you want even finer grained control of the block sizes
1035 issued, not just an even split between them. This option allows
1036 you to weight various block sizes, so that you are able to
1037 define a specific amount of block sizes issued. The format for
1038 this option is:
1039 bssplit=blocksize/percentage:blocksize/percentage
1041 for as many block sizes as needed. So if you want to define a
1043 workload that has 50% 64k blocks, 10% 4k blocks, and 40% 32k
1044 blocks, you would write:
1045 bssplit=4k/10:64k/50:32k/40
1047 Ordering does not matter. If the percentage is left blank, fio
1049 will fill in the remaining values evenly. So a bssplit option
1050 like this one:
1051 bssplit=4k/50:1k/:32k/
1053 would have 50% 4k ios, and 25% 1k and 32k ios. The percentages
1055 always add up to 100, if bssplit is given a range that adds up
1056 to more, it will error out.
1057 Comma-separated values may be specified for reads, writes, and
1059 trims as described in blocksize.
1060 If you want a workload that has 50% 2k reads and 50% 4k reads,
1062 while having 90% 4k writes and 10% 8k writes, you would specify:
1063 bssplit=2k/50:4k/50,4k/90,8k/10
1065 blocksize_unaligned, bs_unaligned
1067 If set, fio will issue I/O units with any size within block‐
1068 size_range, not just multiples of the minimum size. This typi‐
1069 cally won't work with direct I/O, as that normally requires sec‐
1070 tor alignment.
1071 bs_is_seq_rand=bool
1073 If this option is set, fio will use the normal read,write block‐
1074 size settings as sequential,random blocksize settings instead.
1075 Any random read or write will use the WRITE blocksize settings,
1076 and any sequential read or write will use the READ blocksize
1077 settings.
1078 blockalign=int[,int][,int], ba=int[,int][,int]
1080 Boundary to which fio will align random I/O units. Default:
1081 blocksize. Minimum alignment is typically 512b for using direct
1082 I/O, though it usually depends on the hardware block size. This
1083 option is mutually exclusive with using a random map for files,
1084 so it will turn off that option. Comma-separated values may be
1085 specified for reads, writes, and trims as described in block‐
1086 size.
1087 Buffers and memory
1089 zero_buffers
1090 Initialize buffers with all zeros. Default: fill buffers with
1091 random data.
1092 refill_buffers
1094 If this option is given, fio will refill the I/O buffers on
1095 every submit. The default is to only fill it at init time and
1096 reuse that data. Only makes sense if zero_buffers isn't speci‐
1097 fied, naturally. If data verification is enabled, refill_buffers
1098 is also automatically enabled.
1099 scramble_buffers=bool
1101 If refill_buffers is too costly and the target is using data
1102 deduplication, then setting this option will slightly modify the
1103 I/O buffer contents to defeat normal de-dupe attempts. This is
1104 not enough to defeat more clever block compression attempts, but
1105 it will stop naive dedupe of blocks. Default: true.
1106 buffer_compress_percentage=int
1108 If this is set, then fio will attempt to provide I/O buffer con‐
1109 tent (on WRITEs) that compresses to the specified level. Fio
1110 does this by providing a mix of random data and a fixed pattern.
1111 The fixed pattern is either zeros, or the pattern specified by
1112 buffer_pattern. If the pattern option is used, it might skew the
1113 compression ratio slightly. Note that this is per block size
1114 unit, for file/disk wide compression level that matches this
1115 setting, you'll also want to set refill_buffers.
1116 buffer_compress_chunk=int
1118 See buffer_compress_percentage. This setting allows fio to man‐
1119 age how big the ranges of random data and zeroed data is. With‐
1120 out this set, fio will provide buffer_compress_percentage of
1121 blocksize random data, followed by the remaining zeroed. With
1122 this set to some chunk size smaller than the block size, fio can
1123 alternate random and zeroed data throughout the I/O buffer.
1124 buffer_pattern=str
1126 If set, fio will fill the I/O buffers with this pattern or with
1127 the contents of a file. If not set, the contents of I/O buffers
1128 are defined by the other options related to buffer contents. The
1129 setting can be any pattern of bytes, and can be prefixed with 0x
1130 for hex values. It may also be a string, where the string must
1131 then be wrapped with "". Or it may also be a filename, where the
1132 filename must be wrapped with '' in which case the file is
1133 opened and read. Note that not all the file contents will be
1134 read if that would cause the buffers to overflow. So, for exam‐
1135 ple:
1136 buffer_pattern='filename'
1138 or:
1139 buffer_pattern="abcd"
1140 or:
1141 buffer_pattern=-12
1142 or:
1143 buffer_pattern=0xdeadface
1144 Also you can combine everything together in any order:
1146 buffer_pattern=0xdeadface"abcd"-12'filename'
1148 dedupe_percentage=int
1150 If set, fio will generate this percentage of identical buffers
1151 when writing. These buffers will be naturally dedupable. The
1152 contents of the buffers depend on what other buffer compression
1153 settings have been set. It's possible to have the individual
1154 buffers either fully compressible, or not at all. This option
1155 only controls the distribution of unique buffers.
1156 invalidate=bool
1158 Invalidate the buffer/page cache parts of the files to be used
1159 prior to starting I/O if the platform and file type support it.
1160 Defaults to true. This will be ignored if pre_read is also
1161 specified for the same job.
1162 sync=bool
1164 Use synchronous I/O for buffered writes. For the majority of I/O
1165 engines, this means using O_SYNC. Default: false.
1166 iomem=str, mem=str
1168 Fio can use various types of memory as the I/O unit buffer. The
1169 allowed values are:
1170 malloc Use memory from malloc(3) as the buffers. Default
1172 memory type.
1173 shm Use shared memory as the buffers. Allocated
1175 through shmget(2).
1176 shmhuge
1178 Same as shm, but use huge pages as backing.
1179 mmap Use mmap(2) to allocate buffers. May either be
1181 anonymous memory, or can be file backed if a file‐
1182 name is given after the option. The format is
1183 `mem=mmap:/path/to/file'.
1184 mmaphuge
1186 Use a memory mapped huge file as the buffer back‐
1187 ing. Append filename after mmaphuge, ala `mem=mma‐
1188 phuge:/hugetlbfs/file'.
1189 mmapshared
1191 Same as mmap, but use a MMAP_SHARED mapping.
1192 cudamalloc
1194 Use GPU memory as the buffers for GPUDirect RDMA
1195 benchmark. The ioengine must be rdma.
1196 The area allocated is a function of the maximum allowed bs size
1198 for the job, multiplied by the I/O depth given. Note that for
1199 shmhuge and mmaphuge to work, the system must have free huge
1200 pages allocated. This can normally be checked and set by read‐
1201 ing/writing `/proc/sys/vm/nr_hugepages' on a Linux system. Fio
1202 assumes a huge page is 4MiB in size. So to calculate the number
1203 of huge pages you need for a given job file, add up the I/O
1204 depth of all jobs (normally one unless iodepth is used) and mul‐
1205 tiply by the maximum bs set. Then divide that number by the huge
1206 page size. You can see the size of the huge pages in `/proc/mem‐
1207 info'. If no huge pages are allocated by having a non-zero num‐
1208 ber in `nr_hugepages', using mmaphuge or shmhuge will fail. Also
1209 see hugepage-size.
1210 mmaphuge also needs to have hugetlbfs mounted and the file loca‐
1212 tion should point there. So if it's mounted in `/huge', you
1213 would use `mem=mmaphuge:/huge/somefile'.
1214 iomem_align=int, mem_align=int
1216 This indicates the memory alignment of the I/O memory buffers.
1217 Note that the given alignment is applied to the first I/O unit
1218 buffer, if using iodepth the alignment of the following buffers
1219 are given by the bs used. In other words, if using a bs that is
1220 a multiple of the page sized in the system, all buffers will be
1221 aligned to this value. If using a bs that is not page aligned,
1222 the alignment of subsequent I/O memory buffers is the sum of the
1223 iomem_align and bs used.
1224 hugepage-size=int
1226 Defines the size of a huge page. Must at least be equal to the
1227 system setting, see `/proc/meminfo'. Defaults to 4MiB. Should
1228 probably always be a multiple of megabytes, so using
1229 `hugepage-size=Xm' is the preferred way to set this to avoid
1230 setting a non-pow-2 bad value.
1231 lockmem=int
1233 Pin the specified amount of memory with mlock(2). Can be used to
1234 simulate a smaller amount of memory. The amount specified is per
1235 worker.
1236 I/O size
1238 size=int
1239 The total size of file I/O for each thread of this job. Fio will
1240 run until this many bytes has been transferred, unless runtime
1241 is limited by other options (such as runtime, for instance, or
1242 increased/decreased by io_size). Fio will divide this size
1243 between the available files determined by options such as
1244 nrfiles, filename, unless filesize is specified by the job. If
1245 the result of division happens to be 0, the size is set to the
1246 physical size of the given files or devices if they exist. If
1247 this option is not specified, fio will use the full size of the
1248 given files or devices. If the files do not exist, size must be
1249 given. It is also possible to give size as a percentage between
1250 1 and 100. If `size=20%' is given, fio will use 20% of the full
1251 size of the given files or devices. Can be combined with offset
1252 to constrain the start and end range that I/O will be done
1253 within.
1254 io_size=int, io_limit=int
1256 Normally fio operates within the region set by size, which means
1257 that the size option sets both the region and size of I/O to be
1258 performed. Sometimes that is not what you want. With this
1259 option, it is possible to define just the amount of I/O that fio
1260 should do. For instance, if size is set to 20GiB and io_size is
1261 set to 5GiB, fio will perform I/O within the first 20GiB but
1262 exit when 5GiB have been done. The opposite is also possible --
1263 if size is set to 20GiB, and io_size is set to 40GiB, then fio
1264 will do 40GiB of I/O within the 0..20GiB region.
1265 filesize=irange(int)
1267 Individual file sizes. May be a range, in which case fio will
1268 select sizes for files at random within the given range and lim‐
1269 ited to size in total (if that is given). If not given, each
1270 created file is the same size. This option overrides size in
1271 terms of file size, which means this value is used as a fixed
1272 size or possible range of each file.
1273 file_append=bool
1275 Perform I/O after the end of the file. Normally fio will operate
1276 within the size of a file. If this option is set, then fio will
1277 append to the file instead. This has identical behavior to set‐
1278 ting offset to the size of a file. This option is ignored on
1279 non-regular files.
1280 fill_device=bool, fill_fs=bool
1282 Sets size to something really large and waits for ENOSPC (no
1283 space left on device) as the terminating condition. Only makes
1284 sense with sequential write. For a read workload, the mount
1285 point will be filled first then I/O started on the result. This
1286 option doesn't make sense if operating on a raw device node,
1287 since the size of that is already known by the file system.
1288 Additionally, writing beyond end-of-device will not return
1289 ENOSPC there.
1290 I/O engine
1292 ioengine=str
1293 Defines how the job issues I/O to the file. The following types
1294 are defined:
1295 sync Basic read(2) or write(2) I/O. lseek(2) is used to
1297 position the I/O location. See fsync and fdata‐
1298 sync for syncing write I/Os.
1299 psync Basic pread(2) or pwrite(2) I/O. Default on all
1301 supported operating systems except for Windows.
1302 vsync Basic readv(2) or writev(2) I/O. Will emulate
1304 queuing by coalescing adjacent I/Os into a single
1305 submission.
1306 pvsync Basic preadv(2) or pwritev(2) I/O.
1308 pvsync2
1310 Basic preadv2(2) or pwritev2(2) I/O.
1311 libaio Linux native asynchronous I/O. Note that Linux may
1313 only support queued behavior with non-buffered I/O
1314 (set `direct=1' or `buffered=0'). This engine
1315 defines engine specific options.
1316 posixaio
1318 POSIX asynchronous I/O using aio_read(3) and
1319 aio_write(3).
1320 solarisaio
1322 Solaris native asynchronous I/O.
1323 windowsaio
1325 Windows native asynchronous I/O. Default on Win‐
1326 dows.
1327 mmap File is memory mapped with mmap(2) and data copied
1329 to/from using memcpy(3).
1330 splice splice(2) is used to transfer the data and
1332 vmsplice(2) to transfer data from user space to
1333 the kernel.
1334 sg SCSI generic sg v3 I/O. May either be synchronous
1336 using the SG_IO ioctl, or if the target is an sg
1337 character device we use read(2) and write(2) for
1338 asynchronous I/O. Requires filename option to
1339 specify either block or character devices.
1340 null Doesn't transfer any data, just pretends to. This
1342 is mainly used to exercise fio itself and for
1343 debugging/testing purposes.
1344 net Transfer over the network to given `host:port'.
1346 Depending on the protocol used, the hostname,
1347 port, listen and filename options are used to
1348 specify what sort of connection to make, while the
1349 protocol option determines which protocol will be
1350 used. This engine defines engine specific options.
1351 netsplice
1353 Like net, but uses splice(2) and vmsplice(2) to
1354 map data and send/receive. This engine defines
1355 engine specific options.
1356 cpuio Doesn't transfer any data, but burns CPU cycles
1358 according to the cpuload and cpuchunks options.
1359 Setting cpuload=85 will cause that job to do noth‐
1360 ing but burn 85% of the CPU. In case of SMP
1361 machines, use `numjobs=<nr_of_cpu>' to get desired
1362 CPU usage, as the cpuload only loads a single CPU
1363 at the desired rate. A job never finishes unless
1364 there is at least one non-cpuio job.
1365 guasi The GUASI I/O engine is the Generic Userspace
1367 Asyncronous Syscall Interface approach to async
1368 I/O. See http://www.xmailserver.org/guasi-lib.html
1369 for more info on GUASI.
1370 rdma The RDMA I/O engine supports both RDMA memory
1372 semantics (RDMA_WRITE/RDMA_READ) and channel
1373 semantics (Send/Recv) for the InfiniBand, RoCE and
1374 iWARP protocols.
1375 falloc I/O engine that does regular fallocate to simulate
1377 data transfer as fio ioengine.
1378 DDIR_READ does fallocate(,mode = FAL‐
1380 LOC_FL_KEEP_SIZE,).
1381 DIR_WRITE does fallocate(,mode = 0).
1382 DDIR_TRIM does fallocate(,mode = FAL‐
1383 LOC_FL_KEEP_SIZE|FALLOC_FL_PUNCH_HOLE).
1384 ftruncate
1386 I/O engine that sends ftruncate(2) operations in
1387 response to write (DDIR_WRITE) events. Each ftrun‐
1388 cate issued sets the file's size to the current
1389 block offset. blocksize is ignored.
1390 e4defrag
1392 I/O engine that does regular EXT4_IOC_MOVE_EXT
1393 ioctls to simulate defragment activity in request
1394 to DDIR_WRITE event.
1395 rbd I/O engine supporting direct access to Ceph Rados
1397 Block Devices (RBD) via librbd without the need to
1398 use the kernel rbd driver. This ioengine defines
1399 engine specific options.
1400 gfapi Using GlusterFS libgfapi sync interface to direct
1402 access to GlusterFS volumes without having to go
1403 through FUSE. This ioengine defines engine spe‐
1404 cific options.
1405 gfapi_async
1407 Using GlusterFS libgfapi async interface to direct
1408 access to GlusterFS volumes without having to go
1409 through FUSE. This ioengine defines engine spe‐
1410 cific options.
1411 libhdfs
1413 Read and write through Hadoop (HDFS). The filename
1414 option is used to specify host,port of the hdfs
1415 name-node to connect. This engine interprets off‐
1416 sets a little differently. In HDFS, files once
1417 created cannot be modified so random writes are
1418 not possible. To imitate this the libhdfs engine
1419 expects a bunch of small files to be created over
1420 HDFS and will randomly pick a file from them based
1421 on the offset generated by fio backend (see the
1422 example job file to create such files, use
1423 `rw=write' option). Please note, it may be neces‐
1424 sary to set environment variables to work with
1425 HDFS/libhdfs properly. Each job uses its own con‐
1426 nection to HDFS.
1427 mtd Read, write and erase an MTD character device
1429 (e.g., `/dev/mtd0'). Discards are treated as
1430 erases. Depending on the underlying device type,
1431 the I/O may have to go in a certain pattern, e.g.,
1432 on NAND, writing sequentially to erase blocks and
1433 discarding before overwriting. The trimwrite mode
1434 works well for this constraint.
1435 pmemblk
1437 Read and write using filesystem DAX to a file on a
1438 filesystem mounted with DAX on a persistent memory
1439 device through the NVML libpmemblk library.
1440 dev-dax
1442 Read and write using device DAX to a persistent
1443 memory device (e.g., /dev/dax0.0) through the NVML
1444 libpmem library.
1445 external
1447 Prefix to specify loading an external I/O engine
1448 object file. Append the engine filename, e.g.
1449 `ioengine=external:/tmp/foo.o' to load ioengine
1450 `foo.o' in `/tmp'. The path can be either absolute
1451 or relative. See `engines/skeleton_external.c' in
1452 the fio source for details of writing an external
1453 I/O engine.
1454 I/O engine specific parameters
1456 In addition, there are some parameters which are only valid when a spe‐
1457 cific ioengine is in use. These are used identically to normal parame‐
1458 ters, with the caveat that when used on the command line, they must
1459 come after the ioengine that defines them is selected.
1460 (libaio)userspace_reap
1462 Normally, with the libaio engine in use, fio will use the
1463 io_getevents(3) system call to reap newly returned events. With
1464 this flag turned on, the AIO ring will be read directly from
1465 user-space to reap events. The reaping mode is only enabled when
1466 polling for a minimum of 0 events (e.g. when `iodepth_batch_com‐
1467 plete=0').
1468 (pvsync2)hipri
1470 Set RWF_HIPRI on I/O, indicating to the kernel that it's of
1471 higher priority than normal.
1472 (pvsync2)hipri_percentage
1474 When hipri is set this determines the probability of a pvsync2
1475 I/O being high priority. The default is 100%.
1476 (cpuio)cpuload=int
1478 Attempt to use the specified percentage of CPU cycles. This is a
1479 mandatory option when using cpuio I/O engine.
1480 (cpuio)cpuchunks=int
1482 Split the load into cycles of the given time. In microseconds.
1483 (cpuio)exit_on_io_done=bool
1485 Detect when I/O threads are done, then exit.
1486 (libhdfs)namenode=str
1488 The hostname or IP address of a HDFS cluster namenode to con‐
1489 tact.
1490 (libhdfs)port
1492 The listening port of the HFDS cluster namenode.
1493 (netsplice,net)port
1495 The TCP or UDP port to bind to or connect to. If this is used
1496 with numjobs to spawn multiple instances of the same job type,
1497 then this will be the starting port number since fio will use a
1498 range of ports.
1499 (netsplice,net)hostname=str
1501 The hostname or IP address to use for TCP or UDP based I/O. If
1502 the job is a TCP listener or UDP reader, the hostname is not
1503 used and must be omitted unless it is a valid UDP multicast
1504 address.
1505 (netsplice,net)interface=str
1507 The IP address of the network interface used to send or receive
1508 UDP multicast.
1509 (netsplice,net)ttl=int
1511 Time-to-live value for outgoing UDP multicast packets. Default:
1512 1.
1513 (netsplice,net)nodelay=bool
1515 Set TCP_NODELAY on TCP connections.
1516 (netsplice,net)protocol=str, proto=str
1518 The network protocol to use. Accepted values are:
1519 tcp Transmission control protocol.
1521 tcpv6 Transmission control protocol V6.
1523 udp User datagram protocol.
1525 udpv6 User datagram protocol V6.
1527 unix UNIX domain socket.
1529 When the protocol is TCP or UDP, the port must also be given, as
1531 well as the hostname if the job is a TCP listener or UDP reader.
1532 For unix sockets, the normal filename option should be used and
1533 the port is invalid.
1534 (netsplice,net)listen
1536 For TCP network connections, tell fio to listen for incoming
1537 connections rather than initiating an outgoing connection. The
1538 hostname must be omitted if this option is used.
1539 (netsplice,net)pingpong
1541 Normally a network writer will just continue writing data, and a
1542 network reader will just consume packages. If `pingpong=1' is
1543 set, a writer will send its normal payload to the reader, then
1544 wait for the reader to send the same payload back. This allows
1545 fio to measure network latencies. The submission and completion
1546 latencies then measure local time spent sending or receiving,
1547 and the completion latency measures how long it took for the
1548 other end to receive and send back. For UDP multicast traffic
1549 `pingpong=1' should only be set for a single reader when multi‐
1550 ple readers are listening to the same address.
1551 (netsplice,net)window_size=int
1553 Set the desired socket buffer size for the connection.
1554 (netsplice,net)mss=int
1556 Set the TCP maximum segment size (TCP_MAXSEG).
1557 (e4defrag)donorname=str
1559 File will be used as a block donor (swap extents between files).
1560 (e4defrag)inplace=int
1562 Configure donor file blocks allocation strategy:
1563 0 Default. Preallocate donor's file on init.
1565 1 Allocate space immediately inside defragment
1567 event, and free right after event.
1568 (rbd)clustername=str
1570 Specifies the name of the Ceph cluster.
1571 (rbd)rbdname=str
1573 Specifies the name of the RBD.
1574 (rbd)pool=str
1576 Specifies the name of the Ceph pool containing RBD.
1577 (rbd)clientname=str
1579 Specifies the username (without the 'client.' prefix) used to
1580 access the Ceph cluster. If the clustername is specified, the
1581 clientname shall be the full *type.id* string. If no type. pre‐
1582 fix is given, fio will add 'client.' by default.
1583 (mtd)skip_bad=bool
1585 Skip operations against known bad blocks.
1586 (libhdfs)hdfsdirectory
1588 libhdfs will create chunk in this HDFS directory.
1589 (libhdfs)chunk_size
1591 The size of the chunk to use for each file.
1592 I/O depth
1594 iodepth=int
1595 Number of I/O units to keep in flight against the file. Note
1596 that increasing iodepth beyond 1 will not affect synchronous
1597 ioengines (except for small degrees when verify_async is in
1598 use). Even async engines may impose OS restrictions causing the
1599 desired depth not to be achieved. This may happen on Linux when
1600 using libaio and not setting `direct=1', since buffered I/O is
1601 not async on that OS. Keep an eye on the I/O depth distribution
1602 in the fio output to verify that the achieved depth is as
1603 expected. Default: 1.
1604 iodepth_batch_submit=int, iodepth_batch=int
1606 This defines how many pieces of I/O to submit at once. It
1607 defaults to 1 which means that we submit each I/O as soon as it
1608 is available, but can be raised to submit bigger batches of I/O
1609 at the time. If it is set to 0 the iodepth value will be used.
1610 iodepth_batch_complete_min=int, iodepth_batch_complete=int
1612 This defines how many pieces of I/O to retrieve at once. It
1613 defaults to 1 which means that we'll ask for a minimum of 1 I/O
1614 in the retrieval process from the kernel. The I/O retrieval will
1615 go on until we hit the limit set by iodepth_low. If this vari‐
1616 able is set to 0, then fio will always check for completed
1617 events before queuing more I/O. This helps reduce I/O latency,
1618 at the cost of more retrieval system calls.
1619 iodepth_batch_complete_max=int
1621 This defines maximum pieces of I/O to retrieve at once. This
1622 variable should be used along with iodepth_batch_com‐
1623 plete_min=int variable, specifying the range of min and max
1624 amount of I/O which should be retrieved. By default it is equal
1625 to iodepth_batch_complete_min value. Example #1:
1626 iodepth_batch_complete_min=1
1628 iodepth_batch_complete_max=<iodepth>
1629 which means that we will retrieve at least 1 I/O and up to the
1631 whole submitted queue depth. If none of I/O has been completed
1632 yet, we will wait. Example #2:
1633 iodepth_batch_complete_min=0
1635 iodepth_batch_complete_max=<iodepth>
1636 which means that we can retrieve up to the whole submitted queue
1638 depth, but if none of I/O has been completed yet, we will NOT
1639 wait and immediately exit the system call. In this example we
1640 simply do polling.
1641 iodepth_low=int
1643 The low water mark indicating when to start filling the queue
1644 again. Defaults to the same as iodepth, meaning that fio will
1645 attempt to keep the queue full at all times. If iodepth is set
1646 to e.g. 16 and iodepth_low is set to 4, then after fio has
1647 filled the queue of 16 requests, it will let the depth drain
1648 down to 4 before starting to fill it again.
1649 serialize_overlap=bool
1651 Serialize in-flight I/Os that might otherwise cause or suffer
1652 from data races. When two or more I/Os are submitted simultane‐
1653 ously, there is no guarantee that the I/Os will be processed or
1654 completed in the submitted order. Further, if two or more of
1655 those I/Os are writes, any overlapping region between them can
1656 become indeterminate/undefined on certain storage. These issues
1657 can cause verification to fail erratically when at least one of
1658 the racing I/Os is changing data and the overlapping region has
1659 a non-zero size. Setting serialize_overlap tells fio to avoid
1660 provoking this behavior by explicitly serializing in-flight I/Os
1661 that have a non-zero overlap. Note that setting this option can
1662 reduce both performance and the iodepth achieved. Additionally
1663 this option does not work when io_submit_mode is set to offload.
1664 Default: false.
1665 io_submit_mode=str
1667 This option controls how fio submits the I/O to the I/O engine.
1668 The default is `inline', which means that the fio job threads
1669 submit and reap I/O directly. If set to `offload', the job
1670 threads will offload I/O submission to a dedicated pool of I/O
1671 threads. This requires some coordination and thus has a bit of
1672 extra overhead, especially for lower queue depth I/O where it
1673 can increase latencies. The benefit is that fio can manage sub‐
1674 mission rates independently of the device completion rates. This
1675 avoids skewed latency reporting if I/O gets backed up on the
1676 device side (the coordinated omission problem).
1677 I/O rate
1679 thinktime=time
1680 Stall the job for the specified period of time after an I/O has
1681 completed before issuing the next. May be used to simulate pro‐
1682 cessing being done by an application. When the unit is omitted,
1683 the value is interpreted in microseconds. See thinktime_blocks
1684 and thinktime_spin.
1685 thinktime_spin=time
1687 Only valid if thinktime is set - pretend to spend CPU time doing
1688 something with the data received, before falling back to sleep‐
1689 ing for the rest of the period specified by thinktime. When the
1690 unit is omitted, the value is interpreted in microseconds.
1691 thinktime_blocks=int
1693 Only valid if thinktime is set - control how many blocks to
1694 issue, before waiting thinktime usecs. If not set, defaults to 1
1695 which will make fio wait thinktime usecs after every block. This
1696 effectively makes any queue depth setting redundant, since no
1697 more than 1 I/O will be queued before we have to complete it and
1698 do our thinktime. In other words, this setting effectively caps
1699 the queue depth if the latter is larger.
1700 rate=int[,int][,int]
1702 Cap the bandwidth used by this job. The number is in bytes/sec,
1703 the normal suffix rules apply. Comma-separated values may be
1704 specified for reads, writes, and trims as described in block‐
1705 size.
1706 For example, using `rate=1m,500k' would limit reads to 1MiB/sec
1708 and writes to 500KiB/sec. Capping only reads or writes can be
1709 done with `rate=,500k' or `rate=500k,' where the former will
1710 only limit writes (to 500KiB/sec) and the latter will only limit
1711 reads.
1712 rate_min=int[,int][,int]
1714 Tell fio to do whatever it can to maintain at least this band‐
1715 width. Failing to meet this requirement will cause the job to
1716 exit. Comma-separated values may be specified for reads, writes,
1717 and trims as described in blocksize.
1718 rate_iops=int[,int][,int]
1720 Cap the bandwidth to this number of IOPS. Basically the same as
1721 rate, just specified independently of bandwidth. If the job is
1722 given a block size range instead of a fixed value, the smallest
1723 block size is used as the metric. Comma-separated values may be
1724 specified for reads, writes, and trims as described in block‐
1725 size.
1726 rate_iops_min=int[,int][,int]
1728 If fio doesn't meet this rate of I/O, it will cause the job to
1729 exit. Comma-separated values may be specified for reads,
1730 writes, and trims as described in blocksize.
1731 rate_process=str
1733 This option controls how fio manages rated I/O submissions. The
1734 default is `linear', which submits I/O in a linear fashion with
1735 fixed delays between I/Os that gets adjusted based on I/O com‐
1736 pletion rates. If this is set to `poisson', fio will submit I/O
1737 based on a more real world random request flow, known as the
1738 Poisson process (https://en.wikipedia.org/wiki/Pois‐
1739 son_point_process). The lambda will be 10^6 / IOPS for the given
1740 workload.
1741 I/O latency
1743 latency_target=time
1744 If set, fio will attempt to find the max performance point that
1745 the given workload will run at while maintaining a latency below
1746 this target. When the unit is omitted, the value is interpreted
1747 in microseconds. See latency_window and latency_percentile.
1748 latency_window=time
1750 Used with latency_target to specify the sample window that the
1751 job is run at varying queue depths to test the performance. When
1752 the unit is omitted, the value is interpreted in microseconds.
1753 latency_percentile=float
1755 The percentage of I/Os that must fall within the criteria speci‐
1756 fied by latency_target and latency_window. If not set, this
1757 defaults to 100.0, meaning that all I/Os must be equal or below
1758 to the value set by latency_target.
1759 max_latency=time
1761 If set, fio will exit the job with an ETIMEDOUT error if it
1762 exceeds this maximum latency. When the unit is omitted, the
1763 value is interpreted in microseconds.
1764 rate_cycle=int
1766 Average bandwidth for rate and rate_min over this number of mil‐
1767 liseconds. Defaults to 1000.
1768 I/O replay
1770 write_iolog=str
1771 Write the issued I/O patterns to the specified file. See
1772 read_iolog. Specify a separate file for each job, otherwise the
1773 iologs will be interspersed and the file may be corrupt.
1774 read_iolog=str
1776 Open an iolog with the specified filename and replay the I/O
1777 patterns it contains. This can be used to store a workload and
1778 replay it sometime later. The iolog given may also be a blktrace
1779 binary file, which allows fio to replay a workload captured by
1780 blktrace. See blktrace(8) for how to capture such logging data.
1781 For blktrace replay, the file needs to be turned into a blkparse
1782 binary data file first (`blkparse <device> -o /dev/null -d
1783 file_for_fio.bin').
1784 replay_no_stall=bool
1786 When replaying I/O with read_iolog the default behavior is to
1787 attempt to respect the timestamps within the log and replay them
1788 with the appropriate delay between IOPS. By setting this vari‐
1789 able fio will not respect the timestamps and attempt to replay
1790 them as fast as possible while still respecting ordering. The
1791 result is the same I/O pattern to a given device, but different
1792 timings.
1793 replay_redirect=str
1795 While replaying I/O patterns using read_iolog the default behav‐
1796 ior is to replay the IOPS onto the major/minor device that each
1797 IOP was recorded from. This is sometimes undesirable because on
1798 a different machine those major/minor numbers can map to a dif‐
1799 ferent device. Changing hardware on the same system can also
1800 result in a different major/minor mapping. replay_redirect
1801 causes all I/Os to be replayed onto the single specified device
1802 regardless of the device it was recorded from. i.e. `replay_re‐
1803 direct=/dev/sdc' would cause all I/O in the blktrace or iolog to
1804 be replayed onto `/dev/sdc'. This means multiple devices will be
1805 replayed onto a single device, if the trace contains multiple
1806 devices. If you want multiple devices to be replayed concur‐
1807 rently to multiple redirected devices you must blkparse your
1808 trace into separate traces and replay them with independent fio
1809 invocations. Unfortunately this also breaks the strict time
1810 ordering between multiple device accesses.
1811 replay_align=int
1813 Force alignment of I/O offsets and lengths in a trace to this
1814 power of 2 value.
1815 replay_scale=int
1817 Scale sector offsets down by this factor when replaying traces.
1818 Threads, processes and job synchronization
1820 thread Fio defaults to creating jobs by using fork, however if this
1821 option is given, fio will create jobs by using POSIX Threads'
1822 function pthread_create(3) to create threads instead.
1823 wait_for=str
1825 If set, the current job won't be started until all workers of
1826 the specified waitee job are done. wait_for operates on the job
1827 name basis, so there are a few limitations. First, the waitee
1828 must be defined prior to the waiter job (meaning no forward ref‐
1829 erences). Second, if a job is being referenced as a waitee, it
1830 must have a unique name (no duplicate waitees).
1831 nice=int
1833 Run the job with the given nice value. See man nice(2). On Win‐
1834 dows, values less than -15 set the process class to "High"; -1
1835 through -15 set "Above Normal"; 1 through 15 "Below Normal"; and
1836 above 15 "Idle" priority class.
1837 prio=int
1839 Set the I/O priority value of this job. Linux limits us to a
1840 positive value between 0 and 7, with 0 being the highest. See
1841 man ionice(1). Refer to an appropriate manpage for other operat‐
1842 ing systems since meaning of priority may differ.
1843 prioclass=int
1845 Set the I/O priority class. See man ionice(1).
1846 cpumask=int
1848 Set the CPU affinity of this job. The parameter given is a bit
1849 mask of allowed CPUs the job may run on. So if you want the
1850 allowed CPUs to be 1 and 5, you would pass the decimal value of
1851 (1 << 1 | 1 << 5), or 34. See man sched_setaffinity(2). This may
1852 not work on all supported operating systems or kernel versions.
1853 This option doesn't work well for a higher CPU count than what
1854 you can store in an integer mask, so it can only control cpus
1855 1-32. For boxes with larger CPU counts, use cpus_allowed.
1856 cpus_allowed=str
1858 Controls the same options as cpumask, but accepts a textual
1859 specification of the permitted CPUs instead. So to use CPUs 1
1860 and 5 you would specify `cpus_allowed=1,5'. This option also
1861 allows a range of CPUs to be specified -- say you wanted a bind‐
1862 ing to CPUs 1, 5, and 8 to 15, you would set
1863 `cpus_allowed=1,5,8-15'.
1864 cpus_allowed_policy=str
1866 Set the policy of how fio distributes the CPUs specified by
1867 cpus_allowed or cpumask. Two policies are supported:
1868 shared All jobs will share the CPU set specified.
1870 split Each job will get a unique CPU from the CPU set.
1872 shared is the default behavior, if the option isn't specified.
1874 If split is specified, then fio will will assign one cpu per
1875 job. If not enough CPUs are given for the jobs listed, then fio
1876 will roundrobin the CPUs in the set.
1877 numa_cpu_nodes=str
1879 Set this job running on specified NUMA nodes' CPUs. The argu‐
1880 ments allow comma delimited list of cpu numbers, A-B ranges, or
1881 `all'. Note, to enable NUMA options support, fio must be built
1882 on a system with libnuma-dev(el) installed.
1883 numa_mem_policy=str
1885 Set this job's memory policy and corresponding NUMA nodes. For‐
1886 mat of the arguments:
1887 <mode>[:<nodelist>]
1889 `mode' is one of the following memory poicies: `default', `pre‐
1891 fer', `bind', `interleave' or `local'. For `default' and `local'
1892 memory policies, no node needs to be specified. For `prefer',
1893 only one node is allowed. For `bind' and `interleave' the
1894 `nodelist' may be as follows: a comma delimited list of numbers,
1895 A-B ranges, or `all'.
1896 cgroup=str
1898 Add job to this control group. If it doesn't exist, it will be
1899 created. The system must have a mounted cgroup blkio mount point
1900 for this to work. If your system doesn't have it mounted, you
1901 can do so with:
1902 # mount -t cgroup -o blkio none /cgroup
1904 cgroup_weight=int
1906 Set the weight of the cgroup to this value. See the documenta‐
1907 tion that comes with the kernel, allowed values are in the range
1908 of 100..1000.
1909 cgroup_nodelete=bool
1911 Normally fio will delete the cgroups it has created after the
1912 job completion. To override this behavior and to leave cgroups
1913 around after the job completion, set `cgroup_nodelete=1'. This
1914 can be useful if one wants to inspect various cgroup files after
1915 job completion. Default: false.
1916 flow_id=int
1918 The ID of the flow. If not specified, it defaults to being a
1919 global flow. See flow.
1920 flow=int
1922 Weight in token-based flow control. If this value is used, then
1923 there is a 'flow counter' which is used to regulate the propor‐
1924 tion of activity between two or more jobs. Fio attempts to keep
1925 this flow counter near zero. The flow parameter stands for how
1926 much should be added or subtracted to the flow counter on each
1927 iteration of the main I/O loop. That is, if one job has `flow=8'
1928 and another job has `flow=-1', then there will be a roughly 1:8
1929 ratio in how much one runs vs the other.
1930 flow_watermark=int
1932 The maximum value that the absolute value of the flow counter is
1933 allowed to reach before the job must wait for a lower value of
1934 the counter.
1935 flow_sleep=int
1937 The period of time, in microseconds, to wait after the flow
1938 watermark has been exceeded before retrying operations.
1939 stonewall, wait_for_previous
1941 Wait for preceding jobs in the job file to exit, before starting
1942 this one. Can be used to insert serialization points in the job
1943 file. A stone wall also implies starting a new reporting group,
1944 see group_reporting.
1945 exitall
1947 By default, fio will continue running all other jobs when one
1948 job finishes but sometimes this is not the desired action. Set‐
1949 ting exitall will instead make fio terminate all other jobs when
1950 one job finishes.
1951 exec_prerun=str
1953 Before running this job, issue the command specified through
1954 system(3). Output is redirected in a file called `jobname.pre‐
1955 run.txt'.
1956 exec_postrun=str
1958 After the job completes, issue the command specified though sys‐
1959 tem(3). Output is redirected in a file called `job‐
1960 name.postrun.txt'.
1961 uid=int
1963 Instead of running as the invoking user, set the user ID to this
1964 value before the thread/process does any work.
1965 gid=int
1967 Set group ID, see uid.
1968 Verification
1970 verify_only
1971 Do not perform specified workload, only verify data still
1972 matches previous invocation of this workload. This option allows
1973 one to check data multiple times at a later date without over‐
1974 writing it. This option makes sense only for workloads that
1975 write data, and does not support workloads with the time_based
1976 option set.
1977 do_verify=bool
1979 Run the verify phase after a write phase. Only valid if verify
1980 is set. Default: true.
1981 verify=str
1983 If writing to a file, fio can verify the file contents after
1984 each iteration of the job. Each verification method also implies
1985 verification of special header, which is written to the begin‐
1986 ning of each block. This header also includes meta information,
1987 like offset of the block, block number, timestamp when block was
1988 written, etc. verify can be combined with verify_pattern option.
1989 The allowed values are:
1990 md5 Use an md5 sum of the data area and store it in
1992 the header of each block.
1993 crc64 Use an experimental crc64 sum of the data area and
1995 store it in the header of each block.
1996 crc32c Use a crc32c sum of the data area and store it in
1998 the header of each block. This will automatically
1999 use hardware acceleration (e.g. SSE4.2 on an x86
2000 or CRC crypto extensions on ARM64) but will fall
2001 back to software crc32c if none is found. Gener‐
2002 ally the fatest checksum fio supports when hard‐
2003 ware accelerated.
2004 crc32c-intel
2006 Synonym for crc32c.
2007 crc32 Use a crc32 sum of the data area and store it in
2009 the header of each block.
2010 crc16 Use a crc16 sum of the data area and store it in
2012 the header of each block.
2013 crc7 Use a crc7 sum of the data area and store it in
2015 the header of each block.
2016 xxhash Use xxhash as the checksum function. Generally the
2018 fastest software checksum that fio supports.
2019 sha512 Use sha512 as the checksum function.
2021 sha256 Use sha256 as the checksum function.
2023 sha1 Use optimized sha1 as the checksum function.
2025 sha3-224
2027 Use optimized sha3-224 as the checksum function.
2028 sha3-256
2030 Use optimized sha3-256 as the checksum function.
2031 sha3-384
2033 Use optimized sha3-384 as the checksum function.
2034 sha3-512
2036 Use optimized sha3-512 as the checksum function.
2037 meta This option is deprecated, since now meta informa‐
2039 tion is included in generic verification header
2040 and meta verification happens by default. For
2041 detailed information see the description of the
2042 verify setting. This option is kept because of
2043 compatibility's sake with old configurations. Do
2044 not use it.
2045 pattern
2047 Verify a strict pattern. Normally fio includes a
2048 header with some basic information and checksum‐
2049 ming, but if this option is set, only the specific
2050 pattern set with verify_pattern is verified.
2051 null Only pretend to verify. Useful for testing inter‐
2053 nals with `ioengine=null', not for much else.
2054 This option can be used for repeated burn-in tests of a system
2056 to make sure that the written data is also correctly read back.
2057 If the data direction given is a read or random read, fio will
2058 assume that it should verify a previously written file. If the
2059 data direction includes any form of write, the verify will be of
2060 the newly written data.
2061 verifysort=bool
2063 If true, fio will sort written verify blocks when it deems it
2064 faster to read them back in a sorted manner. This is often the
2065 case when overwriting an existing file, since the blocks are
2066 already laid out in the file system. You can ignore this option
2067 unless doing huge amounts of really fast I/O where the red-black
2068 tree sorting CPU time becomes significant. Default: true.
2069 verifysort_nr=int
2071 Pre-load and sort verify blocks for a read workload.
2072 verify_offset=int
2074 Swap the verification header with data somewhere else in the
2075 block before writing. It is swapped back before verifying.
2076 verify_interval=int
2078 Write the verification header at a finer granularity than the
2079 blocksize. It will be written for chunks the size of ver‐
2080 ify_interval. blocksize should divide this evenly.
2081 verify_pattern=str
2083 If set, fio will fill the I/O buffers with this pattern. Fio
2084 defaults to filling with totally random bytes, but sometimes
2085 it's interesting to fill with a known pattern for I/O verifica‐
2086 tion purposes. Depending on the width of the pattern, fio will
2087 fill 1/2/3/4 bytes of the buffer at the time (it can be either a
2088 decimal or a hex number). The verify_pattern if larger than a
2089 32-bit quantity has to be a hex number that starts with either
2090 "0x" or "0X". Use with verify. Also, verify_pattern supports %o
2091 format, which means that for each block offset will be written
2092 and then verified back, e.g.:
2093 verify_pattern=%o
2095 Or use combination of everything:
2097 verify_pattern=0xff%o"abcd"-12
2099 verify_fatal=bool
2101 Normally fio will keep checking the entire contents before quit‐
2102 ting on a block verification failure. If this option is set, fio
2103 will exit the job on the first observed failure. Default: false.
2104 verify_dump=bool
2106 If set, dump the contents of both the original data block and
2107 the data block we read off disk to files. This allows later
2108 analysis to inspect just what kind of data corruption occurred.
2109 Off by default.
2110 verify_async=int
2112 Fio will normally verify I/O inline from the submitting thread.
2113 This option takes an integer describing how many async offload
2114 threads to create for I/O verification instead, causing fio to
2115 offload the duty of verifying I/O contents to one or more sepa‐
2116 rate threads. If using this offload option, even sync I/O
2117 engines can benefit from using an iodepth setting higher than 1,
2118 as it allows them to have I/O in flight while verifies are run‐
2119 ning. Defaults to 0 async threads, i.e. verification is not
2120 asynchronous.
2121 verify_async_cpus=str
2123 Tell fio to set the given CPU affinity on the async I/O verifi‐
2124 cation threads. See cpus_allowed for the format used.
2125 verify_backlog=int
2127 Fio will normally verify the written contents of a job that uti‐
2128 lizes verify once that job has completed. In other words, every‐
2129 thing is written then everything is read back and verified. You
2130 may want to verify continually instead for a variety of reasons.
2131 Fio stores the meta data associated with an I/O block in memory,
2132 so for large verify workloads, quite a bit of memory would be
2133 used up holding this meta data. If this option is enabled, fio
2134 will write only N blocks before verifying these blocks.
2135 verify_backlog_batch=int
2137 Control how many blocks fio will verify if verify_backlog is
2138 set. If not set, will default to the value of verify_backlog
2139 (meaning the entire queue is read back and verified). If ver‐
2140 ify_backlog_batch is less than verify_backlog then not all
2141 blocks will be verified, if verify_backlog_batch is larger than
2142 verify_backlog, some blocks will be verified more than once.
2143 verify_state_save=bool
2145 When a job exits during the write phase of a verify workload,
2146 save its current state. This allows fio to replay up until that
2147 point, if the verify state is loaded for the verify read phase.
2148 The format of the filename is, roughly:
2149 <type>-<jobname>-<jobindex>-verify.state.
2151 <type> is "local" for a local run, "sock" for a client/server
2153 socket connection, and "ip" (192.168.0.1, for instance) for a
2154 networked client/server connection. Defaults to true.
2155 verify_state_load=bool
2157 If a verify termination trigger was used, fio stores the current
2158 write state of each thread. This can be used at verification
2159 time so that fio knows how far it should verify. Without this
2160 information, fio will run a full verification pass, according to
2161 the settings in the job file used. Default false.
2162 trim_percentage=int
2164 Number of verify blocks to discard/trim.
2165 trim_verify_zero=bool
2167 Verify that trim/discarded blocks are returned as zeros.
2168 trim_backlog=int
2170 Verify that trim/discarded blocks are returned as zeros.
2171 trim_backlog_batch=int
2173 Trim this number of I/O blocks.
2174 experimental_verify=bool
2176 Enable experimental verification.
2177 Steady state
2179 steadystate=str:float, ss=str:float
2180 Define the criterion and limit for assessing steady state per‐
2181 formance. The first parameter designates the criterion whereas
2182 the second parameter sets the threshold. When the criterion
2183 falls below the threshold for the specified duration, the job
2184 will stop. For example, `iops_slope:0.1%' will direct fio to
2185 terminate the job when the least squares regression slope falls
2186 below 0.1% of the mean IOPS. If group_reporting is enabled this
2187 will apply to all jobs in the group. Below is the list of avail‐
2188 able steady state assessment criteria. All assessments are car‐
2189 ried out using only data from the rolling collection window.
2190 Threshold limits can be expressed as a fixed value or as a per‐
2191 centage of the mean in the collection window.
2192 iops Collect IOPS data. Stop the job if all individual
2194 IOPS measurements are within the specified limit
2195 of the mean IOPS (e.g., `iops:2' means that all
2196 individual IOPS values must be within 2 of the
2197 mean, whereas `iops:0.2%' means that all individ‐
2198 ual IOPS values must be within 0.2% of the mean
2199 IOPS to terminate the job).
2200 iops_slope
2202 Collect IOPS data and calculate the least squares
2203 regression slope. Stop the job if the slope falls
2204 below the specified limit.
2205 bw Collect bandwidth data. Stop the job if all indi‐
2207 vidual bandwidth measurements are within the spec‐
2208 ified limit of the mean bandwidth.
2209 bw_slope
2211 Collect bandwidth data and calculate the least
2212 squares regression slope. Stop the job if the
2213 slope falls below the specified limit.
2214 steadystate_duration=time, ss_dur=time
2216 A rolling window of this duration will be used to judge whether
2217 steady state has been reached. Data will be collected once per
2218 second. The default is 0 which disables steady state detection.
2219 When the unit is omitted, the value is interpreted in seconds.
2220 steadystate_ramp_time=time, ss_ramp=time
2222 Allow the job to run for the specified duration before beginning
2223 data collection for checking the steady state job termination
2224 criterion. The default is 0. When the unit is omitted, the value
2225 is interpreted in seconds.
2226 Measurements and reporting
2228 per_job_logs=bool
2229 If set, this generates bw/clat/iops log with per file private
2230 filenames. If not set, jobs with identical names will share the
2231 log filename. Default: true.
2232 group_reporting
2234 It may sometimes be interesting to display statistics for groups
2235 of jobs as a whole instead of for each individual job. This is
2236 especially true if numjobs is used; looking at individual
2237 thread/process output quickly becomes unwieldy. To see the final
2238 report per-group instead of per-job, use group_reporting. Jobs
2239 in a file will be part of the same reporting group, unless if
2240 separated by a stonewall, or by using new_group.
2241 new_group
2243 Start a new reporting group. See: group_reporting. If not given,
2244 all jobs in a file will be part of the same reporting group,
2245 unless separated by a stonewall.
2246 stats=bool
2248 By default, fio collects and shows final output results for all
2249 jobs that run. If this option is set to 0, then fio will ignore
2250 it in the final stat output.
2251 write_bw_log=str
2253 If given, write a bandwidth log for this job. Can be used to
2254 store data of the bandwidth of the jobs in their lifetime. The
2255 included fio_generate_plots script uses gnuplot to turn these
2256 text files into nice graphs. See write_lat_log for behavior of
2257 given filename. For this option, the postfix is `_bw.x.log',
2258 where `x' is the index of the job (1..N, where N is the number
2259 of jobs). If per_job_logs is false, then the filename will not
2260 include the job index. See LOG FILE FORMATS section.
2261 write_lat_log=str
2263 Same as write_bw_log, except that this option stores I/O submis‐
2264 sion, completion, and total latencies instead. If no filename is
2265 given with this option, the default filename of `job‐
2266 name_type.log' is used. Even if the filename is given, fio will
2267 still append the type of log. So if one specifies:
2268 write_lat_log=foo
2270 The actual log names will be `foo_slat.x.log', `foo_clat.x.log',
2272 and `foo_lat.x.log', where `x' is the index of the job (1..N,
2273 where N is the number of jobs). This helps fio_generate_plots
2274 find the logs automatically. If per_job_logs is false, then the
2275 filename will not include the job index. See LOG FILE FORMATS
2276 section.
2277 write_hist_log=str
2279 Same as write_lat_log, but writes I/O completion latency his‐
2280 tograms. If no filename is given with this option, the default
2281 filename of `jobname_clat_hist.x.log' is used, where `x' is the
2282 index of the job (1..N, where N is the number of jobs). Even if
2283 the filename is given, fio will still append the type of log. If
2284 per_job_logs is false, then the filename will not include the
2285 job index. See LOG FILE FORMATS section.
2286 write_iops_log=str
2288 Same as write_bw_log, but writes IOPS. If no filename is given
2289 with this option, the default filename of `jobname_type.x.log'
2290 is used, where `x' is the index of the job (1..N, where N is the
2291 number of jobs). Even if the filename is given, fio will still
2292 append the type of log. If per_job_logs is false, then the file‐
2293 name will not include the job index. See LOG FILE FORMATS sec‐
2294 tion.
2295 log_avg_msec=int
2297 By default, fio will log an entry in the iops, latency, or bw
2298 log for every I/O that completes. When writing to the disk log,
2299 that can quickly grow to a very large size. Setting this option
2300 makes fio average the each log entry over the specified period
2301 of time, reducing the resolution of the log. See log_max_value
2302 as well. Defaults to 0, logging all entries. Also see LOG FILE
2303 FORMATS section.
2304 log_hist_msec=int
2306 Same as log_avg_msec, but logs entries for completion latency
2307 histograms. Computing latency percentiles from averages of
2308 intervals using log_avg_msec is inaccurate. Setting this option
2309 makes fio log histogram entries over the specified period of
2310 time, reducing log sizes for high IOPS devices while retaining
2311 percentile accuracy. See log_hist_coarseness as well. Defaults
2312 to 0, meaning histogram logging is disabled.
2313 log_hist_coarseness=int
2315 Integer ranging from 0 to 6, defining the coarseness of the res‐
2316 olution of the histogram logs enabled with log_hist_msec. For
2317 each increment in coarseness, fio outputs half as many bins.
2318 Defaults to 0, for which histogram logs contain 1216 latency
2319 bins. See LOG FILE FORMATS section.
2320 log_max_value=bool
2322 If log_avg_msec is set, fio logs the average over that window.
2323 If you instead want to log the maximum value, set this option to
2324 1. Defaults to 0, meaning that averaged values are logged.
2325 log_offset=bool
2327 If this is set, the iolog options will include the byte offset
2328 for the I/O entry as well as the other data values. Defaults to
2329 0 meaning that offsets are not present in logs. Also see LOG
2330 FILE FORMATS section.
2331 log_compression=int
2333 If this is set, fio will compress the I/O logs as it goes, to
2334 keep the memory footprint lower. When a log reaches the speci‐
2335 fied size, that chunk is removed and compressed in the back‐
2336 ground. Given that I/O logs are fairly highly compressible, this
2337 yields a nice memory savings for longer runs. The downside is
2338 that the compression will consume some background CPU cycles, so
2339 it may impact the run. This, however, is also true if the log‐
2340 ging ends up consuming most of the system memory. So pick your
2341 poison. The I/O logs are saved normally at the end of a run, by
2342 decompressing the chunks and storing them in the specified log
2343 file. This feature depends on the availability of zlib.
2344 log_compression_cpus=str
2346 Define the set of CPUs that are allowed to handle online log
2347 compression for the I/O jobs. This can provide better isolation
2348 between performance sensitive jobs, and background compression
2349 work.
2350 log_store_compressed=bool
2352 If set, fio will store the log files in a compressed format.
2353 They can be decompressed with fio, using the --inflate-log com‐
2354 mand line parameter. The files will be stored with a `.fz' suf‐
2355 fix.
2356 log_unix_epoch=bool
2358 If set, fio will log Unix timestamps to the log files produced
2359 by enabling write_type_log for each log type, instead of the
2360 default zero-based timestamps.
2361 block_error_percentiles=bool
2363 If set, record errors in trim block-sized units from writes and
2364 trims and output a histogram of how many trims it took to get to
2365 errors, and what kind of error was encountered.
2366 bwavgtime=int
2368 Average the calculated bandwidth over the given time. Value is
2369 specified in milliseconds. If the job also does bandwidth log‐
2370 ging through write_bw_log, then the minimum of this option and
2371 log_avg_msec will be used. Default: 500ms.
2372 iopsavgtime=int
2374 Average the calculated IOPS over the given time. Value is speci‐
2375 fied in milliseconds. If the job also does IOPS logging through
2376 write_iops_log, then the minimum of this option and log_avg_msec
2377 will be used. Default: 500ms.
2378 disk_util=bool
2380 Generate disk utilization statistics, if the platform supports
2381 it. Default: true.
2382 disable_lat=bool
2384 Disable measurements of total latency numbers. Useful only for
2385 cutting back the number of calls to gettimeofday(2), as that
2386 does impact performance at really high IOPS rates. Note that to
2387 really get rid of a large amount of these calls, this option
2388 must be used with disable_slat and disable_bw_measurement as
2389 well.
2390 disable_clat=bool
2392 Disable measurements of completion latency numbers. See dis‐
2393 able_lat.
2394 disable_slat=bool
2396 Disable measurements of submission latency numbers. See dis‐
2397 able_lat.
2398 disable_bw_measurement=bool, disable_bw=bool
2400 Disable measurements of throughput/bandwidth numbers. See dis‐
2401 able_lat.
2402 clat_percentiles=bool
2404 Enable the reporting of percentiles of completion latencies.
2405 This option is mutually exclusive with lat_percentiles.
2406 lat_percentiles=bool
2408 Enable the reporting of percentiles of IO latencies. This is
2409 similar to clat_percentiles, except that this includes the sub‐
2410 mission latency. This option is mutually exclusive with
2411 clat_percentiles.
2412 percentile_list=float_list
2414 Overwrite the default list of percentiles for completion laten‐
2415 cies and the block error histogram. Each number is a floating
2416 number in the range (0,100], and the maximum length of the list
2417 is 20. Use ':' to separate the numbers, and list the numbers in
2418 ascending order. For example, `--percentile_list=99.5:99.9' will
2419 cause fio to report the values of completion latency below which
2420 99.5% and 99.9% of the observed latencies fell, respectively.
2421 Error handling
2423 exitall_on_error
2424 When one job finishes in error, terminate the rest. The default
2425 is to wait for each job to finish.
2426 continue_on_error=str
2428 Normally fio will exit the job on the first observed failure. If
2429 this option is set, fio will continue the job when there is a
2430 'non-fatal error' (EIO or EILSEQ) until the runtime is exceeded
2431 or the I/O size specified is completed. If this option is used,
2432 there are two more stats that are appended, the total error
2433 count and the first error. The error field given in the stats is
2434 the first error that was hit during the run. The allowed values
2435 are:
2436 none Exit on any I/O or verify errors.
2438 read Continue on read errors, exit on all others.
2440 write Continue on write errors, exit on all others.
2442 io Continue on any I/O error, exit on all others.
2444 verify Continue on verify errors, exit on all others.
2446 all Continue on all errors.
2448 0 Backward-compatible alias for 'none'.
2450 1 Backward-compatible alias for 'all'.
2452 ignore_error=str
2454 Sometimes you want to ignore some errors during test in that
2455 case you can specify error list for each error type, instead of
2456 only being able to ignore the default 'non-fatal error' using
2457 continue_on_error.
2458 `ignore_error=READ_ERR_LIST,WRITE_ERR_LIST,VERIFY_ERR_LIST'
2459 errors for given error type is separated with ':'. Error may be
2460 symbol ('ENOSPC', 'ENOMEM') or integer. Example:
2461 ignore_error=EAGAIN,ENOSPC:122
2463 This option will ignore EAGAIN from READ, and ENOSPC and
2465 122(EDQUOT) from WRITE. This option works by overriding con‐
2466 tinue_on_error with the list of errors for each error type if
2467 any.
2468 error_dump=bool
2470 If set dump every error even if it is non fatal, true by
2471 default. If disabled only fatal error will be dumped.
2472 Running predefined workloads
2474 Fio includes predefined profiles that mimic the I/O workloads generated
2475 by other tools.
2476 profile=str
2478 The predefined workload to run. Current profiles are:
2479 tiobench
2481 Threaded I/O bench (tiotest/tiobench) like work‐
2482 load.
2483 act Aerospike Certification Tool (ACT) like workload.
2485 To view a profile's additional options use --cmdhelp after specifying
2487 the profile. For example:
2488 $ fio --profile=act --cmdhelp
2490 Act profile options
2492 device-names=str
2493 Devices to use.
2494 load=int
2496 ACT load multiplier. Default: 1.
2497 test-duration=time
2499 How long the entire test takes to run. When the unit is omitted,
2500 the value is given in seconds. Default: 24h.
2501 threads-per-queue=int
2503 Number of read I/O threads per device. Default: 8.
2504 read-req-num-512-blocks=int
2506 Number of 512B blocks to read at the time. Default: 3.
2507 large-block-op-kbytes=int
2509 Size of large block ops in KiB (writes). Default: 131072.
2510 prep Set to run ACT prep phase.
2512 Tiobench profile options
2514 size=str
2515 Size in MiB.
2516 block=int
2518 Block size in bytes. Default: 4096.
2519 numruns=int
2521 Number of runs.
2522 dir=str
2524 Test directory.
2525 threads=int
2527 Number of threads.
2528
2530 Fio spits out a lot of output. While running, fio will display the sta‐
2531 tus of the jobs created. An example of that would be:
2532 Jobs: 1 (f=1): [_(1),M(1)][24.8%][r=20.5MiB/s,w=23.5MiB/s][r=82,w=94 IOPS][eta 01m:31s]
2534 The characters inside the first set of square brackets denote the cur‐
2536 rent status of each thread. The first character is the first job
2537 defined in the job file, and so forth. The possible values (in typical
2538 life cycle order) are:
2539 P Thread setup, but not started.
2541 C Thread created.
2542 I Thread initialized, waiting or generating necessary data.
2543 P Thread running pre-reading file(s).
2544 / Thread is in ramp period.
2545 R Running, doing sequential reads.
2546 r Running, doing random reads.
2547 W Running, doing sequential writes.
2548 w Running, doing random writes.
2549 M Running, doing mixed sequential reads/writes.
2550 m Running, doing mixed random reads/writes.
2551 D Running, doing sequential trims.
2552 d Running, doing random trims.
2553 F Running, currently waiting for fsync(2).
2554 V Running, doing verification of written data.
2555 f Thread finishing.
2556 E Thread exited, not reaped by main thread yet.
2557 - Thread reaped.
2558 X Thread reaped, exited with an error.
2559 K Thread reaped, exited due to signal.
2560 Fio will condense the thread string as not to take up more space on the
2562 command line than needed. For instance, if you have 10 readers and 10
2563 writers running, the output would look like this:
2564 Jobs: 20 (f=20): [R(10),W(10)][4.0%][r=20.5MiB/s,w=23.5MiB/s][r=82,w=94 IOPS][eta 57m:36s]
2566 Note that the status string is displayed in order, so it's possible to
2568 tell which of the jobs are currently doing what. In the example above
2569 this means that jobs 1--10 are readers and 11--20 are writers.
2570 The other values are fairly self explanatory -- number of threads cur‐
2572 rently running and doing I/O, the number of currently open files (f=),
2573 the estimated completion percentage, the rate of I/O since last check
2574 (read speed listed first, then write speed and optionally trim speed)
2575 in terms of bandwidth and IOPS, and time to completion for the current
2576 running group. It's impossible to estimate runtime of the following
2577 groups (if any).
2578 When fio is done (or interrupted by Ctrl-C), it will show the data for
2580 each thread, group of threads, and disks in that order. For each over‐
2581 all thread (or group) the output looks like:
2582 Client1: (groupid=0, jobs=1): err= 0: pid=16109: Sat Jun 24 12:07:54 2017
2584 write: IOPS=88, BW=623KiB/s (638kB/s)(30.4MiB/50032msec)
2585 slat (nsec): min=500, max=145500, avg=8318.00, stdev=4781.50
2586 clat (usec): min=170, max=78367, avg=4019.02, stdev=8293.31
2587 lat (usec): min=174, max=78375, avg=4027.34, stdev=8291.79
2588 clat percentiles (usec):
2589 | 1.00th=[ 302], 5.00th=[ 326], 10.00th=[ 343], 20.00th=[ 363],
2590 | 30.00th=[ 392], 40.00th=[ 404], 50.00th=[ 416], 60.00th=[ 445],
2591 | 70.00th=[ 816], 80.00th=[ 6718], 90.00th=[12911], 95.00th=[21627],
2592 | 99.00th=[43779], 99.50th=[51643], 99.90th=[68682], 99.95th=[72877],
2593 | 99.99th=[78119]
2594 bw ( KiB/s): min= 532, max= 686, per=0.10%, avg=622.87, stdev=24.82, samples= 100
2595 iops : min= 76, max= 98, avg=88.98, stdev= 3.54, samples= 100
2596 lat (usec) : 250=0.04%, 500=64.11%, 750=4.81%, 1000=2.79%
2597 lat (msec) : 2=4.16%, 4=1.84%, 10=4.90%, 20=11.33%, 50=5.37%
2598 lat (msec) : 100=0.65%
2599 cpu : usr=0.27%, sys=0.18%, ctx=12072, majf=0, minf=21
2600 IO depths : 1=85.0%, 2=13.1%, 4=1.8%, 8=0.1%, 16=0.0%, 32=0.0%, >=64=0.0%
2601 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
2602 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
2603 issued rwt: total=0,4450,0, short=0,0,0, dropped=0,0,0
2604 latency : target=0, window=0, percentile=100.00%, depth=8
2605 The job name (or first job's name when using group_reporting) is
2607 printed, along with the group id, count of jobs being aggregated, last
2608 error id seen (which is 0 when there are no errors), pid/tid of that
2609 thread and the time the job/group completed. Below are the I/O statis‐
2610 tics for each data direction performed (showing writes in the example
2611 above). In the order listed, they denote:
2612 read/write/trim
2614 The string before the colon shows the I/O direction the
2615 statistics are for. IOPS is the average I/Os performed
2616 per second. BW is the average bandwidth rate shown as:
2617 value in power of 2 format (value in power of 10 format).
2618 The last two values show: (total I/O performed in power
2619 of 2 format / runtime of that thread).
2620 slat Submission latency (min being the minimum, max being the
2622 maximum, avg being the average, stdev being the standard
2623 deviation). This is the time it took to submit the I/O.
2624 For sync I/O this row is not displayed as the slat is
2625 really the completion latency (since queue/complete is
2626 one operation there). This value can be in nanoseconds,
2627 microseconds or milliseconds --- fio will choose the most
2628 appropriate base and print that (in the example above
2629 nanoseconds was the best scale). Note: in --minimal mode
2630 latencies are always expressed in microseconds.
2631 clat Completion latency. Same names as slat, this denotes the
2633 time from submission to completion of the I/O pieces. For
2634 sync I/O, clat will usually be equal (or very close) to
2635 0, as the time from submit to complete is basically just
2636 CPU time (I/O has already been done, see slat explana‐
2637 tion).
2638 lat Total latency. Same names as slat and clat, this denotes
2640 the time from when fio created the I/O unit to completion
2641 of the I/O operation.
2642 bw Bandwidth statistics based on samples. Same names as the
2644 xlat stats, but also includes the number of samples taken
2645 (samples) and an approximate percentage of total aggre‐
2646 gate bandwidth this thread received in its group (per).
2647 This last value is only really useful if the threads in
2648 this group are on the same disk, since they are then com‐
2649 peting for disk access.
2650 iops IOPS statistics based on samples. Same names as bw.
2652 lat (nsec/usec/msec)
2654 The distribution of I/O completion latencies. This is the
2655 time from when I/O leaves fio and when it gets completed.
2656 Unlike the separate read/write/trim sections above, the
2657 data here and in the remaining sections apply to all I/Os
2658 for the reporting group. 250=0.04% means that 0.04% of
2659 the I/Os completed in under 250us. 500=64.11% means that
2660 64.11% of the I/Os required 250 to 499us for completion.
2661 cpu CPU usage. User and system time, along with the number of
2663 context switches this thread went through, usage of sys‐
2664 tem and user time, and finally the number of major and
2665 minor page faults. The CPU utilization numbers are aver‐
2666 ages for the jobs in that reporting group, while the con‐
2667 text and fault counters are summed.
2668 IO depths
2670 The distribution of I/O depths over the job lifetime. The
2671 numbers are divided into powers of 2 and each entry cov‐
2672 ers depths from that value up to those that are lower
2673 than the next entry -- e.g., 16= covers depths from 16 to
2674 31. Note that the range covered by a depth distribution
2675 entry can be different to the range covered by the equiv‐
2676 alent submit/complete distribution entry.
2677 IO submit
2679 How many pieces of I/O were submitting in a single submit
2680 call. Each entry denotes that amount and below, until the
2681 previous entry -- e.g., 16=100% means that we submitted
2682 anywhere between 9 to 16 I/Os per submit call. Note that
2683 the range covered by a submit distribution entry can be
2684 different to the range covered by the equivalent depth
2685 distribution entry.
2686 IO complete
2688 Like the above submit number, but for completions
2689 instead.
2690 IO issued rwt
2692 The number of read/write/trim requests issued, and how
2693 many of them were short or dropped.
2694 IO latency
2696 These values are for latency-target and related options.
2697 When these options are engaged, this section describes
2698 the I/O depth required to meet the specified latency tar‐
2699 get.
2700 After each client has been listed, the group statistics are printed.
2702 They will look like this:
2703 Run status group 0 (all jobs):
2705 READ: bw=20.9MiB/s (21.9MB/s), 10.4MiB/s-10.8MiB/s (10.9MB/s-11.3MB/s), io=64.0MiB (67.1MB), run=2973-3069msec
2706 WRITE: bw=1231KiB/s (1261kB/s), 616KiB/s-621KiB/s (630kB/s-636kB/s), io=64.0MiB (67.1MB), run=52747-53223msec
2707 For each data direction it prints:
2709 bw Aggregate bandwidth of threads in this group followed by
2711 the minimum and maximum bandwidth of all the threads in
2712 this group. Values outside of brackets are power-of-2
2713 format and those within are the equivalent value in a
2714 power-of-10 format.
2715 io Aggregate I/O performed of all threads in this group. The
2717 format is the same as bw.
2718 run The smallest and longest runtimes of the threads in this
2720 group.
2721 And finally, the disk statistics are printed. This is Linux specific.
2723 They will look like this:
2724 Disk stats (read/write):
2726 sda: ios=16398/16511, merge=30/162, ticks=6853/819634, in_queue=826487, util=100.00%
2727 Each value is printed for both reads and writes, with reads first. The
2729 numbers denote:
2730 ios Number of I/Os performed by all groups.
2732 merge Number of merges performed by the I/O scheduler.
2734 ticks Number of ticks we kept the disk busy.
2736 in_queue
2738 Total time spent in the disk queue.
2739 util The disk utilization. A value of 100% means we kept the
2741 disk busy constantly, 50% would be a disk idling half of
2742 the time.
2743 It is also possible to get fio to dump the current output while it is
2745 running, without terminating the job. To do that, send fio the USR1
2746 signal. You can also get regularly timed dumps by using the --sta‐
2747 tus-interval parameter, or by creating a file in `/tmp' named
2748 `fio-dump-status'. If fio sees this file, it will unlink it and dump
2749 the current output status.
2750
2752 For scripted usage where you typically want to generate tables or
2753 graphs of the results, fio can output the results in a semicolon sepa‐
2754 rated format. The format is one long line of values, such as:
2755 2;card0;0;0;7139336;121836;60004;1;10109;27.932460;116.933948;220;126861;3495.446807;1085.368601;226;126864;3523.635629;1089.012448;24063;99944;50.275485%;59818.274627;5540.657370;7155060;122104;60004;1;8338;29.086342;117.839068;388;128077;5032.488518;1234.785715;391;128085;5061.839412;1236.909129;23436;100928;50.287926%;59964.832030;5644.844189;14.595833%;19.394167%;123706;0;7313;0.1%;0.1%;0.1%;0.1%;0.1%;0.1%;100.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.02%;0.05%;0.16%;6.04%;40.40%;52.68%;0.64%;0.01%;0.00%;0.01%;0.00%;0.00%;0.00%;0.00%;0.00%
2757 A description of this job goes here.
2758 The job description (if provided) follows on a second line.
2760 To enable terse output, use the --minimal or `--output-format=terse'
2762 command line options. The first value is the version of the terse out‐
2763 put format. If the output has to be changed for some reason, this num‐
2764 ber will be incremented by 1 to signify that change.
2765 Split up, the format is as follows (comments in brackets denote when a
2767 field was introduced or whether it's specific to some terse version):
2768 terse version, fio version [v3], jobname, groupid, error
2770 READ status:
2772 Total IO (KiB), bandwidth (KiB/sec), IOPS, runtime (msec)
2774 Submission latency: min, max, mean, stdev (usec)
2775 Completion latency: min, max, mean, stdev (usec)
2776 Completion latency percentiles: 20 fields (see below)
2777 Total latency: min, max, mean, stdev (usec)
2778 Bw (KiB/s): min, max, aggregate percentage of total, mean, stdev, number of samples [v5]
2779 IOPS [v5]: min, max, mean, stdev, number of samples
2780 WRITE status:
2782 Total IO (KiB), bandwidth (KiB/sec), IOPS, runtime (msec)
2784 Submission latency: min, max, mean, stdev (usec)
2785 Completion latency: min, max, mean, stdev (usec)
2786 Completion latency percentiles: 20 fields (see below)
2787 Total latency: min, max, mean, stdev (usec)
2788 Bw (KiB/s): min, max, aggregate percentage of total, mean, stdev, number of samples [v5]
2789 IOPS [v5]: min, max, mean, stdev, number of samples
2790 TRIM status [all but version 3]:
2792 Fields are similar to READ/WRITE status.
2794 CPU usage:
2796 user, system, context switches, major faults, minor faults
2798 I/O depths:
2800 <=1, 2, 4, 8, 16, 32, >=64
2802 I/O latencies microseconds:
2804 <=2, 4, 10, 20, 50, 100, 250, 500, 750, 1000
2806 I/O latencies milliseconds:
2808 <=2, 4, 10, 20, 50, 100, 250, 500, 750, 1000, 2000, >=2000
2810 Disk utilization [v3]:
2812 disk name, read ios, write ios, read merges, write merges, read ticks, write ticks, time spent in queue, disk utilization percentage
2814 Additional Info (dependent on continue_on_error, default off):
2816 total # errors, first error code
2818 Additional Info (dependent on description being set):
2820 Text description
2822 Completion latency percentiles can be a grouping of up to 20 sets, so
2824 for the terse output fio writes all of them. Each field will look like
2825 this:
2826 1.00%=6112
2828 which is the Xth percentile, and the `usec' latency associated with it.
2830 For Disk utilization, all disks used by fio are shown. So for each disk
2832 there will be a disk utilization section.
2833 Below is a single line containing short names for each of the fields in
2835 the minimal output v3, separated by semicolons:
2836 terse_version_3;fio_version;jobname;groupid;error;read_kb;read_bandwidth;read_iops;read_runtime_ms;read_slat_min;read_slat_max;read_slat_mean;read_slat_dev;read_clat_min;read_clat_max;read_clat_mean;read_clat_dev;read_clat_pct01;read_clat_pct02;read_clat_pct03;read_clat_pct04;read_clat_pct05;read_clat_pct06;read_clat_pct07;read_clat_pct08;read_clat_pct09;read_clat_pct10;read_clat_pct11;read_clat_pct12;read_clat_pct13;read_clat_pct14;read_clat_pct15;read_clat_pct16;read_clat_pct17;read_clat_pct18;read_clat_pct19;read_clat_pct20;read_tlat_min;read_lat_max;read_lat_mean;read_lat_dev;read_bw_min;read_bw_max;read_bw_agg_pct;read_bw_mean;read_bw_dev;write_kb;write_bandwidth;write_iops;write_runtime_ms;write_slat_min;write_slat_max;write_slat_mean;write_slat_dev;write_clat_min;write_clat_max;write_clat_mean;write_clat_dev;write_clat_pct01;write_clat_pct02;write_clat_pct03;write_clat_pct04;write_clat_pct05;write_clat_pct06;write_clat_pct07;write_clat_pct08;write_clat_pct09;write_clat_pct10;write_clat_pct11;write_clat_pct12;write_clat_pct13;write_clat_pct14;write_clat_pct15;write_clat_pct16;write_clat_pct17;write_clat_pct18;write_clat_pct19;write_clat_pct20;write_tlat_min;write_lat_max;write_lat_mean;write_lat_dev;write_bw_min;write_bw_max;write_bw_agg_pct;write_bw_mean;write_bw_dev;cpu_user;cpu_sys;cpu_csw;cpu_mjf;cpu_minf;iodepth_1;iodepth_2;iodepth_4;iodepth_8;iodepth_16;iodepth_32;iodepth_64;lat_2us;lat_4us;lat_10us;lat_20us;lat_50us;lat_100us;lat_250us;lat_500us;lat_750us;lat_1000us;lat_2ms;lat_4ms;lat_10ms;lat_20ms;lat_50ms;lat_100ms;lat_250ms;lat_500ms;lat_750ms;lat_1000ms;lat_2000ms;lat_over_2000ms;disk_name;disk_read_iops;disk_write_iops;disk_read_merges;disk_write_merges;disk_read_ticks;write_ticks;disk_queue_time;disk_util
2838
2840 The json output format is intended to be both human readable and conve‐
2841 nient for automated parsing. For the most part its sections mirror
2842 those of the normal output. The runtime value is reported in msec and
2843 the bw value is reported in 1024 bytes per second units.
2844
2846 The json+ output format is identical to the json output format except
2847 that it adds a full dump of the completion latency bins. Each bins
2848 object contains a set of (key, value) pairs where keys are latency
2849 durations and values count how many I/Os had completion latencies of
2850 the corresponding duration. For example, consider:
2851 "bins" : { "87552" : 1, "89600" : 1, "94720" : 1, "96768" : 1,
2853 "97792" : 1, "99840" : 1, "100864" : 2, "103936" : 6, "104960" :
2854 534, "105984" : 5995, "107008" : 7529, ... }
2855 This data indicates that one I/O required 87,552ns to complete, two
2857 I/Os required 100,864ns to complete, and 7529 I/Os required 107,008ns
2858 to complete.
2859 Also included with fio is a Python script fio_jsonplus_clat2csv that
2861 takes json+ output and generates CSV-formatted latency data suitable
2862 for plotting.
2863 The latency durations actually represent the midpoints of latency
2865 intervals. For details refer to `stat.h' in the fio source.
2866
2868 There are two trace file format that you can encounter. The older (v1)
2869 format is unsupported since version 1.20-rc3 (March 2008). It will
2870 still be described below in case that you get an old trace and want to
2871 understand it.
2872 In any case the trace is a simple text file with a single action per
2874 line.
2875 Trace file format v1
2877 Each line represents a single I/O action in the following for‐
2878 mat:
2879 rw, offset, length
2881 where `rw=0/1' for read/write, and the `offset' and `length'
2883 entries being in bytes.
2884 This format is not supported in fio versions >= 1.20-rc3.
2886 Trace file format v2
2888 The second version of the trace file format was added in fio
2889 version 1.17. It allows to access more then one file per trace
2890 and has a bigger set of possible file actions.
2891 The first line of the trace file has to be:
2893 "fio version 2 iolog"
2895 Following this can be lines in two different formats, which are
2897 described below.
2898 The file management format:
2900 filename action
2901 The `filename' is given as an absolute path. The `action'
2903 can be one of these:
2904 add Add the given `filename' to the trace.
2906 open Open the file with the given `filename'.
2908 The `filename' has to have been added with
2909 the add action before.
2910 close Close the file with the given `filename'.
2912 The file has to have been opened before.
2913 The file I/O action format:
2915 filename action offset length
2916 The `filename' is given as an absolute path, and has to
2918 have been added and opened before it can be used with
2919 this format. The `offset' and `length' are given in
2920 bytes. The `action' can be one of these:
2921 wait Wait for `offset' microseconds. Everything
2923 below 100 is discarded. The time is rela‐
2924 tive to the previous `wait' statement.
2925 read Read `length' bytes beginning from `off‐
2927 set'.
2928 write Write `length' bytes beginning from `off‐
2930 set'.
2931 sync fsync(2) the file.
2933 datasync
2935 fdatasync(2) the file.
2936 trim Trim the given file from the given `offset'
2938 for `length' bytes.
2939
2941 In some cases, we want to understand CPU overhead in a test. For exam‐
2942 ple, we test patches for the specific goodness of whether they reduce
2943 CPU usage. Fio implements a balloon approach to create a thread per
2944 CPU that runs at idle priority, meaning that it only runs when nobody
2945 else needs the cpu. By measuring the amount of work completed by the
2946 thread, idleness of each CPU can be derived accordingly.
2947 An unit work is defined as touching a full page of unsigned characters.
2949 Mean and standard deviation of time to complete an unit work is
2950 reported in "unit work" section. Options can be chosen to report
2951 detailed percpu idleness or overall system idleness by aggregating per‐
2952 cpu stats.
2953
2955 Fio is usually run in one of two ways, when data verification is done.
2956 The first is a normal write job of some sort with verify enabled. When
2957 the write phase has completed, fio switches to reads and verifies
2958 everything it wrote. The second model is running just the write phase,
2959 and then later on running the same job (but with reads instead of
2960 writes) to repeat the same I/O patterns and verify the contents. Both
2961 of these methods depend on the write phase being completed, as fio oth‐
2962 erwise has no idea how much data was written.
2963 With verification triggers, fio supports dumping the current write
2965 state to local files. Then a subsequent read verify workload can load
2966 this state and know exactly where to stop. This is useful for testing
2967 cases where power is cut to a server in a managed fashion, for
2968 instance.
2969 A verification trigger consists of two things:
2971 1) Storing the write state of each job.
2973 2) Executing a trigger command.
2975 The write state is relatively small, on the order of hundreds of bytes
2977 to single kilobytes. It contains information on the number of comple‐
2978 tions done, the last X completions, etc.
2979 A trigger is invoked either through creation ('touch') of a specified
2981 file in the system, or through a timeout setting. If fio is run with
2982 `--trigger-file=/tmp/trigger-file', then it will continually check for
2983 the existence of `/tmp/trigger-file'. When it sees this file, it will
2984 fire off the trigger (thus saving state, and executing the trigger com‐
2985 mand).
2986 For client/server runs, there's both a local and remote trigger. If fio
2988 is running as a server backend, it will send the job states back to the
2989 client for safe storage, then execute the remote trigger, if specified.
2990 If a local trigger is specified, the server will still send back the
2991 write state, but the client will then execute the trigger.
2992 Verification trigger example
2994 Let's say we want to run a powercut test on the remote Linux
2995 machine 'server'. Our write workload is in `write-test.fio'. We
2996 want to cut power to 'server' at some point during the run, and
2997 we'll run this test from the safety or our local machine,
2998 'localbox'. On the server, we'll start the fio backend normally:
2999 server# fio --server
3001 and on the client, we'll fire off the workload:
3003 localbox$ fio --client=server --trig‐
3005 ger-file=/tmp/my-trigger --trigger-remote="bash -c "echo
3006 b > /proc/sysrq-triger""
3007 We set `/tmp/my-trigger' as the trigger file, and we tell fio to
3009 execute:
3010 echo b > /proc/sysrq-trigger
3012 on the server once it has received the trigger and sent us the
3014 write state. This will work, but it's not really cutting power
3015 to the server, it's merely abruptly rebooting it. If we have a
3016 remote way of cutting power to the server through IPMI or simi‐
3017 lar, we could do that through a local trigger command instead.
3018 Let's assume we have a script that does IPMI reboot of a given
3019 hostname, ipmi-reboot. On localbox, we could then have run fio
3020 with a local trigger instead:
3021 localbox$ fio --client=server --trig‐
3023 ger-file=/tmp/my-trigger --trigger="ipmi-reboot server"
3024 For this case, fio would wait for the server to send us the
3026 write state, then execute `ipmi-reboot server' when that hap‐
3027 pened.
3028 Loading verify state
3030 To load stored write state, a read verification job file must
3031 contain the verify_state_load option. If that is set, fio will
3032 load the previously stored state. For a local fio run this is
3033 done by loading the files directly, and on a client/server run,
3034 the server backend will ask the client to send the files over
3035 and load them from there.
3036
3038 Fio supports a variety of log file formats, for logging latencies,
3039 bandwidth, and IOPS. The logs share a common format, which looks like
3040 this:
3041 time (msec), value, data direction, block size (bytes), offset
3043 (bytes)
3044 `Time' for the log entry is always in milliseconds. The `value' logged
3046 depends on the type of log, it will be one of the following:
3047 Latency log
3049 Value is latency in nsecs
3050 Bandwidth log
3052 Value is in KiB/sec
3053 IOPS log
3055 Value is IOPS
3056 `Data direction' is one of the following:
3058 0 I/O is a READ
3060 1 I/O is a WRITE
3062 2 I/O is a TRIM
3064 The entry's `block size' is always in bytes. The `offset' is the off‐
3066 set, in bytes, from the start of the file, for that particular I/O. The
3067 logging of the offset can be toggled with log_offset.
3068 Fio defaults to logging every individual I/O. When IOPS are logged for
3070 individual I/Os the `value' entry will always be 1. If windowed logging
3071 is enabled through log_avg_msec, fio logs the average values over the
3072 specified period of time. If windowed logging is enabled and
3073 log_max_value is set, then fio logs maximum values in that window
3074 instead of averages. Since `data direction', `block size' and `offset'
3075 are per-I/O values, if windowed logging is enabled they aren't applica‐
3076 ble and will be 0.
3077
3079 Normally fio is invoked as a stand-alone application on the machine
3080 where the I/O workload should be generated. However, the backend and
3081 frontend of fio can be run separately i.e., the fio server can generate
3082 an I/O workload on the "Device Under Test" while being controlled by a
3083 client on another machine.
3084 Start the server on the machine which has access to the storage DUT:
3086 $ fio --server=args
3088 where `args' defines what fio listens to. The arguments are of the form
3090 `type,hostname' or `IP,port'. `type' is either `ip' (or ip4) for TCP/IP
3091 v4, `ip6' for TCP/IP v6, or `sock' for a local unix domain socket.
3092 `hostname' is either a hostname or IP address, and `port' is the port
3093 to listen to (only valid for TCP/IP, not a local socket). Some exam‐
3094 ples:
3095 1) fio --server
3097 Start a fio server, listening on all interfaces on the
3098 default port (8765).
3099 2) fio --server=ip:hostname,4444
3101 Start a fio server, listening on IP belonging to hostname
3102 and on port 4444.
3103 3) fio --server=ip6:::1,4444
3105 Start a fio server, listening on IPv6 localhost ::1 and
3106 on port 4444.
3107 4) fio --server=,4444
3109 Start a fio server, listening on all interfaces on port
3110 4444.
3111 5) fio --server=1.2.3.4
3113 Start a fio server, listening on IP 1.2.3.4 on the
3114 default port.
3115 6) fio --server=sock:/tmp/fio.sock
3117 Start a fio server, listening on the local socket
3118 `/tmp/fio.sock'.
3119 Once a server is running, a "client" can connect to the fio server
3121 with:
3122 $ fio <local-args> --client=<server> <remote-args> <job file(s)>
3124 where `local-args' are arguments for the client where it is running,
3126 `server' is the connect string, and `remote-args' and `job file(s)' are
3127 sent to the server. The `server' string follows the same format as it
3128 does on the server side, to allow IP/hostname/socket and port strings.
3129 Fio can connect to multiple servers this way:
3131 $ fio --client=<server1> <job file(s)> --client=<server2> <job
3133 file(s)>
3134 If the job file is located on the fio server, then you can tell the
3136 server to load a local file as well. This is done by using
3137 --remote-config:
3138 $ fio --client=server --remote-config /path/to/file.fio
3140 Then fio will open this local (to the server) job file instead of being
3142 passed one from the client.
3143 If you have many servers (example: 100 VMs/containers), you can input a
3145 pathname of a file containing host IPs/names as the parameter value for
3146 the --client option. For example, here is an example `host.list' file
3147 containing 2 hostnames:
3148 host1.your.dns.domain
3150 host2.your.dns.domain
3151 The fio command would then be:
3153 $ fio --client=host.list <job file(s)>
3155 In this mode, you cannot input server-specific parameters or job files
3157 -- all servers receive the same job file.
3158 In order to let `fio --client' runs use a shared filesystem from multi‐
3160 ple hosts, `fio --client' now prepends the IP address of the server to
3161 the filename. For example, if fio is using the directory `/mnt/nfs/fio'
3162 and is writing filename `fileio.tmp', with a --client `hostfile' con‐
3163 taining two hostnames `h1' and `h2' with IP addresses 192.168.10.120
3164 and 192.168.10.121, then fio will create two files:
3165 /mnt/nfs/fio/192.168.10.120.fileio.tmp
3167 /mnt/nfs/fio/192.168.10.121.fileio.tmp
3168
3170 fio was written by Jens Axboe <jens.axboe@oracle.com>, now Jens Axboe
3171 <axboe@fb.com>.
3172 This man page was written by Aaron Carroll <aaronc@cse.unsw.edu.au>
3173 based on documentation by Jens Axboe.
3174 This man page was rewritten by Tomohiro Kusumi <tkusumi@tuxera.com>
3175 based on documentation by Jens Axboe.
3176
3178 Report bugs to the fio mailing list <fio@vger.kernel.org>.
3179 See REPORTING-BUGS.
3180 REPORTING-BUGS: http://git.kernel.dk/cgit/fio/plain/REPORTING-BUGS
3182
3184 For further documentation see HOWTO and README.
3185 Sample jobfiles are available in the `examples/' directory.
3186 These are typically located under `/usr/share/doc/fio'.
3187 HOWTO: http://git.kernel.dk/cgit/fio/plain/HOWTO
3189 README: http://git.kernel.dk/cgit/fio/plain/README
3190User Manual August 2017 fio(1)