1fio(1)                      General Commands Manual                     fio(1)
2
3
4

NAME

6       fio - flexible I/O tester
7

SYNOPSIS

9       fio [options] [jobfile]...
10

DESCRIPTION

12       fio  is a tool that will spawn a number of threads or processes doing a
13       particular type of I/O action as specified by the  user.   The  typical
14       use  of  fio  is to write a job file matching the I/O load one wants to
15       simulate.
16

OPTIONS

18       --debug=type
19              Enable verbose tracing type of various fio actions. May be `all'
20              for  all  types  or  individual types separated by a comma (e.g.
21              `--debug=file,mem'  will  enable  file  and  memory  debugging).
22              `help' will list all available tracing options.
23
24       --parse-only
25              Parse options only, don't start any I/O.
26
27       --merge-blktrace-only
28              Merge blktraces only, don't start any I/O.
29
30       --output=filename
31              Write output to filename.
32
33       --output-format=format
34              Set  the  reporting  format  to  `normal',  `terse',  `json', or
35              `json+'. Multiple formats can be selected, separate by a  comma.
36              `terse' is a CSV based format. `json+' is like `json', except it
37              adds a full dump of the latency buckets.
38
39       --bandwidth-log
40              Generate aggregate bandwidth logs.
41
42       --minimal
43              Print statistics in a terse, semicolon-delimited format.
44
45       --append-terse
46              Print statistics in selected mode AND terse, semicolon-delimited
47              format.   Deprecated, use --output-format instead to select mul‐
48              tiple formats.
49
50       --terse-version=version
51              Set terse version output format (default `3', or `2', `4', `5').
52
53       --version
54              Print version information and exit.
55
56       --help Print a summary of the command line options and exit.
57
58       --cpuclock-test
59              Perform test and validation of internal CPU clock.
60
61       --crctest=[test]
62              Test the speed of the built-in checksumming functions. If no ar‐
63              gument  is given, all of them are tested. Alternatively, a comma
64              separated list can be passed, in which case the given  ones  are
65              tested.
66
67       --cmdhelp=command
68              Print  help  information  for command. May be `all' for all com‐
69              mands.
70
71       --enghelp=[ioengine[,command]]
72              List all commands defined by ioengine, or print help for command
73              defined by ioengine. If no ioengine is given, list all available
74              ioengines.
75
76       --showcmd=jobfile
77              Convert jobfile to a set of command-line options.
78
79       --readonly
80              Turn on safety read-only checks, preventing  writes  and  trims.
81              The  --readonly option is an extra safety guard to prevent users
82              from accidentally starting a write or trim workload when that is
83              not  desired.  Fio  will  only  modify  the device under test if
84              `rw=write/randwrite/rw/randrw/trim/randtrim/trimwrite' is given.
85              This safety net can be used as an extra precaution.
86
87       --eta=when
88              Specifies  when  real-time  ETA estimate should be printed. when
89              may be `always', `never' or `auto'. `auto' is  the  default,  it
90              prints  ETA when requested if the output is a TTY. `always' dis‐
91              regards the output type, and prints ETA when requested.  `never'
92              never prints ETA.
93
94       --eta-interval=time
95              By default, fio requests client ETA status roughly every second.
96              With this option, the interval is configurable.  Fio  imposes  a
97              minimum  allowed  time  to avoid flooding the console, less than
98              250 msec is not supported.
99
100       --eta-newline=time
101              Force a new line for every time period passed. When the unit  is
102              omitted, the value is interpreted in seconds.
103
104       --status-interval=time
105              Force  a  full status dump of cumulative (from job start) values
106              at time intervals. This option  does  *not*  provide  per-period
107              measurements.  So values such as bandwidth are running averages.
108              When the time unit is omitted, time is interpreted  in  seconds.
109              Note  that  using  this  option with `--output-format=json' will
110              yield output that technically isn't valid json, since the output
111              will  be  collated  sets of valid json. It will need to be split
112              into valid sets of json after the run.
113
114       --section=name
115              Only run specified section name in job file.  Multiple  sections
116              can  be  specified.   The --section option allows one to combine
117              related jobs into one file.  E.g.  one  job  file  could  define
118              light,  moderate,  and  heavy sections. Tell fio to run only the
119              "heavy" section by giving `--section=heavy' command line option.
120              One  can  also specify the "write" operations in one section and
121              "verify" operation in another section. The --section option only
122              applies to job sections. The reserved *global* section is always
123              parsed and used.
124
125       --alloc-size=kb
126              Allocate additional internal smalloc pools of size  kb  in  KiB.
127              The  --alloc-size  option  increases shared memory set aside for
128              use by fio.  If running large jobs with randommap  enabled,  fio
129              can  run  out  of  memory.  Smalloc is an internal allocator for
130              shared structures from a fixed size memory pool and can grow  to
131              16  pools. The pool size defaults to 16MiB.  NOTE: While running
132              `.fio_smalloc.*' backing store files are visible in `/tmp'.
133
134       --warnings-fatal
135              All fio parser warnings are fatal, causing fio to exit  with  an
136              error.
137
138       --max-jobs=nr
139              Set  the  maximum  number of threads/processes to support to nr.
140              NOTE: On Linux, it may be necessary to increase the  shared-mem‐
141              ory  limit  (`/proc/sys/kernel/shmmax')  if fio runs into errors
142              while creating jobs.
143
144       --server=args
145              Start a backend server, with args specifying what to listen  to.
146              See CLIENT/SERVER section.
147
148       --daemonize=pidfile
149              Background  a  fio  server, writing the pid to the given pidfile
150              file.
151
152       --client=hostname
153              Instead of running the jobs locally, send and run  them  on  the
154              given hostname or set of hostnames. See CLIENT/SERVER section.
155
156       --remote-config=file
157              Tell fio server to load this local file.
158
159       --idle-prof=option
160              Report CPU idleness. option is one of the following:
161
162                     calibrate
163                            Run unit work calibration only and exit.
164
165                     system Show aggregate system idleness and unit work.
166
167                     percpu As system but also show per CPU idleness.
168
169       --inflate-log=log
170              Inflate and output compressed log.
171
172       --trigger-file=file
173              Execute trigger command when file exists.
174
175       --trigger-timeout=time
176              Execute trigger at this time.
177
178       --trigger=command
179              Set this command as local trigger.
180
181       --trigger-remote=command
182              Set this command as remote trigger.
183
184       --aux-path=path
185              Use  the  directory  specified by path for generated state files
186              instead of the current working directory.
187

JOB FILE FORMAT

189       Any parameters following the options will be assumed to be  job  files,
190       unless  they  match  a  job  file  parameter. Multiple job files can be
191       listed and each job file will be regarded as a separate group. Fio will
192       stonewall execution between each group.
193
194       Fio accepts one or more job files describing what it is supposed to do.
195       The job file format is the classic ini file, where the  names  enclosed
196       in  [] brackets define the job name. You are free to use any ASCII name
197       you want, except *global* which has special meaning. Following the  job
198       name  is  a sequence of zero or more parameters, one per line, that de‐
199       fine the behavior of the job. If the first character in a line is a ';'
200       or a '#', the entire line is discarded as a comment.
201
202       A *global* section sets defaults for the jobs described in that file. A
203       job may override a *global* section parameter, and a job file may  even
204       have several *global* sections if so desired. A job is only affected by
205       a *global* section residing above it.
206
207       The --cmdhelp option also lists all options. If used  with  an  command
208       argument, --cmdhelp will detail the given command.
209
210       See  the  `examples/'  directory  for  inspiration  on how to write job
211       files. Note the copyright and license requirements currently  apply  to
212       `examples/' files.
213
214       Note that the maximum length of a line in the job file is 8192 bytes.
215

JOB FILE PARAMETERS

217       Some parameters take an option of a given type, such as an integer or a
218       string. Anywhere a numeric value is required, an arithmetic  expression
219       may be used, provided it is surrounded by parentheses. Supported opera‐
220       tors are:
221
222              addition (+)
223
224              subtraction (-)
225
226              multiplication (*)
227
228              division (/)
229
230              modulus (%)
231
232              exponentiation (^)
233
234       For time values in expressions, units are microseconds by default. This
235       is  different  than for time values not in expressions (not enclosed in
236       parentheses).
237

PARAMETER TYPES

239       The following parameter types are used.
240
241       str    String. A sequence of alphanumeric characters.
242
243       time   Integer with possible time suffix. Without a unit value  is  in‐
244              terpreted  as seconds unless otherwise specified. Accepts a suf‐
245              fix of 'd' for days, 'h' for hours, 'm'  for  minutes,  's'  for
246              seconds,  'ms' (or 'msec') for milliseconds and 'us' (or 'usec')
247              for microseconds. For example, use 10m for 10 minutes.
248
249       int    Integer. A whole number value, which may contain an integer pre‐
250              fix and an integer suffix.
251
252                     [*integer prefix*] **number** [*integer suffix*]
253
254              The  optional  *integer prefix* specifies the number's base. The
255              default is decimal. *0x* specifies hexadecimal.
256
257              The optional *integer suffix* specifies the number's units,  and
258              includes an optional unit prefix and an optional unit. For quan‐
259              tities of data, the default unit is  bytes.  For  quantities  of
260              time, the default unit is seconds unless otherwise specified.
261
262              With  `kb_base=1000',  fio  follows  international standards for
263              unit prefixes. To specify power-of-10 decimal values defined  in
264              the International System of Units (SI):
265
266                     K means kilo (K) or 1000
267                     M means mega (M) or 1000**2
268                     G means giga (G) or 1000**3
269                     T means tera (T) or 1000**4
270                     P means peta (P) or 1000**5
271
272              To specify power-of-2 binary values defined in IEC 80000-13:
273
274                     Ki means kibi (Ki) or 1024
275                     Mi means mebi (Mi) or 1024**2
276                     Gi means gibi (Gi) or 1024**3
277                     Ti means tebi (Ti) or 1024**4
278                     Pi means pebi (Pi) or 1024**5
279
280              For Zone Block Device Mode:
281
282                     z means Zone
283              With  `kb_base=1024'  (the default), the unit prefixes are oppo‐
284              site from those specified in the SI and IEC  80000-13  standards
285              to provide compatibility with old scripts. For example, 4k means
286              4096.
287
288              For quantities of data, an optional unit of 'B' may be  included
289              (e.g., 'kB' is the same as 'k').
290
291              The  *integer  suffix*  is  not  case sensitive (e.g., m/mi mean
292              mebi/mega, not milli). 'b' and 'B' both mean byte, not bit.
293
294              Examples with `kb_base=1000':
295
296                     4 KiB: 4096, 4096b, 4096B, 4k, 4kb, 4kB, 4K, 4KB
297                     1 MiB: 1048576, 1m, 1024k
298                     1 MB: 1000000, 1mi, 1000ki
299                     1 TiB: 1073741824, 1t, 1024m, 1048576k
300                     1 TB: 1000000000, 1ti, 1000mi, 1000000ki
301
302              Examples with `kb_base=1024' (default):
303
304                     4 KiB: 4096, 4096b, 4096B, 4k, 4kb, 4kB, 4K, 4KB
305                     1 MiB: 1048576, 1m, 1024k
306                     1 MB: 1000000, 1mi, 1000ki
307                     1 TiB: 1073741824, 1t, 1024m, 1048576k
308                     1 TB: 1000000000, 1ti, 1000mi, 1000000ki
309
310              To specify times (units are not case sensitive):
311
312                     D means days
313                     H means hours
314                     M mean minutes
315                     s or sec means seconds (default)
316                     ms or msec means milliseconds
317                     us or usec means microseconds
318
319              `z' suffix specifies that the value is measured in zones.  Value
320              is recalculated once block device's zone size becomes known.
321
322              If  the option accepts an upper and lower range, use a colon ':'
323              or minus '-' to separate such values. See irange parameter type.
324              If the lower value specified happens to be larger than the upper
325              value the two values are swapped.
326
327       bool   Boolean. Usually parsed as an integer, however only defined  for
328              true and false (1 and 0).
329
330       irange Integer  range with suffix. Allows value range to be given, such
331              as 1024-4096. A colon may also be used as  the  separator,  e.g.
332              1k:4k.  If  the  option  allows  two sets of ranges, they can be
333              specified with a ',' or '/' delimiter:  1k-4k/8k-32k.  Also  see
334              int parameter type.
335
336       float_list
337              A list of floating point numbers, separated by a ':' character.
338

JOB PARAMETERS

340       With  the  above in mind, here follows the complete list of fio job pa‐
341       rameters.
342
343   Units
344       kb_base=int
345              Select the interpretation of unit prefixes in input parameters.
346
347                     1000   Inputs comply with IEC 80000-13 and  the  Interna‐
348                            tional System of Units (SI). Use:
349
350                            - power-of-2 values with IEC prefixes (e.g., KiB)
351                            - power-of-10 values with SI prefixes (e.g., kB)
352
353                     1024   Compatibility  mode  (default).  To avoid breaking
354                            old scripts:
355
356                            - power-of-2 values with SI prefixes
357                            - power-of-10 values with IEC prefixes
358
359              See bs for more details on input parameters.
360
361              Outputs always use correct prefixes. Most outputs  include  both
362              side-by-side, like:
363
364                     bw=2383.3kB/s (2327.4KiB/s)
365
366              If  only  one value is reported, then kb_base selects the one to
367              use:
368
369                     1000 -- SI prefixes
370                     1024 -- IEC prefixes
371
372       unit_base=int
373              Base unit for reporting. Allowed values are:
374
375                     0      Use auto-detection (default).
376
377                     8      Byte based.
378
379                     1      Bit based.
380
381   Job description
382       name=str
383              ASCII name of the job. This may be used  to  override  the  name
384              printed  by fio for this job. Otherwise the job name is used. On
385              the command line this parameter has the special purpose of  also
386              signaling the start of a new job.
387
388       description=str
389              Text  description  of  the  job. Doesn't do anything except dump
390              this text description when this job is run. It's not parsed.
391
392       loops=int
393              Run the specified number of iterations of this job. Used to  re‐
394              peat the same workload a given number of times. Defaults to 1.
395
396       numjobs=int
397              Create the specified number of clones of this job. Each clone of
398              job is spawned as an independent thread or process. May be  used
399              to  setup  a  larger  number of threads/processes doing the same
400              thing. Each thread is reported separately; to see statistics for
401              all  clones  as a whole, use group_reporting in conjunction with
402              new_group.  See --max-jobs. Default: 1.
403
404   Time related parameters
405       runtime=time
406              Tell fio to terminate processing after the specified  period  of
407              time. It can be quite hard to determine for how long a specified
408              job will run, so this parameter is handy to cap the  total  run‐
409              time to a given time. When the unit is omitted, the value is in‐
410              terpreted in seconds.
411
412       time_based
413              If set, fio will run for the duration of the  runtime  specified
414              even if the file(s) are completely read or written. It will sim‐
415              ply loop over the same workload as many times as the runtime al‐
416              lows.
417
418       startdelay=irange(int)
419              Delay  the start of job for the specified amount of time. Can be
420              a single value or a range. When given as a  range,  each  thread
421              will  choose a value randomly from within the range. Value is in
422              seconds if a unit is omitted.
423
424       ramp_time=time
425              If set, fio will run the specified workload for this  amount  of
426              time  before logging any performance numbers. Useful for letting
427              performance settle before logging results, thus  minimizing  the
428              runtime  required for stable results. Note that the ramp_time is
429              considered lead in time for a job, thus it will increase the to‐
430              tal  runtime  if a special timeout or runtime is specified. When
431              the unit is omitted, the value is given in seconds.
432
433       clocksource=str
434              Use the given clocksource as the base of timing.  The  supported
435              options are:
436
437                     gettimeofday
438                            gettimeofday(2)
439
440                     clock_gettime
441                            clock_gettime(2)
442
443                     cpu    Internal CPU clock source
444
445              cpu  is  the  preferred  clocksource if it is reliable, as it is
446              very fast (and fio is heavy on time calls). Fio  will  automati‐
447              cally  use this clocksource if it's supported and considered re‐
448              liable on the system it is running  on,  unless  another  clock‐
449              source is specifically set. For x86/x86-64 CPUs, this means sup‐
450              porting TSC Invariant.
451
452       gtod_reduce=bool
453              Enable  all  of  the  gettimeofday(2)  reducing  options   (dis‐
454              able_clat,  disable_slat,  disable_bw_measurement)  plus  reduce
455              precision of the timeout somewhat to really shrink the  gettime‐
456              ofday(2)  call count. With this option enabled, we only do about
457              0.4% of the gettimeofday(2) calls we would have done if all time
458              keeping was enabled.
459
460       gtod_cpu=int
461              Sometimes  it's cheaper to dedicate a single thread of execution
462              to just getting the current time. Fio (and  databases,  for  in‐
463              stance)  are  very intensive on gettimeofday(2) calls. With this
464              option, you can set one CPU aside for doing nothing but  logging
465              current  time  to  a  shared  memory  location.  Then  the other
466              threads/processes that run I/O workloads  need  only  copy  that
467              segment,  instead  of entering the kernel with a gettimeofday(2)
468              call. The CPU set aside for doing these time calls will  be  ex‐
469              cluded  from other uses. Fio will manually clear it from the CPU
470              mask of other jobs.
471
472   Target file/device
473       directory=str
474              Prefix filenames with this directory. Used to place files  in  a
475              different location than `./'. You can specify a number of direc‐
476              tories by separating the names with a ':' character.  These  di‐
477              rectories  will  be  assigned  equally distributed to job clones
478              created by numjobs as long as they  are  using  generated  file‐
479              names.  If  specific  filename(s) are set fio will use the first
480              listed directory, and thereby  matching  the  filename  semantic
481              (which  generates  a  file  for each clone if not specified, but
482              lets all clones use the same file if set).
483
484              See the filename option for information on  how  to  escape  ':'
485              characters within the directory path itself.
486
487              Note:  To  control the directory fio will use for internal state
488              files use --aux-path.
489
490       filename=str
491              Fio normally makes up a filename based on the job  name,  thread
492              number,  and  file  number (see filename_format). If you want to
493              share files between threads in a job or several jobs with  fixed
494              file  paths, specify a filename for each of them to override the
495              default. If the ioengine is file based, you can specify a number
496              of  files  by  separating  the names with a ':' colon. So if you
497              wanted a job to open `/dev/sda' and `/dev/sdb' as the two  work‐
498              ing files, you would use `filename=/dev/sda:/dev/sdb'. This also
499              means that whenever this option is  specified,  nrfiles  is  ig‐
500              nored.  The  size of regular files specified by this option will
501              be size divided by number of files unless an  explicit  size  is
502              specified by filesize.
503
504              Each colon in the wanted path must be escaped with a '\' charac‐
505              ter. For instance, if the path is `/dev/dsk/foo@3,0:c' then  you
506              would  use  `filename=/dev/dsk/foo@3,0\:c'  and  if  the path is
507              `F:\filename' then you would use `filename=F\:\filename'.
508
509              On Windows, disk devices are  accessed  as  `\\.\PhysicalDrive0'
510              for  the  first device, `\\.\PhysicalDrive1' for the second etc.
511              Note: Windows and FreeBSD prevent write access to areas  of  the
512              disk containing in-use data (e.g. filesystems).
513
514              The  filename  `-'  is a reserved name, meaning *stdin* or *std‐
515              out*. Which of the two depends on the read/write direction set.
516
517       filename_format=str
518              If sharing multiple files between jobs, it is usually  necessary
519              to  have fio generate the exact names that you want. By default,
520              fio will name a file based on the default file format specifica‐
521              tion  of  `jobname.jobnumber.filenumber'. With this option, that
522              can be customized. Fio will recognize and replace the  following
523              keywords in this string:
524
525                     $jobname
526                            The name of the worker thread or process.
527
528                     $clientuid
529                            IP  of  the  fio  process when using client/server
530                            mode.
531
532                     $jobnum
533                            The incremental number of  the  worker  thread  or
534                            process.
535
536                     $filenum
537                            The incremental number of the file for that worker
538                            thread or process.
539
540              To have dependent jobs share a set of files, this option can  be
541              set  to  have fio generate filenames that are shared between the
542              two. For instance, if `testfiles.$filenum'  is  specified,  file
543              number 4 for any job will be named `testfiles.4'. The default of
544              `$jobname.$jobnum.$filenum' will be  used  if  no  other  format
545              specifier is given.
546
547              If you specify a path then the directories will be created up to
548              the main directory for the file.  So for example if you  specify
549              `a/b/c/$jobnum`  then  the directories a/b/c will be created be‐
550              fore the file setup part of the job.  If you  specify  directory
551              then  the  path will be relative that directory, otherwise it is
552              treated as the absolute path.
553
554       unique_filename=bool
555              To avoid collisions between networked clients, fio  defaults  to
556              prefixing  any  generated filenames (with a directory specified)
557              with the source of the client connecting. To disable this behav‐
558              ior, set this option to 0.
559
560       opendir=str
561              Recursively open any files below directory str.
562
563       lockfile=str
564              Fio  defaults  to  not  locking  any files before it does I/O to
565              them. If a file or file descriptor is shared, fio can  serialize
566              I/O  to  that  file  to  make the end result consistent. This is
567              usual for emulating real workloads that share  files.  The  lock
568              modes are:
569
570                     none   No locking. The default.
571
572                     exclusive
573                            Only  one  thread or process may do I/O at a time,
574                            excluding all others.
575
576                     readwrite
577                            Read-write locking on the file. Many  readers  may
578                            access  the  file at the same time, but writes get
579                            exclusive access.
580
581       nrfiles=int
582              Number of files to use for this job. Defaults to 1. The size  of
583              files will be size divided by this unless explicit size is spec‐
584              ified by filesize. Files are created for each thread separately,
585              and  each  file  will  have a file number within its name by de‐
586              fault, as explained in filename section.
587
588       openfiles=int
589              Number of files to keep open at the same time. Defaults  to  the
590              same as nrfiles, can be set smaller to limit the number simulta‐
591              neous opens.
592
593       file_service_type=str
594              Defines how fio decides which file from a job to  service  next.
595              The following types are defined:
596
597                     random Choose a file at random.
598
599                     roundrobin
600                            Round  robin  over  opened  files. This is the de‐
601                            fault.
602
603                     sequential
604                            Finish one file before moving on to the next. Mul‐
605                            tiple  files  can still be open depending on open‐
606                            files.
607
608                     zipf   Use a Zipf distribution to decide what file to ac‐
609                            cess.
610
611                     pareto Use  a  Pareto distribution to decide what file to
612                            access.
613
614                     normal Use a Gaussian  (normal)  distribution  to  decide
615                            what file to access.
616
617                     gauss  Alias for normal.
618
619              For  random,  roundrobin,  and  sequential, a postfix can be ap‐
620              pended to tell fio how many I/Os to issue before switching to  a
621              new  file.  For example, specifying `file_service_type=random:8'
622              would cause fio to issue 8 I/Os before selecting a new  file  at
623              random.  For  the  non-uniform  distributions,  a floating point
624              postfix can be  given  to  influence  how  the  distribution  is
625              skewed.  See  random_distribution  for a description of how that
626              would work.
627
628       ioscheduler=str
629              Attempt to switch the device hosting the file to  the  specified
630              I/O scheduler before running. If the file is a pipe, a character
631              device file or if device hosting the file could  not  be  deter‐
632              mined, this option is ignored.
633
634       create_serialize=bool
635              If  true,  serialize the file creation for the jobs. This may be
636              handy to avoid interleaving of data files, which may greatly de‐
637              pend on the filesystem used and even the number of processors in
638              the system. Default: true.
639
640       create_fsync=bool
641              fsync(2) the data file after creation. This is the default.
642
643       create_on_open=bool
644              If true, don't pre-create files but allow the  job's  open()  to
645              create  a  file when it's time to do I/O. Default: false -- pre-
646              create all necessary files when the job starts.
647
648       create_only=bool
649              If true, fio will only run the setup phase of the job. If  files
650              need  to  be laid out or updated on disk, only that will be done
651              -- the actual job contents are not executed. Default: false.
652
653       allow_file_create=bool
654              If true, fio is permitted to create files as part of  its  work‐
655              load.  If  this  option is false, then fio will error out if the
656              files it needs to use don't already exist. Default: true.
657
658       allow_mounted_write=bool
659              If this isn't set, fio will  abort  jobs  that  are  destructive
660              (e.g. that write) to what appears to be a mounted device or par‐
661              tition. This should help catch creating  inadvertently  destruc‐
662              tive tests, not realizing that the test will destroy data on the
663              mounted file system. Note that some platforms don't allow  writ‐
664              ing against a mounted device regardless of this option. Default:
665              false.
666
667       pre_read=bool
668              If this is given, files will  be  pre-read  into  memory  before
669              starting  the  given I/O operation. This will also clear the in‐
670              validate flag, since it is pointless to pre-read and  then  drop
671              the  cache.  This  will only work for I/O engines that are seek-
672              able, since they allow you to read the same data multiple times.
673              Thus it will not work on non-seekable I/O engines (e.g. network,
674              splice). Default: false.
675
676       unlink=bool
677              Unlink the job files when done. Not  the  default,  as  repeated
678              runs  of  that job would then waste time recreating the file set
679              again and again. Default: false.
680
681       unlink_each_loop=bool
682              Unlink job files after each iteration or loop. Default: false.
683
684       zonemode=str
685              Accepted values are:
686
687                     none   The zonerange, zonesize zonecapacity and  zoneskip
688                            parameters are ignored.
689
690                     strided
691                            I/O  happens in a single zone until zonesize bytes
692                            have been transferred.  After that number of bytes
693                            has  been  transferred processing of the next zone
694                            starts. The zonecapacity parameter is ignored.
695
696                     zbd    Zoned block device mode. I/O happens  sequentially
697                            in  each  zone,  even  if  random I/O has been se‐
698                            lected. Random I/O happens across  all  zones  in‐
699                            stead  of being restricted to a single zone.  Trim
700                            is handled using a zone reset operation. Trim only
701                            considers  non-empty sequential write required and
702                            sequential write preferred zones.
703
704       zonerange=int
705              For zonemode=strided, this is the size of  a  single  zone.  See
706              also zonesize and zoneskip.
707
708              For zonemode=zbd, this parameter is ignored.
709
710       zonesize=int
711              For  zonemode=strided,  this  is the number of bytes to transfer
712              before skipping zoneskip bytes. If  this  parameter  is  smaller
713              than  zonerange then only a fraction of each zone with zonerange
714              bytes will be accessed.  If this parameter is larger  than  zon‐
715              erange  then  each  zone  will be accessed multiple times before
716              skipping to the next zone.
717
718              For zonemode=zbd, this is the size of a single  zone.  The  zon‐
719              erange  parameter is ignored in this mode. For a job accessing a
720              zoned block device, the specified zonesize must be 0 or equal to
721              the  device  zone  size. For a regular block device or file, the
722              specified zonesize must be at least 512B.
723
724       zonecapacity=int
725              For zonemode=zbd, this defines the capacity of  a  single  zone,
726              which  is  the  accessible area starting from the zone start ad‐
727              dress. This parameter only applies when  using  zonemode=zbd  in
728              combination with regular block devices.  If not specified it de‐
729              faults to the zone size. If the target device is a  zoned  block
730              device,  the  zone capacity is obtained from the device informa‐
731              tion and this option is ignored.
732
733       zoneskip=int[z]
734              For zonemode=strided, the number of bytes to skip after zonesize
735              bytes of data have been transferred.
736
737              For  zonemode=zbd,  the zonesize aligned number of bytes to skip
738              once a zone is fully written (write workloads)  or  all  written
739              data in the zone have been read (read workloads). This parameter
740              is valid only for sequential workloads and  ignored  for  random
741              workloads. For read workloads, see also read_beyond_wp.
742
743
744       read_beyond_wp=bool
745              This parameter applies to zonemode=zbd only.
746
747              Zoned  block  devices are block devices that consist of multiple
748              zones. Each zone has a type, e.g. conventional or sequential.  A
749              conventional  zone can be written at any offset that is a multi‐
750              ple of the block size. Sequential zones must be written  sequen‐
751              tially.  The  position at which a write must occur is called the
752              write pointer. A zoned block device can be either  host  managed
753              or  host  aware.  For  host managed devices the host must ensure
754              that writes happen sequentially. Fio recognizes host managed de‐
755              vices  and  serializes  writes to sequential zones for these de‐
756              vices.
757
758              If a read occurs in a sequential zone beyond the  write  pointer
759              then the zoned block device will complete the read without read‐
760              ing any data from the storage medium. Since such reads  lead  to
761              unrealistically  high  bandwidth and IOPS numbers fio only reads
762              beyond the write pointer if explicitly told to do  so.  Default:
763              false.
764
765       max_open_zones=int
766              When  running  a  random  write test across an entire drive many
767              more zones will be open than in a typical application  workload.
768              Hence  this  command line option that allows to limit the number
769              of open zones. The number of open zones is defined as the number
770              of  zones to which write commands are issued by all threads/pro‐
771              cesses.
772
773       job_max_open_zones=int
774              Limit on the number of simultaneously opened  zones  per  single
775              thread/process.
776
777       ignore_zone_limits=bool
778              If  this  option  is used, fio will ignore the maximum number of
779              open zones limit of the zoned block device in use, thus allowing
780              the option max_open_zones value to be larger than the device re‐
781              ported limit. Default: false.
782
783       zone_reset_threshold=float
784              A number between zero and one that indicates the ratio of  logi‐
785              cal  blocks  with  data to the total number of logical blocks in
786              the test above which zones should be reset periodically.
787
788       zone_reset_frequency=float
789              A number between zero and one that indicates how  often  a  zone
790              reset  should be issued if the zone reset threshold has been ex‐
791              ceeded. A zone reset is  submitted  after  each  (1  /  zone_re‐
792              set_frequency)  write  requests. This and the previous parameter
793              can be used to simulate garbage collection activity.
794
795
796   I/O type
797       direct=bool
798              If value is true, use non-buffered I/O. This  is  usually  O_DI‐
799              RECT.  Note that OpenBSD and ZFS on Solaris don't support direct
800              I/O. On Windows the synchronous ioengines don't  support  direct
801              I/O. Default: false.
802
803       atomic=bool
804              If  value  is  true,  attempt  to  use atomic direct I/O. Atomic
805              writes are guaranteed to be stable once acknowledged by the  op‐
806              erating system. Only Linux supports O_ATOMIC right now.
807
808       buffered=bool
809              If  value is true, use buffered I/O. This is the opposite of the
810              direct option. Defaults to true.
811
812       readwrite=str, rw=str
813              Type of I/O pattern. Accepted values are:
814
815                     read   Sequential reads.
816
817                     write  Sequential writes.
818
819                     trim   Sequential trims (Linux  block  devices  and  SCSI
820                            character devices only).
821
822                     randread
823                            Random reads.
824
825                     randwrite
826                            Random writes.
827
828                     randtrim
829                            Random trims (Linux block devices and SCSI charac‐
830                            ter devices only).
831
832                     rw,readwrite
833                            Sequential mixed reads and writes.
834
835                     randrw Random mixed reads and writes.
836
837                     trimwrite
838                            Sequential trim+write sequences.  Blocks  will  be
839                            trimmed  first, then the same blocks will be writ‐
840                            ten to.
841
842              Fio defaults to read if the option is  not  specified.  For  the
843              mixed I/O types, the default is to split them 50/50. For certain
844              types of I/O the result may still be skewed  a  bit,  since  the
845              speed may be different.
846
847              It  is  possible to specify the number of I/Os to do before get‐
848              ting a new offset by appending `:<nr>' to the end of the  string
849              given. For a random read, it would look like `rw=randread:8' for
850              passing in an offset modifier with a value of 8. If  the  suffix
851              is  used  with  a  sequential I/O pattern, then the `<nr>' value
852              specified will be added to the generated  offset  for  each  I/O
853              turning  sequential I/O into sequential I/O with holes.  For in‐
854              stance, using `rw=write:4k' will skip 4k for every  write.  Also
855              see the rw_sequencer option.
856
857       rw_sequencer=str
858              If  an  offset  modifier  is  given by appending a number to the
859              `rw=str' line, then this option controls how that  number  modi‐
860              fies the I/O offset being generated. Accepted values are:
861
862                     sequential
863                            Generate sequential offset.
864
865                     identical
866                            Generate the same offset.
867
868              sequential  is  only useful for random I/O, where fio would nor‐
869              mally generate a new random offset for every I/O. If you  append
870              e.g.  8 to randread, you would get a new random offset for every
871              8 I/Os. The result would be a seek for only every  8  I/Os,  in‐
872              stead  of for every I/O. Use `rw=randread:8' to specify that. As
873              sequential I/O is already  sequential,  setting  sequential  for
874              that would not result in any differences. identical behaves in a
875              similar fashion, except it sends the same  offset  8  number  of
876              times before generating a new offset.
877
878       unified_rw_reporting=str
879              Fio  normally  reports statistics on a per data direction basis,
880              meaning that reads, writes, and trims are accounted and reported
881              separately.  This  option determines whether fio reports the re‐
882              sults normally, summed together, or as both  options.   Accepted
883              values are:
884
885              none   Normal statistics reporting.
886
887              mixed  Statistics are summed per data direction and reported to‐
888                     gether.
889
890              both   Statistics are reported normally, followed by  the  mixed
891                     statistics.
892
893              0      Backward-compatible alias for none.
894
895              1      Backward-compatible alias for mixed.
896
897              2      Alias for both.
898
899       randrepeat=bool
900              Seed the random number generator used for random I/O patterns in
901              a predictable way so the pattern is repeatable across runs.  De‐
902              fault: true.
903
904       allrandrepeat=bool
905              Seed  all  random  number generators in a predictable way so re‐
906              sults are repeatable across runs. Default: false.
907
908       randseed=int
909              Seed the random number generators based on this seed  value,  to
910              be  able  to control what sequence of output is being generated.
911              If not set, the random sequence depends on the  randrepeat  set‐
912              ting.
913
914       fallocate=str
915              Whether pre-allocation is performed when laying down files.  Ac‐
916              cepted values are:
917
918                     none   Do not pre-allocate space.
919
920                     native Use a platform's native  pre-allocation  call  but
921                            fall  back to none behavior if it fails/is not im‐
922                            plemented.
923
924                     posix  Pre-allocate via posix_fallocate(3).
925
926                     keep   Pre-allocate   via    fallocate(2)    with    FAL‐
927                            LOC_FL_KEEP_SIZE set.
928
929                     truncate
930                            Extend  file to final size using ftruncate|(2) in‐
931                            stead of allocating.
932
933                     0      Backward-compatible alias for none.
934
935                     1      Backward-compatible alias for posix.
936
937              May not be available on all supported platforms.  keep  is  only
938              available  on  Linux. If using ZFS on Solaris this cannot be set
939              to posix because ZFS doesn't  support  pre-allocation.  Default:
940              native  if any pre-allocation methods except truncate are avail‐
941              able, none if not.
942
943              Note that using truncate on Windows will  interact  surprisingly
944              with  non-sequential write patterns. When writing to a file that
945              has been extended by setting the end-of-file  information,  Win‐
946              dows  will backfill the unwritten portion of the file up to that
947              offset with zeroes before issuing the new write. This means that
948              a  single  small write to the end of an extended file will stall
949              until the entire file has been filled with zeroes.
950
951       fadvise_hint=str
952              Use posix_fadvise(2) or posix_madvise(2) to  advise  the  kernel
953              what I/O patterns are likely to be issued. Accepted values are:
954
955                     0      Backwards compatible hint for "no hint".
956
957                     1      Backwards  compatible  hint  for  "advise with fio
958                            workload type". This uses FADV_RANDOM for a random
959                            workload,  and  FADV_SEQUENTIAL  for  a sequential
960                            workload.
961
962                     sequential
963                            Advise using FADV_SEQUENTIAL.
964
965                     random Advise using FADV_RANDOM.
966
967       write_hint=str
968              Use fcntl(2) to advise the kernel what life time to expect  from
969              a  write.  Only supported on Linux, as of version 4.13. Accepted
970              values are:
971
972                     none   No particular life time associated with this file.
973
974                     short  Data written to this file has a short life time.
975
976                     medium Data written to this file has a medium life time.
977
978                     long   Data written to this file has a long life time.
979
980                     extreme
981                            Data written to this file has  a  very  long  life
982                            time.
983
984              The values are all relative to each other, and no absolute mean‐
985              ing should be associated with them.
986
987       offset=int[%|z]
988              Start I/O at the provided offset in the file, given as either  a
989              fixed  size  in bytes, zones or a percentage. If a percentage is
990              given, the generated offset  will  be  aligned  to  the  minimum
991              blocksize  or to the value of offset_align if provided. Data be‐
992              fore the given offset will not be touched. This effectively caps
993              the file size at `real_size - offset'. Can be combined with size
994              to constrain the start and end range of  the  I/O  workload.   A
995              percentage  can  be specified by a number between 1 and 100 fol‐
996              lowed by '%', for example, `offset=20%' to specify 20%.  In  ZBD
997              mode, value can be set as number of zones using 'z'.
998
999       offset_align=int
1000              If  set  to  non-zero value, the byte offset generated by a per‐
1001              centage offset is aligned upwards to this value. Defaults  to  0
1002              meaning that a percentage offset is aligned to the minimum block
1003              size.
1004
1005       offset_increment=int[%|z]
1006              If this is provided, then the real offset becomes `offset + off‐
1007              set_increment  *  thread_number',  where  the thread number is a
1008              counter that starts at 0 and is  incremented  for  each  sub-job
1009              (i.e.  when  numjobs option is specified). This option is useful
1010              if there are several jobs which are intended  to  operate  on  a
1011              file  in  parallel  disjoint segments, with even spacing between
1012              the starting points. Percentages can be used  for  this  option.
1013              If  a  percentage is given, the generated offset will be aligned
1014              to the minimum blocksize or to the value of offset_align if pro‐
1015              vided.In  ZBD  mode,  value  can be set as number of zones using
1016              'z'.
1017
1018       number_ios=int
1019              Fio will normally perform I/Os until it has exhausted  the  size
1020              of  the  region set by size, or if it exhaust the allocated time
1021              (or hits an error condition). With this setting, the  range/size
1022              can  be set independently of the number of I/Os to perform. When
1023              fio reaches this number, it will exit normally and  report  sta‐
1024              tus.  Note that this does not extend the amount of I/O that will
1025              be done, it will only stop fio if this condition is  met  before
1026              other end-of-job criteria.
1027
1028       fsync=int
1029              If  writing  to a file, issue an fsync(2) (or its equivalent) of
1030              the dirty data for every number of blocks given. For example, if
1031              you  give  32 as a parameter, fio will sync the file after every
1032              32 writes issued. If fio is using non-buffered I/O, we  may  not
1033              sync  the  file.  The exception is the sg I/O engine, which syn‐
1034              chronizes the disk cache anyway. Defaults to 0, which means  fio
1035              does  not  periodically  issue  and wait for a sync to complete.
1036              Also see end_fsync and fsync_on_close.
1037
1038       fdatasync=int
1039              Like fsync but uses fdatasync(2) to only sync data and not meta‐
1040              data  blocks.  In Windows, FreeBSD, DragonFlyBSD or OSX there is
1041              no fdatasync(2) so this falls back to using fsync(2).   Defaults
1042              to 0, which means fio does not periodically issue and wait for a
1043              data-only sync to complete.
1044
1045       write_barrier=int
1046              Make every N-th write a barrier write.
1047
1048       sync_file_range=str:int
1049              Use sync_file_range(2) for every int number of write operations.
1050              Fio will track range of writes that have happened since the last
1051              sync_file_range(2) call. str can currently be one or more of:
1052
1053                     wait_before
1054                            SYNC_FILE_RANGE_WAIT_BEFORE
1055
1056                     write  SYNC_FILE_RANGE_WRITE
1057
1058                     wait_after
1059                            SYNC_FILE_RANGE_WRITE_AFTER
1060
1061              So if you do  `sync_file_range=wait_before,write:8',  fio  would
1062              use  `SYNC_FILE_RANGE_WAIT_BEFORE  |  SYNC_FILE_RANGE_WRITE' for
1063              every 8 writes. Also see the sync_file_range(2) man  page.  This
1064              option is Linux specific.
1065
1066       overwrite=bool
1067              If  true,  writes to a file will always overwrite existing data.
1068              If the file doesn't already exist, it will be created before the
1069              write  phase  begins. If the file exists and is large enough for
1070              the specified write phase, nothing will be done. Default: false.
1071
1072       end_fsync=bool
1073              If true, fsync(2) file contents when  a  write  stage  has  com‐
1074              pleted.  Default: false.
1075
1076       fsync_on_close=bool
1077              If  true,  fio will fsync(2) a dirty file on close. This differs
1078              from end_fsync in that it will happen on every file  close,  not
1079              just at the end of the job. Default: false.
1080
1081       rwmixread=int
1082              Percentage  of  a  mixed workload that should be reads. Default:
1083              50.
1084
1085       rwmixwrite=int
1086              Percentage of a mixed workload that should be  writes.  If  both
1087              rwmixread  and  rwmixwrite is given and the values do not add up
1088              to 100%, the latter of the two will  be  used  to  override  the
1089              first.  This  may interfere with a given rate setting, if fio is
1090              asked to limit reads or writes to a certain rate. If that is the
1091              case, then the distribution may be skewed. Default: 50.
1092
1093       random_distribution=str:float[:float][,str:float][,str:float]
1094              By  default,  fio will use a completely uniform random distribu‐
1095              tion when asked to perform random I/O. Sometimes it is useful to
1096              skew the distribution in specific ways, ensuring that some parts
1097              of the data is more hot than others.  fio includes the following
1098              distribution models:
1099
1100                     random Uniform random distribution
1101
1102                     zipf   Zipf distribution
1103
1104                     pareto Pareto distribution
1105
1106                     normal Normal (Gaussian) distribution
1107
1108                     zoned  Zoned random distribution zoned_abs Zoned absolute
1109                            random distribution
1110
1111              When using a zipf or pareto distribution, an input value is also
1112              needed to define the access pattern. For zipf, this is the `Zipf
1113              theta'.  For pareto, it's the `Pareto  power'.  Fio  includes  a
1114              test  program,  fio-genzipf, that can be used visualize what the
1115              given input values will yield in terms  of  hit  rates.  If  you
1116              wanted  to  use  zipf with a `theta' of 1.2, you would use `ran‐
1117              dom_distribution=zipf:1.2' as the option. If a non-uniform model
1118              is  used, fio will disable use of the random map. For the normal
1119              distribution, a normal (Gaussian) deviation  is  supplied  as  a
1120              value between 0 and 100.
1121
1122              The  second, optional float is allowed for pareto, zipf and nor‐
1123              mal distributions. It allows to set base of distribution in non-
1124              default  place,  giving more control over most probable outcome.
1125              This value is in range [0-1] which maps  linearly  to  range  of
1126              possible  random  values.   Defaults  are: random for pareto and
1127              zipf, and 0.5 for normal.  If you wanted  to  use  zipf  with  a
1128              `theta` of 1.2 centered on 1/4 of allowed value range, you would
1129              use `random_distibution=zipf:1.2:0.25`.
1130
1131              For a zoned distribution, fio supports specifying percentages of
1132              I/O access that should fall within what range of the file or de‐
1133              vice. For example, given a criteria of:
1134
1135                     60% of accesses should be to the first 10%
1136                     30% of accesses should be to the next 20%
1137                     8% of accesses should be to the next 30%
1138                     2% of accesses should be to the next 40%
1139
1140              we can define that through zoning of the  random  accesses.  For
1141              the above example, the user would do:
1142
1143                     random_distribution=zoned:60/10:30/20:8/30:2/40
1144
1145              A  zoned_abs  distribution  works  exactly like thezoned, except
1146              that it takes absolute sizes. For example, let's say you  wanted
1147              to define access according to the following criteria:
1148
1149                     60% of accesses should be to the first 20G
1150                     30% of accesses should be to the next 100G
1151                     10% of accesses should be to the next 500G
1152
1153              we can define an absolute zoning distribution with:
1154
1155                     random_distribution=zoned:60/10:30/20:8/30:2/40
1156
1157              For  both  zoned  and zoned_abs, fio supports defining up to 256
1158              separate zones.
1159
1160              Similarly to how bssplit works for setting ranges  and  percent‐
1161              ages of block sizes. Like bssplit, it's possible to specify sep‐
1162              arate zones for reads, writes, and trims. If  just  one  set  is
1163              given, it'll apply to all of them.
1164
1165       percentage_random=int[,int][,int]
1166              For  a  random workload, set how big a percentage should be ran‐
1167              dom. This defaults to 100%, in which case the workload is  fully
1168              random. It can be set from anywhere from 0 to 100. Setting it to
1169              0 would make the workload fully sequential. Any setting  in  be‐
1170              tween  will result in a random mix of sequential and random I/O,
1171              at the given percentages. Comma-separated values may  be  speci‐
1172              fied for reads, writes, and trims as described in blocksize.
1173
1174       norandommap
1175              Normally  fio will cover every block of the file when doing ran‐
1176              dom I/O. If this option is given, fio will just get a new random
1177              offset without looking at past I/O history. This means that some
1178              blocks may not be read or written, and that some blocks  may  be
1179              read/written  more than once. If this option is used with verify
1180              and multiple blocksizes (via bsrange), only  intact  blocks  are
1181              verified,  i.e., partially-overwritten blocks are ignored.  With
1182              an async I/O engine and an I/O depth > 1, it is possible for the
1183              same  block  to be overwritten, which can cause verification er‐
1184              rors.  Either do not use norandommap in this case, or  also  use
1185              the lfsr random generator.
1186
1187       softrandommap=bool
1188              See  norandommap.  If fio runs with the random block map enabled
1189              and it fails to allocate the map, if this option is set it  will
1190              continue  without a random block map. As coverage will not be as
1191              complete as with random maps, this option  is  disabled  by  de‐
1192              fault.
1193
1194       random_generator=str
1195              Fio  supports  the  following engines for generating I/O offsets
1196              for random I/O:
1197
1198                     tausworthe
1199                            Strong 2^88 cycle random number generator.
1200
1201                     lfsr   Linear feedback shift register generator.
1202
1203                     tausworthe64
1204                            Strong 64-bit 2^258 cycle random number generator.
1205
1206              tausworthe is a strong random number generator, but it  requires
1207              tracking  on  the side if we want to ensure that blocks are only
1208              read or written once. lfsr guarantees that we never generate the
1209              same offset twice, and it's also less computationally expensive.
1210              It's not a true random generator, however, though for  I/O  pur‐
1211              poses  it's  typically  good enough. lfsr only works with single
1212              block sizes, not with workloads that use multiple  block  sizes.
1213              If  used with such a workload, fio may read or write some blocks
1214              multiple times. The default value is tausworthe, unless the  re‐
1215              quired  space exceeds 2^32 blocks. If it does, then tausworthe64
1216              is selected automatically.
1217
1218   Block size
1219       blocksize=int[,int][,int], bs=int[,int][,int]
1220              The block size in bytes used for I/O  units.  Default:  4096.  A
1221              single  value  applies  to reads, writes, and trims. Comma-sepa‐
1222              rated values may be specified for reads, writes,  and  trims.  A
1223              value not terminated in a comma applies to subsequent types. Ex‐
1224              amples:
1225
1226                     bs=256k        means 256k for reads, writes and trims.
1227                     bs=8k,32k      means 8k for reads,  32k  for  writes  and
1228                     trims.
1229                     bs=8k,32k,      means  8k  for reads, 32k for writes, and
1230                     default for trims.
1231                     bs=,8k         means default for reads, 8k for writes and
1232                     trims.
1233                     bs=,8k,         means  default  for reads, 8k for writes,
1234                     and default for trims.
1235
1236       blocksize_range=irange[,irange][,irange],
1237       bsrange=irange[,irange][,irange]
1238              A  range  of  block sizes in bytes for I/O units. The issued I/O
1239              unit will always be a  multiple  of  the  minimum  size,  unless
1240              blocksize_unaligned is set.  Comma-separated ranges may be spec‐
1241              ified for reads, writes, and trims as  described  in  blocksize.
1242              Example:
1243
1244                     bsrange=1k-4k,2k-8k
1245
1246       bssplit=str[,str][,str]
1247              Sometimes you want even finer grained control of the block sizes
1248              issued, not just an even split between them. This option  allows
1249              you  to  weight various block sizes, so that you are able to de‐
1250              fine a specific amount of block sizes  issued.  The  format  for
1251              this option is:
1252
1253                     bssplit=blocksize/percentage:blocksize/percentage
1254
1255              for  as  many  block sizes as needed. So if you want to define a
1256              workload that has 50% 64k blocks, 10% 4k  blocks,  and  40%  32k
1257              blocks, you would write:
1258
1259                     bssplit=4k/10:64k/50:32k/40
1260
1261              Ordering  does  not matter. If the percentage is left blank, fio
1262              will fill in the remaining values evenly. So  a  bssplit  option
1263              like this one:
1264
1265                     bssplit=4k/50:1k/:32k/
1266
1267              would  have  50% 4k ios, and 25% 1k and 32k ios. The percentages
1268              always add up to 100, if bssplit is given a range that  adds  up
1269              to more, it will error out.
1270
1271              Comma-separated  values  may be specified for reads, writes, and
1272              trims as described in blocksize.
1273
1274              If you want a workload that has 50% 2k reads and 50%  4k  reads,
1275              while having 90% 4k writes and 10% 8k writes, you would specify:
1276
1277                     bssplit=2k/50:4k/50,4k/90:8k/10
1278
1279              Fio  supports  defining up to 64 different weights for each data
1280              direction.
1281
1282       blocksize_unaligned, bs_unaligned
1283              If set, fio will issue I/O units with  any  size  within  block‐
1284              size_range,  not  just multiples of the minimum size. This typi‐
1285              cally won't work with direct I/O, as that normally requires sec‐
1286              tor alignment.
1287
1288       bs_is_seq_rand=bool
1289              If this option is set, fio will use the normal read,write block‐
1290              size settings as sequential,random blocksize  settings  instead.
1291              Any  random read or write will use the WRITE blocksize settings,
1292              and any sequential read or write will  use  the  READ  blocksize
1293              settings.
1294
1295       blockalign=int[,int][,int], ba=int[,int][,int]
1296              Boundary  to  which  fio  will  align random I/O units. Default:
1297              blocksize. Minimum alignment is typically 512b for using  direct
1298              I/O,  though it usually depends on the hardware block size. This
1299              option is mutually exclusive with using a random map for  files,
1300              so  it  will turn off that option. Comma-separated values may be
1301              specified for reads, writes, and trims as  described  in  block‐
1302              size.
1303
1304   Buffers and memory
1305       zero_buffers
1306              Initialize  buffers  with  all zeros. Default: fill buffers with
1307              random data.
1308
1309       refill_buffers
1310              If this option is given, fio will refill the I/O buffers on  ev‐
1311              ery  submit. The default is to only fill it at init time and re‐
1312              use that data. Only makes sense if zero_buffers isn't specified,
1313              naturally.  If  data  verification is enabled, refill_buffers is
1314              also automatically enabled.
1315
1316       scramble_buffers=bool
1317              If refill_buffers is too costly and the  target  is  using  data
1318              deduplication, then setting this option will slightly modify the
1319              I/O buffer contents to defeat normal de-dupe attempts.  This  is
1320              not enough to defeat more clever block compression attempts, but
1321              it will stop naive dedupe of blocks. Default: true.
1322
1323       buffer_compress_percentage=int
1324              If this is set, then fio will attempt to provide I/O buffer con‐
1325              tent  (on  WRITEs)  that  compresses to the specified level. Fio
1326              does this by providing a mix of random data  followed  by  fixed
1327              pattern  data. The fixed pattern is either zeros, or the pattern
1328              specified by buffer_pattern. If  the  buffer_pattern  option  is
1329              used, it might skew the compression ratio slightly. Setting buf‐
1330              fer_compress_percentage to a value other than 100 will also  en‐
1331              able refill_buffers in order to reduce the likelihood that adja‐
1332              cent blocks are so similar that they over compress when seen to‐
1333              gether.  See  buffer_compress_chunk  for  how  to set a finer or
1334              coarser granularity of the random/fixed data  regions.  Defaults
1335              to  unset  i.e.,  buffer data will not adhere to any compression
1336              level.
1337
1338       buffer_compress_chunk=int
1339              This setting allows fio to manage how big the random/fixed  data
1340              region  is  when  using  buffer_compress_percentage.  When  buf‐
1341              fer_compress_chunk is set to some non-zero  value  smaller  than
1342              the  block size, fio can repeat the random/fixed region through‐
1343              out the I/O buffer at the specified interval (which particularly
1344              useful  when bigger block sizes are used for a job). When set to
1345              0, fio will use a chunk size that matches the block size result‐
1346              ing  in  a single random/fixed region within the I/O buffer. De‐
1347              faults to 512. When the unit is omitted,  the  value  is  inter‐
1348              preted in bytes.
1349
1350       buffer_pattern=str
1351              If  set, fio will fill the I/O buffers with this pattern or with
1352              the contents of a file. If not set, the contents of I/O  buffers
1353              are defined by the other options related to buffer contents. The
1354              setting can be any pattern of bytes, and can be prefixed with 0x
1355              for  hex  values. It may also be a string, where the string must
1356              then be wrapped with "". Or it may also be a filename, where the
1357              filename  must  be  wrapped  with  ''  in which case the file is
1358              opened and read. Note that not all the  file  contents  will  be
1359              read  if that would cause the buffers to overflow. So, for exam‐
1360              ple:
1361
1362                     buffer_pattern='filename'
1363                     or:
1364                     buffer_pattern="abcd"
1365                     or:
1366                     buffer_pattern=-12
1367                     or:
1368                     buffer_pattern=0xdeadface
1369
1370              Also you can combine everything together in any order:
1371
1372                     buffer_pattern=0xdeadface"abcd"-12'filename'
1373
1374       dedupe_percentage=int
1375              If set, fio will generate this percentage of  identical  buffers
1376              when  writing.  These  buffers  will be naturally dedupable. The
1377              contents of the buffers depend on what other buffer  compression
1378              settings  have  been  set.  It's possible to have the individual
1379              buffers either fully compressible, or not at all -- this  option
1380              only  controls  the distribution of unique buffers. Setting this
1381              option will also enable refill_buffers to prevent  every  buffer
1382              being identical.
1383
1384       dedupe_mode=str
1385              If dedupe_percentage is given, then this option controls how fio
1386              generates the dedupe buffers.
1387
1388                     repeat
1389
1390                            Generate  dedupe  buffers  by  repeating  previous
1391                            writes
1392
1393                     working_set
1394
1395                            Generate dedupe buffers from working set
1396
1397              repeat  is the default option for fio. Dedupe buffers are gener‐
1398              ated by repeating previous unique write.
1399
1400              working_set is a more  realistic  workload.   With  working_set,
1401              dedupe_working_set_percentage  should  be provided.  Given that,
1402              fio will use the initial unique write  buffers  as  its  working
1403              set.  Upon deciding to dedupe, fio will randomly choose a buffer
1404              from the working set.  Note that by using working_set the dedupe
1405              percentage  will  converge to the desired over time while repeat
1406              maintains the desired percentage throughout the job.
1407
1408       dedupe_working_set_percentage=int
1409              If dedupe_mode is set to working_set,  then  this  controls  the
1410              percentage of size of the file or device used as the buffers fio
1411              will choose to generate the dedupe buffers from
1412
1413              Note that size needs to be explicitly provided and only  1  file
1414              per job is supported
1415
1416       invalidate=bool
1417              Invalidate  the  buffer/page cache parts of the files to be used
1418              prior to starting I/O if the platform and file type support  it.
1419              Defaults  to  true.   This  will  be ignored if pre_read is also
1420              specified for the same job.
1421
1422       sync=str
1423              Whether, and what type, of synchronous I/O to  use  for  writes.
1424              The allowed values are:
1425
1426                     none   Do not use synchronous IO, the default.
1427
1428                     0      Same as none.
1429
1430                     sync   Use  synchronous  file IO. For the majority of I/O
1431                            engines, this means using O_SYNC.
1432
1433                     1      Same as sync.
1434
1435                     dsync  Use synchronous data IO. For the majority  of  I/O
1436                            engines, this means using O_DSYNC.
1437
1438       iomem=str, mem=str
1439              Fio  can use various types of memory as the I/O unit buffer. The
1440              allowed values are:
1441
1442                     malloc Use memory from malloc(3) as the buffers.  Default
1443                            memory type.
1444
1445                     shm    Use   shared  memory  as  the  buffers.  Allocated
1446                            through shmget(2).
1447
1448                     shmhuge
1449                            Same as shm, but use huge pages as backing.
1450
1451                     mmap   Use mmap(2) to allocate  buffers.  May  either  be
1452                            anonymous memory, or can be file backed if a file‐
1453                            name is given after  the  option.  The  format  is
1454                            `mem=mmap:/path/to/file'.
1455
1456                     mmaphuge
1457                            Use  a memory mapped huge file as the buffer back‐
1458                            ing. Append filename after mmaphuge, ala `mem=mma‐
1459                            phuge:/hugetlbfs/file'.
1460
1461                     mmapshared
1462                            Same as mmap, but use a MMAP_SHARED mapping.
1463
1464                     cudamalloc
1465                            Use  GPU  memory as the buffers for GPUDirect RDMA
1466                            benchmark.  The ioengine must be rdma.
1467
1468              The area allocated is a function of the maximum allowed bs  size
1469              for  the  job,  multiplied by the I/O depth given. Note that for
1470              shmhuge and mmaphuge to work, the system  must  have  free  huge
1471              pages  allocated.  This can normally be checked and set by read‐
1472              ing/writing `/proc/sys/vm/nr_hugepages' on a Linux  system.  Fio
1473              assumes  a huge page is 4MiB in size. So to calculate the number
1474              of huge pages you need for a given job  file,  add  up  the  I/O
1475              depth of all jobs (normally one unless iodepth is used) and mul‐
1476              tiply by the maximum bs set. Then divide that number by the huge
1477              page size. You can see the size of the huge pages in `/proc/mem‐
1478              info'. If no huge pages are allocated by having a non-zero  num‐
1479              ber in `nr_hugepages', using mmaphuge or shmhuge will fail. Also
1480              see hugepage-size.
1481
1482              mmaphuge also needs to have hugetlbfs mounted and the file loca‐
1483              tion  should  point  there.  So  if it's mounted in `/huge', you
1484              would use `mem=mmaphuge:/huge/somefile'.
1485
1486       iomem_align=int, mem_align=int
1487              This indicates the memory alignment of the I/O  memory  buffers.
1488              Note  that  the given alignment is applied to the first I/O unit
1489              buffer, if using iodepth the alignment of the following  buffers
1490              are  given by the bs used. In other words, if using a bs that is
1491              a multiple of the page sized in the system, all buffers will  be
1492              aligned  to  this value. If using a bs that is not page aligned,
1493              the alignment of subsequent I/O memory buffers is the sum of the
1494              iomem_align and bs used.
1495
1496       hugepage-size=int
1497              Defines  the  size of a huge page. Must at least be equal to the
1498              system setting, see `/proc/meminfo'. Defaults  to  4MiB.  Should
1499              probably   always   be   a   multiple  of  megabytes,  so  using
1500              `hugepage-size=Xm' is the preferred way to  set  this  to  avoid
1501              setting a non-pow-2 bad value.
1502
1503       lockmem=int
1504              Pin the specified amount of memory with mlock(2). Can be used to
1505              simulate a smaller amount of memory. The amount specified is per
1506              worker.
1507
1508   I/O size
1509       size=int[%|z]
1510              The total size of file I/O for each thread of this job. Fio will
1511              run until this many bytes has been transferred,  unless  runtime
1512              is  limited  by other options (such as runtime, for instance, or
1513              increased/decreased by io_size).  Fio will divide this size  be‐
1514              tween the available files determined by options such as nrfiles,
1515              filename, unless filesize is specified by the job. If the result
1516              of  division  happens  to  be 0, the size is set to the physical
1517              size of the given files or devices if they exist.  If  this  op‐
1518              tion  is  not specified, fio will use the full size of the given
1519              files or devices. If the files do not exist, size must be given.
1520              It  is  also possible to give size as a percentage between 1 and
1521              100. If `size=20%' is given, fio will use 20% of the  full  size
1522              of the given files or devices. In ZBD mode, size can be given in
1523              units of number of zones using 'z'. Can be combined with  offset
1524              to  constrain  the  start  and  end  range that I/O will be done
1525              within.
1526
1527       io_size=int[%|z], io_limit=int[%|z]
1528              Normally fio operates within the region set by size, which means
1529              that  the size option sets both the region and size of I/O to be
1530              performed. Sometimes that is not what you want.  With  this  op‐
1531              tion,  it  is possible to define just the amount of I/O that fio
1532              should do. For instance, if size is set to 20GiB and io_size  is
1533              set  to  5GiB,  fio  will perform I/O within the first 20GiB but
1534              exit when 5GiB have been done. The opposite is also possible  --
1535              if  size  is set to 20GiB, and io_size is set to 40GiB, then fio
1536              will do 40GiB of I/O within the 0..20GiB region.  Value  can  be
1537              set  as percentage: io_size=N%.  In this case io_size multiplies
1538              size= value. In ZBD mode, value can also be  set  as  number  of
1539              zones using 'z'.
1540
1541       filesize=irange(int)
1542              Individual  file  sizes.  May be a range, in which case fio will
1543              select sizes for files at random within the given range and lim‐
1544              ited  to  size  in  total (if that is given). If not given, each
1545              created file is the same size.  This option  overrides  size  in
1546              terms  of  file  size, which means this value is used as a fixed
1547              size or possible range of each file.
1548
1549       file_append=bool
1550              Perform I/O after the end of the file. Normally fio will operate
1551              within  the size of a file. If this option is set, then fio will
1552              append to the file instead. This has identical behavior to  set‐
1553              ting  offset  to  the  size of a file. This option is ignored on
1554              non-regular files.
1555
1556       fill_device=bool, fill_fs=bool
1557              Sets size to something really large and  waits  for  ENOSPC  (no
1558              space  left  on  device)  or EDQUOT (disk quota exceeded) as the
1559              terminating condition. Only makes sense with  sequential  write.
1560              For  a  read workload, the mount point will be filled first then
1561              I/O started on the result.
1562
1563   I/O engine
1564       ioengine=str
1565              Defines how the job issues I/O to the file. The following  types
1566              are defined:
1567
1568                     sync   Basic read(2) or write(2) I/O. lseek(2) is used to
1569                            position the I/O location.  See fsync  and  fdata‐
1570                            sync for syncing write I/Os.
1571
1572                     psync  Basic  pread(2)  or  pwrite(2) I/O. Default on all
1573                            supported operating systems except for Windows.
1574
1575                     vsync  Basic readv(2)  or  writev(2)  I/O.  Will  emulate
1576                            queuing  by coalescing adjacent I/Os into a single
1577                            submission.
1578
1579                     pvsync Basic preadv(2) or pwritev(2) I/O.
1580
1581                     pvsync2
1582                            Basic preadv2(2) or pwritev2(2) I/O.
1583
1584                     libaio Linux native asynchronous I/O. Note that Linux may
1585                            only support queued behavior with non-buffered I/O
1586                            (set `direct=1' or `buffered=0').  This engine de‐
1587                            fines engine specific options.
1588
1589                     posixaio
1590                            POSIX   asynchronous  I/O  using  aio_read(3)  and
1591                            aio_write(3).
1592
1593                     solarisaio
1594                            Solaris native asynchronous I/O.
1595
1596                     windowsaio
1597                            Windows native asynchronous I/O. Default  on  Win‐
1598                            dows.
1599
1600                     mmap   File is memory mapped with mmap(2) and data copied
1601                            to/from using memcpy(3).
1602
1603                     splice splice(2) is used to transfer  the  data  and  vm‐
1604                            splice(2)  to transfer data from user space to the
1605                            kernel.
1606
1607                     sg     SCSI generic sg v3 I/O. May either be  synchronous
1608                            using  the  SG_IO ioctl, or if the target is an sg
1609                            character device we use read(2) and  write(2)  for
1610                            asynchronous  I/O.  Requires  filename  option  to
1611                            specify either block or  character  devices.  This
1612                            engine supports trim operations. The sg engine in‐
1613                            cludes engine specific options.
1614
1615                     libzbc Read, write, trim  and  ZBC/ZAC  operations  to  a
1616                            zoned  block device using libzbc library. The tar‐
1617                            get can be either an  SG  character  device  or  a
1618                            block device file.
1619
1620                     null   Doesn't  transfer any data, just pretends to. This
1621                            is mainly used to exercise fio itself and for  de‐
1622                            bugging/testing purposes.
1623
1624                     net    Transfer  over  the  network to given `host:port'.
1625                            Depending on  the  protocol  used,  the  hostname,
1626                            port,  listen  and  filename  options  are used to
1627                            specify what sort of connection to make, while the
1628                            protocol  option determines which protocol will be
1629                            used. This engine defines engine specific options.
1630
1631                     netsplice
1632                            Like net, but uses splice(2)  and  vmsplice(2)  to
1633                            map  data  and  send/receive.  This engine defines
1634                            engine specific options.
1635
1636                     cpuio  Doesn't transfer any data, but  burns  CPU  cycles
1637                            according  to  the  cpuload, cpuchunks and cpumode
1638                            options.  A job never finishes unless there is  at
1639                            least one non-cpuio job.
1640
1641                            cpuload=85  will  cause that job to do nothing but
1642                            burn 85% of the CPU.  In case of SMP machines, use
1643                            numjobs=<nr_of_cpu> to  get  desired CPU usage, as
1644                            the cpuload only loads a single CPU at the desired
1645                            rate.
1646
1647                            cpumode=qsort  replace  the  default noop instruc‐
1648                            tions loop by a qsort algorithm  to  consume  more
1649                            energy.
1650
1651                     rdma   The  RDMA I/O engine supports both RDMA memory se‐
1652                            mantics (RDMA_WRITE/RDMA_READ) and channel  seman‐
1653                            tics  (Send/Recv)  for  the  InfiniBand,  RoCE and
1654                            iWARP protocols. This engine defines  engine  spe‐
1655                            cific options.
1656                     falloc I/O engine that does regular fallocate to simulate
1657                            data transfer as fio ioengine.
1658                            DDIR_READ        does   fallocate(,mode   =   FAL‐
1659                            LOC_FL_KEEP_SIZE,).
1660                            DIR_WRITE      does fallocate(,mode = 0).
1661                            DDIR_TRIM        does   fallocate(,mode   =   FAL‐
1662                            LOC_FL_KEEP_SIZE|FALLOC_FL_PUNCH_HOLE).
1663
1664                     ftruncate
1665                            I/O engine that sends ftruncate(2)  operations  in
1666                            response to write (DDIR_WRITE) events. Each ftrun‐
1667                            cate issued sets the file's size  to  the  current
1668                            block offset. blocksize is ignored.
1669
1670                     e4defrag
1671                            I/O  engine  that  does  regular EXT4_IOC_MOVE_EXT
1672                            ioctls to simulate defragment activity in  request
1673                            to DDIR_WRITE event.
1674
1675                     rados  I/O  engine supporting direct access to Ceph Reli‐
1676                            able Autonomic Distributed  Object  Store  (RADOS)
1677                            via  librados.  This  ioengine defines engine spe‐
1678                            cific options.
1679
1680                     rbd    I/O engine supporting direct access to Ceph  Rados
1681                            Block Devices (RBD) via librbd without the need to
1682                            use the kernel rbd driver. This  ioengine  defines
1683                            engine specific options.
1684
1685                     http   I/O   engine   supporting  GET/PUT  requests  over
1686                            HTTP(S) with libcurl to a WebDAV or  S3  endpoint.
1687                            This ioengine defines engine specific options.
1688
1689                            This  engine only supports direct IO of iodepth=1;
1690                            you need to scale this via numjobs. blocksize  de‐
1691                            fines the size of the objects to be created.
1692
1693                            TRIM is translated to object deletion.
1694
1695                     gfapi  Using  GlusterFS libgfapi sync interface to direct
1696                            access to GlusterFS volumes without having  to  go
1697                            through  FUSE.  This  ioengine defines engine spe‐
1698                            cific options.
1699
1700                     gfapi_async
1701                            Using GlusterFS libgfapi async interface to direct
1702                            access  to  GlusterFS volumes without having to go
1703                            through FUSE. This ioengine  defines  engine  spe‐
1704                            cific options.
1705
1706                     libhdfs
1707                            Read and write through Hadoop (HDFS). The filename
1708                            option is used to specify host,port  of  the  hdfs
1709                            name-node  to connect. This engine interprets off‐
1710                            sets a little differently.  In  HDFS,  files  once
1711                            created  cannot  be  modified so random writes are
1712                            not possible. To imitate this the  libhdfs  engine
1713                            expects  a bunch of small files to be created over
1714                            HDFS and will randomly pick a file from them based
1715                            on  the  offset  generated by fio backend (see the
1716                            example  job  file  to  create  such  files,   use
1717                            `rw=write'  option). Please note, it may be neces‐
1718                            sary to set environment  variables  to  work  with
1719                            HDFS/libhdfs  properly. Each job uses its own con‐
1720                            nection to HDFS.
1721
1722                     mtd    Read, write and  erase  an  MTD  character  device
1723                            (e.g.,   `/dev/mtd0').  Discards  are  treated  as
1724                            erases. Depending on the underlying  device  type,
1725                            the I/O may have to go in a certain pattern, e.g.,
1726                            on NAND, writing sequentially to erase blocks  and
1727                            discarding  before overwriting. The trimwrite mode
1728                            works well for this constraint.
1729
1730                     pmemblk
1731                            Read and write using filesystem DAX to a file on a
1732                            filesystem mounted with DAX on a persistent memory
1733                            device through the PMDK libpmemblk library.
1734
1735                     dev-dax
1736                            Read and write using device DAX  to  a  persistent
1737                            memory device (e.g., /dev/dax0.0) through the PMDK
1738                            libpmem library.
1739
1740                     external
1741                            Prefix to specify loading an external  I/O  engine
1742                            object file. Append the engine filename, e.g. `io‐
1743                            engine=external:/tmp/foo.o'   to   load   ioengine
1744                            `foo.o' in `/tmp'. The path can be either absolute
1745                            or relative. See `engines/skeleton_external.c'  in
1746                            the  fio source for details of writing an external
1747                            I/O engine.
1748
1749                     filecreate
1750                            Simply create the files and do  no  I/O  to  them.
1751                            You still need to set filesize so that all the ac‐
1752                            counting still occurs, but no actual I/O  will  be
1753                            done other than creating the file.
1754
1755                     filestat
1756                            Simply  do  stat()  and do no I/O to the file. You
1757                            need to set  'filesize'  and  'nrfiles',  so  that
1758                            files  will be created.  This engine is to measure
1759                            file lookup and meta data access.
1760
1761                     filedelete
1762                            Simply delete files by unlink() and do no  I/O  to
1763                            the  file.  You  need  to  set 'filesize' and 'nr‐
1764                            files', so that files will be created.   This  en‐
1765                            gine is to measure file delete.
1766
1767                     libpmem
1768                            Read  and  write  using  mmap  I/O  to a file on a
1769                            filesystem mounted with DAX on a persistent memory
1770                            device through the PMDK libpmem library.
1771
1772                     ime_psync
1773                            Synchronous  read  and  write using DDN's Infinite
1774                            Memory Engine (IME). This engine is very basic and
1775                            issues calls to IME whenever an IO is queued.
1776
1777                     ime_psyncv
1778                            Synchronous  read  and  write using DDN's Infinite
1779                            Memory Engine (IME). This engine uses  iovecs  and
1780                            will  try to stack as much IOs as possible (if the
1781                            IOs are "contiguous" and the IO depth is  not  ex‐
1782                            ceeded) before issuing a call to IME.
1783
1784                     ime_aio
1785                            Asynchronous  read  and write using DDN's Infinite
1786                            Memory Engine (IME). This engine will try to stack
1787                            as  much  IOs as possible by creating requests for
1788                            IME.  FIO will then decide when  to  commit  these
1789                            requests.
1790
1791                     libiscsi
1792                            Read and write iscsi lun with libiscsi.
1793
1794                     nbd    Synchronous  read and write a Network Block Device
1795                            (NBD).
1796
1797                     libcufile
1798                            I/O engine supporting libcufile synchronous access
1799                            to  nvidia-fs  and  a  GPUDirect Storage-supported
1800                            filesystem.  This  engine  performs  I/O   without
1801                            transferring  buffers  between  user-space and the
1802                            kernel, unless verify is set or cuda_io is  posix.
1803                            iomem  must  not  be cudamalloc. This ioengine de‐
1804                            fines engine specific options.
1805
1806                     dfs    I/O engine supporting asynchronous read and  write
1807                            operations  to  the  DAOS  File  System  (DFS) via
1808                            libdfs.
1809
1810                     nfs    I/O engine supporting asynchronous read and  write
1811                            operations  to  NFS filesystems from userspace via
1812                            libnfs. This is useful for achieving  higher  con‐
1813                            currency  and thus throughput than is possible via
1814                            kernel NFS.
1815
1816                     exec   Execute 3rd party tools. Could be used to  perform
1817                            monitoring during jobs runtime.
1818
1819   I/O engine specific parameters
1820       In addition, there are some parameters which are only valid when a spe‐
1821       cific ioengine is in use. These are used identically to normal  parame‐
1822       ters,  with  the  caveat  that when used on the command line, they must
1823       come after the ioengine that defines them is selected.
1824
1825       (io_uring,libaio)cmdprio_percentage=int[,int]
1826              Set the percentage of I/O that will be issued with  the  highest
1827              priority.   Default:  0.  A  single  value  applies to reads and
1828              writes. Comma-separated values may be specified  for  reads  and
1829              writes.  For  this  option to be effective, NCQ priority must be
1830              supported and enabled, and `direct=1' option must be  used.  fio
1831              must  also  be run as the root user. Unlike slat/clat/lat stats,
1832              which can be tracked and reported  independently,  per  priority
1833              stats  only  track  and  report a single type of latency. By de‐
1834              fault, completion latency (clat) will be reported,  if  lat_per‐
1835              centiles is set, total latency (lat) will be reported.
1836
1837       (io_uring,libaio)cmdprio_class=int[,int]
1838              Set  the  I/O priority class to use for I/Os that must be issued
1839              with a priority when cmdprio_percentage  or  cmdprio_bssplit  is
1840              set.   If  not specified when cmdprio_percentage or cmdprio_bss‐
1841              plit is set, this defaults to the highest priority class. A sin‐
1842              gle  value  applies  to reads and writes. Comma-separated values
1843              may be specified for reads and writes. See  man  ionice(1).  See
1844              also the prioclass option.
1845
1846       (io_uring,libaio)cmdprio=int[,int]
1847              Set  the  I/O priority value to use for I/Os that must be issued
1848              with a priority when cmdprio_percentage  or  cmdprio_bssplit  is
1849              set.   If  not specified when cmdprio_percentage or cmdprio_bss‐
1850              plit is set, this defaults to 0. Linux limits us to  a  positive
1851              value  between 0 and 7, with 0 being the highest. A single value
1852              applies to reads and  writes.   Comma-separated  values  may  be
1853              specified  for  reads and writes. See man ionice(1). Refer to an
1854              appropriate manpage for other operating systems since the  mean‐
1855              ing of priority may differ. See also the prio option.
1856
1857       (io_uring,libaio)cmdprio_bssplit=str[,str]
1858              To  get  a  finer  control over I/O priority, this option allows
1859              specifying the percentage of IOs that must have a  priority  set
1860              depending  on  the  block  size of the IO. This option is useful
1861              only when used together with the option bssplit, that is, multi‐
1862              ple  different  block  sizes  are used for reads and writes. The
1863              format for this option is the same as the format of the  bssplit
1864              option, with the exception that values for trim IOs are ignored.
1865              This option is mutually exclusive  with  the  cmdprio_percentage
1866              option.
1867
1868       (io_uring)fixedbufs
1869              If  fio  is asked to do direct IO, then Linux will map pages for
1870              each IO call, and release them when IO is done. If  this  option
1871              is  set,  the  pages  are  pre-mapped before IO is started. This
1872              eliminates the need to map and release for  each  IO.   This  is
1873              more efficient, and reduces the IO latency as well.
1874
1875       (io_uring)hipri
1876              If this option is set, fio will attempt to use polled IO comple‐
1877              tions. Normal IO completions generate interrupts to  signal  the
1878              completion  of IO, polled completions do not. Hence they are re‐
1879              quire active reaping by the application.  The benefits are  more
1880              efficient  IO  for  high IOPS scenarios, and lower latencies for
1881              low queue depth IO.
1882
1883       (io_uring)registerfiles
1884              With this option, fio registers the set of files being used with
1885              the kernel.  This avoids the overhead of managing file counts in
1886              the kernel, making  the  submission  and  completion  part  more
1887              lightweight. Required for the below sqthread_poll option.
1888
1889       (io_uring)sqthread_poll
1890              Normally  fio  will submit IO by issuing a system call to notify
1891              the kernel of available items in the SQ ring. If this option  is
1892              set,  the  act of submitting IO will be done by a polling thread
1893              in the kernel. This frees up cycles for fio, at the cost of  us‐
1894              ing more CPU in the system.
1895
1896       (io_uring)sqthread_poll_cpu
1897              When  `sqthread_poll`  is set, this option provides a way to de‐
1898              fine which CPU should be used for the polling thread.
1899
1900       (libaio)userspace_reap
1901              Normally, with the libaio  engine  in  use,  fio  will  use  the
1902              io_getevents(3)  system call to reap newly returned events. With
1903              this flag turned on, the AIO ring will  be  read  directly  from
1904              user-space to reap events. The reaping mode is only enabled when
1905              polling for a minimum of 0 events (e.g. when `iodepth_batch_com‐
1906              plete=0').
1907
1908       (pvsync2)hipri
1909              Set  RWF_HIPRI  on  I/O,  indicating  to the kernel that it's of
1910              higher priority than normal.
1911
1912       (pvsync2)hipri_percentage
1913              When hipri is set this determines the probability of  a  pvsync2
1914              I/O being high priority. The default is 100%.
1915
1916       (pvsync2,libaio,io_uring)nowait
1917              By default if a request cannot be executed immediately (e.g. re‐
1918              source starvation, waiting on locks) it is queued and the initi‐
1919              ating  process  will  be blocked until the required resource be‐
1920              comes free.  This option sets  the  RWF_NOWAIT  flag  (supported
1921              from  the  4.14 Linux kernel) and the call will return instantly
1922              with EAGAIN or a partial result rather than waiting.
1923
1924              It is useful to also use ignore_error=EAGAIN when using this op‐
1925              tion.   Note:  glibc  2.27,  2.28 have a bug in syscall wrappers
1926              preadv2, pwritev2.  They return EOPNOTSUP instead of EAGAIN.
1927
1928              For cached I/O, using this option usually means a request  oper‐
1929              ates  only  with cached data. Currently the RWF_NOWAIT flag does
1930              not supported for cached write.  For direct I/O,  requests  will
1931              only  succeed  if cache invalidation isn't required, file blocks
1932              are fully allocated and the disk request could be issued immedi‐
1933              ately.
1934
1935       (cpuio)cpuload=int
1936              Attempt to use the specified percentage of CPU cycles. This is a
1937              mandatory option when using cpuio I/O engine.
1938
1939       (cpuio)cpuchunks=int
1940              Split the load into cycles of the given time. In microseconds.
1941
1942       (cpuio)exit_on_io_done=bool
1943              Detect when I/O threads are done, then exit.
1944
1945       (libhdfs)namenode=str
1946              The hostname or IP address of a HDFS cluster  namenode  to  con‐
1947              tact.
1948
1949       (libhdfs)port=int
1950              The listening port of the HFDS cluster namenode.
1951
1952       (netsplice,net)port=int
1953              The  TCP  or  UDP port to bind to or connect to. If this is used
1954              with numjobs to spawn multiple instances of the same  job  type,
1955              then  this will be the starting port number since fio will use a
1956              range of ports.
1957
1958       (rdma,librpma_*)port=int
1959              The port to use for RDMA-CM communication. This  should  be  the
1960              same value on the client and the server side.
1961
1962       (netsplice,net,rdma)hostname=str
1963              The  hostname or IP address to use for TCP, UDP or RDMA-CM based
1964              I/O.  If the job is a TCP listener or UDP reader,  the  hostname
1965              is  not used and must be omitted unless it is a valid UDP multi‐
1966              cast address.
1967
1968       (librpma_*)serverip=str
1969              The IP address to be used for RDMA-CM based I/O.
1970
1971       (librpma_*_server)direct_write_to_pmem=bool
1972              Set to 1 only when Direct Write to PMem from the remote host  is
1973              possible. Otherwise, set to 0.
1974
1975       (librpma_*_server)busy_wait_polling=bool
1976              Set  to  0  to  wait for completion instead of busy-wait polling
1977              completion.  Default: 1.
1978
1979       (netsplice,net)interface=str
1980              The IP address of the network interface used to send or  receive
1981              UDP multicast.
1982
1983       (netsplice,net)ttl=int
1984              Time-to-live  value for outgoing UDP multicast packets. Default:
1985              1.
1986
1987       (netsplice,net)nodelay=bool
1988              Set TCP_NODELAY on TCP connections.
1989
1990       (netsplice,net)protocol=str, proto=str
1991              The network protocol to use. Accepted values are:
1992
1993                     tcp    Transmission control protocol.
1994
1995                     tcpv6  Transmission control protocol V6.
1996
1997                     udp    User datagram protocol.
1998
1999                     udpv6  User datagram protocol V6.
2000
2001                     unix   UNIX domain socket.
2002
2003              When the protocol is TCP or UDP, the port must also be given, as
2004              well as the hostname if the job is a TCP listener or UDP reader.
2005              For unix sockets, the normal filename option should be used  and
2006              the port is invalid.
2007
2008       (netsplice,net)listen
2009              For  TCP  network  connections,  tell fio to listen for incoming
2010              connections rather than initiating an outgoing  connection.  The
2011              hostname must be omitted if this option is used.
2012
2013       (netsplice,net)pingpong
2014              Normally a network writer will just continue writing data, and a
2015              network reader will just consume packages.  If  `pingpong=1'  is
2016              set,  a  writer will send its normal payload to the reader, then
2017              wait for the reader to send the same payload back.  This  allows
2018              fio  to measure network latencies. The submission and completion
2019              latencies then measure local time spent  sending  or  receiving,
2020              and  the  completion  latency  measures how long it took for the
2021              other end to receive and send back. For  UDP  multicast  traffic
2022              `pingpong=1'  should only be set for a single reader when multi‐
2023              ple readers are listening to the same address.
2024
2025       (netsplice,net)window_size=int
2026              Set the desired socket buffer size for the connection.
2027
2028       (netsplice,net)mss=int
2029              Set the TCP maximum segment size (TCP_MAXSEG).
2030
2031       (e4defrag)donorname=str
2032              File will be used as a block donor (swap extents between files).
2033
2034       (e4defrag)inplace=int
2035              Configure donor file blocks allocation strategy:
2036
2037                     0      Default. Preallocate donor's file on init.
2038
2039                     1      Allocate  space  immediately   inside   defragment
2040                            event, and free right after event.
2041
2042       (rbd,rados)clustername=str
2043              Specifies the name of the Ceph cluster.
2044
2045       (rbd)rbdname=str
2046              Specifies the name of the RBD.
2047
2048       (rbd,rados)pool=str
2049              Specifies  the  name  of  the  Ceph pool containing RBD or RADOS
2050              data.
2051
2052       (rbd,rados)clientname=str
2053              Specifies the username (without the 'client.'  prefix)  used  to
2054              access  the  Ceph  cluster. If the clustername is specified, the
2055              clientname shall be the full *type.id* string. If no type.  pre‐
2056              fix is given, fio will add 'client.'  by default.
2057
2058       (rbd,rados)busy_poll=bool
2059              Poll  store instead of waiting for completion. Usually this pro‐
2060              vides better throughput at cost of higher(up to 100%)  CPU  uti‐
2061              lization.
2062
2063       (rados)touch_objects=bool
2064              During  initialization,  touch  (create if do not exist) all ob‐
2065              jects (files).  Touching all objects  affects  ceph  caches  and
2066              likely impacts test results.  Enabled by default.
2067
2068       (http)http_host=str
2069              Hostname  to  connect to. For S3, this could be the bucket name.
2070              Default is localhost
2071
2072       (http)http_user=str
2073              Username for HTTP authentication.
2074
2075       (http)http_pass=str
2076              Password for HTTP authentication.
2077
2078       (http)https=str
2079              Whether to use HTTPS instead of plain HTTP.  on  enables  HTTPS;
2080              insecure  will  enable  HTTPS, but disable SSL peer verification
2081              (use with caution!).  Default is off.
2082
2083       (http)http_mode=str
2084              Which HTTP access mode to use: webdav, swift, or s3. Default  is
2085              webdav.
2086
2087       (http)http_s3_region=str
2088              The  S3  region/zone  to  include in the request. Default is us-
2089              east-1.
2090
2091       (http)http_s3_key=str
2092              The S3 secret key.
2093
2094       (http)http_s3_keyid=str
2095              The S3 key/access id.
2096
2097       (http)http_swift_auth_token=str
2098              The Swift auth token. See the example configuration file on  how
2099              to retrieve this.
2100
2101       (http)http_verbose=int
2102              Enable  verbose  requests  from libcurl. Useful for debugging. 1
2103              turns on verbose logging from libcurl,  2  additionally  enables
2104              HTTP IO tracing.  Default is 0
2105
2106       (mtd)skip_bad=bool
2107              Skip operations against known bad blocks.
2108
2109       (libhdfs)hdfsdirectory
2110              libhdfs will create chunk in this HDFS directory.
2111
2112       (libhdfs)chunk_size
2113              The size of the chunk to use for each file.
2114
2115       (rdma)verb=str
2116              The  RDMA  verb to use on this side of the RDMA ioengine connec‐
2117              tion. Valid values are write, read, send and recv. These  corre‐
2118              spond  to  the  equivalent  RDMA  verbs (e.g. write = rdma_write
2119              etc.). Note that this only needs to be specified on  the  client
2120              side of the connection. See the examples folder.
2121
2122       (rdma)bindname=str
2123              The  name to use to bind the local RDMA-CM connection to a local
2124              RDMA device. This could be a hostname or an  IPv4  or  IPv6  ad‐
2125              dress.  On  the  server  side  this  will  be  passed  into  the
2126              rdma_bind_addr() function and on the client site it will be used
2127              in the rdma_resolve_add() function. This can be useful when mul‐
2128              tiple paths exist between the client and the server or  in  cer‐
2129              tain loopback configurations.
2130
2131       (filestat)stat_type=str
2132              Specify  stat system call type to measure lookup/getattr perfor‐
2133              mance.  Default is stat for stat(2).
2134
2135       (sg)hipri
2136              If this option is set, fio will attempt to use polled IO comple‐
2137              tions.  This will have a similar effect as (io_uring)hipri. Only
2138              SCSI READ and WRITE commands will have the  SGV4_FLAG_HIPRI  set
2139              (not  UNMAP  (trim) nor VERIFY).  Older versions of the Linux sg
2140              driver that do not support hipri will simply  ignore  this  flag
2141              and  do  normal  IO.  The Linux SCSI Low Level Driver (LLD) that
2142              "owns" the device also needs to support  hipri  (also  known  as
2143              iopoll and mq_poll). The MegaRAID driver is an example of a SCSI
2144              LLD.  Default: clear (0) which does normal  (interrupted  based)
2145              IO.
2146
2147       (sg)readfua=bool
2148              With  readfua option set to 1, read operations include the force
2149              unit access (fua) flag. Default: 0.
2150
2151       (sg)writefua=bool
2152              With writefua option set to  1,  write  operations  include  the
2153              force unit access (fua) flag. Default: 0.
2154
2155       (sg)sg_write_mode=str
2156              Specify  the  type  of  write commands to issue. This option can
2157              take three values:
2158
2159                     write (default)
2160                            Write opcodes are issued as usual
2161
2162                     verify Issue WRITE AND VERIFY commands. The BYTCHK bit is
2163                            set  to  0. This directs the device to carry out a
2164                            medium verification with no data  comparison.  The
2165                            writefua option is ignored with this selection.
2166
2167                     same   Issue WRITE SAME commands. This transfers a single
2168                            block to the device and writes this same block  of
2169                            data to a contiguous sequence of LBAs beginning at
2170                            the specified offset. fio's block  size  parameter
2171                            specifies  the  amount  of  data written with each
2172                            command. However,  the  amount  of  data  actually
2173                            transferred to the device is equal to the device's
2174                            block (sector) size. For a device  with  512  byte
2175                            sectors,  blocksize=8k  will write 16 sectors with
2176                            each command. fio will still generate 8k  of  data
2177                            for  each command butonly the first 512 bytes will
2178                            be used and transferred to the device. The  write‐
2179                            fua option is ignored with this selection.
2180
2181       (nbd)uri=str
2182              Specify  the  NBD  URI  of  the server to test.  The string is a
2183              standard   NBD   URI   (see   https://github.com/NetworkBlockDe
2184              vice/nbd/tree/master/doc).  Example URIs:
2185
2186                     nbd://localhost:10809
2187
2188                     nbd+unix:///?socket=/tmp/socket
2189
2190                     nbds://tlshost/exportname
2191
2192       (libcufile)gpu_dev_ids=str
2193              Specify  the GPU IDs to use with CUDA. This is a colon-separated
2194              list of int.  GPUs are assigned to workers  roundrobin.  Default
2195              is 0.
2196
2197       (libcufile)cuda_io=str
2198              Specify  the type of I/O to use with CUDA. This option takes the
2199              following values:
2200
2201                     cufile (default)
2202                            Use libcufile and nvidia-fs. This option  performs
2203                            I/O  directly between a GPUDirect Storage filesys‐
2204                            tem and GPU buffers, avoiding use of a bounce buf‐
2205                            fer.  If verify is set, cudaMemcpy is used to copy
2206                            verification data between RAM and GPU(s).  Verifi‐
2207                            cation  data  is  copied  from RAM to GPU before a
2208                            write and from GPU to RAM after  a  read.   direct
2209                            must be 1.
2210
2211                     posix  Use  POSIX  to  perform I/O with a RAM buffer, and
2212                            use cudaMemcpy to transfer data  between  RAM  and
2213                            the GPU(s).  Data is copied from GPU to RAM before
2214                            a write and copied from RAM to GPU after  a  read.
2215                            verify does not affect the use of cudaMemcpy.
2216
2217       (dfs)pool
2218              Specify the label or UUID of the DAOS pool to connect to.
2219
2220       (dfs)cont
2221              Specify the label or UUID of the DAOS container to open.
2222
2223       (dfs)chunk_size
2224              Specificy  a  different  chunk size (in bytes) for the dfs file.
2225              Use DAOS container's chunk size by default.
2226
2227       (dfs)object_class
2228              Specificy a different object class for the dfs file.   Use  DAOS
2229              container's object class by default.
2230
2231       (nfs)nfs_url
2232              URL           in           libnfs           format,           eg
2233              nfs://<server|ipv4|ipv6>/path[?arg=val[&arg=val]*] Refer to  the
2234              libnfs README for more details.
2235
2236       (exec)program=str
2237              Specify the program to execute.  Note the program will receive a
2238              SIGTERM when the job is reaching the time limit.  A  SIGKILL  is
2239              sent  once the job is over. The delay between the two signals is
2240              defined by grace_time option.
2241
2242       (exec)arguments=str
2243              Specify arguments to pass to program.   Some  special  variables
2244              can be expanded to pass fio's job details to the program :
2245
2246                     %r     replaced by the duration of the job in seconds
2247
2248                     %n     replaced by the name of the job
2249
2250       (exec)grace_time=int
2251              Defines  the  time  between the SIGTERM and SIGKILL signals. De‐
2252              fault is 1 second.
2253
2254       (exec)std_redirect=ool
2255              If set, stdout and stderr streams are redirected to files  named
2256              from the job name. Default is true.
2257
2258   I/O depth
2259       iodepth=int
2260              Number  of  I/O  units  to keep in flight against the file. Note
2261              that increasing iodepth beyond 1 will not affect synchronous io‐
2262              engines  (except for small degrees when verify_async is in use).
2263              Even async engines may impose OS restrictions  causing  the  de‐
2264              sired  depth  not  to be achieved. This may happen on Linux when
2265              using libaio and not setting `direct=1', since buffered  I/O  is
2266              not  async on that OS. Keep an eye on the I/O depth distribution
2267              in the fio output to verify that the achieved depth  is  as  ex‐
2268              pected. Default: 1.
2269
2270       iodepth_batch_submit=int, iodepth_batch=int
2271              This  defines  how  many pieces of I/O to submit at once. It de‐
2272              faults to 1 which means that we submit each I/O as soon as it is
2273              available,  but can be raised to submit bigger batches of I/O at
2274              the time. If it is set to 0 the iodepth value will be used.
2275
2276       iodepth_batch_complete_min=int, iodepth_batch_complete=int
2277              This defines how many pieces of I/O to retrieve at once. It  de‐
2278              faults to 1 which means that we'll ask for a minimum of 1 I/O in
2279              the retrieval process from the kernel. The I/O retrieval will go
2280              on  until  we hit the limit set by iodepth_low. If this variable
2281              is set to 0, then fio will always check for completed events be‐
2282              fore  queuing  more  I/O.  This helps reduce I/O latency, at the
2283              cost of more retrieval system calls.
2284
2285       iodepth_batch_complete_max=int
2286              This defines maximum pieces of I/O to  retrieve  at  once.  This
2287              variable   should   be   used   along   with  iodepth_batch_com‐
2288              plete_min=int variable, specifying the  range  of  min  and  max
2289              amount  of I/O which should be retrieved. By default it is equal
2290              to iodepth_batch_complete_min value. Example #1:
2291
2292                     iodepth_batch_complete_min=1
2293                     iodepth_batch_complete_max=<iodepth>
2294
2295              which means that we will retrieve at least 1 I/O and up  to  the
2296              whole  submitted  queue depth. If none of I/O has been completed
2297              yet, we will wait.  Example #2:
2298
2299                     iodepth_batch_complete_min=0
2300                     iodepth_batch_complete_max=<iodepth>
2301
2302              which means that we can retrieve up to the whole submitted queue
2303              depth,  but  if  none of I/O has been completed yet, we will NOT
2304              wait and immediately exit the system call. In  this  example  we
2305              simply do polling.
2306
2307       iodepth_low=int
2308              The  low  water  mark indicating when to start filling the queue
2309              again. Defaults to the same as iodepth, meaning  that  fio  will
2310              attempt  to  keep the queue full at all times. If iodepth is set
2311              to e.g. 16 and iodepth_low is set  to  4,  then  after  fio  has
2312              filled  the  queue  of  16 requests, it will let the depth drain
2313              down to 4 before starting to fill it again.
2314
2315       serialize_overlap=bool
2316              Serialize in-flight I/Os that might otherwise  cause  or  suffer
2317              from data races.  When two or more I/Os are submitted simultane‐
2318              ously, there is no guarantee that the I/Os will be processed  or
2319              completed  in  the  submitted  order. Further, if two or more of
2320              those I/Os are writes, any overlapping region between  them  can
2321              become  indeterminate/undefined on certain storage. These issues
2322              can cause verification to fail erratically when at least one  of
2323              the  racing I/Os is changing data and the overlapping region has
2324              a non-zero size. Setting serialize_overlap tells  fio  to  avoid
2325              provoking this behavior by explicitly serializing in-flight I/Os
2326              that have a non-zero overlap. Note that setting this option  can
2327              reduce both performance and the iodepth achieved.
2328
2329              This  option only applies to I/Os issued for a single job except
2330              when it is enabled along with io_submit_mode=offload. In offload
2331              mode,  fio  will  check  for overlap among all I/Os submitted by
2332              offload jobs with serialize_overlap enabled.
2333
2334              Default: false.
2335
2336       io_submit_mode=str
2337              This option controls how fio submits the I/O to the I/O  engine.
2338              The  default  is  `inline', which means that the fio job threads
2339              submit and reap I/O directly.  If  set  to  `offload',  the  job
2340              threads  will  offload I/O submission to a dedicated pool of I/O
2341              threads. This requires some coordination and thus has a  bit  of
2342              extra  overhead,  especially  for lower queue depth I/O where it
2343              can increase latencies. The benefit is that fio can manage  sub‐
2344              mission rates independently of the device completion rates. This
2345              avoids skewed latency reporting if I/O gets backed up on the de‐
2346              vice side (the coordinated omission problem). Note that this op‐
2347              tion cannot reliably be used with async IO engines.
2348
2349   I/O rate
2350       thinktime=time
2351              Stall the job for the specified period of time after an I/O  has
2352              completed  before issuing the next. May be used to simulate pro‐
2353              cessing being done by an application.  When the unit is omitted,
2354              the  value is interpreted in microseconds. See thinktime_blocks,
2355              thinktime_iotime and thinktime_spin.
2356
2357       thinktime_spin=time
2358              Only valid if thinktime is set - pretend to spend CPU time doing
2359              something  with the data received, before falling back to sleep‐
2360              ing for the rest of the period specified by thinktime. When  the
2361              unit is omitted, the value is interpreted in microseconds.
2362
2363       thinktime_blocks=int
2364              Only  valid if thinktime is set - control how many blocks to is‐
2365              sue, before waiting thinktime usecs. If not set, defaults  to  1
2366              which will make fio wait thinktime usecs after every block. This
2367              effectively makes any queue depth setting  redundant,  since  no
2368              more than 1 I/O will be queued before we have to complete it and
2369              do our thinktime. In other words, this setting effectively  caps
2370              the queue depth if the latter is larger.
2371
2372       thinktime_blocks_type=str
2373              Only  valid  if  thinktime is set - control how thinktime_blocks
2374              triggers.  The default is `complete', which  triggers  thinktime
2375              when  fio  completes  thinktime_blocks blocks. If this is set to
2376              `issue', then the trigger happens at the issue side.
2377
2378       thinktime_iotime=time
2379              Only valid if thinktime is set - control thinktime  interval  by
2380              time.   The  thinktime  stall is repeated after IOs are executed
2381              for  thinktime_iotime.   For   example,   `--thinktime_iotime=9s
2382              --thinktime=1s'  repeat  10-second  cycle with IOs for 9 seconds
2383              and stall for 1 second. When the unit is omitted,  thinktime_io‐
2384              time  is  interpreted as a number of seconds.  If this option is
2385              used together with thinktime_blocks, the thinktime stall is  re‐
2386              peated  after  thinktime_iotime  or  after thinktime_blocks IOs,
2387              whichever happens first.
2388
2389
2390       rate=int[,int][,int]
2391              Cap the bandwidth used by this job. The number is in  bytes/sec,
2392              the  normal  suffix  rules  apply. Comma-separated values may be
2393              specified for reads, writes, and trims as  described  in  block‐
2394              size.
2395
2396              For  example, using `rate=1m,500k' would limit reads to 1MiB/sec
2397              and writes to 500KiB/sec. Capping only reads or  writes  can  be
2398              done  with  `rate=,500k'  or  `rate=500k,' where the former will
2399              only limit writes (to 500KiB/sec) and the latter will only limit
2400              reads.
2401
2402       rate_min=int[,int][,int]
2403              Tell  fio  to do whatever it can to maintain at least this band‐
2404              width. Failing to meet this requirement will cause  the  job  to
2405              exit. Comma-separated values may be specified for reads, writes,
2406              and trims as described in blocksize.
2407
2408       rate_iops=int[,int][,int]
2409              Cap the bandwidth to this number of IOPS. Basically the same  as
2410              rate,  just  specified independently of bandwidth. If the job is
2411              given a block size range instead of a fixed value, the  smallest
2412              block  size is used as the metric. Comma-separated values may be
2413              specified for reads, writes, and trims as  described  in  block‐
2414              size.
2415
2416       rate_iops_min=int[,int][,int]
2417              If  fio  doesn't meet this rate of I/O, it will cause the job to
2418              exit.   Comma-separated  values  may  be  specified  for  reads,
2419              writes, and trims as described in blocksize.
2420
2421       rate_process=str
2422              This  option controls how fio manages rated I/O submissions. The
2423              default is `linear', which submits I/O in a linear fashion  with
2424              fixed  delays  between I/Os that gets adjusted based on I/O com‐
2425              pletion rates. If this is set to `poisson', fio will submit  I/O
2426              based  on  a  more  real world random request flow, known as the
2427              Poisson       process       (https://en.wikipedia.org/wiki/Pois
2428              son_point_process). The lambda will be 10^6 / IOPS for the given
2429              workload.
2430
2431       rate_ignore_thinktime=bool
2432              By default, fio will attempt to catch up to the  specified  rate
2433              setting,  if any kind of thinktime setting was used. If this op‐
2434              tion is set, then fio will ignore the thinktime and continue do‐
2435              ing  IO  at  the  specified rate, instead of entering a catch-up
2436              mode after thinktime is done.
2437
2438   I/O latency
2439       latency_target=time
2440              If set, fio will attempt to find the max performance point  that
2441              the given workload will run at while maintaining a latency below
2442              this target. When the unit is omitted, the value is  interpreted
2443              in microseconds. See latency_window and latency_percentile.
2444
2445       latency_window=time
2446              Used  with  latency_target to specify the sample window that the
2447              job is run at varying queue depths to test the performance. When
2448              the unit is omitted, the value is interpreted in microseconds.
2449
2450       latency_percentile=float
2451              The percentage of I/Os that must fall within the criteria speci‐
2452              fied by latency_target and latency_window. If not set, this  de‐
2453              faults to 100.0, meaning that all I/Os must be equal or below to
2454              the value set by latency_target.
2455
2456       latency_run=bool
2457              Used with latency_target. If false (default), fio will find  the
2458              highest queue depth that meets latency_target and exit. If true,
2459              fio will continue running and try to meet latency_target by  ad‐
2460              justing queue depth.
2461
2462       max_latency=time[,time][,time]
2463              If  set, fio will exit the job with an ETIMEDOUT error if it ex‐
2464              ceeds this maximum latency. When the unit is omitted, the  value
2465              is  interpreted  in  microseconds. Comma-separated values may be
2466              specified for reads, writes, and trims as  described  in  block‐
2467              size.
2468
2469       rate_cycle=int
2470              Average bandwidth for rate and rate_min over this number of mil‐
2471              liseconds. Defaults to 1000.
2472
2473   I/O replay
2474       write_iolog=str
2475              Write the  issued  I/O  patterns  to  the  specified  file.  See
2476              read_iolog.  Specify a separate file for each job, otherwise the
2477              iologs will be interspersed and the file may be corrupt.
2478
2479       read_iolog=str
2480              Open an iolog with the specified filename  and  replay  the  I/O
2481              patterns  it  contains. This can be used to store a workload and
2482              replay it sometime later. The iolog given may also be a blktrace
2483              binary  file,  which allows fio to replay a workload captured by
2484              blktrace. See blktrace(8) for how to capture such logging  data.
2485              For blktrace replay, the file needs to be turned into a blkparse
2486              binary data file  first  (`blkparse  <device>  -o  /dev/null  -d
2487              file_for_fio.bin').   You can specify a number of files by sepa‐
2488              rating the names with a ':' character.  See the filename  option
2489              for  information on how to escape ':' characters within the file
2490              names. These files will be sequentially assigned to  job  clones
2491              created  by  numjobs.  '-' is a reserved name, meaning read from
2492              stdin, notably if filename is set to '-' which  means  stdin  as
2493              well, then this flag can't be set to '-'.
2494
2495       read_iolog_chunked=bool
2496              Determines   how  iolog  is  read.  If  false  (default)  entire
2497              read_iolog will be read at once. If selected  true,  input  from
2498              iolog  will be read gradually.  Useful when iolog is very large,
2499              or it is generated.
2500
2501       merge_blktrace_file=str
2502              When  specified,  rather  than  replaying  the  logs  passed  to
2503              read_iolog,  the  logs go through a merge phase which aggregates
2504              them into a single blktrace.  The resulting file is then  passed
2505              on  as  the  read_iolog parameter. The intention here is to make
2506              the order of events consistent. This limits the influence of the
2507              scheduler  compared  to replaying multiple blktraces via concur‐
2508              rent jobs.
2509
2510       merge_blktrace_scalars=float_list
2511              This is a percentage based option that is index paired with  the
2512              list  of  files passed to read_iolog. When merging is performed,
2513              scale the time of each event by the  corresponding  amount.  For
2514              example,   `--merge_blktrace_scalars="50:100"'  runs  the  first
2515              trace in halftime and the second trace in realtime. This knob is
2516              separately tunable from replay_time_scale which scales the trace
2517              during runtime and will not change the output of the  merge  un‐
2518              like this option.
2519
2520       merge_blktrace_iters=float_list
2521              This is a whole number option that is index paired with the list
2522              of files passed to read_iolog. When merging  is  performed,  run
2523              each  trace for the specified number of iterations. For example,
2524              `--merge_blktrace_iters="2:1"' runs the first trace for two  it‐
2525              erations and the second trace for one iteration.
2526
2527       replay_no_stall=bool
2528              When  replaying  I/O  with read_iolog the default behavior is to
2529              attempt to respect the timestamps within the log and replay them
2530              with  the  appropriate delay between IOPS. By setting this vari‐
2531              able fio will not respect the timestamps and attempt  to  replay
2532              them  as  fast  as possible while still respecting ordering. The
2533              result is the same I/O pattern to a given device, but  different
2534              timings.
2535
2536       replay_time_scale=int
2537              When  replaying I/O with read_iolog, fio will honor the original
2538              timing in the trace. With this option, it's  possible  to  scale
2539              the time. It's a percentage option, if set to 50 it means run at
2540              50% the original IO rate in the trace. If set  to  200,  run  at
2541              twice the original IO rate. Defaults to 100.
2542
2543       replay_redirect=str
2544              While replaying I/O patterns using read_iolog the default behav‐
2545              ior is to replay the IOPS onto the major/minor device that  each
2546              IOP  was recorded from. This is sometimes undesirable because on
2547              a different machine those major/minor numbers can map to a  dif‐
2548              ferent device. Changing hardware on the same system can also re‐
2549              sult in a different major/minor mapping.  replay_redirect causes
2550              all I/Os to be replayed onto the single specified device regard‐
2551              less of the device it  was  recorded  from.  i.e.  `replay_redi‐
2552              rect=/dev/sdc'  would  cause all I/O in the blktrace or iolog to
2553              be replayed onto `/dev/sdc'. This means multiple devices will be
2554              replayed  onto  a  single device, if the trace contains multiple
2555              devices. If you want multiple devices  to  be  replayed  concur‐
2556              rently  to  multiple  redirected  devices you must blkparse your
2557              trace into separate traces and replay them with independent  fio
2558              invocations.  Unfortunately this also breaks the strict time or‐
2559              dering between multiple device accesses.
2560
2561       replay_align=int
2562              Force alignment of the byte offsets in a trace  to  this  value.
2563              The value must be a power of 2.
2564
2565       replay_scale=int
2566              Scale  bye  offsets  down  by this factor when replaying traces.
2567              Should most likely use replay_align as well.
2568
2569   Threads, processes and job synchronization
2570       replay_skip=str
2571              Sometimes it's useful to skip  certain  IO  types  in  a  replay
2572              trace.  This  could  be, for instance, eliminating the writes in
2573              the trace. Or not replaying the trims/discards, if you are redi‐
2574              recting  to  a  device  that  doesn't support them.  This option
2575              takes a comma separated list of read, write, trim, sync.
2576
2577       thread Fio defaults to creating jobs by using fork, however if this op‐
2578              tion  is  given,  fio  will  create jobs by using POSIX Threads'
2579              function pthread_create(3) to create threads instead.
2580
2581       wait_for=str
2582              If set, the current job won't be started until  all  workers  of
2583              the specified waitee job are done.  wait_for operates on the job
2584              name basis, so there are a few limitations.  First,  the  waitee
2585              must be defined prior to the waiter job (meaning no forward ref‐
2586              erences). Second, if a job is being referenced as a  waitee,  it
2587              must have a unique name (no duplicate waitees).
2588
2589       nice=int
2590              Run the job with the given nice value. See man nice(2).  On Win‐
2591              dows, values less than -15 set the process class to  "High";  -1
2592              through -15 set "Above Normal"; 1 through 15 "Below Normal"; and
2593              above 15 "Idle" priority class.
2594
2595       prio=int
2596              Set the I/O priority value of this job. Linux  limits  us  to  a
2597              positive  value  between  0 and 7, with 0 being the highest. See
2598              man ionice(1). Refer to an appropriate manpage for other operat‐
2599              ing  systems  since meaning of priority may differ. For per-com‐
2600              mand priority setting, see the I/O engine specific `cmdprio_per‐
2601              centage` and `cmdprio` options.
2602
2603       prioclass=int
2604              Set  the  I/O priority class. See man ionice(1). For per-command
2605              priority setting, see the I/O engine specific  `cmdprio_percent‐
2606              age` and `cmdprio_class` options.
2607
2608       cpus_allowed=str
2609              Controls  the  same  options  as  cpumask, but accepts a textual
2610              specification of the permitted CPUs instead and CPUs are indexed
2611              from  0.  So  to  use  CPUs  0 and 5 you would specify `cpus_al‐
2612              lowed=0,5'. This option also allows a range of CPUs to be speci‐
2613              fied  -- say you wanted a binding to CPUs 0, 5, and 8 to 15, you
2614              would set `cpus_allowed=0,5,8-15'.
2615
2616              On Windows, when `cpus_allowed' is unset only  CPUs  from  fio's
2617              current  processor  group will be used and affinity settings are
2618              inherited from the system. An fio  build  configured  to  target
2619              Windows  7 makes options that set CPUs processor group aware and
2620              values will set both the processor group and a CPU  from  within
2621              that group. For example, on a system where processor group 0 has
2622              40 CPUs and processor group 1 has 32 CPUs, `cpus_allowed' values
2623              between  0  and  39  will  bind  CPUs from processor group 0 and
2624              `cpus_allowed' values between 40 and 71 will bind CPUs from pro‐
2625              cessor group 1. When using `cpus_allowed_policy=shared' all CPUs
2626              specified by a single `cpus_allowed' option  must  be  from  the
2627              same  processor group. For Windows fio builds not built for Win‐
2628              dows 7, CPUs will only be selected from  (and  be  relative  to)
2629              whatever  processor  group fio happens to be running in and CPUs
2630              from other processor groups cannot be used.
2631
2632       cpus_allowed_policy=str
2633              Set the policy of how fio  distributes  the  CPUs  specified  by
2634              cpus_allowed or cpumask. Two policies are supported:
2635
2636                     shared All jobs will share the CPU set specified.
2637
2638                     split  Each job will get a unique CPU from the CPU set.
2639
2640              shared  is  the default behavior, if the option isn't specified.
2641              If split is specified, then fio will assign one cpu per job.  If
2642              not  enough  CPUs  are  given for the jobs listed, then fio will
2643              roundrobin the CPUs in the set.
2644
2645       cpumask=int
2646              Set the CPU affinity of this job. The parameter given is  a  bit
2647              mask  of allowed CPUs the job may run on. So if you want the al‐
2648              lowed CPUs to be 1 and 5, you would pass the decimal value of (1
2649              <<  1  |  1 << 5), or 34. See man sched_setaffinity(2). This may
2650              not work on all supported operating systems or kernel  versions.
2651              This  option  doesn't work well for a higher CPU count than what
2652              you can store in an integer mask, so it can  only  control  cpus
2653              1-32. For boxes with larger CPU counts, use cpus_allowed.
2654
2655       numa_cpu_nodes=str
2656              Set  this  job  running on specified NUMA nodes' CPUs. The argu‐
2657              ments allow comma delimited list of cpu numbers, A-B ranges,  or
2658              `all'.  Note,  to enable NUMA options support, fio must be built
2659              on a system with libnuma-dev(el) installed.
2660
2661       numa_mem_policy=str
2662              Set this job's memory policy and corresponding NUMA nodes.  For‐
2663              mat of the arguments:
2664
2665                     <mode>[:<nodelist>]
2666
2667              `mode' is one of the following memory policies: `default', `pre‐
2668              fer', `bind', `interleave' or `local'. For `default' and `local'
2669              memory  policies,  no  node needs to be specified. For `prefer',
2670              only one node is allowed. For `bind' and `interleave' the `node‐
2671              list'  may be as follows: a comma delimited list of numbers, A-B
2672              ranges, or `all'.
2673
2674       cgroup=str
2675              Add job to this control group. If it doesn't exist, it  will  be
2676              created. The system must have a mounted cgroup blkio mount point
2677              for this to work. If your system doesn't have  it  mounted,  you
2678              can do so with:
2679
2680                     # mount -t cgroup -o blkio none /cgroup
2681
2682       cgroup_weight=int
2683              Set  the  weight of the cgroup to this value. See the documenta‐
2684              tion that comes with the kernel, allowed values are in the range
2685              of 100..1000.
2686
2687       cgroup_nodelete=bool
2688              Normally  fio  will  delete the cgroups it has created after the
2689              job completion. To override this behavior and to  leave  cgroups
2690              around  after  the job completion, set `cgroup_nodelete=1'. This
2691              can be useful if one wants to inspect various cgroup files after
2692              job completion. Default: false.
2693
2694       flow_id=int
2695              The  ID  of  the  flow. If not specified, it defaults to being a
2696              global flow. See flow.
2697
2698       flow=int
2699              Weight in token-based flow control. If this value is used,  then
2700              fio  regulates the activity between two or more jobs sharing the
2701              same flow_id.  Fio attempts to keep each  job  activity  propor‐
2702              tional to other jobs' activities in the same flow_id group, with
2703              respect to requested weight per job.  That is, if  one  job  has
2704              `flow=3',  another  job  has `flow=2' and another with `flow=1`,
2705              then there will be a roughly 3:2:1 ratio in how much one runs vs
2706              the others.
2707
2708       flow_sleep=int
2709              The  period  of  time,  in  microseconds, to wait after the flow
2710              counter has exceeded its proportion before retrying operations.
2711
2712       stonewall, wait_for_previous
2713              Wait for preceding jobs in the job file to exit, before starting
2714              this  one. Can be used to insert serialization points in the job
2715              file. A stone wall also implies starting a new reporting  group,
2716              see  group_reporting.  Optionally  you  can use `stonewall=0` to
2717              disable or `stonewall=1` to enable it.
2718
2719       exitall
2720              By default, fio will continue running all other  jobs  when  one
2721              job finishes.  Sometimes this is not the desired action. Setting
2722              exitall will instead make fio terminate all  jobs  in  the  same
2723              group, as soon as one job of that group finishes.
2724
2725       exit_what=str
2726              By  default,  fio  will continue running all other jobs when one
2727              job finishes.  Sometimes this is not the desired action. Setting
2728              exitall  will  instead  make  fio terminate all jobs in the same
2729              group. The option exit_what allows you to control which jobs get
2730              terminated when exitall is enabled.  The default value is group.
2731              The allowed values are:
2732
2733                     all    terminates all jobs.
2734
2735                     group  is the default and does not change  the  behaviour
2736                            of exitall.
2737
2738                     stonewall
2739                            terminates  all  currently running jobs across all
2740                            groups  and  continues  execution  with  the  next
2741                            stonewalled group.
2742
2743       exec_prerun=str
2744              Before  running  this  job,  issue the command specified through
2745              system(3). Output is redirected in a file  called  `jobname.pre‐
2746              run.txt'.
2747
2748       exec_postrun=str
2749              After the job completes, issue the command specified though sys‐
2750              tem(3).  Output  is  redirected   in   a   file   called   `job‐
2751              name.postrun.txt'.
2752
2753       uid=int
2754              Instead of running as the invoking user, set the user ID to this
2755              value before the thread/process does any work.
2756
2757       gid=int
2758              Set group ID, see uid.
2759
2760   Verification
2761       verify_only
2762              Do not  perform  specified  workload,  only  verify  data  still
2763              matches previous invocation of this workload. This option allows
2764              one to check data multiple times at a later date  without  over‐
2765              writing  it.  This  option  makes  sense only for workloads that
2766              write data, and does not support workloads with  the  time_based
2767              option set.
2768
2769       do_verify=bool
2770              Run  the  verify phase after a write phase. Only valid if verify
2771              is set. Default: true.
2772
2773       verify=str
2774              If writing to a file, fio can verify  the  file  contents  after
2775              each iteration of the job. Each verification method also implies
2776              verification of special header, which is written to  the  begin‐
2777              ning  of each block. This header also includes meta information,
2778              like offset of the block, block number, timestamp when block was
2779              written, etc. verify can be combined with verify_pattern option.
2780              The allowed values are:
2781
2782                     md5    Use an md5 sum of the data area and  store  it  in
2783                            the header of each block.
2784
2785                     crc64  Use an experimental crc64 sum of the data area and
2786                            store it in the header of each block.
2787
2788                     crc32c Use a crc32c sum of the data area and store it  in
2789                            the  header of each block. This will automatically
2790                            use hardware acceleration (e.g. SSE4.2 on  an  x86
2791                            or  CRC  crypto extensions on ARM64) but will fall
2792                            back to software crc32c if none is  found.  Gener‐
2793                            ally  the fastest checksum fio supports when hard‐
2794                            ware accelerated.
2795
2796                     crc32c-intel
2797                            Synonym for crc32c.
2798
2799                     crc32  Use a crc32 sum of the data area and store  it  in
2800                            the header of each block.
2801
2802                     crc16  Use  a  crc16 sum of the data area and store it in
2803                            the header of each block.
2804
2805                     crc7   Use a crc7 sum of the data area and  store  it  in
2806                            the header of each block.
2807
2808                     xxhash Use xxhash as the checksum function. Generally the
2809                            fastest software checksum that fio supports.
2810
2811                     sha512 Use sha512 as the checksum function.
2812
2813                     sha256 Use sha256 as the checksum function.
2814
2815                     sha1   Use optimized sha1 as the checksum function.
2816
2817                     sha3-224
2818                            Use optimized sha3-224 as the checksum function.
2819
2820                     sha3-256
2821                            Use optimized sha3-256 as the checksum function.
2822
2823                     sha3-384
2824                            Use optimized sha3-384 as the checksum function.
2825
2826                     sha3-512
2827                            Use optimized sha3-512 as the checksum function.
2828
2829                     meta   This option is deprecated, since now meta informa‐
2830                            tion  is  included  in generic verification header
2831                            and meta verification happens by default. For  de‐
2832                            tailed information see the description of the ver‐
2833                            ify setting. This option is kept because  of  com‐
2834                            patibility's  sake with old configurations. Do not
2835                            use it.
2836
2837                     pattern
2838                            Verify a strict pattern. Normally fio  includes  a
2839                            header  with  some basic information and checksum‐
2840                            ming, but if this option is set, only the specific
2841                            pattern set with verify_pattern is verified.
2842
2843                     null   Only  pretend to verify. Useful for testing inter‐
2844                            nals with `ioengine=null', not for much else.
2845
2846              This option can be used for repeated burn-in tests of  a  system
2847              to  make sure that the written data is also correctly read back.
2848              If the data direction given is a read or random read,  fio  will
2849              assume  that  it should verify a previously written file. If the
2850              data direction includes any form of write, the verify will be of
2851              the newly written data.
2852
2853              To  avoid  false verification errors, do not use the norandommap
2854              option when verifying data with async I/O engines and I/O depths
2855              >  1.   Or use the norandommap and the lfsr random generator to‐
2856              gether to avoid writing to the same  offset  with  muliple  out‐
2857              standing I/Os.
2858
2859       verify_offset=int
2860              Swap  the  verification  header  with data somewhere else in the
2861              block before writing. It is swapped back before verifying.
2862
2863       verify_interval=int
2864              Write the verification header at a finer  granularity  than  the
2865              blocksize.  It will be written for chunks the size of verify_in‐
2866              terval. blocksize should divide this evenly.
2867
2868       verify_pattern=str
2869              If set, fio will fill the I/O buffers with this pattern. Fio de‐
2870              faults  to filling with totally random bytes, but sometimes it's
2871              interesting to fill with a known pattern  for  I/O  verification
2872              purposes.  Depending  on the width of the pattern, fio will fill
2873              1/2/3/4 bytes of the buffer at the time (it can be either a dec‐
2874              imal  or  a  hex  number).  The  verify_pattern if larger than a
2875              32-bit quantity has to be a hex number that starts  with  either
2876              "0x"  or "0X". Use with verify. Also, verify_pattern supports %o
2877              format, which means that for each block offset will  be  written
2878              and then verified back, e.g.:
2879
2880                     verify_pattern=%o
2881
2882              Or use combination of everything:
2883
2884                     verify_pattern=0xff%o"abcd"-12
2885
2886       verify_fatal=bool
2887              Normally fio will keep checking the entire contents before quit‐
2888              ting on a block verification failure. If this option is set, fio
2889              will exit the job on the first observed failure. Default: false.
2890
2891       verify_dump=bool
2892              If  set,  dump  the contents of both the original data block and
2893              the data block we read off disk  to  files.  This  allows  later
2894              analysis  to inspect just what kind of data corruption occurred.
2895              Off by default.
2896
2897       verify_async=int
2898              Fio will normally verify I/O inline from the submitting  thread.
2899              This  option  takes an integer describing how many async offload
2900              threads to create for I/O verification instead, causing  fio  to
2901              offload  the duty of verifying I/O contents to one or more sepa‐
2902              rate threads. If using this offload option, even  sync  I/O  en‐
2903              gines  can  benefit from using an iodepth setting higher than 1,
2904              as it allows them to have I/O in flight while verifies are  run‐
2905              ning.   Defaults  to  0  async threads, i.e. verification is not
2906              asynchronous.
2907
2908       verify_async_cpus=str
2909              Tell fio to set the given CPU affinity on the async I/O  verifi‐
2910              cation threads. See cpus_allowed for the format used.
2911
2912       verify_backlog=int
2913              Fio will normally verify the written contents of a job that uti‐
2914              lizes verify once that job has completed. In other words, every‐
2915              thing  is written then everything is read back and verified. You
2916              may want to verify continually instead for a variety of reasons.
2917              Fio stores the meta data associated with an I/O block in memory,
2918              so for large verify workloads, quite a bit of  memory  would  be
2919              used  up  holding this meta data. If this option is enabled, fio
2920              will write only N blocks before verifying these blocks.
2921
2922       verify_backlog_batch=int
2923              Control how many blocks fio will  verify  if  verify_backlog  is
2924              set.  If  not  set,  will default to the value of verify_backlog
2925              (meaning the entire queue is read back and  verified).  If  ver‐
2926              ify_backlog_batch  is  less  than  verify_backlog  then  not all
2927              blocks will be verified, if verify_backlog_batch is larger  than
2928              verify_backlog, some blocks will be verified more than once.
2929
2930       verify_state_save=bool
2931              When  a  job  exits during the write phase of a verify workload,
2932              save its current state. This allows fio to replay up until  that
2933              point,  if the verify state is loaded for the verify read phase.
2934              The format of the filename is, roughly:
2935
2936                     <type>-<jobname>-<jobindex>-verify.state.
2937
2938              <type> is "local" for a local run, "sock"  for  a  client/server
2939              socket  connection,  and  "ip" (192.168.0.1, for instance) for a
2940              networked client/server connection. Defaults to true.
2941
2942       verify_state_load=bool
2943              If a verify termination trigger was used, fio stores the current
2944              write  state  of  each  thread. This can be used at verification
2945              time so that fio knows how far it should  verify.  Without  this
2946              information, fio will run a full verification pass, according to
2947              the settings in the job file used. Default false.
2948
2949       trim_percentage=int
2950              Number of verify blocks to discard/trim.
2951
2952       trim_verify_zero=bool
2953              Verify that trim/discarded blocks are returned as zeros.
2954
2955       trim_backlog=int
2956              Verify that trim/discarded blocks are returned as zeros.
2957
2958       trim_backlog_batch=int
2959              Trim this number of I/O blocks.
2960
2961       experimental_verify=bool
2962              Enable experimental verification.
2963
2964   Steady state
2965       steadystate=str:float, ss=str:float
2966              Define the criterion and limit for assessing steady  state  per‐
2967              formance.  The  first parameter designates the criterion whereas
2968              the second parameter sets  the  threshold.  When  the  criterion
2969              falls  below  the  threshold for the specified duration, the job
2970              will stop. For example, `iops_slope:0.1%'  will  direct  fio  to
2971              terminate  the job when the least squares regression slope falls
2972              below 0.1% of the mean IOPS. If group_reporting is enabled  this
2973              will apply to all jobs in the group. Below is the list of avail‐
2974              able steady state assessment criteria. All assessments are  car‐
2975              ried  out  using  only  data from the rolling collection window.
2976              Threshold limits can be expressed as a fixed value or as a  per‐
2977              centage of the mean in the collection window.
2978
2979              When using this feature, most jobs should include the time_based
2980              and runtime options or the loops option so  that  fio  does  not
2981              stop running after it has covered the full size of the specified
2982              file(s) or device(s).
2983
2984                            iops   Collect IOPS data. Stop the job if all  in‐
2985                                   dividual  IOPS  measurements are within the
2986                                   specified limit of  the  mean  IOPS  (e.g.,
2987                                   `iops:2'  means  that  all  individual IOPS
2988                                   values  must  be  within  2  of  the  mean,
2989                                   whereas `iops:0.2%' means that all individ‐
2990                                   ual IOPS values must be within 0.2% of  the
2991                                   mean IOPS to terminate the job).
2992
2993                            iops_slope
2994                                   Collect  IOPS  data and calculate the least
2995                                   squares regression slope. Stop the  job  if
2996                                   the slope falls below the specified limit.
2997
2998                            bw     Collect bandwidth data. Stop the job if all
2999                                   individual   bandwidth   measurements   are
3000                                   within  the  specified  limit  of  the mean
3001                                   bandwidth.
3002
3003                            bw_slope
3004                                   Collect bandwidth data  and  calculate  the
3005                                   least  squares  regression  slope. Stop the
3006                                   job if the slope falls below the  specified
3007                                   limit.
3008
3009              steadystate_duration=time, ss_dur=time
3010                     A  rolling  window of this duration will be used to judge
3011                     whether steady state has been reached. Data will be  col‐
3012                     lected  once  per second. The default is 0 which disables
3013                     steady state detection. When the  unit  is  omitted,  the
3014                     value is interpreted in seconds.
3015
3016              steadystate_ramp_time=time, ss_ramp=time
3017                     Allow  the  job  to run for the specified duration before
3018                     beginning data collection for checking the  steady  state
3019                     job  termination  criterion.  The  default is 0. When the
3020                     unit is omitted, the value is interpreted in seconds.
3021
3022   Measurements and reporting
3023       per_job_logs=bool
3024              If set, this generates bw/clat/iops log with  per  file  private
3025              filenames.  If not set, jobs with identical names will share the
3026              log filename. Default: true.
3027
3028       group_reporting
3029              It may sometimes be interesting to display statistics for groups
3030              of  jobs  as a whole instead of for each individual job. This is
3031              especially true  if  numjobs  is  used;  looking  at  individual
3032              thread/process output quickly becomes unwieldy. To see the final
3033              report per-group instead of per-job, use  group_reporting.  Jobs
3034              in  a  file  will be part of the same reporting group, unless if
3035              separated by a stonewall, or by using new_group.
3036
3037       new_group
3038              Start a new reporting group. See: group_reporting. If not given,
3039              all jobs in a file will be part of the same reporting group, un‐
3040              less separated by a stonewall.
3041
3042       stats=bool
3043              By default, fio collects and shows final output results for  all
3044              jobs  that run. If this option is set to 0, then fio will ignore
3045              it in the final stat output.
3046
3047       write_bw_log=str
3048              If given, write a bandwidth log for this job.  Can  be  used  to
3049              store data of the bandwidth of the jobs in their lifetime.
3050
3051              If  no  str  argument  is  given,  the default filename of `job‐
3052              name_type.x.log' is used. Even when the argument is  given,  fio
3053              will still append the type of log. So if one specifies:
3054
3055                     write_bw_log=foo
3056
3057              The  actual log name will be `foo_bw.x.log' where `x' is the in‐
3058              dex of the job (1..N,  where  N  is  the  number  of  jobs).  If
3059              per_job_logs  is  false,  then the filename will not include the
3060              `.x` job index.
3061
3062              The included fio_generate_plots  script  uses  gnuplot  to  turn
3063              these text files into nice graphs. See the LOG FILE FORMATS sec‐
3064              tion for how data is structured within the file.
3065
3066       write_lat_log=str
3067              Same as write_bw_log, except this option creates I/O  submission
3068              (e.g., `name_slat.x.log'), completion (e.g., `name_clat.x.log'),
3069              and total (e.g., `name_lat.x.log') latency  files  instead.  See
3070              write_bw_log  for  details about the filename format and the LOG
3071              FILE FORMATS section for  how  data  is  structured  within  the
3072              files.
3073
3074       write_hist_log=str
3075              Same  as  write_bw_log but writes an I/O completion latency his‐
3076              togram file (e.g., `name_hist.x.log') instead.  Note  that  this
3077              file  will be empty unless log_hist_msec has also been set.  See
3078              write_bw_log for details about the filename format and  the  LOG
3079              FILE FORMATS section for how data is structured within the file.
3080
3081       write_iops_log=str
3082              Same   as   write_bw_log,   but   writes   an  IOPS  file  (e.g.
3083              `name_iops.x.log`) instead. Because fio defaults  to  individual
3084              I/O  logging,  the  value entry in the IOPS log will be 1 unless
3085              windowed  logging  (see  log_avg_msec)  has  been  enabled.  See
3086              write_bw_log  for details about the filename format and LOG FILE
3087              FORMATS for how data is structured within the file.
3088
3089       log_entries=int
3090              By default, fio will log an entry in the iops,  latency,  or  bw
3091              log  for every I/O that completes. The initial number of I/O log
3092              entries is 1024.  When the log entries are all used, new log en‐
3093              tries are dynamically allocated.  This dynamic log entry alloca‐
3094              tion may negatively impact time-related statistics such  as  I/O
3095              tail latencies (e.g. 99.9th percentile completion latency). This
3096              option allows specifying a larger initial number of log  entries
3097              to  avoid  run-time  allocation of new log entries, resulting in
3098              more precise time-related I/O statistics.  Also see log_avg_msec
3099              as well. Defaults to 1024.
3100
3101       log_avg_msec=int
3102              By  default,  fio  will log an entry in the iops, latency, or bw
3103              log for every I/O that completes. When writing to the disk  log,
3104              that  can quickly grow to a very large size. Setting this option
3105              makes fio average the each log entry over the  specified  period
3106              of  time,  reducing the resolution of the log. See log_max_value
3107              as well. Defaults to 0, logging all entries.  Also see LOG  FILE
3108              FORMATS section.
3109
3110       log_hist_msec=int
3111              Same  as  log_avg_msec,  but logs entries for completion latency
3112              histograms. Computing latency percentiles from averages  of  in‐
3113              tervals  using  log_avg_msec  is inaccurate. Setting this option
3114              makes fio log histogram entries over  the  specified  period  of
3115              time,  reducing  log sizes for high IOPS devices while retaining
3116              percentile accuracy. See log_hist_coarseness and  write_hist_log
3117              as well.  Defaults to 0, meaning histogram logging is disabled.
3118
3119       log_hist_coarseness=int
3120              Integer ranging from 0 to 6, defining the coarseness of the res‐
3121              olution of the histogram logs enabled  with  log_hist_msec.  For
3122              each increment in coarseness, fio outputs half as many bins. De‐
3123              faults to 0, for which histogram logs contain 1216 latency bins.
3124              See LOG FILE FORMATS section.
3125
3126       log_max_value=bool
3127              If  log_avg_msec  is set, fio logs the average over that window.
3128              If you instead want to log the maximum value, set this option to
3129              1. Defaults to 0, meaning that averaged values are logged.
3130
3131       log_offset=bool
3132              If  this  is set, the iolog options will include the byte offset
3133              for the I/O entry as well as the other data values. Defaults  to
3134              0  meaning  that  offsets  are not present in logs. Also see LOG
3135              FILE FORMATS section.
3136
3137       log_prio=bool
3138              If this is set, the iolog options will include the I/O  priority
3139              for  the I/O entry as well as the other data values. Defaults to
3140              0 meaning that I/O priorities are not present in logs. Also  see
3141              LOG FILE FORMATS section.
3142
3143       log_compression=int
3144              If  this  is  set, fio will compress the I/O logs as it goes, to
3145              keep the memory footprint lower. When a log reaches  the  speci‐
3146              fied  size,  that  chunk  is removed and compressed in the back‐
3147              ground. Given that I/O logs are fairly highly compressible, this
3148              yields  a  nice  memory savings for longer runs. The downside is
3149              that the compression will consume some background CPU cycles, so
3150              it  may  impact the run. This, however, is also true if the log‐
3151              ging ends up consuming most of the system memory. So  pick  your
3152              poison.  The I/O logs are saved normally at the end of a run, by
3153              decompressing the chunks and storing them in the  specified  log
3154              file. This feature depends on the availability of zlib.
3155
3156       log_compression_cpus=str
3157              Define  the  set  of  CPUs that are allowed to handle online log
3158              compression for the I/O jobs. This can provide better  isolation
3159              between  performance  sensitive jobs, and background compression
3160              work. See cpus_allowed for the format used.
3161
3162       log_store_compressed=bool
3163              If set, fio will store the log files  in  a  compressed  format.
3164              They  can be decompressed with fio, using the --inflate-log com‐
3165              mand line parameter. The files will be stored with a `.fz'  suf‐
3166              fix.
3167
3168       log_unix_epoch=bool
3169              If  set,  fio will log Unix timestamps to the log files produced
3170              by enabling write_type_log for each log type, instead of the de‐
3171              fault zero-based timestamps.
3172
3173       block_error_percentiles=bool
3174              If  set, record errors in trim block-sized units from writes and
3175              trims and output a histogram of how many trims it took to get to
3176              errors, and what kind of error was encountered.
3177
3178       bwavgtime=int
3179              Average  the  calculated bandwidth over the given time. Value is
3180              specified in milliseconds. If the job also does  bandwidth  log‐
3181              ging  through  write_bw_log, then the minimum of this option and
3182              log_avg_msec will be used. Default: 500ms.
3183
3184       iopsavgtime=int
3185              Average the calculated IOPS over the given time. Value is speci‐
3186              fied  in milliseconds. If the job also does IOPS logging through
3187              write_iops_log, then the minimum of this option and log_avg_msec
3188              will be used. Default: 500ms.
3189
3190       disk_util=bool
3191              Generate  disk  utilization statistics, if the platform supports
3192              it.  Default: true.
3193
3194       disable_lat=bool
3195              Disable measurements of total latency numbers. Useful  only  for
3196              cutting  back  the  number  of calls to gettimeofday(2), as that
3197              does impact performance at really high IOPS rates. Note that  to
3198              really  get  rid  of  a large amount of these calls, this option
3199              must be used with  disable_slat  and  disable_bw_measurement  as
3200              well.
3201
3202       disable_clat=bool
3203              Disable  measurements  of  completion  latency numbers. See dis‐
3204              able_lat.
3205
3206       disable_slat=bool
3207              Disable measurements of submission  latency  numbers.  See  dis‐
3208              able_lat.
3209
3210       disable_bw_measurement=bool, disable_bw=bool
3211              Disable  measurements  of throughput/bandwidth numbers. See dis‐
3212              able_lat.
3213
3214       slat_percentiles=bool
3215              Report submission latency percentiles. Submission latency is not
3216              recorded for synchronous ioengines.
3217
3218       clat_percentiles=bool
3219              Report completion latency percentiles.
3220
3221       lat_percentiles=bool
3222              Report  total  latency  percentiles. Total latency is the sum of
3223              submission latency and completion latency.
3224
3225       percentile_list=float_list
3226              Overwrite the default list of percentiles for latencies and  the
3227              block error histogram. Each number is a floating point number in
3228              the range (0,100], and the maximum length of the list is 20. Use
3229              ':'    to   separate   the   numbers.   For   example,   `--per‐
3230              centile_list=99.5:99.9' will cause fio to report the latency du‐
3231              rations  below  which  99.5% and 99.9% of the observed latencies
3232              fell, respectively.
3233
3234       significant_figures=int
3235              If using --output-format of `normal', set the  significant  fig‐
3236              ures  to  this value. Higher values will yield more precise IOPS
3237              and throughput units, while lower values will round. Requires  a
3238              minimum value of 1 and a maximum value of 10. Defaults to 4.
3239
3240   Error handling
3241       exitall_on_error
3242              When  one job finishes in error, terminate the rest. The default
3243              is to wait for each job to finish.
3244
3245       continue_on_error=str
3246              Normally fio will exit the job on the first observed failure. If
3247              this  option  is  set, fio will continue the job when there is a
3248              'non-fatal error' (EIO or EILSEQ) until the runtime is  exceeded
3249              or  the I/O size specified is completed. If this option is used,
3250              there are two more stats that  are  appended,  the  total  error
3251              count and the first error. The error field given in the stats is
3252              the first error that was hit during the run.  The allowed values
3253              are:
3254
3255                     none   Exit on any I/O or verify errors.
3256
3257                     read   Continue on read errors, exit on all others.
3258
3259                     write  Continue on write errors, exit on all others.
3260
3261                     io     Continue on any I/O error, exit on all others.
3262
3263                     verify Continue on verify errors, exit on all others.
3264
3265                     all    Continue on all errors.
3266
3267                     0      Backward-compatible alias for 'none'.
3268
3269                     1      Backward-compatible alias for 'all'.
3270
3271       ignore_error=str
3272              Sometimes  you  want  to  ignore some errors during test in that
3273              case you can specify error list for each error type, instead  of
3274              only  being  able  to ignore the default 'non-fatal error' using
3275              continue_on_error.                                   `ignore_er‐
3276              ror=READ_ERR_LIST,WRITE_ERR_LIST,VERIFY_ERR_LIST'   errors   for
3277              given error type is separated with  ':'.  Error  may  be  symbol
3278              ('ENOSPC', 'ENOMEM') or integer. Example:
3279
3280                     ignore_error=EAGAIN,ENOSPC:122
3281
3282              This  option  will  ignore  EAGAIN  from  READ,  and  ENOSPC and
3283              122(EDQUOT) from WRITE. This option  works  by  overriding  con‐
3284              tinue_on_error  with  the  list of errors for each error type if
3285              any.
3286
3287       error_dump=bool
3288              If set dump every error even if it is non  fatal,  true  by  de‐
3289              fault. If disabled only fatal error will be dumped.
3290
3291   Running predefined workloads
3292       Fio includes predefined profiles that mimic the I/O workloads generated
3293       by other tools.
3294
3295       profile=str
3296              The predefined workload to run. Current profiles are:
3297
3298                     tiobench
3299                            Threaded I/O bench (tiotest/tiobench)  like  work‐
3300                            load.
3301
3302                     act    Aerospike Certification Tool (ACT) like workload.
3303
3304       To  view  a profile's additional options use --cmdhelp after specifying
3305       the profile. For example:
3306
3307              $ fio --profile=act --cmdhelp
3308
3309   Act profile options
3310       device-names=str
3311              Devices to use.
3312
3313       load=int
3314              ACT load multiplier. Default: 1.
3315
3316       test-duration=time
3317              How long the entire test takes to run. When the unit is omitted,
3318              the value is given in seconds. Default: 24h.
3319
3320       threads-per-queue=int
3321              Number of read I/O threads per device. Default: 8.
3322
3323       read-req-num-512-blocks=int
3324              Number of 512B blocks to read at the time. Default: 3.
3325
3326       large-block-op-kbytes=int
3327              Size of large block ops in KiB (writes). Default: 131072.
3328
3329       prep   Set to run ACT prep phase.
3330
3331   Tiobench profile options
3332       size=str
3333              Size in MiB.
3334
3335       block=int
3336              Block size in bytes. Default: 4096.
3337
3338       numruns=int
3339              Number of runs.
3340
3341       dir=str
3342              Test directory.
3343
3344       threads=int
3345              Number of threads.
3346

OUTPUT

3348       Fio spits out a lot of output. While running, fio will display the sta‐
3349       tus of the jobs created. An example of that would be:
3350
3351                 Jobs: 1 (f=1): [_(1),M(1)][24.8%][r=20.5MiB/s,w=23.5MiB/s][r=82,w=94 IOPS][eta 01m:31s]
3352
3353       The characters inside the first set of square brackets denote the  cur‐
3354       rent  status  of  each thread. The first character is the first job de‐
3355       fined in the job file, and so forth. The possible  values  (in  typical
3356       life cycle order) are:
3357
3358              P      Thread setup, but not started.
3359              C      Thread created.
3360              I      Thread initialized, waiting or generating necessary data.
3361              p      Thread running pre-reading file(s).
3362              /      Thread is in ramp period.
3363              R      Running, doing sequential reads.
3364              r      Running, doing random reads.
3365              W      Running, doing sequential writes.
3366              w      Running, doing random writes.
3367              M      Running, doing mixed sequential reads/writes.
3368              m      Running, doing mixed random reads/writes.
3369              D      Running, doing sequential trims.
3370              d      Running, doing random trims.
3371              F      Running, currently waiting for fsync(2).
3372              V      Running, doing verification of written data.
3373              f      Thread finishing.
3374              E      Thread exited, not reaped by main thread yet.
3375              -      Thread reaped.
3376              X      Thread reaped, exited with an error.
3377              K      Thread reaped, exited due to signal.
3378
3379       Fio will condense the thread string as not to take up more space on the
3380       command line than needed. For instance, if you have 10 readers  and  10
3381       writers running, the output would look like this:
3382
3383                 Jobs: 20 (f=20): [R(10),W(10)][4.0%][r=20.5MiB/s,w=23.5MiB/s][r=82,w=94 IOPS][eta 57m:36s]
3384
3385       Note  that the status string is displayed in order, so it's possible to
3386       tell which of the jobs are currently doing what. In the  example  above
3387       this means that jobs 1--10 are readers and 11--20 are writers.
3388
3389       The  other values are fairly self explanatory -- number of threads cur‐
3390       rently running and doing I/O, the number of currently open files  (f=),
3391       the  estimated  completion percentage, the rate of I/O since last check
3392       (read speed listed first, then write speed and optionally  trim  speed)
3393       in  terms of bandwidth and IOPS, and time to completion for the current
3394       running group. It's impossible to estimate  runtime  of  the  following
3395       groups (if any).
3396
3397       When  fio is done (or interrupted by Ctrl-C), it will show the data for
3398       each thread, group of threads, and disks in that order. For each  over‐
3399       all thread (or group) the output looks like:
3400
3401                 Client1: (groupid=0, jobs=1): err= 0: pid=16109: Sat Jun 24 12:07:54 2017
3402                   write: IOPS=88, BW=623KiB/s (638kB/s)(30.4MiB/50032msec)
3403                     slat (nsec): min=500, max=145500, avg=8318.00, stdev=4781.50
3404                     clat (usec): min=170, max=78367, avg=4019.02, stdev=8293.31
3405                      lat (usec): min=174, max=78375, avg=4027.34, stdev=8291.79
3406                     clat percentiles (usec):
3407                      |  1.00th=[  302],  5.00th=[  326], 10.00th=[  343], 20.00th=[  363],
3408                      | 30.00th=[  392], 40.00th=[  404], 50.00th=[  416], 60.00th=[  445],
3409                      | 70.00th=[  816], 80.00th=[ 6718], 90.00th=[12911], 95.00th=[21627],
3410                      | 99.00th=[43779], 99.50th=[51643], 99.90th=[68682], 99.95th=[72877],
3411                      | 99.99th=[78119]
3412                    bw (  KiB/s): min=  532, max=  686, per=0.10%, avg=622.87, stdev=24.82, samples=  100
3413                    iops        : min=   76, max=   98, avg=88.98, stdev= 3.54, samples=  100
3414                   lat (usec)   : 250=0.04%, 500=64.11%, 750=4.81%, 1000=2.79%
3415                   lat (msec)   : 2=4.16%, 4=1.84%, 10=4.90%, 20=11.33%, 50=5.37%
3416                   lat (msec)   : 100=0.65%
3417                   cpu          : usr=0.27%, sys=0.18%, ctx=12072, majf=0, minf=21
3418                   IO depths    : 1=85.0%, 2=13.1%, 4=1.8%, 8=0.1%, 16=0.0%, 32=0.0%, >=64=0.0%
3419                      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
3420                      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
3421                      issued rwt: total=0,4450,0, short=0,0,0, dropped=0,0,0
3422                      latency   : target=0, window=0, percentile=100.00%, depth=8
3423
3424       The  job  name  (or  first  job's  name  when using group_reporting) is
3425       printed, along with the group id, count of jobs being aggregated,  last
3426       error  id  seen  (which is 0 when there are no errors), pid/tid of that
3427       thread and the time the job/group completed. Below are the I/O  statis‐
3428       tics  for  each data direction performed (showing writes in the example
3429       above). In the order listed, they denote:
3430
3431              read/write/trim
3432                     The string before the colon shows the I/O  direction  the
3433                     statistics  are  for.  IOPS is the average I/Os performed
3434                     per second. BW is the average bandwidth  rate  shown  as:
3435                     value in power of 2 format (value in power of 10 format).
3436                     The last two values show: (total I/O performed  in  power
3437                     of 2 format / runtime of that thread).
3438
3439              slat   Submission  latency (min being the minimum, max being the
3440                     maximum, avg being the average, stdev being the  standard
3441                     deviation).  This  is the time it took to submit the I/O.
3442                     For sync I/O this row is not displayed as the slat is re‐
3443                     ally  the completion latency (since queue/complete is one
3444                     operation there).  This value can be in nanoseconds,  mi‐
3445                     croseconds  or  milliseconds --- fio will choose the most
3446                     appropriate base and print that  (in  the  example  above
3447                     nanoseconds  was the best scale). Note: in --minimal mode
3448                     latencies are always expressed in microseconds.
3449
3450              clat   Completion latency. Same names as slat, this denotes  the
3451                     time from submission to completion of the I/O pieces. For
3452                     sync I/O, clat will usually be equal (or very  close)  to
3453                     0,  as the time from submit to complete is basically just
3454                     CPU time (I/O has already been done,  see  slat  explana‐
3455                     tion).
3456
3457              lat    Total  latency. Same names as slat and clat, this denotes
3458                     the time from when fio created the I/O unit to completion
3459                     of the I/O operation.
3460
3461              bw     Bandwidth  statistics based on samples. Same names as the
3462                     xlat stats, but also includes the number of samples taken
3463                     (samples)  and  an approximate percentage of total aggre‐
3464                     gate bandwidth this thread received in its  group  (per).
3465                     This  last  value is only really useful if the threads in
3466                     this group are on the same disk, since they are then com‐
3467                     peting for disk access.
3468
3469              iops   IOPS statistics based on samples. Same names as bw.
3470
3471              lat (nsec/usec/msec)
3472                     The distribution of I/O completion latencies. This is the
3473                     time from when I/O leaves fio and when it gets completed.
3474                     Unlike  the  separate read/write/trim sections above, the
3475                     data here and in the remaining sections apply to all I/Os
3476                     for  the  reporting  group. 250=0.04% means that 0.04% of
3477                     the I/Os completed in under 250us. 500=64.11% means  that
3478                     64.11% of the I/Os required 250 to 499us for completion.
3479
3480              cpu    CPU usage. User and system time, along with the number of
3481                     context switches this thread went through, usage of  sys‐
3482                     tem  and  user  time, and finally the number of major and
3483                     minor page faults. The CPU utilization numbers are  aver‐
3484                     ages for the jobs in that reporting group, while the con‐
3485                     text and fault counters are summed.
3486
3487              IO depths
3488                     The distribution of I/O depths over the job lifetime. The
3489                     numbers  are divided into powers of 2 and each entry cov‐
3490                     ers depths from that value up to  those  that  are  lower
3491                     than the next entry -- e.g., 16= covers depths from 16 to
3492                     31. Note that the range covered by a  depth  distribution
3493                     entry can be different to the range covered by the equiv‐
3494                     alent submit/complete distribution entry.
3495
3496              IO submit
3497                     How many pieces of I/O were submitting in a single submit
3498                     call. Each entry denotes that amount and below, until the
3499                     previous entry -- e.g., 16=100% means that  we  submitted
3500                     anywhere  between 9 to 16 I/Os per submit call. Note that
3501                     the range covered by a submit distribution entry  can  be
3502                     different  to  the  range covered by the equivalent depth
3503                     distribution entry.
3504
3505              IO complete
3506                     Like the above submit number,  but  for  completions  in‐
3507                     stead.
3508
3509              IO issued rwt
3510                     The  number  of  read/write/trim requests issued, and how
3511                     many of them were short or dropped.
3512
3513              IO latency
3514                     These values are for latency_target and related  options.
3515                     When  these  options  are engaged, this section describes
3516                     the I/O depth required to meet the specified latency tar‐
3517                     get.
3518
3519       After  each  client  has been listed, the group statistics are printed.
3520       They will look like this:
3521
3522                 Run status group 0 (all jobs):
3523                    READ: bw=20.9MiB/s (21.9MB/s), 10.4MiB/s-10.8MiB/s (10.9MB/s-11.3MB/s), io=64.0MiB (67.1MB), run=2973-3069msec
3524                   WRITE: bw=1231KiB/s (1261kB/s), 616KiB/s-621KiB/s (630kB/s-636kB/s), io=64.0MiB (67.1MB), run=52747-53223msec
3525
3526       For each data direction it prints:
3527
3528              bw     Aggregate bandwidth of threads in this group followed  by
3529                     the  minimum  and maximum bandwidth of all the threads in
3530                     this group.  Values outside of  brackets  are  power-of-2
3531                     format  and  those  within  are the equivalent value in a
3532                     power-of-10 format.
3533
3534              io     Aggregate I/O performed of all threads in this group. The
3535                     format is the same as bw.
3536
3537              run    The  smallest and longest runtimes of the threads in this
3538                     group.
3539
3540       And finally, the disk statistics are printed. This is  Linux  specific.
3541       They will look like this:
3542
3543                   Disk stats (read/write):
3544                     sda: ios=16398/16511, merge=30/162, ticks=6853/819634, in_queue=826487, util=100.00%
3545
3546       Each  value is printed for both reads and writes, with reads first. The
3547       numbers denote:
3548
3549              ios    Number of I/Os performed by all groups.
3550
3551              merge  Number of merges performed by the I/O scheduler.
3552
3553              ticks  Number of ticks we kept the disk busy.
3554
3555              in_queue
3556                     Total time spent in the disk queue.
3557
3558              util   The disk utilization. A value of 100% means we  kept  the
3559                     disk  busy constantly, 50% would be a disk idling half of
3560                     the time.
3561
3562       It is also possible to get fio to dump the current output while  it  is
3563       running,  without  terminating  the  job. To do that, send fio the USR1
3564       signal. You can also get regularly timed  dumps  by  using  the  --sta‐
3565       tus-interval   parameter,  or  by  creating  a  file  in  `/tmp'  named
3566       `fio-dump-status'. If fio sees this file, it will unlink  it  and  dump
3567       the current output status.
3568

TERSE OUTPUT

3570       For  scripted  usage  where  you  typically  want to generate tables or
3571       graphs of the results, fio can output the results in a semicolon  sepa‐
3572       rated format. The format is one long line of values, such as:
3573
3574                 2;card0;0;0;7139336;121836;60004;1;10109;27.932460;116.933948;220;126861;3495.446807;1085.368601;226;126864;3523.635629;1089.012448;24063;99944;50.275485%;59818.274627;5540.657370;7155060;122104;60004;1;8338;29.086342;117.839068;388;128077;5032.488518;1234.785715;391;128085;5061.839412;1236.909129;23436;100928;50.287926%;59964.832030;5644.844189;14.595833%;19.394167%;123706;0;7313;0.1%;0.1%;0.1%;0.1%;0.1%;0.1%;100.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.02%;0.05%;0.16%;6.04%;40.40%;52.68%;0.64%;0.01%;0.00%;0.01%;0.00%;0.00%;0.00%;0.00%;0.00%
3575                 A description of this job goes here.
3576
3577       The  job  description  (if provided) follows on a second line for terse
3578       v2.  It appears on the same line for other terse versions.
3579
3580       To enable terse output, use the  --minimal  or  `--output-format=terse'
3581       command  line options. The first value is the version of the terse out‐
3582       put format. If the output has to be changed for some reason, this  num‐
3583       ber will be incremented by 1 to signify that change.
3584
3585       Split  up, the format is as follows (comments in brackets denote when a
3586       field was introduced or whether it's specific to some terse version):
3587
3588                      terse version, fio version [v3], jobname, groupid, error
3589
3590              READ status:
3591
3592                      Total IO (KiB), bandwidth (KiB/sec), IOPS, runtime (msec)
3593                      Submission latency: min, max, mean, stdev (usec)
3594                      Completion latency: min, max, mean, stdev (usec)
3595                      Completion latency percentiles: 20 fields (see below)
3596                      Total latency: min, max, mean, stdev (usec)
3597                      Bw (KiB/s): min, max, aggregate percentage of total, mean, stdev, number of samples [v5]
3598                      IOPS [v5]: min, max, mean, stdev, number of samples
3599
3600              WRITE status:
3601
3602                      Total IO (KiB), bandwidth (KiB/sec), IOPS, runtime (msec)
3603                      Submission latency: min, max, mean, stdev (usec)
3604                      Completion latency: min, max, mean, stdev (usec)
3605                      Completion latency percentiles: 20 fields (see below)
3606                      Total latency: min, max, mean, stdev (usec)
3607                      Bw (KiB/s): min, max, aggregate percentage of total, mean, stdev, number of samples [v5]
3608                      IOPS [v5]: min, max, mean, stdev, number of samples
3609
3610              TRIM status [all but version 3]:
3611
3612                      Fields are similar to READ/WRITE status.
3613
3614              CPU usage:
3615
3616                      user, system, context switches, major faults, minor faults
3617
3618              I/O depths:
3619
3620                      <=1, 2, 4, 8, 16, 32, >=64
3621
3622              I/O latencies microseconds:
3623
3624                      <=2, 4, 10, 20, 50, 100, 250, 500, 750, 1000
3625
3626              I/O latencies milliseconds:
3627
3628                      <=2, 4, 10, 20, 50, 100, 250, 500, 750, 1000, 2000, >=2000
3629
3630              Disk utilization [v3]:
3631
3632                      disk name, read ios, write ios, read merges, write merges, read ticks, write ticks, time spent in queue, disk utilization percentage
3633
3634              Additional Info (dependent on continue_on_error, default off):
3635
3636                      total # errors, first error code
3637
3638              Additional Info (dependent on description being set):
3639
3640                      Text description
3641
3642       Completion latency percentiles can be a grouping of up to 20  sets,  so
3643       for  the terse output fio writes all of them. Each field will look like
3644       this:
3645
3646                 1.00%=6112
3647
3648       which is the Xth percentile, and the `usec' latency associated with it.
3649
3650       For Disk utilization, all disks used by fio are shown. So for each disk
3651       there will be a disk utilization section.
3652
3653       Below is a single line containing short names for each of the fields in
3654       the minimal output v3, separated by semicolons:
3655
3656                 terse_version_3;fio_version;jobname;groupid;error;read_kb;read_bandwidth_kb;read_iops;read_runtime_ms;read_slat_min_us;read_slat_max_us;read_slat_mean_us;read_slat_dev_us;read_clat_min_us;read_clat_max_us;read_clat_mean_us;read_clat_dev_us;read_clat_pct01;read_clat_pct02;read_clat_pct03;read_clat_pct04;read_clat_pct05;read_clat_pct06;read_clat_pct07;read_clat_pct08;read_clat_pct09;read_clat_pct10;read_clat_pct11;read_clat_pct12;read_clat_pct13;read_clat_pct14;read_clat_pct15;read_clat_pct16;read_clat_pct17;read_clat_pct18;read_clat_pct19;read_clat_pct20;read_tlat_min_us;read_lat_max_us;read_lat_mean_us;read_lat_dev_us;read_bw_min_kb;read_bw_max_kb;read_bw_agg_pct;read_bw_mean_kb;read_bw_dev_kb;write_kb;write_bandwidth_kb;write_iops;write_runtime_ms;write_slat_min_us;write_slat_max_us;write_slat_mean_us;write_slat_dev_us;write_clat_min_us;write_clat_max_us;write_clat_mean_us;write_clat_dev_us;write_clat_pct01;write_clat_pct02;write_clat_pct03;write_clat_pct04;write_clat_pct05;write_clat_pct06;write_clat_pct07;write_clat_pct08;write_clat_pct09;write_clat_pct10;write_clat_pct11;write_clat_pct12;write_clat_pct13;write_clat_pct14;write_clat_pct15;write_clat_pct16;write_clat_pct17;write_clat_pct18;write_clat_pct19;write_clat_pct20;write_tlat_min_us;write_lat_max_us;write_lat_mean_us;write_lat_dev_us;write_bw_min_kb;write_bw_max_kb;write_bw_agg_pct;write_bw_mean_kb;write_bw_dev_kb;cpu_user;cpu_sys;cpu_csw;cpu_mjf;cpu_minf;iodepth_1;iodepth_2;iodepth_4;iodepth_8;iodepth_16;iodepth_32;iodepth_64;lat_2us;lat_4us;lat_10us;lat_20us;lat_50us;lat_100us;lat_250us;lat_500us;lat_750us;lat_1000us;lat_2ms;lat_4ms;lat_10ms;lat_20ms;lat_50ms;lat_100ms;lat_250ms;lat_500ms;lat_750ms;lat_1000ms;lat_2000ms;lat_over_2000ms;disk_name;disk_read_iops;disk_write_iops;disk_read_merges;disk_write_merges;disk_read_ticks;write_ticks;disk_queue_time;disk_util
3657
3658       In client/server mode terse output differs from what appears when  jobs
3659       are  run  locally.  Disk  utilization data is omitted from the standard
3660       terse output and for v3 and later appears on its own separate  line  at
3661       the end of each terse reporting cycle.
3662

JSON OUTPUT

3664       The json output format is intended to be both human readable and conve‐
3665       nient for automated parsing. For the  most  part  its  sections  mirror
3666       those  of  the normal output. The runtime value is reported in msec and
3667       the bw value is reported in 1024 bytes per second units.
3668

JSON+ OUTPUT

3670       The json+ output format is identical to the json output  format  except
3671       that  it adds a full dump of the completion latency bins. Each bins ob‐
3672       ject contains a set of (key, value) pairs where keys are latency  dura‐
3673       tions  and  values  count how many I/Os had completion latencies of the
3674       corresponding duration. For example, consider:
3675
3676              "bins" : { "87552" : 1, "89600" : 1, "94720" : 1, "96768"  :  1,
3677              "97792" : 1, "99840" : 1, "100864" : 2, "103936" : 6, "104960" :
3678              534, "105984" : 5995, "107008" : 7529, ... }
3679
3680       This data indicates that one I/O required  87,552ns  to  complete,  two
3681       I/Os  required  100,864ns to complete, and 7529 I/Os required 107,008ns
3682       to complete.
3683
3684       Also included with fio is a Python  script  fio_jsonplus_clat2csv  that
3685       takes  json+  output  and generates CSV-formatted latency data suitable
3686       for plotting.
3687
3688       The latency durations actually represent the midpoints of  latency  in‐
3689       tervals.  For details refer to `stat.h' in the fio source.
3690

TRACE FILE FORMAT

3692       There  are two trace file format that you can encounter. The older (v1)
3693       format is unsupported since version  1.20-rc3  (March  2008).  It  will
3694       still  be described below in case that you get an old trace and want to
3695       understand it.
3696
3697       In any case the trace is a simple text file with a  single  action  per
3698       line.
3699
3700       Trace file format v1
3701              Each  line  represents a single I/O action in the following for‐
3702              mat:
3703
3704                     rw, offset, length
3705
3706              where `rw=0/1' for read/write, and the `offset' and `length' en‐
3707              tries being in bytes.
3708
3709              This format is not supported in fio versions >= 1.20-rc3.
3710
3711       Trace file format v2
3712              The  second  version  of  the trace file format was added in fio
3713              version 1.17. It allows to access more then one file  per  trace
3714              and has a bigger set of possible file actions.
3715
3716              The first line of the trace file has to be:
3717
3718                     "fio version 2 iolog"
3719
3720              Following  this can be lines in two different formats, which are
3721              described below.
3722
3723              The file management format:
3724                     filename action
3725
3726                     The `filename' is given as an absolute path. The `action'
3727                     can be one of these:
3728
3729                            add    Add the given `filename' to the trace.
3730
3731                            open   Open  the  file  with the given `filename'.
3732                                   The `filename' has to have been added  with
3733                                   the add action before.
3734
3735                            close  Close  the  file with the given `filename'.
3736                                   The file has to have been opened before.
3737
3738              The file I/O action format:
3739                     filename action offset length
3740
3741                     The `filename' is given as an absolute path, and  has  to
3742                     have  been  added  and  opened before it can be used with
3743                     this format. The  `offset'  and  `length'  are  given  in
3744                     bytes. The `action' can be one of these:
3745
3746                            wait   Wait  for `offset' microseconds. Everything
3747                                   below 100 is discarded.  The time is  rela‐
3748                                   tive to the previous `wait' statement.
3749
3750                            read   Read  `length'  bytes  beginning from `off‐
3751                                   set'.
3752
3753                            write  Write `length' bytes beginning  from  `off‐
3754                                   set'.
3755
3756                            sync   fsync(2) the file.
3757
3758                            datasync
3759                                   fdatasync(2) the file.
3760
3761                            trim   Trim the given file from the given `offset'
3762                                   for `length' bytes.
3763

I/O REPLAY - MERGING TRACES

3765       Colocation is a common practice used to get the most out of a  machine.
3766       Knowing  which  workloads  play  nicely  with each other and which ones
3767       don't is a much harder task. While fio  can  replay  workloads  concur‐
3768       rently  via  multiple jobs, it leaves some variability up to the sched‐
3769       uler making results harder to reproduce. Merging is a way to  make  the
3770       order of events consistent.
3771
3772       Merging  is  integrated  into  I/O  replay  and  done when a merge_blk‐
3773       trace_file is specified. The list of  files  passed  to  read_iolog  go
3774       through the merge process and output a single file stored to the speci‐
3775       fied file. The output file is passed on as if it  were  the  only  file
3776       passed to read_iolog. An example would look like:
3777
3778              $      fio      --read_iolog="<file1>:<file2>"      --merge_blk‐
3779              trace_file="<output_file>"
3780
3781       Creating only the merged file can be done by passing the  command  line
3782       argument merge-blktrace-only.
3783
3784       Scaling traces can be done to see the relative impact of any particular
3785       trace being slowed down or sped up. merge_blktrace_scalars takes  in  a
3786       colon separated list of percentage scalars. It is index paired with the
3787       files passed to read_iolog.
3788
3789       With scaling, it may be desirable to match  the  running  time  of  all
3790       traces.  This can be done with merge_blktrace_iters. It is index paired
3791       with read_iolog just like merge_blktrace_scalars.
3792
3793       In an example, given two traces, A and B, each 60s long. If we want  to
3794       see  the impact of trace A issuing IOs twice as fast and repeat trace A
3795       over the runtime of trace B, the following can be done:
3796
3797              $    fio    --read_iolog="<trace_a>:"<trace_b>"     --merge_blk‐
3798              trace_file"<output_file>"      --merge_blktrace_scalars="50:100"
3799              --merge_blktrace_iters="2:1"
3800
3801       This runs trace A at 2x the speed twice for approximately the same run‐
3802       time as a single run of trace B.
3803

CPU IDLENESS PROFILING

3805       In  some cases, we want to understand CPU overhead in a test. For exam‐
3806       ple, we test patches for the specific goodness of whether  they  reduce
3807       CPU  usage.   Fio  implements a balloon approach to create a thread per
3808       CPU that runs at idle priority, meaning that it only runs  when  nobody
3809       else  needs  the cpu.  By measuring the amount of work completed by the
3810       thread, idleness of each CPU can be derived accordingly.
3811
3812       An unit work is defined as touching a full page of unsigned characters.
3813       Mean  and  standard  deviation  of time to complete an unit work is re‐
3814       ported in "unit work" section. Options can be chosen to report detailed
3815       percpu idleness or overall system idleness by aggregating percpu stats.
3816

VERIFICATION AND TRIGGERS

3818       Fio  is usually run in one of two ways, when data verification is done.
3819       The first is a normal write job of some sort with verify enabled.  When
3820       the  write  phase has completed, fio switches to reads and verifies ev‐
3821       erything it wrote. The second model is running just  the  write  phase,
3822       and  then  later  on  running  the  same job (but with reads instead of
3823       writes) to repeat the same I/O patterns and verify the  contents.  Both
3824       of these methods depend on the write phase being completed, as fio oth‐
3825       erwise has no idea how much data was written.
3826
3827       With verification triggers, fio  supports  dumping  the  current  write
3828       state  to  local files. Then a subsequent read verify workload can load
3829       this state and know exactly where to stop. This is useful  for  testing
3830       cases  where  power  is  cut  to a server in a managed fashion, for in‐
3831       stance.
3832
3833       A verification trigger consists of two things:
3834
3835              1) Storing the write state of each job.
3836
3837              2) Executing a trigger command.
3838
3839       The write state is relatively small, on the order of hundreds of  bytes
3840       to  single  kilobytes. It contains information on the number of comple‐
3841       tions done, the last X completions, etc.
3842
3843       A trigger is invoked either through creation ('touch') of  a  specified
3844       file  in  the  system, or through a timeout setting. If fio is run with
3845       `--trigger-file=/tmp/trigger-file', then it will continually check  for
3846       the  existence  of `/tmp/trigger-file'. When it sees this file, it will
3847       fire off the trigger (thus saving state, and executing the trigger com‐
3848       mand).
3849
3850       For client/server runs, there's both a local and remote trigger. If fio
3851       is running as a server backend, it will send the job states back to the
3852       client for safe storage, then execute the remote trigger, if specified.
3853       If a local trigger is specified, the server will still  send  back  the
3854       write state, but the client will then execute the trigger.
3855
3856       Verification trigger example
3857              Let's say we want to run a powercut test on the remote Linux ma‐
3858              chine 'server'.  Our write workload is in  `write-test.fio'.  We
3859              want  to cut power to 'server' at some point during the run, and
3860              we'll run this test from the safety or our local  machine,  'lo‐
3861              calbox'. On the server, we'll start the fio backend normally:
3862
3863                     server# fio --server
3864
3865              and on the client, we'll fire off the workload:
3866
3867                     localbox$        fio        --client=server       --trig‐
3868                     ger-file=/tmp/my-trigger --trigger-remote="bash -c  "echo
3869                     b > /proc/sysrq-triger""
3870
3871              We set `/tmp/my-trigger' as the trigger file, and we tell fio to
3872              execute:
3873
3874                     echo b > /proc/sysrq-trigger
3875
3876              on the server once it has received the trigger and sent  us  the
3877              write  state.  This will work, but it's not really cutting power
3878              to the server, it's merely abruptly rebooting it. If we  have  a
3879              remote  way of cutting power to the server through IPMI or simi‐
3880              lar, we could do that through a local trigger  command  instead.
3881              Let's  assume  we have a script that does IPMI reboot of a given
3882              hostname, ipmi-reboot. On localbox, we could then have  run  fio
3883              with a local trigger instead:
3884
3885                     localbox$        fio        --client=server       --trig‐
3886                     ger-file=/tmp/my-trigger --trigger="ipmi-reboot server"
3887
3888              For this case, fio would wait for the  server  to  send  us  the
3889              write  state,  then  execute `ipmi-reboot server' when that hap‐
3890              pened.
3891
3892       Loading verify state
3893              To load stored write state, a read verification  job  file  must
3894              contain  the  verify_state_load option. If that is set, fio will
3895              load the previously stored state. For a local fio  run  this  is
3896              done  by loading the files directly, and on a client/server run,
3897              the server backend will ask the client to send  the  files  over
3898              and load them from there.
3899

LOG FILE FORMATS

3901       Fio  supports  a  variety  of  log file formats, for logging latencies,
3902       bandwidth, and IOPS. The logs share a common format, which  looks  like
3903       this:
3904
3905              time  (msec),  value, data direction, block size (bytes), offset
3906              (bytes), command priority
3907
3908       `Time' for the log entry is always in milliseconds. The `value'  logged
3909       depends on the type of log, it will be one of the following:
3910
3911              Latency log
3912                     Value is latency in nsecs
3913
3914              Bandwidth log
3915                     Value is in KiB/sec
3916
3917              IOPS log
3918                     Value is IOPS
3919
3920       `Data direction' is one of the following:
3921
3922              0      I/O is a READ
3923
3924              1      I/O is a WRITE
3925
3926              2      I/O is a TRIM
3927
3928       The  entry's `block size' is always in bytes. The `offset' is the posi‐
3929       tion in bytes from the start of the file for that particular  I/O.  The
3930       logging of the offset can be toggled with log_offset.
3931
3932       If  log_prio  is not set, the entry's `Command priority` is 1 for an IO
3933       executed with the  highest  RT  priority  class  (prioclass=1  or  cmd‐
3934       prio_class=1)  and 0 otherwise. This is controlled by the prioclass op‐
3935       tion and the ioengine  specific  cmdprio_percentage  cmdprio_class  op‐
3936       tions. If log_prio is set, the entry's `Command priority` is the prior‐
3937       ity set for the IO, as a 16-bits hexadecimal number with the lowest  13
3938       bits  indicating  the priority value (prio and cmdprio options) and the
3939       highest 3 bits indicating the IO priority  class  (prioclass  and  cmd‐
3940       prio_class options).
3941
3942       Fio  defaults to logging every individual I/O but when windowed logging
3943       is set through log_avg_msec, either the average  (by  default)  or  the
3944       maximum  (log_max_value  is set) `value' seen over the specified period
3945       of time is recorded. Each `data direction' seen within the  window  pe‐
3946       riod  will  aggregate its values in a separate row. Further, when using
3947       windowed logging the `block size' and `offset' entries will always con‐
3948       tain 0.
3949

CLIENT / SERVER

3951       Normally  fio  is  invoked  as a stand-alone application on the machine
3952       where the I/O workload should be generated. However,  the  backend  and
3953       frontend of fio can be run separately i.e., the fio server can generate
3954       an I/O workload on the "Device Under Test" while being controlled by  a
3955       client on another machine.
3956
3957       Start the server on the machine which has access to the storage DUT:
3958
3959              $ fio --server=args
3960
3961       where `args' defines what fio listens to. The arguments are of the form
3962       `type,hostname' or `IP,port'. `type' is either `ip' (or ip4) for TCP/IP
3963       v4,  `ip6'  for  TCP/IP  v6,  or `sock' for a local unix domain socket.
3964       `hostname' is either a hostname or IP address, and `port' is  the  port
3965       to  listen  to  (only valid for TCP/IP, not a local socket). Some exam‐
3966       ples:
3967
3968              1) fio --server
3969                     Start a fio server, listening on all  interfaces  on  the
3970                     default port (8765).
3971
3972              2) fio --server=ip:hostname,4444
3973                     Start a fio server, listening on IP belonging to hostname
3974                     and on port 4444.
3975
3976              3) fio --server=ip6:::1,4444
3977                     Start a fio server, listening on IPv6 localhost  ::1  and
3978                     on port 4444.
3979
3980              4) fio --server=,4444
3981                     Start  a  fio server, listening on all interfaces on port
3982                     4444.
3983
3984              5) fio --server=1.2.3.4
3985                     Start a fio server, listening on IP 1.2.3.4  on  the  de‐
3986                     fault port.
3987
3988              6) fio --server=sock:/tmp/fio.sock
3989                     Start  a  fio  server,  listening  on  the  local  socket
3990                     `/tmp/fio.sock'.
3991
3992       Once a server is running, a "client" can  connect  to  the  fio  server
3993       with:
3994
3995              $ fio <local-args> --client=<server> <remote-args> <job file(s)>
3996
3997       where  `local-args'  are  arguments for the client where it is running,
3998       `server' is the connect string, and `remote-args' and `job file(s)' are
3999       sent  to  the server. The `server' string follows the same format as it
4000       does on the server side, to allow IP/hostname/socket and port strings.
4001
4002       Fio can connect to multiple servers this way:
4003
4004              $ fio --client=<server1> <job file(s)>  --client=<server2>  <job
4005              file(s)>
4006
4007       If  the  job  file  is located on the fio server, then you can tell the
4008       server to load a local file as  well.  This  is  done  by  using  --re‐
4009       mote-config:
4010
4011              $ fio --client=server --remote-config /path/to/file.fio
4012
4013       Then fio will open this local (to the server) job file instead of being
4014       passed one from the client.
4015
4016       If you have many servers (example: 100 VMs/containers), you can input a
4017       pathname of a file containing host IPs/names as the parameter value for
4018       the --client option. For example, here is an example  `host.list'  file
4019       containing 2 hostnames:
4020
4021              host1.your.dns.domain
4022              host2.your.dns.domain
4023
4024       The fio command would then be:
4025
4026              $ fio --client=host.list <job file(s)>
4027
4028       In  this mode, you cannot input server-specific parameters or job files
4029       -- all servers receive the same job file.
4030
4031       In order to let `fio --client' runs use a shared filesystem from multi‐
4032       ple  hosts, `fio --client' now prepends the IP address of the server to
4033       the filename. For example, if fio is using the directory `/mnt/nfs/fio'
4034       and  is  writing filename `fileio.tmp', with a --client `hostfile' con‐
4035       taining two hostnames `h1' and `h2' with  IP  addresses  192.168.10.120
4036       and 192.168.10.121, then fio will create two files:
4037
4038              /mnt/nfs/fio/192.168.10.120.fileio.tmp
4039              /mnt/nfs/fio/192.168.10.121.fileio.tmp
4040
4041       Terse  output  in  client/server mode will differ slightly from what is
4042       produced when fio is run in stand-alone mode. See the terse output sec‐
4043       tion for details.
4044

AUTHORS

4046       fio was written by Jens Axboe <axboe@kernel.dk>.
4047       This  man  page  was  written by Aaron Carroll <aaronc@cse.unsw.edu.au>
4048       based on documentation by Jens Axboe.
4049       This man page was rewritten  by  Tomohiro  Kusumi  <tkusumi@tuxera.com>
4050       based on documentation by Jens Axboe.
4051

REPORTING BUGS

4053       Report bugs to the fio mailing list <fio@vger.kernel.org>.
4054       See REPORTING-BUGS.
4055
4056       REPORTING-BUGS: http://git.kernel.dk/cgit/fio/plain/REPORTING-BUGS
4057

SEE ALSO

4059       For further documentation see HOWTO and README.
4060       Sample jobfiles are available in the `examples/' directory.
4061       These are typically located under `/usr/share/doc/fio'.
4062
4063       HOWTO: http://git.kernel.dk/cgit/fio/plain/HOWTO
4064       README: http://git.kernel.dk/cgit/fio/plain/README
4065
4066
4067
4068User Manual                       August 2017                           fio(1)
Impressum