fio(1) General Commands Manual fio(1) (NAME) fio - / (I/O) (SYNOPSIS) fio [options] [jobfile]... (DESCRIPTION) fio (thread) I/O . fio (jobfile) / . (OPTIONS) --debug=type type fio . "all" type ( "--debug=file,mem" ). "help" . --parse-only I/O . --merge-blktrace-only blktrace I/O . --output=filename filename . --output-format=format (format) "normal" "terse" "json" "json+" . . "terse" CSV . "json+" "json" (dump) (latency buckets) . --bandwidth-log . `filename` . Fio "agg-{read,write,trim}_bw.log" . filename Fio "{filename}-{read,write,trim}_bw.log" . --minimal . --append-terse . --output-format . --terse-version=version (version) ( "3" "2" "4" "5"). --version . --help . --cpuclock-test CPU . --crctest=[test] (checksum) . . . --cmdhelp=command command . "all" . --enghelp=[ioengine[,command]] ioengine command ioengine . ioengine I/O . --showcmd jobfile . --readonly trim . --readonly trim . Fio "rw=write/randwrite/rw/randrw/trim/randtrim/trimwrite" . . --eta=when (ETA) . when "always" "never" "auto" . "auto" TTY ETA . "always" ETA . "never" ETA . --eta-interval=time fio ETA . . Fio . --eta-newline=time time . . --status-interval=time ( ) time . **. . time . `--output-format=json` JSON JSON . JSON . --section=name name . . --section . jobfile (light) (moderate) (heavy) . `--section=heavy` fio "heavy" . "write" "verify" . --section . *global* . --alloc-size=kb smalloc kb KiB . --alloc-size fio . randommap fio . Smalloc . 16MiB . : `.fio_smalloc.*` `/tmp` . --warnings-fatal fio fio . --max-jobs=nr / nr . : fio (`/proc/sys/kernel/shmmax`) . --server=args (server) args . CLIENT/SERVER . --daemonize=pidfile fio pid pidfile . --client=hostname hostname hostname . CLIENT/SERVER . --remote-config=file fio file . --idle-prof=option CPU . option : calibrate . system . percpu system CPU . --inflate-log=log log . --trigger-file=file file (trigger) . --trigger-timeout=time (time) . --trigger=command command . --trigger-remote=command command . --aux-path=path path ..SH " (JOB FILE FORMAT)" . . Fio stonewall () . Fio . ini [] . ASCII *global* . . ';' '#' . *global* . *global* *global* . *global* . --cmdhelp . command --cmdhelp command . `examples/' . `examples/' . . (JOB FILE PARAMETERS) . . : (+) (-) (*) (/) (%) (^) . ( ) . (PARAMETER TYPES) : str . -. time . . 'd' 'h' ( 'usec') . 10m . int . . [*integer prefix*] **number** [*integer suffix*] *integer prefix* . (decimal) . *0x* (hexadecimal) . *integer suffix* . . . `kb_base=1000' fio . actor (SI): K (K) 1000 M (M) 1000**2 G (G) 1000**3 T (T) 1000**4 P (P) 1000**5 IEC 80000-13: Ki (Ki) 1024 Mi (Mi) 1024**2 Gi (Gi) 1024**3 Ti (Ti) 1024**4 Pi (Pi) 1024**5 (Zone Block Device Mode): z Zone () `kb_base=1024' () SI IEC 80000-13 . 4k 4096 . 'B' ( 'kB' 'k' ). *integer suffix* ( m/mi mebi/mega milli). 'b' 'B' . `kb_base=1000': 4 KiB: 4096, 4096b, 4096B, 4k, 4kb, 4kB, 4K, 4KB 1 MiB: 1048576, 1m, 1024k 1 MB: 1000000, 1mi, 1000ki 1 TiB: 1073741824, 1t, 1024m, 1048576k 1 TB: 1000000000, 1ti, 1000mi, 1000000ki `kb_base=1024' (): 4 KiB: 4096, 4096b, 4096B, 4k, 4kb, 4kB, 4K, 4KB 1 MiB: 1048576, 1m, 1024k 1 MB: 1000000, 1mi, 1000ki 1 TiB: 1073741824, 1t, 1024m, 1048576k 1 TB: 1000000000, 1ti, 1000mi, 1000000ki ( ): D (days) H (hours) M (minutes) s sec () ms msec us usec `z' (zones) . . ':' '-' . irange . . bool (Boolean). (1 0) . irange . 1024-4096. 1k:4k. float_list ':' ..SH " (JOB PARAMETERS)" fio . (Units) kb_base=int . 1000 IEC 80000-13 (SI) . : - IEC ( KiB) - SI ( kB) 1024 (). : - SI - IEC bs . . : bw=2383.3kB/s (2327.4KiB/s) kb_base : 1000 -- SI 1024 -- IEC unit_base=int . : 0 (). 8 . 1 . (Job description) name=str ASCII . fio . . . description=str . . . loops=int . . 1 . numjobs=int (clone) . . / . group_reporting new_group . --max-jobs . : 1. (Time related parameters) runtime=time . I/O . . . time_based fio runtime () . runtime . startdelay=irange(int) . . . . ramp_time=time fio . . ramp_time runtime . . ramp_size=size fio I/O . group_reporting I/O . ramp_time runtime . . clocksource=str (clocksource) . : gettimeofday gettimeofday(2) clock_gettime clock_gettime(2) cpu (CPU) cpu ( fio ). Fio . x86/x86-64 TSC Invariant . gtod_reduce=bool gettimeofday(2) (disable_clat disable_slat disable_bw_measurement) (timeout) gettimeofday(2). 0.4% gettimeofday(2) . gtod_cpu=int . Fio ( ) gettimeofday(2) . (CPU) . / I/O gettimeofday(2) . . Fio CPU . job_start_clock_id=int clock_id clock_gettime job_start json . 0 CLOCK_REALTIME ..SS "/ (Target file/device)" directory=str filename . `./' . ':' . (job clones) numjobs . filename() fio filename ( ). (escape) ':' filename . : fio --aux-path . filename=str fio filename (thread) ( filename_format ). filename . ioengine ':' . `/dev/sda' `/dev/sdb' `filename=/dev/sda:/dev/sdb' . nrfiles . size filesize . '\' . `/dev/dsk/foo@3,0:c' `filename=/dev/dsk/foo@3,0\:c' `F:\filename' `filename=F\:\filename' . Windows `\\.\PhysicalDrive0' `\\.\PhysicalDrive1' . : Windows FreeBSD ( ) . HTTP S3 URL S3 . S3 path- style (`/bucket/k/e.y') S3 virtual-hosted- style (`/k/e.y') http_host . `-' *stdin* *stdout* . / . filename_format=str fio . fio `jobname.jobnumber.filenumber' . . Fio : $jobname (worker). $clientuid IP fio client/server. $jobnum . $filenum . fio . `testfiles.$filenum' `testfiles.4' . `$jobname.$jobnum.$filenum' . . `a/b/c/$jobnum` a/b/c . directory . unique_filename=bool fio ( ) . 0 . filetype=str . fio stat(2) . . : none . . file (regular). block (Block device). char (Char device). opendir=str str . . lockfile=str fio I/O . fio I/O () . . : none . . exclusive I/O ( ). readwrite - (Read-write) . . nrfiles=int . 1 . size filesize . filename . openfiles=int . nrfiles . file_service_type=str fio . : random . roundrobin (Round robin) . . sequential . openfiles . zipf (Zipf) . pareto (Pareto) . normal () . gauss normal. random roundrobin sequential fio I/O . `file_service_type=random:8' fio I/O . . random_distribution . ioscheduler=str / (I/O scheduler) . (pipe) . create_serialize=bool true () . (interleaving) . : true..TP create_fsync=bool fsync(2) . . create_on_open=bool true open() I/O . : false -- . create_only=bool true fio (setup) . -- . : false. allow_file_create=bool true fio . false fio . : true. allow_mounted_write=bool fio ( ) (mounted) . . . : false. pre_read=bool I/O . invalidate . I/O (seek-able) . I/O ( network splice) . : false. unlink=bool (unlink) . . : false. unlink_each_loop=bool (unlink) . : false. zonemode=str : none zonerange zonesize zonecapacity zoneskip . strided I/O (zone) zonesize . . zonecapacity . zbd (Zoned block device). I/O I/O . I/O . zoneskip . zonerange zonesize . (open) . fio (zone finish) . Trim (zone reset) . Trim sequential write required sequential write preferred . zonerange=int zonemode=strided . zonesize zoneskip . zonemode=zbd . zonesize=int zonemode=strided (skip) zoneskip . zonerange zonerange . zonerange . zonemode=zbd . zonerange . zonesize 0 . zonesize 512B . zonecapacity=int zonemode=zbd . zonemode=zbd . . . zoneskip=int[z] zonemode=strided zonesize . zonemode=zbd zonesize ( ) ( ) . . read_beyond_wp . read_beyond_wp=bool zonemode=zbd . . conventional () sequential (). conventional . sequential . << >> (write pointer) . host managed host aware . host managed . fio host managed . . IOPS fio . : false. max_open_zones=int ( ) : "implicit open" ( ) "explicit open" ( ) "closed" (). "max_open_zones" ( ) "implicit open" "explicit open" . "max_active_zones" . max_open_zones fio . max_open_zones max_active_zones max_open_zones . fio . max_open_zones max_active_zones max_open_zones . fio . zonemode=zbd . max_open_zones ignore_zone_limits . ignore_zone_limits max_open_zones max_open_zones 0 . job_max_open_zones=int max_open_zones fio . . : . ignore_zone_limits=bool fio max_open_zones . : false. zone_reset_threshold=float IO IO . zone_reset_frequency . IO . zone_reset_frequency=float . (1 / zone_reset_frequency) . (garbage collection) . recover_zbd_write_error=bool continue_on_error sequential write required . . continue_on_error . (asynchronous) . IO verify . : false. write_zone_remainders=bool . fio zone finish (full) . zone finish (in-flight) . fio zone finish . . fio . norandommap . norandommap . : false..SS " I/O (I/O type)" direct=bool true I/O . O_DIRECT . OpenBSD ZFS Solaris direct I/O . Windows / (ioengines) direct I/O . : false. buffered=bool true I/O . direct . : true. readwrite=str, rw=str I/O. : read . write . trim (trim) ( Linux SCSI). randread . randwrite . randtrim (trim) ( Linux SCSI). rw,readwrite . randrw . trimwrite trim+write. . `io_size=64K' Fio . `number_ios' Fio I/O . randtrimwrite trimwrite . Fio read . I/O / . I/O . I/O `:' . `rw=randread:8' . I/O `' I/O I/O I/O () . `rw=write:4k' 4k (). rw_sequencer . verify verify_header_seed . rw_sequencer=str `rw=str' I/O . : sequential . identical . sequential I/O fio I/O . randread `rw=randread:8' I/O . . I/O . I/O sequential . identical . : rw=randread:8 rw_sequencer=sequential bs=4k : 4k, 8k, 12k, 16k, 20k, 24k, 28k, 32k, 92k, 96k, 100k, 104k, 108k, 112k, 116k, 120k, 48k, 52k ... : rw=randread:8 rw_sequencer=identical bs=4k : 4k, 4k, 4k, 4k, 4k, 4k, 4k, 4k, 92k, 92k, 92k, 92k, 92k, 92k, 92k, 92k, 48k, 48k, 48k ... unified_rw_reporting=str Fio (data direction) . fio . : none . mixed . both . 0 none. 1 mixed. 2 both. randrepeat=bool (seed) . : true. allrandrepeat=bool randrepeat. : true. randseed=int (seed) seed . randrepeat . fallocate=str . : none . native none. posix posix_fallocate(3). keep fallocate(2) FALLOC_FL_KEEP_SIZE. truncate ftruncate|(2) . 0 none. 1 posix. . keep Linux . ZFS Solaris posix ZFS . : native truncate none. truncate Windows . (end-of-file) Windows . . fadvise_hint=str posix_fadvise(2) posix_madvise(2) I/O . : 0 << >> (no hint). 1 << fio>>. FADV_RANDOM FADV_SEQUENTIAL . sequential FADV_SEQUENTIAL. random FADV_RANDOM. noreuse FADV_NOREUSE. (no-op) . . LRU . posix_fadvise(2) . write_hint=str fcntl(2) . . . : none . short . medium . long . extreme . . offset=int[%|z] I/O (zones) . blocksize offset_align ( ) . . `real_size - offset' . size I/O . '%' `offset=20%' . ZBD 'z' . offset_align=int offset . 0 . offset_increment=int[%|z] `offset + offset_increment * thread_number' thread_number 0 ( numjobs ) . . . blocksize offset_align ( ) . ZBD 'z' . number_ios=int fio I/O size ( ). / I/O . fio . I/O fio . fsync=int fsync(2) ( ) (dirty) . fio . fio I/O . sg I/O . 0 fio . end_fsync end_syncfs fsync_on_close . fdatasync=int fsync fdatasync(2) (metadata) . Windows DragonFlyBSD OSX fdatasync(2) fsync(2) . 0 fio . write_barrier=int N- (barrier write) . sync_file_range=str:int int sync_file_range(2) . fio sync_file_range(2) . str : wait_before SYNC_FILE_RANGE_WAIT_BEFORE write SYNC_FILE_RANGE_WRITE wait_after SYNC_FILE_RANGE_WRITE_AFTER `sync_file_range=wait_before,write:8' fio `SYNC_FILE_RANGE_WAIT_BEFORE | SYNC_FILE_RANGE_WRITE' . sync_file_range(2) . . overwrite=bool true . . . : false. end_fsync=bool true fsync(2) . : false. end_syncfs=bool end_fsync fsync(2) syncfs(2) . : false. fsync_on_close=bool true fio (dirty) fsync(2) . end_fsync . : false. rwmixread=int . : 50. rwmixwrite=int . rwmixread rwmixwrite . fio . . : 50. random_distribution=str:float[:float][,str:float][,str:float] fio I/O . () . fio : random zipf (Zipf) pareto (Pareto) normal () zoned zoned_abs zipf pareto . zipf `Zipf theta' . pareto `Pareto power' . fio fio-genzipf . zipf `theta' 1.2 `random_distribution=zipf:1.2' . fio (random map) . normal () 0 100 . pareto zipf normal . . [0-1] . : pareto zipf 0.5 normal. zipf `theta` 1.2 / `random_distribution=zipf:1.2:0.25' . zoned fio I/O . : . : random_distribution=zoned:60/10:30/20:8/30:2/40 zoned_abs zoned . : 20G 100G 500G : random_distribution=zoned:60/10:30/20:8/30:2/40 zoned zoned_abs fio . bssplit . bssplit trim . . percentage_random=int[,int][,int] . %100 . 0 100 . 0 . I/O . trim blocksize . norandommap fio I/O . fio I/O . / . verify verify_header_seed . verify ( bsrange) (partially- overwritten) . I/O (async) I/O 1 (I/O depth > 1) . verify_write_sequence . softrandommap=bool norandommap . fio . . random_generator=str fio I/O : tausworthe 2^88. lfsr (Linear feedback shift register). tausworthe64 2^258. tausworthe . lfsr . I/O . lfsr . fio . tausworthe 2^32 tausworthe64 . sprandom=bool SPRandom (precondition) SSD (steady-state) . . : false. random_generator=lfsr fio . . spr_num_regions=int sprandom . SPRandom . : 100 . . spr_cs=int sprandom . () SSD . (flush) . sprandom . . : 0. spr_op=float sprandom . (Over- provisioning) (0, 1) SSD. : 0.15. (Block size) blocksize=int[,int][,int], bs=int[,int][,int] I/O . : 4096. trim . trim . . : bs=256k 256k trim . bs=8k,32k 8k 32k trim . bs=8k,32k, 8k 32k trim . bs=,8k 8k trim . bs=,8k, 8k trim . blocksize_range=irange[,irange][,irange], bsrange=irange[,irange][,irange] I/O. I/O blocksize_unaligned . trim blocksize . : bsrange=1k-4k,2k-8k bsrange=1k:4k,2k:8k bssplit=str[,str][,str] . . : bssplit=blocksize/percentage:blocksize/percentage . %50 64k %10 4k %40 32k : bssplit=4k/10:64k/50:32k/40 . fio . : bssplit=4k/50:1k/:32k/ %50 4k %25 1k 32k . 100 bssplit . trim blocksize . %50 2k %50 4k %90 4k %10 8k : bssplit=2k/50:4k/50,4k/90:8k/10 fio 64 . blocksize_unaligned, bs_unaligned fio I/O blocksize_range . direct I/O direct I/O (sector alignment) . bs_is_seq_rand=bool fio read,write sequential,random . WRITE READ . blockalign=int[,int][,int], ba=int[,int][,int] fio I/O . : blocksize. direct I/O 512b . . trim blocksize ..SS " (Buffers and memory)" zero_buffers . : . refill_buffers fio I/O ( submit) I/O . . zero_buffers . (data verification) refill_buffers . scramble_buffers=bool refill_buffers (deduplication) I/O . dedupe . : true. buffer_compress_percentage=int fio I/O ( /WRITE) . fio . buffer_pattern . buffer_pattern . buffer_compress_percentage refill_buffers . (granularity) / buffer_compress_chunk . . buffer_compress_chunk=int fio / buffer_compress_percentage . buffer_compress_chunk (block size) fio / I/O ( ). fio (chunk size) / I/O . : . . buffer_pattern=str fio I/O . I/O . 0x . "" . '' . . : buffer_pattern='filename' : buffer_pattern="abcd" : buffer_pattern=-12 : buffer_pattern=0xdeadface : buffer_pattern=0xdeadface"abcd"-12'filename' dedupe_percentage=int fio . (dedupe) . . -- (unique) . refill_buffers . dedupe_mode=str dedupe_percentage dedupe fio . repeat dedupe working_set dedupe (working set) repeat fio . dedupe . working_set . working_set dedupe_working_set_percentage . fio . dedupe fio . working_set dedupe repeat . dedupe_working_set_percentage=int dedupe_mode working_set fio dedupe . size (job) . dedupe_global=bool (deduplication) . . dedupe_mode working_set . . invalidate=bool / (buffer/page cache) I/O (invalidate) . : true. pre_read . sync=str I/O (synchronous I/O) . : none I/O . 0 none. sync I/O . I/O O_SYNC . 1 sync. dsync I/O . I/O O_DSYNC . iomem=str, mem=str fio I/O . : malloc malloc(3) . . shm (shared memory) . shmget(2). shmhuge shm (huge pages) . mmap mmap(2) . (anonymous memory) (file-backed). `mem=mmap:/path/to/file' . mmaphuge . mmaphuge `mem=mmaphuge:/hugetlbfs/file'. mmapshared mmap MMAP_SHARED. cudamalloc GPU GPUDirect RDMA. ioengine rdma . bs I/O (I/O depth) . shmhuge mmaphuge . / `/proc/sys/vm/nr_hugepages' . fio huge page (MiB) . I/O ( iodepth ) bs . huge page . huge page `/proc/meminfo' . `nr_hugepages' huge page mmaphuge shmhuge . hugepage-size . mmaphuge hugetlbfs (mount) . `/huge' `mem=mmaphuge:/huge/somefile' . iomem_align=int, mem_align=int I/O . I/O iodepth bs . bs (page size) . bs I/O iomem_align bs . hugepage-size=int huge page . `/proc/meminfo' `/sys/kernel/mm/hugepages/' . (2MiB 4MiB) . `hugepage-size=Xm' . lockmem=int (Pin) mlock(2). . worker . I/O (I/O size) size=int[%|z] I/O (thread) . fio () runtime () io_size () number_ios () / I/O `rw=read:16K' () I/O percentage_random . fio nrfiles filename filesize . . fio . . . `size=20%' fio . ZBD zone 'z' . offset I/O . io_size=int[%|z], io_limit=int[%|z] fio size size I/O . . I/O fio . size 20GiB io_size 5GiB fio I/O 20GiB 5GiB . -- size 20GiB io_size 40GiB fio 40GiB I/O 0..20GiB . : io_size=N%. io_size =size . ZBD 'z' zone . filesize=irange(int) . fio . . size size io_size ( io_size ). file_append=bool I/O . fio . fio (append) . offset . (non-regular) . fill_device=bool, fill_fs=bool ENOSPC ( ) EDQUOT ( ) . . (mount point) I/O . I/O (I/O engine) ioengine=str fio : I/O /. I/O I/O . : sync I/O read(2) write(2). lseek(2) I/O . I/O fsync fdatasync . psync I/O pread(2) pwrite(2). Windows. vsync I/O readv(2) writev(2). (coalescing) I/O . pvsync I/O preadv(2) pwritev(2). pvsync2 I/O preadv2(2) pwritev2(2). io_uring / . I/O direct IO buffered IO . . io_uring_cmd / passthrough. . libaio / . I/O (`direct=1' `buffered=0' ). . posixaio / POSIX aio_read(3) aio_write(3). solarisaio / Solaris. windowsaio / Windows. Windows. mmap mmap(2) memcpy(3) / . splice splice(2) vmsplice(2) . sg / SCSI 3 (sg v3). SG_IO ioctl sg read(2) write(2) I/O . filename . trim . sg . libzbc trim ZBC/ZAC libzbc. SG . null . fio / . net `host:port' . protocol hostname port listen filename protocol . . netsplice net splice(2) vmsplice(2) / . . cpuio CPU cpuload cpuchunks cpumode . -cpuio . cpuload=85 CPU . SMP numjobs= cpuload . cpumode=qsort noop qsort . rdma I/O RDMA RDMA ( RDMA_WRITE/RDMA_READ) (Send/Recv) InfiniBand RoCE iWARP . . falloc I/O fallocate ioengine fio . DDIR_READ fallocate(,mode = FALLOC_FL_KEEP_SIZE,) . DIR_WRITE fallocate(,mode = 0) . DDIR_TRIM fallocate(,mode = FALLOC_FL_KEEP_SIZE|FALLOC_FL_PUNCH_HOLE) . ftruncate I/O (DDIR_WRITE) ftruncate(2) . ftruncate . blocksize . e4defrag I/O ioctl EXT4_IOC_MOVE_EXT (defragment) DDIR_WRITE . rados I/O Ceph Reliable Autonomic Distributed Object Store (RADOS) librados. ioengine . rbd I/O Ceph Rados Block Devices (RBD) librbd rbd . ioengine . http I/O GET/PUT HTTP(S) libcurl WebDAV S3. ioengine . I/O iodepth=1 numjobs . blocksize . TRIM . gfapi libgfapi GlusterFS GlusterFS FUSE. ioengine . gfapi_async libgfapi GlusterFS GlusterFS FUSE. ioengine . libhdfs Hadoop (HDFS). filename host,port name-node hdfs . . HDFS . libhdfs HDFS fio ( job `rw=write' ). HDFS/libhdfs . job HDFS . mtd MTD ( `/dev/mtd0'). discard erase . I/O NAND erase discard (overwriting). trimwrite . dev-dax device DAX ( /dev/dax0.0) PMDK libpmem. external (object file) I/O . `ioengine=external:/tmp/foo.o' ioengine `foo.o' `/tmp'. . I/O `engines/skeleton_external.c' fio . libpmem mmap I/O (mounted) DAX PMDK libpmem. ime_psync Infinite Memory Engine (IME) DDN. IO IME . ime_psyncv Infinite Memory Engine (IME) DDN. iovec IME IO ( IO <<>> IO ). ime_aio Infinite Memory Engine (IME) DDN. IME IO . FIO (commit) . libiscsi iscsi lun libiscsi. nbd Network Block Device (NBD). libcufile I/O libcufile nvidia-fs GPUDirect Storage. I/O verify cuda_io posix . iomem cudamalloc . ioengine . dfs I/O DAOS File System (DFS) libdfs. nfs I/O NFS libnfs. (throughput) NFS . exec (3rd party). job . xnvme I/O API C xNVMe NVMe. xnvme NVMe GNU/Linux libaio IOCTL io_uring NVMe SPDK . xnvme . (: https://xnvme.io). libblkio libblkio (https://gitlab.com/libblkio/libblkio). libblkio_driver . mem/iomem libblkio ( ). libblkio job thread ( thread) . thread . / job . : filecreate I/O . filesize I/O . job : filecreate-ioengine.fio. filestat stat() I/O . filesize nrfiles . (metadata) . job : filestat- ioengine.fio. filedelete unlink() I/O . filesize nrfiles . . job : filedelete-ioengine.fio. dircreate I/O . filesize I/O . job : dircreate-ioengine.fio. dirstat stat() I/O . filesize nrfiles . (metadata) . job : dirstat-ioengine.fio. dirdelete rmdir() I/O . filesize nrfiles . . I/O . : iops clat. bw . <> . ioengine . ioengine . (io_uring,libaio)cmdprio_percentage=int[,int] I/O . : 0. . . NCQ `direct=1' . fio root . slat/clat/lat . (clat) lat_percentiles (lat) . (io_uring,libaio)cmdprio_class=int[,int] I/O I/O cmdprio_percentage cmdprio_bssplit . cmdprio_percentage cmdprio_bssplit . . . ionice(1) . prioclass . (io_uring,libaio)cmdprio_hint=int[,int] I/O (priority hint) I/O cmdprio_percentage cmdprio_bssplit . cmdprio_percentage cmdprio_bssplit 0 ( ) . . . priohint . (io_uring,libaio)cmdprio=int[,int] I/O I/O cmdprio_percentage cmdprio_bssplit . cmdprio_percentage cmdprio_bssplit 0 . 0 7 0 . . . ionice(1) . . prio . (io_uring,libaio)cmdprio_bssplit=str[,str] I/O I/O (block size) / . bssplit . bssplit : cmdprio_bssplit=blocksize/percentage:blocksize/percentage cmdprio_class cmdprio cmdprio_hint . : cmdprio_bssplit=blocksize/percentage/class/level:blocksize/percentage/class/level . . : cmdprio_bssplit=blocksize/percentage/class/level/hint:... (priority hint) . I/O trim . cmdprio_percentage . (io_uring,io_uring_cmd)fixedbufs fio I/O (direct IO) IO (map) IO . IO . IO . IO . (io_uring,io_uring_cmd)nonvectored=int fio / (non-vectored) . -1 . (io_uring,io_uring_cmd)force_async io_uring sqe (non- blocking) (async) . N N fio sqe . 0 . (io_uring,io_uring_cmd,xnvme)hipri fio / (polled IO completions) . IO (interrupt) (active reaping) . IO IOPS IO . (io_uring,io_uring_cmd)registerfiles fio . (submission) (completion) . sqthread_poll . (io_uring,io_uring_cmd,xnvme)sqthread_poll fio / SQ . IO (polling thread) . fio CPU . sqe fio (submission latencies) . (io_uring,io_uring_cmd)sqthread_poll_cpu=int `sqthread_poll` CPU . (io_uring_cmd)cmd_type=str uring passthrough . nvme . nvme . (libaio)userspace_reap libaio fio io_getevents(3) . AIO . 0 ( `iodepth_batch_complete=0`). (pvsync2)hipri RWF_HIPRI I/O . (pvsync2)hipri_percentage hipri I/O pvsync2 . 100% . (pvsync2,libaio,io_uring,io_uring_cmd)nowait=bool ( ) . RWF_NOWAIT ( 4.14 ) EAGAIN . ignore_error=EAGAIN . : preadv2 pwritev2 glibc 2.27 2.28 EAGAIN EOPNOTSUP . I/O (cached I/O) . RWF_NOWAIT . I/O . (pvsync2,io_uring)uncached=int I/O (page cache) . (evict) . writeback . writeback . RWF_UNCACHED ( 6.14 ) I/O . (pvsync2,libaio,io_uring)atomic=bool (torn-write protection) . torn-write (atomic writes) . RWF_ATOMIC ( 6.11 ) I/O . RWF_ATOMIC torn-write . torn-write statx STATX_WRITE_ATOMIC . (io_uring_cmd,xnvme)fdp=bool Flexible Data Placement . (io_uring_cmd,xnvme)dataplacement=str data placement . : none data placement . . fdp Flexible Data Placement . fdp=1 ..TP streams Streams (write). (io_uring_cmd,xnvme)plid_select=str,fdp_pli_select=str fio (placement ID) . : random (placement ID) (). roundrobin (Round robin) . . scheme () dp_scheme. () plids fdp_pli scheme. (io_uring_cmd,xnvme)plids=str,fdp_pli=str (job) (FDP) (streams) . ( 1,2-4,5,6-8). FDP . nvme-cli `nvme fdp status' . fio FDP 0 2 5 `plids=0,2,5' . streams Stream ID . (io_uring_cmd,xnvme)dp_scheme=str () (LBA) . : 0, 10737418240, 0 10737418240, 21474836480, 1 21474836480, 32212254720, 2 ... () (,) . (start offset <= offset < end offset) () . . dspec 0 RUH . (: . plids fdp_pli .) (io_uring_cmd,xnvme)md_per_io_size=int IO. io_uring_cmd iomem malloc . : 0. (io_uring_cmd,xnvme)pi_act=int nvme (protection information) . 1 fio . 1 fio . 0 fio . : 1. CRC (16 bit) fio isa-l . (: https://github.com/intel/isa-l) (io_uring_cmd,xnvme)pi_chk=str[,str][,str] . . : none. GUARD guard . REFTAG (logical block reference tag) . APPTAG (application tag) . (io_uring_cmd,xnvme)apptag=int (application tag) . : 0x1234. (io_uring_cmd,xnvme)apptag_mask=int (application tag mask) . : 0xffff. (io_uring_cmd)num_range=int trim I/O . bs . . 1 (> 1) log_offset . : 1. (cpuio)cpuload=int CPU. / cpuio . (cpuio)cpuchunks=int . . (cpuio)cpumode=str CPU . : noop CPU noop . qsort noop qsort . (cpuio)exit_on_io_done=bool I/O . (libhdfs)namenode=str (hostname) IP namenode HDFS . (libhdfs)port=int namenode HDFS. (netsplice,net)port=int TCP UDP bind . numjobs fio . (rdma)port=int RDMA-CM. (client) (server) . (netsplice,net,rdma)hostname=str IP I/O TCP UDP RDMA-CM. TCP UDP hostname (multicast) UDP . (netsplice,net)interface=str IP UDP . (netsplice,net)ttl=int Time-to-live UDP. : 1. (netsplice,net)nodelay=bool TCP_NODELAY TCP. (netsplice,net)protocol=str, proto=str . : tcp (Transmission control protocol). tcpv6 (Transmission control protocol V6). udp (User datagram protocol). udpv6 (User datagram protocol V6). unix (UNIX domain socket). vsock VSOCK. TCP UDP VSOCK (port) TCP VSOCK UDP hostname . unix filename . VSOCK hostname CID . (netsplice,net)listen TCP fio (listen ). hostname . (netsplice,net)pingpong . `pingpong=1' (payload) . fio . (submission) (completion) . (multicast) UDP `pingpong=1' . (netsplice,net)window_size=int . (netsplice,net)mss=int TCP (TCP_MAXSEG) . (e4defrag)donorname=str (donor) (extents) . (e4defrag)inplace=int : 0 . . 1 (defragment) . (rbd,rados)clustername=str Ceph . (rbd)rbdname=str RBD . (rbd,rados)pool=str (pool) Ceph RBD RADOS . (rbd,rados)clientname=str ( 'client.') Ceph . clustername clientname *type.id* . type. fio 'client.' . (rados)conf=str ceph conf /etc/ceph/ceph.conf . (rbd,rados)busy_poll=bool (poll) . ( ) . (rados)touch_objects=bool () ( ) . ceph . . (rbd)rbd_encryption_format=str RBD . `luks1' `luks2' . rbd_encryption_passphrase . `rbd encryption format ' rbd fio . RBD . . librbd (Ceph Pacific) . (rbd)rbd_encryption_passphrase=str RBD. rbd_encryption_format . (http)http_host=str . https off HTTP (on) HTTPS . S3 virtual-hosted S3 path-style . localhost . (http)http_user=str HTTP. (http)http_pass=str HTTP. (http)https=str HTTP HTTPS . on HTTPS insecure HTTPS SSL ( !). off . (http)http_mode=str HTTP : webdav swift s3. webdav . (http)http_s3_region=str / (region/zone) S3 . us-east-1 . (http)http_s3_key=str (secret key) S3. (http)http_s3_keyid=str / (access id) S3. (http)http_s3_security_token=str S3. (http)http_s3_sse_customer_key=str SSE. (http)http_s3_sse_customer_algorithm=str SSE. AES256 . (http)http_s3_storage_class=str . . STANDARD . (http)http_swift_auth_token=str Swift. . (http)http_verbose=int (verbose) libcurl. . 1 libcurl 2 / HTTP . 0 . (http)http_object_mode=str / HTTP: block range. block . block . HTTP blocksize / filename . . range . I/O . blocksize offset "Range" . block blocksize . (mtd)skip_bad=bool (bad blocks). (libhdfs)hdfsdirectory libhdfs (chunks) HDFS . (libhdfs)chunk_size . (rdma)verb=str (verb) RDMA / RDMA. write read send recv. RDMA ( write = rdma_write ). . . (rdma)bindname=str (bind) RDMA-CM RDMA. IPv4 IPv6 . rdma_bind_addr() rdma_resolve_add() . (loopback) . (filestat)stat_type=str stat lookup/getattr . stat(2) stat . (sg)hipri fio / (polled IO completions) . (io_uring)hipri . READ WRITE SCSI SGV4_FLAG_HIPRI ( UNMAP (trim) VERIFY). sg hipri / . SCSI (LLD) <<>> hipri ( iopoll mq_poll) . MegaRAID SCSI LLD . : (0) / ( ) . (sg,io_uring_cmd)readfua=bool readfua 1 (fua) . : 0. (sg,io_uring_cmd)writefua=bool writefua 1 (fua) . : 0. (io_uring_cmd)write_mode=str . 'write' . write Write uncor Write Uncorrectable zeroes Write Zeroes verify Verify (io_uring_cmd)verify_mode=str . "read" . read Read compare Compare . () ** `verify=pattern` `verify_pattern=` . (sg)sg_write_mode=str . : write (default) (opcode) Write write_and_verify WRITE AND VERIFY . BYTCHK 00b . . writefua . verify . write_and_verify . write_same WRITE SAME . LBA . fio . () . blocksize=8k . fio . writefua . same . write_same . write_same_ndob WRITE SAME(16) No Data Output Buffer (NDOB) . . . write_stream WRITE STREAM(16) . stream_id . verify_bytchk_00 VERIFY BYTCHK 00 . . verify_bytchk_01 VERIFY BYTCHK 01 . . verify_bytchk_11 VERIFY BYTCHK 11 . . fio . () . WRITE SAME . (sg)stream_id=int WRITE STREAM . 0 ( ) fio . 0 . (nbd)uri=str NBD URI . NBD URI (: https://github.com/NetworkBlockDevice/nbd/tree/master/doc). URI: nbd://localhost:10809 nbd+unix:///?socket=/tmp/socket nbds://tlshost/exportname (libcufile)gpu_dev_ids=str GPU CUDA . . GPU (roundrobin) worker . 0 . (libcufile)cuda_io=str I/O CUDA . : cufile (default) libcufile nvidia-fs. I/O GPUDirect Storage GPU (bounce buffer) . verify cudaMemcpy RAM GPU() . RAM GPU GPU RAM . direct 1 . posix POSIX I/O RAM cudaMemcpy RAM GPU(). GPU RAM RAM GPU . verify cudaMemcpy . (dfs)pool UUID DAOS . (dfs)cont UUID DAOS . (dfs)chunk_size (chunk size) ( ) dfs . DAOS . (dfs)object_class dfs . DAOS . (nfs)nfs_url URL libnfs nfs:///path[?arg=val[&arg=val]*] README libnfs . (exec)program=str . SIGTERM . SIGKILL . grace_time . (exec)arguments=str . fio : %r %n (exec)grace_time=int SIGTERM SIGKILL . . (exec)std_redirect=bool stdout stderr . true . (xnvme)xnvme_async=str xnvme . : emu I/O (thread) IOCTL NVMe (queue pair) I/O . thrpool I/O I/O IOCTL NVMe . . io_uring I/O I/O (direct) (buffered) ..TP libaio aio I/O (Asynchronous I/O). posix I/O posix I/O . vfio (backend) VFIO libvfn SPDK. nil . . (xnvme)xnvme_sync=str xnvme. . nvme ()ioctl NVMe I/O . psync ()pread ()pwrite . block psync (zone management) IOCTL . (xnvme)xnvme_admin=str (admin) xnvme. . nvme ()ioctl NVMe . block ()ioctl sysfs . (xnvme)xnvme_dev_nsid=int (namespace) xnvme NVMe SPDK vfio. (xnvme)xnvme_dev_subnqn=str NQN fabrics. xNVMe fabrics . (xnvme)xnvme_mem=str (backend) xnvme. . posix posix NVMe . hugepage hugepage posix. hugetlbfs . hugepage (mount) hugetlbfs XNVME_HUGETLB_PATH . spdk SPDK . vfio libvfn . libvfn SPDK . (xnvme)xnvme_iovec xnvme / (vectored) . (libblkio)libblkio_driver=str libblkio . . libblkio https://libblkio.gitlab.io/libblkio/blkio.html#drivers . (libblkio)libblkio_path=str "path" libblkio I/O . : https://libblkio.gitlab.io/libblkio/blkio.html#drivers (libblkio)libblkio_pre_connect_props=str libblkio (:) libblkio . = . \: . . libblkio https://libblkio.gitlab.io/libblkio/blkio.html#properties . (libblkio)libblkio_num_entries=int "num-entries" libblkio . : https://libblkio.gitlab.io/libblkio/blkio.html#drivers (libblkio)libblkio_queue_size=int "queue-size" libblkio . : https://libblkio.gitlab.io/libblkio/blkio.html#drivers (libblkio)libblkio_pre_start_props=str libblkio (:) libblkio . = . \: . . libblkio https://libblkio.gitlab.io/libblkio/blkio.html#properties . (libblkio)hipri (poll queues). libblkio_wait_mode=eventfd libblkio_force_enable_completion_eventfd . (libblkio)libblkio_vectored (vectored). (libblkio)libblkio_write_zeroes_on_trim trim "write zeroes" discard. (libblkio)libblkio_wait_mode=str : block () blkioq_do_io(). eventfd ()read eventfd . loop (busy loop) blkioq_do_io(). (libblkio)libblkio_force_enable_completion_eventfd eventfd . . libblkio_wait_mode=eventfd . (windowsaio)no_completion_thread (thread) (completion polling)..SS " / (I/O depth)" iodepth=int I/O (in flight) . iodepth ioengine ( verify_async ). (async) . libaio `direct=1' / . I/O fio . : 1. iodepth_batch_submit=int, iodepth_batch=int I/O . I/O I/O . 0 iodepth . iodepth_batch_complete_min=int, iodepth_batch_complete=int I/O . I/O . I/O iodepth_low . 0 fio I/O . I/O . iodepth_batch_complete_max=int I/O . iodepth_batch_complete_min=int I/O . iodepth_batch_complete_min . : iodepth_batch_complete_min=1 iodepth_batch_complete_max= / . I/O . : iodepth_batch_complete_min=0 iodepth_batch_complete_max= I/O . (polling) . iodepth_low=int (low water mark) . iodepth fio . iodepth 16 iodepth_low 4 fio . serialize_overlap=bool I/O (data races) . I/O I/O . I/O . I/O . serialize_overlap fio I/O . iodepth . I/O io_submit_mode=offload . offload fio I/O offload serialize_overlap . : false. io_submit_mode=str I/O fio / . `inline' fio I/O . `offload' I/O I/O . I/O . fio . I/O ( omission ). I/O . / (I/O rate) thinkcycles=int I/O . . thinktime . thinkcycles . thinktime=time I/O . . . thinktime_blocks thinktime_iotime thinktime_spin . thinktime_spin=time thinktime - (CPU) thinktime. . thinktime_blocks=int thinktime - thinktime . fio thinktime . thinktime I/O . . thinktime_blocks_type=str thinktime - thinktime_blocks . `complete' fio thinktime_blocks thinktime . `issue' . thinktime_iotime=time thinktime - thinktime . thinktime IO thinktime_iotime . `--thinktime_iotime=9s --thinktime=1s' IO . thinktime_iotime . thinktime_blocks thinktime thinktime_iotime thinktime_blocks . rate=int[,int][,int] . (bytes/sec) . trim blocksize . `rate=1m,500k' 1MiB/sec 500KiB/sec . `rate=,500k' `rate=500k,' ( 500KiB/sec) . rate_min=int[,int][,int] fio . . trim blocksize . rate_iops=int[,int][,int] IOPS . rate . . trim blocksize . rate_iops_min=int[,int][,int] fio I/O . trim blocksize . rate_process=str I/O fio . `linear' I/O I/O I/O . `poisson' fio / (https://en.wikipedia.org/wiki/Poisson_point_process) . 10^6 / IOPS . rate_ignore_thinktime=bool thinktime fio ( ). fio thinktime thinktime IO . rate_cycle=int rate_min rate_iops_min . : 1000..SS " / (I/O latency)" latency_target=time fio . . latency_window latency_percentile . latency_window=time latency_target . . latency_percentile=float I/O latency_target latency_window . 100.0 I/O latency_target . latency_run=bool latency_target . false () fio latency_target . true fio latency_target . max_latency=time[,time][,time] fio ETIMEDOUT . . trim blocksize . / (I/O replay) write_iolog=str I/O . read_iolog . iolog . (append mode) . read_iolog=str iolog I/O . . iolog blktrace fio blktrace . blktrace(8) . blktrace blkparse ("blkparse -o /dev/null -d file_for_fio.bin"). ':' . (escape) ':' filename . numjobs . '-' stdin filename '-' ( stdin ) '-' . read_iolog_chunked=bool iolog . false () read_iolog . true iolog . iolog . merge_blktrace_file=str read_iolog blktrace . read_iolog . . blktrace (scheduler) . merge_blktrace_scalars=float_list read_iolog . . "--merge_blktrace_scalars="50:100"" (halftime) (realtime) . replay_time_scale . merge_blktrace_iters=float_list read_iolog . . "--merge_blktrace_iters="2:1"" . replay_no_stall=bool I/O read_iolog IOPS . fio . I/O . replay_time_scale=int I/O read_iolog fio . . 50 50% I/O . 200 I/O . 100 . replay_redirect=str I/O read_iolog IOPS major/minor IOP . major/minor . major/minor . replay_redirect I/O "replay_redirect=/dev/sdc" I/O blktrace iolog "/dev/sdc" . . blkparse fio . . replay_align=int . 2 . replay_scale=int . replay_align . (Threads, processes and job synchronization) replay_skip=str I/O . trim/discard . read write trim sync . thread fio fork fio (POSIX Threads) pthread_create(3) . wait_for=str (workers) waitee . wait_for : (waitee) (waiter) ( ). waitee ( waitee ). comm=str comm . prctrl(2) . : . nice=int nice . nice(2) . -15 "High" -1 -15 "Above Normal" 1 15 "Below Normal" 15 "Idle" . prio=int I/O . 0 7 0 . ionice(1) . . I/O `cmdprio_percentage` `cmdprio` . prioclass=int I/O . ionice(1) . I/O `cmdprio_percentage` `cmdprio_class` . priohint=int I/O . I/O hint CDL . CDL I/O . I/O I/O cmdprio_hint . cpus_allowed=str cpumask 0 . 0 5 `cpus_allowed=0,5` . -- (binding) 0 5 8 15 `cpus_allowed=0,5,8-15` . `cpus_allowed` fio (affinity) . fio 7 . 0 40 1 32 `cpus_allowed` 0 39 0 `cpus_allowed` 40 71 1 . `cpus_allowed_policy=shared` `cpus_allowed` . fio 7 fio ( ) . cpus_allowed_policy=str cpus_allowed cpumask fio . : shared job CPU . split job CPU CPU . shared . split fio job CPU . CPU job fio (roundrobin) . cpumask=int (CPU affinity) job . job . (1 << 1 | 1 << 5) 34 . sched_setaffinity(2) . . 1-32 . CPU cpus_allowed . numa_cpu_nodes=str job NUMA . A-B `all' . NUMA fio () libnuma-dev(el) . numa_mem_policy=str job NUMA . : [:] `mode' : `default' `prefer' `bind' `interleave' `local'. `default' `local' . `prefer' . `bind' `interleave' `nodelist' : A-B `all'. cgroup=str job (cgroup) . . cgroup blkio . : # mount -t cgroup -o blkio none /cgroup cgroup_weight=int cgroup . 100..1000 . cgroup_nodelete=bool fio job (cgroups) . cgroup job `cgroup_nodelete=1' . cgroup job . : false. flow_id=int (flow). (global) . flow . flow=int (token-based flow control). fio job flow_id . fio job job flow_id job . job `flow=3' `flow=2' `flow=1` :: . flow_sleep=int . stonewall, wait_for_previous job job job . (serialization points) job . (stone wall) group_reporting . `stonewall=0` `stonewall=1` . exitall job fio job . . exitall fio job job . exit_what=str job fio job . . exitall fio job . exit_what job exitall . group . : all job . group exitall . stonewall job stonewall . exec_prerun=str job system(3) . `jobname.prerun.txt' . exec_postrun=str job system(3) . `jobname.postrun.txt' . uid=int (user ID) / (thread/process) . gid=int (group ID) uid ..SS " (Verification)" verify_only . . time_based . verify_write_sequence verify_header_seed . (phantom writes) . do_verify=bool . verify . : true. verify=str fio . (header) . (seed) . verify verify_pattern . : md5 md5 . crc64 crc64 . crc32c crc32c . ( SSE4.2 x86 CRC ARM64) crc32c . fio . crc32c-intel crc32c. crc32 crc32 . crc16 crc16 . crc7 crc7 . xxhash xxhash . fio . sha512 sha512 . sha256 sha256 . sha1 sha1 . sha3-224 sha3-224 . sha3-256 sha3-256 . sha3-384 sha3-384 . sha3-512 sha3-512 . meta . verify . . . pattern . fio verify_pattern . pattern_hdr . null . 'ioengine=null' . (burn-in tests) . (read) (random read) fio . fio . verify_header_seed . . . I/O I/O (I/O depths > 1) norandommap norandommap lfsr I/O . verify_offset=int . . verify_interval . verify_interval=int blocksize. verify_interval . blocksize . verify_pattern=str fio I/O . fio I/O . fio /// ( ). verify_pattern "0x" "0X" . verify . verify_pattern %o : verify_pattern=%o : verify_pattern=0xff%o"abcd"-12 verify_pattern_interval=bool verify_pattern verify_pattern_interval . verify_pattern %o . : 0 (). verify_fatal=bool fio . fio . : false. verify_dump=bool (dump) . . . verify_async=int fio I/O (inline) . I/O fio I/O . I/O iodepth I/O . : 0 . verify_async_cpus=str fio (CPU affinity) I/O . cpus_allowed . verify_backlog=int fio . . . fio I/O . fio N . verify_backlog_batch=int verify_backlog fio . verify_backlog ( ). verify_backlog_batch verify_backlog verify_backlog_batch verify_backlog ..TP verify_state_save=bool . fio (replay) . : ---verify.state. "local" / "sock" / "ip" ( 192.168.0.1) . true . verify_state_load=bool fio (thread) . fio . fio . false . experimental_verify=bool . I/O . I/O . verify_write_sequence=bool (write sequence number) . . . (atomic writes) checksum . atomic checksum ( ). verify_header_seed=bool seed . / norandommap ( readwrite rw_sequencer ) seed . seed . true . trim_percentage=int /trim . trim_verify_zero=bool trim/ . trim_backlog=int trim/ . trim_backlog_batch=int I/O trim . (Steady state) steadystate=str:float, ss=str:float . . . `iops_slope:0.1%' fio IOPS . group_reporting . . . . time_based runtime loops fio () () . iops IOPS. IOPS IOPS ( `iops:2' IOPS `iops:0.2%' IOPS IOPS ). iops_slope IOPS . . bw . . bw_slope . . steadystate_duration=time, ss_dur=time . ss_interval . 0 . . steadystate_ramp_time=time, ss_ramp=time . 0 . . steadystate_check_interval=time, ss_interval=time . ss_interval 30s ss_dur 300s . 1s . . (Measurements and reporting) per_job_logs=bool true fio bw/clat/iops . false . false (append mode) . : true. group_reporting . numjobs / . group_reporting . stonewall new_group . : group_reporting json . kb_base unit_base sig_figs thread_number pid job_start . . percentile_list unified_rw_reporting . . new_group . : group_reporting. stonewall . stats=bool fio . 0 fio . write_bw_log=str . . str `jobname_type.x.log' . fio . : write_bw_log=foo `foo_bw.x.log' `x' (1..N N ). per_job_logs false `.x` . fio_generate_plots gnuplot . LOG FILE FORMATS . write_lat_log=str write_bw_log I/O ( `name_slat.x.log') ( `name_clat.x.log') ( `name_lat.x.log') . write_bw_log LOG FILE FORMATS . write_hist_log=str write_bw_log I/O ( `name_hist.x.log') . log_hist_msec . write_bw_log LOG FILE FORMATS . write_iops_log=str write_bw_log IOPS ( `name_iops.x.log`) . fio I/O IOPS 1 (windowed logging - log_avg_msec) . write_bw_log LOG FILE FORMATS ..TP log_entries=int fio I/O iops latency bw . I/O . . (tail latencies) I/O ( ) . I/O . log_avg_msec . . log_avg_msec=int fio I/O iops latency bw . . fio . fio (flush) . log_window_value . 0 ( I/O). LOG FILE FORMATS . log_hist_msec=int log_avg_msec . log_avg_msec . fio IOPS . log_hist_coarseness write_hist_log . 0 . log_hist_coarseness=int 0 6 (coarseness) log_hist_msec . fio (bins) . 0 1216 . LOG FILE FORMATS . log_window_value=str, log_max_value=str log_avg_msec fio . fio . iops bw . : avg . max . both . 0 avg. 1 max. log_offset=bool iolog (byte offset) I/O . 0 . LOG FILE FORMATS . log_prio=bool `Command priority` LOG FILE FORMATS IO . RT . LOG FILE FORMATS . log_issue_time=bool iolog (command issue time) I/O . 0 . LOG FILE FORMATS . write_lat_log log_offset . log_compression=int fio I/O . (chunk) . I/O . CPU . I/O . zlib . log_compression_cpus=str CPU I/O . . cpus_allowed . log_store_compressed=bool fio . fio --inflate-log . `.fz' . log_unix_epoch=bool log_alternate_epoch. log_alternate_epoch=bool fio write_type_log (epoch) log_alternate_epoch_clock_id . log_alternate_epoch_clock_id=int clock_id clock_gettime log_alternate_epoch . . 0 CLOCK_REALTIME . block_error_percentiles=bool trim trim trim . bwavgtime=int . . write_bw_log log_avg_msec . : 500ms. iopsavgtime=int IOPS . . IOPS write_iops_log log_avg_msec . : 500ms. disk_util=bool (disk utilization) . : true. disable_lat=bool (total latency) . gettimeofday(2) IOPS . disable_slat disable_bw_measurement . disable_clat=bool (completion latency) . disable_lat . disable_slat=bool (submission latency) . disable_lat . disable_bw_measurement=bool, disable_bw=bool / . disable_lat . slat_percentiles=bool . ioengine (synchronous) . clat_percentiles=bool . lat_percentiles=bool . . percentile_list=float_list . (0,100] . ':' . `--percentile_list=99.5:99.9' fio . significant_figures=int --output-format `normal' (significant figures) . IOPS . . ..SS " " exitall_on_error . . continue_on_error=str fio . fio << >> (EIO EILSEQ) I/O . : . . : I/O () fio write() ( ) (dirty) `sync` `direct` . I/O . `fsync` `fdatasync` . (synchronous) . : none I/O (verify). read . write . io I/O . verify . all . 0 'none'. 1 'all'. ignore_error=str << >> continue_on_error . `ignore_error=READ_ERR_LIST,WRITE_ERR_LIST,VERIFY_ERR_LIST` ':' . ('ENOSPC' 'ENOMEM') . : ignore_error=EAGAIN,ENOSPC:122 EAGAIN READ ENOSPC 122(EDQUOT) WRITE . continue_on_error ( ) . error_dump=bool (dump) true . . fio I/O . profile=str . : tiobench I/O (tiotest/tiobench). act Aerospike Certification Tool (ACT). --cmdhelp . : $ fio --profile=act --cmdhelp Act device-names=str . load=int ACT. : 1. test-duration=time . . : 24h. threads-per-queue=int I/O . : 8. read-req-num-512-blocks=int 512 . : 3. large-block-op-kbytes=int KiB (). : 131072. prep (prep) ACT..SS " Tiobench" size=str (MiB). block=int . : 4096. numruns=int . dir=str . threads=int . (OUTPUT) fio . fio . : Jobs: 1 (f=1): [_(1),M(1)][24.8%][r=20.5MiB/s,w=23.5MiB/s][r=82,w=94 IOPS][eta 01m:31s] . (job file) . ( ) : P . C . I . p () . / (ramp period) . R . r . W . w . M / . m / . D trim . d trim . F fsync(2). V (verification) . f . E (reap) . - . X . K . fio . : Jobs: 20 (f=20): [R(10),W(10)][4.0%][r=20.5MiB/s,w=23.5MiB/s][r=82,w=94 IOPS][eta 57m:36s] . 1--10 11--20 . -- I/O (f=) I/O ( trim) IOPS . ( ) . fio ( Ctrl-C ) . ( ) : Client1: (groupid=0, jobs=1): err= 0: pid=16109: Sat Jun 24 12:07:54 2017 write: IOPS=88, BW=623KiB/s (638kB/s)(30.4MiB/50032msec) slat (nsec): min=500, max=145500, avg=8318.00, stdev=4781.50 clat (usec): min=170, max=78367, avg=4019.02, stdev=8293.31 lat (usec): min=174, max=78375, avg=4027.34, stdev=8291.79 clat percentiles (usec): | 1.00th=[ 302], 5.00th=[ 326], 10.00th=[ 343], 20.00th=[ 363], | 30.00th=[ 392], 40.00th=[ 404], 50.00th=[ 416], 60.00th=[ 445], | 70.00th=[ 816], 80.00th=[ 6718], 90.00th=[12911], 95.00th=[21627], | 99.00th=[43779], 99.50th=[51643], 99.90th=[68682], 99.95th=[72877], | 99.99th=[78119] bw ( KiB/s): min= 532, max= 686, per=0.10%, avg=622.87, stdev=24.82, samples= 100 iops : min= 76, max= 98, avg=88.98, stdev= 3.54, samples= 100 lat (usec) : 250=0.04%, 500=64.11%, 750=4.81%, 1000=2.79% lat (msec) : 2=4.16%, 4=1.84%, 10=4.90%, 20=11.33%, 50=5.37% lat (msec) : 100=0.65% cpu : usr=0.27%, sys=0.18%, ctx=12072, majf=0, minf=21 IO depths : 1=85.0%, 2=13.1%, 4=1.8%, 8=0.1%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,4450,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=8 ( group_reporting) (group id) ( 0 ) pid/tid / . I/O ( ). : read/write/trim I/O . IOPS I/O . BW : ( ). : ( I/O / runtime ) . slat (Submission latency) min max avg stdev . I/O . I/O (sync) slat ( / ). --- fio ( ). : --minimal . clat (Completion latency). slat I/O . I/O clat ( ) 0 (CPU) (I/O slat ). clat . filecreate engine: filestat engine: filedelete engine: dircreate engine: dirstat engine: dirdelete engine: lat (Total latency). slat clat fio I/O I/O. bw . fio I/O . fio ( bwavgtime ) . xlat (samples) (per) . . bw ..TP iops IOPS . bw . iopsavgtime . bw . iops . . filecreate engine: filestat engine: (look up) filedelete engine: dircreate engine: dirstat engine: (look up) dirdelete engine: lat (nsec/usec/msec) I/O. I/O fio . read/write/trim I/O . 250=0.04% 0.04% I/O 250us . 500=64.11% 64.11% I/O 250 499us . cpu CPU. (user) (system) (context switches) (page faults) . CPU . IO depths I/O (job). -- =16 . submit/complete . IO submit I/O (submit call) . -- 16=100% I/O . submit . IO complete submit . IO issued rwt read/write/trim (dropped) . IO latency latency_target . I/O . . : Run status group 0 (all jobs): READ: bw=20.9MiB/s (21.9MB/s), 10.4MiB/s-10.8MiB/s (10.9MB/s-11.3MB/s), io=64.0MiB (67.1MB), run=2973-3069msec WRITE: bw=1231KiB/s (1261kB/s), 616KiB/s-621KiB/s (630kB/s-636kB/s), io=64.0MiB (67.1MB), run=52747-53223msec : bw . . io I/O . bw . run . . . : Disk stats (read/write): sda: ios=16398/16511, sectors=32321/65472, merge=30/162, ticks=6853/819634, in_queue=826487, util=100.00% . : ios I/O . merge I/O. ticks . in_queue . util . 100% 50% . fio (dump) . USR1 fio . --status-interval `/tmp' `fio-dump-status' . fio (unlink) ..SH " (TERSE OUTPUT)" fio - . : 2;card0;0;0;7139336;121836;60004;1;10109;27.932460;116.933948;220;126861;3495.446807;1085.368601;226;126864;3523.635629;1089.012448;24063;99944;50.275485%;59818.274627;5540.657370;7155060;122104;60004;1;8338;29.086342;117.839068;388;128077;5032.488518;1234.785715;391;128085;5061.839412;1236.909129;23436;100928;50.287926%;59964.832030;5644.844189;14.595833%;19.394167%;123706;0;7313;0.1%;0.1%;0.1%;0.1%;0.1%;0.1%;100.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.02%;0.05%;0.16%;6.04%;40.40%;52.68%;0.64%;0.01%;0.00%;0.01%;0.00%;0.00%;0.00%;0.00%;0.00% A description of this job goes here. ( ) (terse v2) . . --minimal '--output-format=terse' . . . ( ): (terse version) fio [v3] (jobname) (groupid) (error) (READ): IO (KiB) (KiB/sec) IOPS (msec) (Submission latency): min, max, mean, stdev (usec) (Completion latency): min, max, mean, stdev (usec) : ( ) (Total latency): min, max, mean, stdev (usec) (KiB/s): min, max mean, stdev [v5] IOPS [v5]: min, max, mean, stdev (WRITE): IO (KiB) (KiB/sec) IOPS (msec) (Submission latency): min, max, mean, stdev (usec) (Completion latency): min, max, mean, stdev (usec) : ( ) (Total latency): min, max, mean, stdev (usec) (KiB/s): min, max mean, stdev [v5] IOPS [v5]: min, max, mean, stdev TRIM [ ]: READ/WRITE . (CPU): (user) (system) (context switches) (major faults) (minor faults) I/O (I/O depths): <=1, 2, 4, 8, 16, 32, >=64 I/O : <=2, 4, 10, 20, 50, 100, 250, 500, 750, 1000 I/O : <=2, 4, 10, 20, 50, 100, 250, 500, 750, 1000, 2000, >=2000 [ ]: I/O I/O ( continue_on_error ): ( description): fio . : 1.00%=6112 X 'usec' . (Disk utilization) fio . . (minimal) - : terse_version_3;fio_version;jobname;groupid;error;read_kb;read_bandwidth_kb;read_iops;read_runtime_ms;read_slat_min_us;read_slat_max_us;read_slat_mean_us;read_slat_dev_us;read_clat_min_us;read_clat_max_us;read_clat_mean_us;read_clat_dev_us;read_clat_pct01;read_clat_pct02;read_clat_pct03;read_clat_pct04;read_clat_pct05;read_clat_pct06;read_clat_pct07;read_clat_pct08;read_clat_pct09;read_clat_pct10;read_clat_pct11;read_clat_pct12;read_clat_pct13;read_clat_pct14;read_clat_pct15;read_clat_pct16;read_clat_pct17;read_clat_pct18;read_clat_pct19;read_clat_pct20;read_tlat_min_us;read_lat_max_us;read_lat_mean_us;read_lat_dev_us;read_bw_min_kb;read_bw_max_kb;read_bw_agg_pct;read_bw_mean_kb;read_bw_dev_kb;write_kb;write_bandwidth_kb;write_iops;write_runtime_ms;write_slat_min_us;write_slat_max_us;write_slat_mean_us;write_slat_dev_us;write_clat_min_us;write_clat_max_us;write_clat_mean_us;write_clat_dev_us;write_clat_pct01;write_clat_pct02;write_clat_pct03;write_clat_pct04;write_clat_pct05;write_clat_pct06;write_clat_pct07;write_clat_pct08;write_clat_pct09;write_clat_pct10;write_clat_pct11;write_clat_pct12;write_clat_pct13;write_clat_pct14;write_clat_pct15;write_clat_pct16;write_clat_pct17;write_clat_pct18;write_clat_pct19;write_clat_pct20;write_tlat_min_us;write_lat_max_us;write_lat_mean_us;write_lat_dev_us;write_bw_min_kb;write_bw_max_kb;write_bw_agg_pct;write_bw_mean_kb;write_bw_dev_kb;cpu_user;cpu_sys;cpu_csw;cpu_mjf;cpu_minf;iodepth_1;iodepth_2;iodepth_4;iodepth_8;iodepth_16;iodepth_32;iodepth_64;lat_2us;lat_4us;lat_10us;lat_20us;lat_50us;lat_100us;lat_250us;lat_500us;lat_750us;lat_1000us;lat_2ms;lat_4ms;lat_10ms;lat_20ms;lat_50ms;lat_100ms;lat_250ms;lat_500ms;lat_750ms;lat_1000ms;lat_2000ms;lat_over_2000ms;disk_name;disk_read_iops;disk_write_iops;disk_read_merges;disk_write_merges;disk_read_ticks;write_ticks;disk_queue_time;disk_util / (/) . ..SH " JSON (JSON OUTPUT)" json . normal . runtime bw . JSON+ (JSON+ OUTPUT) json+ json (completion latency bins) . bins ( ) I/O . : "bins" : { "87552" : 1, "89600" : 1, "94720" : 1, "96768" : 1, "97792" : 1, "99840" : 1, "100864" : 2, "103936" : 6, "104960" : 534, "105984" : 5995, "107008" : 7529, ... } I/O 87,552ns I/O 100,864ns I/O 107,008ns . fio fio_jsonplus_clat2csv json+ CSV . . `stat.h' fio . (TRACE FILE FORMAT) . (v1) 1.20-rc3 ( ) . . . (Trace file format v1) I/O : rw, offset, length `rw=0/1' / (read/write) `offset' `length' . fio >= 1.20-rc3 . (Trace file format v2) fio . . . : "fio version 2 iolog" . (The file management format): filename action `filename' . `action' : add `filename' . open `filename' . `filename' add . close `filename' . open . I/O (The file I/O action format): filename action offset length `filename' (add) (open) . `offset' `length' . `action' : wait `offset' . . `wait' . `wait` (timestamps) . read `length' `offset'. write `length' `offset'. sync fsync(2) . datasync fdatasync(2) . trim trim `offset' `length' . (Trace file format v3) fio . . (timestamp) . : "fio version 3 iolog" . (The file management format): timestamp filename action I/O (The file I/O action format): timestamp filename action offset length `timestamp` ( ). `filename` `action` `offset` `length` `wait` . (Colocation) . . fio (job) (scheduler) . . I/O merge_blktrace_file . read_iolog . read_iolog . : $ fio --read_iolog=":" --merge_blktrace_file="" merge-blktrace-only . (traces) . merge_blktrace_scalars . read_iolog . . merge_blktrace_iters . merge_blktrace_scalars read_iolog . A B I/O A A B : $ fio --read_iolog=":"" --merge_blktrace_file"" --merge_blktrace_scalars="50:100" --merge_blktrace_iters="2:1" A B . (CPU IDLENESS PROFILING) (CPU overhead) . . fio (balloon approach) (thread) CPU (idle) CPU . CPU . (unsigned characters) . "unit work" . CPU CPU . (VERIFICATION AND TRIGGERS) fio . (verify) . fio . ( ) I/O . fio . fio . . . : ) . ) . . X . ('touch') (timeout) . fio `--trigger-file=/tmp/trigger-file' `/tmp/trigger-file' . ( ). / . fio backend . . (Verification trigger example) 'server' . `write-test.fio' . 'server' 'localbox' . backend fio : server# fio --server : localbox$ fio --client=server --trigger-file=/tmp/my-trigger --trigger-remote="bash -c "echo b > /proc/sysrq-triger"" `/tmp/my-trigger' fio : echo b > /proc/sysrq-trigger . (reboot) . IPMI . ipmi-reboot IPMI . localbox fio : localbox$ fio --client=server --trigger-file=/tmp/my-trigger --trigger="ipmi-reboot server" fio `ipmi-reboot server' . (Loading verify state) verify_state_load . fio . fio / backend . fio IOPS . : time (msec), value, data direction, block size (bytes), offset (bytes), command priority, issue time (nsec) <