t4k: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 t16k: (g=0): rw=rw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=16 t64k: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=16 t256k: (g=0): rw=rw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=16 fio-3.1 Starting 4 processes t4k: Laying out IO file (1 file / 10240MiB) Jobs: 4 (f=4): [M(4)][100.0%][r=54.1MiB/s,w=58.6MiB/s][r=594,w=625 IOPS][eta 00m:00s] t4k: (groupid=0, jobs=1): err= 0: pid=2791: Sat Oct 29 12:04:57 2022 read: IOPS=55, BW=221KiB/s (226kB/s)(194MiB/900083msec) slat (usec): min=10, max=3964, avg=46.44, stdev=71.60 clat (usec): min=738, max=1982.7k, avg=135991.58, stdev=93203.27 lat (usec): min=758, max=1982.7k, avg=136038.52, stdev=93205.78 clat percentiles (usec): | 1.00th=[ 1614], 5.00th=[ 6652], 10.00th=[ 47449], | 20.00th=[ 73925], 30.00th=[ 88605], 40.00th=[ 102237], | 50.00th=[ 116917], 60.00th=[ 135267], 70.00th=[ 158335], | 80.00th=[ 189793], 90.00th=[ 244319], 95.00th=[ 299893], | 99.00th=[ 459277], 99.50th=[ 541066], 99.90th=[ 759170], | 99.95th=[ 884999], 99.99th=[1350566] bw ( KiB/s): min= 8, max= 2008, per=0.41%, avg=221.26, stdev=125.89, samples=1797 iops : min= 2, max= 502, avg=55.27, stdev=31.47, samples=1797 write: IOPS=54, BW=220KiB/s (225kB/s)(193MiB/900083msec) slat (usec): min=10, max=5498, avg=50.84, stdev=72.48 clat (msec): min=5, max=2066, avg=154.61, stdev=97.86 lat (msec): min=5, max=2066, avg=154.66, stdev=97.86 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 46], 10.00th=[ 67], 20.00th=[ 87], | 30.00th=[ 103], 40.00th=[ 116], 50.00th=[ 132], 60.00th=[ 153], | 70.00th=[ 176], 80.00th=[ 209], 90.00th=[ 266], 95.00th=[ 330], | 99.00th=[ 498], 99.50th=[ 584], 99.90th=[ 827], 99.95th=[ 1083], | 99.99th=[ 1603] bw ( KiB/s): min= 8, max= 1800, per=0.40%, avg=220.03, stdev=123.03, samples=1796 iops : min= 2, max= 450, avg=54.96, stdev=30.75, samples=1796 lat (usec) : 750=0.01%, 1000=0.06% lat (msec) : 2=0.69%, 4=0.96%, 10=1.39%, 20=0.90%, 50=4.24% lat (msec) : 100=25.10%, 250=55.91%, 500=9.91%, 750=0.71%, 1000=0.09% lat (msec) : 2000=0.04%, >=2000=0.01% cpu : usr=0.10%, sys=0.64%, ctx=70980, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=49699,49396,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 t16k: (groupid=0, jobs=1): err= 0: pid=2792: Sat Oct 29 12:04:57 2022 read: IOPS=245, BW=3927KiB/s (4022kB/s)(3452MiB/900039msec) slat (usec): min=10, max=6245, avg=43.59, stdev=68.83 clat (usec): min=938, max=1982.7k, avg=17906.61, stdev=40701.29 lat (usec): min=980, max=1982.7k, avg=17950.62, stdev=40704.07 clat percentiles (usec): | 1.00th=[ 1237], 5.00th=[ 1565], 10.00th=[ 1942], 20.00th=[ 2704], | 30.00th=[ 3589], 40.00th=[ 4621], 50.00th=[ 6063], 60.00th=[ 8094], | 70.00th=[ 10814], 80.00th=[ 16057], 90.00th=[ 35390], 95.00th=[ 90702], | 99.00th=[204473], 99.50th=[254804], 99.90th=[392168], 99.95th=[492831], | 99.99th=[700449] bw ( KiB/s): min= 96, max=10048, per=7.28%, avg=3936.54, stdev=2180.21, samples=1797 iops : min= 6, max= 628, avg=245.91, stdev=136.23, samples=1797 write: IOPS=246, BW=3942KiB/s (4037kB/s)(3465MiB/900039msec) slat (usec): min=11, max=7490, avg=59.89, stdev=95.91 clat (msec): min=2, max=1890, avg=46.99, stdev=44.77 lat (msec): min=2, max=1890, avg=47.05, stdev=44.77 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 18], 10.00th=[ 19], 20.00th=[ 21], | 30.00th=[ 24], 40.00th=[ 28], 50.00th=[ 33], 60.00th=[ 40], | 70.00th=[ 49], 80.00th=[ 64], 90.00th=[ 91], 95.00th=[ 123], | 99.00th=[ 228], 99.50th=[ 279], 99.90th=[ 422], 99.95th=[ 506], | 99.99th=[ 810] bw ( KiB/s): min= 64, max= 9600, per=7.27%, avg=3949.02, stdev=2155.43, samples=1798 iops : min= 4, max= 600, avg=246.68, stdev=134.68, samples=1798 lat (usec) : 1000=0.01% lat (msec) : 2=5.41%, 4=11.68%, 10=16.56%, 20=16.51%, 50=31.52% lat (msec) : 100=12.07%, 250=5.61%, 500=0.59%, 750=0.04%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.43%, sys=2.82%, ctx=287037, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=220922,221759,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 t64k: (groupid=0, jobs=1): err= 0: pid=2793: Sat Oct 29 12:04:57 2022 read: IOPS=179, BW=11.2MiB/s (11.8MB/s)(9.85GiB/900016msec) slat (usec): min=11, max=5995, avg=50.07, stdev=88.52 clat (usec): min=1569, max=1413.5k, avg=27620.91, stdev=49348.92 lat (usec): min=1621, max=1413.5k, avg=27671.44, stdev=49354.51 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 7], 40.00th=[ 9], 50.00th=[ 11], 60.00th=[ 14], | 70.00th=[ 18], 80.00th=[ 29], 90.00th=[ 81], 95.00th=[ 130], | 99.00th=[ 236], 99.50th=[ 284], 99.90th=[ 430], 99.95th=[ 510], | 99.99th=[ 818] bw ( KiB/s): min= 128, max=31294, per=21.28%, avg=11512.34, stdev=6485.31, samples=1799 iops : min= 2, max= 488, avg=179.51, stdev=101.16, samples=1799 write: IOPS=179, BW=11.2MiB/s (11.7MB/s)(9.85GiB/900016msec) slat (usec): min=12, max=7729, avg=88.84, stdev=162.87 clat (msec): min=3, max=1428, avg=61.47, stdev=55.16 lat (msec): min=3, max=1428, avg=61.56, stdev=55.16 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 19], 10.00th=[ 21], 20.00th=[ 26], | 30.00th=[ 31], 40.00th=[ 37], 50.00th=[ 45], 60.00th=[ 55], | 70.00th=[ 67], 80.00th=[ 86], 90.00th=[ 121], 95.00th=[ 163], | 99.00th=[ 279], 99.50th=[ 334], 99.90th=[ 514], 99.95th=[ 567], | 99.99th=[ 818] bw ( KiB/s): min= 128, max=29114, per=21.16%, avg=11497.68, stdev=6356.04, samples=1800 iops : min= 2, max= 454, avg=179.28, stdev=99.14, samples=1800 lat (msec) : 2=0.19%, 4=6.35%, 10=18.14%, 20=15.67%, 50=30.75% lat (msec) : 100=17.61%, 250=10.20%, 500=1.03%, 750=0.07%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.38%, sys=2.75%, ctx=223000, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=161435,161335,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 t256k: (groupid=0, jobs=1): err= 0: pid=2794: Sat Oct 29 12:04:57 2022 read: IOPS=150, BW=37.6MiB/s (39.4MB/s)(33.0GiB/900036msec) slat (usec): min=12, max=6574, avg=57.96, stdev=105.27 clat (msec): min=4, max=813, avg=27.88, stdev=34.64 lat (msec): min=4, max=814, avg=27.94, stdev=34.64 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 12], | 30.00th=[ 14], 40.00th=[ 17], 50.00th=[ 20], 60.00th=[ 23], | 70.00th=[ 28], 80.00th=[ 34], 90.00th=[ 50], 95.00th=[ 75], | 99.00th=[ 174], 99.50th=[ 224], 99.90th=[ 422], 99.95th=[ 510], | 99.99th=[ 684] bw ( KiB/s): min= 2064, max=74688, per=71.85%, avg=38872.11, stdev=11159.53, samples=1799 iops : min= 8, max= 291, avg=151.35, stdev=43.58, samples=1799 write: IOPS=151, BW=37.8MiB/s (39.6MB/s)(33.2GiB/900036msec) slat (usec): min=20, max=8209, avg=132.44, stdev=312.83 clat (msec): min=7, max=988, avg=77.93, stdev=46.12 lat (msec): min=7, max=988, avg=78.06, stdev=46.12 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 33], 10.00th=[ 39], 20.00th=[ 48], | 30.00th=[ 56], 40.00th=[ 63], 50.00th=[ 69], 60.00th=[ 77], | 70.00th=[ 86], 80.00th=[ 99], 90.00th=[ 124], 95.00th=[ 153], | 99.00th=[ 249], 99.50th=[ 300], 99.90th=[ 506], 99.95th=[ 575], | 99.99th=[ 827] bw ( KiB/s): min= 1551, max=62577, per=71.96%, avg=39092.41, stdev=10538.78, samples=1799 iops : min= 6, max= 244, avg=152.17, stdev=41.13, samples=1799 lat (msec) : 10=7.48%, 20=18.85%, 50=30.45%, 100=32.22%, 250=10.32% lat (msec) : 500=0.61%, 750=0.06%, 1000=0.01% cpu : usr=0.43%, sys=2.96%, ctx=175906, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=135286,136050,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=52.8MiB/s (55.4MB/s), 221KiB/s-37.6MiB/s (226kB/s-39.4MB/s), io=46.4GiB (49.9GB), run=900016-900083msec WRITE: bw=53.1MiB/s (55.6MB/s), 220KiB/s-37.8MiB/s (225kB/s-39.6MB/s), io=46.6GiB (50.1GB), run=900016-900083msec