t4k: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 t16k: (g=0): rw=rw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=16 t64k: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=16 t256k: (g=0): rw=rw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=16 fio-3.1 Starting 4 processes t4k: Laying out IO file (1 file / 10240MiB) Jobs: 4 (f=4): [M(4)][100.0%][r=57.4MiB/s,w=54.4MiB/s][r=676,w=631 IOPS][eta 00m:00s] t4k: (groupid=0, jobs=1): err= 0: pid=3577: Sat Oct 29 12:46:01 2022 read: IOPS=51, BW=207KiB/s (212kB/s)(182MiB/900082msec) slat (usec): min=11, max=3076, avg=47.43, stdev=63.15 clat (usec): min=779, max=1109.8k, avg=145631.38, stdev=93240.86 lat (usec): min=816, max=1109.8k, avg=145679.36, stdev=93243.86 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 11], 10.00th=[ 59], 20.00th=[ 83], | 30.00th=[ 97], 40.00th=[ 111], 50.00th=[ 127], 60.00th=[ 146], | 70.00th=[ 169], 80.00th=[ 201], 90.00th=[ 259], 95.00th=[ 317], | 99.00th=[ 481], 99.50th=[ 542], 99.90th=[ 735], 99.95th=[ 844], | 99.99th=[ 953] bw ( KiB/s): min= 16, max= 1128, per=0.39%, avg=207.39, stdev=106.92, samples=1800 iops : min= 4, max= 282, avg=51.81, stdev=26.73, samples=1800 write: IOPS=51, BW=206KiB/s (211kB/s)(181MiB/900082msec) slat (usec): min=11, max=4203, avg=51.02, stdev=64.65 clat (msec): min=6, max=1042, avg=164.01, stdev=94.81 lat (msec): min=6, max=1042, avg=164.06, stdev=94.81 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 59], 10.00th=[ 78], 20.00th=[ 95], | 30.00th=[ 110], 40.00th=[ 125], 50.00th=[ 142], 60.00th=[ 161], | 70.00th=[ 186], 80.00th=[ 220], 90.00th=[ 279], 95.00th=[ 342], | 99.00th=[ 514], 99.50th=[ 584], 99.90th=[ 793], 99.95th=[ 869], | 99.99th=[ 961] bw ( KiB/s): min= 7, max= 1176, per=0.39%, avg=205.96, stdev=104.64, samples=1800 iops : min= 1, max= 294, avg=51.45, stdev=26.16, samples=1800 lat (usec) : 1000=0.02% lat (msec) : 2=0.36%, 4=0.78%, 10=1.28%, 20=0.62%, 50=2.86% lat (msec) : 100=21.71%, 250=59.96%, 500=11.43%, 750=0.87%, 1000=0.11% lat (msec) : 2000=0.01% cpu : usr=0.10%, sys=0.61%, ctx=67062, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=46662,46343,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 t16k: (groupid=0, jobs=1): err= 0: pid=3578: Sat Oct 29 12:46:01 2022 read: IOPS=229, BW=3678KiB/s (3767kB/s)(3233MiB/900031msec) slat (usec): min=9, max=4981, avg=43.88, stdev=70.83 clat (usec): min=883, max=1136.1k, avg=20111.77, stdev=42333.40 lat (usec): min=921, max=1136.2k, avg=20156.13, stdev=42336.81 clat percentiles (usec): | 1.00th=[ 1287], 5.00th=[ 1696], 10.00th=[ 2147], 20.00th=[ 3064], | 30.00th=[ 4015], 40.00th=[ 5211], 50.00th=[ 6849], 60.00th=[ 9110], | 70.00th=[ 12387], 80.00th=[ 19268], 90.00th=[ 43779], 95.00th=[101188], | 99.00th=[210764], 99.50th=[270533], 99.90th=[417334], 99.95th=[467665], | 99.99th=[641729] bw ( KiB/s): min= 64, max=10997, per=6.95%, avg=3684.17, stdev=2109.10, samples=1799 iops : min= 4, max= 687, avg=230.12, stdev=131.75, samples=1799 write: IOPS=230, BW=3694KiB/s (3783kB/s)(3247MiB/900031msec) slat (usec): min=11, max=4842, avg=57.18, stdev=83.49 clat (msec): min=3, max=1178, avg=49.16, stdev=45.51 lat (msec): min=3, max=1178, avg=49.22, stdev=45.51 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 18], 10.00th=[ 20], 20.00th=[ 23], | 30.00th=[ 26], 40.00th=[ 30], 50.00th=[ 35], 60.00th=[ 42], | 70.00th=[ 52], 80.00th=[ 65], 90.00th=[ 93], 95.00th=[ 130], | 99.00th=[ 234], 99.50th=[ 292], 99.90th=[ 451], 99.95th=[ 493], | 99.99th=[ 885] bw ( KiB/s): min= 64, max= 9632, per=6.96%, avg=3700.09, stdev=2082.85, samples=1799 iops : min= 4, max= 602, avg=231.12, stdev=130.11, samples=1799 lat (usec) : 1000=0.01% lat (msec) : 2=4.13%, 4=10.72%, 10=16.71%, 20=14.81%, 50=33.60% lat (msec) : 100=13.15%, 250=6.18%, 500=0.66%, 750=0.03%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.40%, sys=2.69%, ctx=266265, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=206922,207820,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 t64k: (groupid=0, jobs=1): err= 0: pid=3579: Sat Oct 29 12:46:01 2022 read: IOPS=172, BW=10.8MiB/s (11.3MB/s)(9683MiB/900023msec) slat (usec): min=11, max=7781, avg=50.10, stdev=79.09 clat (usec): min=1589, max=996165, avg=30430.41, stdev=53040.64 lat (usec): min=1623, max=996209, avg=30481.00, stdev=53046.07 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 12], 60.00th=[ 15], | 70.00th=[ 21], 80.00th=[ 33], 90.00th=[ 90], 95.00th=[ 142], | 99.00th=[ 249], 99.50th=[ 305], 99.90th=[ 481], 99.95th=[ 609], | 99.99th=[ 835] bw ( KiB/s): min= 257, max=28985, per=20.86%, avg=11055.55, stdev=6366.57, samples=1799 iops : min= 4, max= 452, avg=172.37, stdev=99.32, samples=1799 write: IOPS=171, BW=10.7MiB/s (11.3MB/s)(9669MiB/900023msec) slat (usec): min=13, max=5643, avg=86.79, stdev=153.50 clat (msec): min=4, max=989, avg=62.46, stdev=54.66 lat (msec): min=4, max=989, avg=62.55, stdev=54.66 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 20], 10.00th=[ 23], 20.00th=[ 27], | 30.00th=[ 32], 40.00th=[ 38], 50.00th=[ 46], 60.00th=[ 55], | 70.00th=[ 66], 80.00th=[ 86], 90.00th=[ 125], 95.00th=[ 167], | 99.00th=[ 279], 99.50th=[ 338], 99.90th=[ 489], 99.95th=[ 550], | 99.99th=[ 718] bw ( KiB/s): min= 128, max=28088, per=20.75%, avg=11038.98, stdev=6249.01, samples=1799 iops : min= 2, max= 438, avg=172.12, stdev=97.49, samples=1799 lat (msec) : 2=0.08%, 4=4.92%, 10=17.43%, 20=15.04%, 50=32.86% lat (msec) : 100=17.70%, 250=10.77%, 500=1.12%, 750=0.07%, 1000=0.01% cpu : usr=0.37%, sys=2.64%, ctx=218107, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=154929,154707,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 t256k: (groupid=0, jobs=1): err= 0: pid=3580: Sat Oct 29 12:46:01 2022 read: IOPS=148, BW=37.2MiB/s (39.0MB/s)(32.7GiB/900034msec) slat (usec): min=13, max=5576, avg=61.20, stdev=99.81 clat (msec): min=5, max=754, avg=28.13, stdev=36.09 lat (msec): min=5, max=754, avg=28.20, stdev=36.09 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 12], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 19], 60.00th=[ 22], | 70.00th=[ 27], 80.00th=[ 34], 90.00th=[ 52], 95.00th=[ 82], | 99.00th=[ 184], 99.50th=[ 236], 99.90th=[ 401], 99.95th=[ 489], | 99.99th=[ 718] bw ( KiB/s): min= 521, max=69818, per=72.66%, avg=38502.51, stdev=11354.35, samples=1799 iops : min= 2, max= 272, avg=149.89, stdev=44.35, samples=1799 write: IOPS=149, BW=37.4MiB/s (39.2MB/s)(32.9GiB/900034msec) slat (usec): min=21, max=8632, avg=131.32, stdev=271.73 clat (msec): min=8, max=747, avg=78.77, stdev=45.87 lat (msec): min=8, max=748, avg=78.90, stdev=45.87 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 37], 10.00th=[ 44], 20.00th=[ 52], | 30.00th=[ 57], 40.00th=[ 63], 50.00th=[ 68], 60.00th=[ 74], | 70.00th=[ 84], 80.00th=[ 97], 90.00th=[ 125], 95.00th=[ 159], | 99.00th=[ 255], 99.50th=[ 313], 99.90th=[ 506], 99.95th=[ 550], | 99.99th=[ 693] bw ( KiB/s): min= 1034, max=62060, per=72.77%, avg=38710.40, stdev=10618.09, samples=1799 iops : min= 4, max= 242, avg=150.68, stdev=41.46, samples=1799 lat (msec) : 10=7.28%, 20=20.51%, 50=26.30%, 100=34.98%, 250=10.19% lat (msec) : 500=0.66%, 750=0.08%, 1000=0.01% cpu : usr=0.49%, sys=2.99%, ctx=194799, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=133923,134643,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=51.7MiB/s (54.3MB/s), 207KiB/s-37.2MiB/s (212kB/s-39.0MB/s), io=45.5GiB (48.8GB), run=900023-900082msec WRITE: bw=51.9MiB/s (54.5MB/s), 206KiB/s-37.4MiB/s (211kB/s-39.2MB/s), io=45.7GiB (49.0GB), run=900023-900082msec