single generic sandisk sata3 ssd; ext4
fio --filename=f --size=1GB --direct=1 --rw=randrw --bs=4k --runtime=120 --numjobs=8 --time_based --group_reporting --name=fio
fio: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.25
Starting 8 processes
fio: Laying out IO file (1 file / 1024MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=48.4MiB/s,w=49.6MiB/s][r=12.4k,w=12.7k IOPS][eta 00m:00s]
fio: (groupid=0, jobs=8): err= 0: pid=227740: Tue May 2 09:48:55 2023
read: IOPS=12.3k, BW=48.2MiB/s (50.5MB/s)(5778MiB/120002msec)
clat (usec): min=97, max=52495, avg=592.40, stdev=977.18
lat (usec): min=97, max=52495, avg=592.43, stdev=977.18
clat percentiles (usec):
| 1.00th=[ 133], 5.00th=[ 149], 10.00th=[ 161], 20.00th=[ 176],
| 30.00th=[ 190], 40.00th=[ 206], 50.00th=[ 223], 60.00th=[ 243],
| 70.00th=[ 277], 80.00th=[ 881], 90.00th=[ 2507], 95.00th=[ 2671],
| 99.00th=[ 2966], 99.50th=[ 3163], 99.90th=[ 8029], 99.95th=[10421],
| 99.99th=[21890]
bw ( KiB/s): min=39496, max=53712, per=100.00%, avg=49326.79, stdev=299.93, samples=1912
iops : min= 9874, max=13428, avg=12331.70, stdev=74.98, samples=1912
write: IOPS=12.3k, BW=48.2MiB/s (50.6MB/s)(5787MiB/120002msec); 0 zone resets
clat (usec): min=26, max=56677, avg=55.34, stdev=350.08
lat (usec): min=26, max=56677, avg=55.44, stdev=350.08
clat percentiles (usec):
| 1.00th=[ 28], 5.00th=[ 30], 10.00th=[ 31], 20.00th=[ 35],
| 30.00th=[ 40], 40.00th=[ 44], 50.00th=[ 49], 60.00th=[ 53],
| 70.00th=[ 59], 80.00th=[ 66], 90.00th=[ 77], 95.00th=[ 87],
| 99.00th=[ 112], 99.50th=[ 122], 99.90th=[ 153], 99.95th=[ 243],
| 99.99th=[ 4752]
bw ( KiB/s): min=37272, max=57248, per=100.00%, avg=49403.92, stdev=382.25, samples=1912
iops : min= 9318, max=14312, avg=12350.98, stdev=95.56, samples=1912
lat (usec) : 50=27.11%, 100=21.88%, 250=32.43%, 500=7.97%, 750=0.15%
lat (usec) : 1000=1.96%
lat (msec) : 2=2.71%, 4=5.63%, 10=0.12%, 20=0.02%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=0.42%, sys=1.47%, ctx=2961065, majf=0, minf=136
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1479212,1481493,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=48.2MiB/s (50.5MB/s), 48.2MiB/s-48.2MiB/s (50.5MB/s-50.5MB/s), io=5778MiB (6059MB), run=120002-120002msec
WRITE: bw=48.2MiB/s (50.6MB/s), 48.2MiB/s-48.2MiB/s (50.6MB/s-50.6MB/s), io=5787MiB (6068MB), run=120002-120002msec
Disk stats (read/write):
sdb: ios=1477022/1480633, merge=0/60, ticks=869483/82187, in_queue=952779, util=99.95%
single hgst 5400rpm 2.5" hdd; ext4; low LBA
fio --filename=f --size=1GB --direct=1 --rw=randrw --bs=4k --runtime=120 --numjobs=8 --time_based --group_reporting --name=fio
fio: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.25
Starting 8 processes
fio: Laying out IO file (1 file / 1024MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=344KiB/s,w=324KiB/s][r=86,w=81 IOPS][eta 00m:00s]
fio: (groupid=0, jobs=8): err= 0: pid=228358: Tue May 2 10:01:25 2023
read: IOPS=69, BW=278KiB/s (285kB/s)(32.7MiB/120553msec)
clat (usec): min=110, max=998739, avg=85820.06, stdev=112433.90
lat (usec): min=110, max=998739, avg=85820.41, stdev=112433.93
clat percentiles (msec):
| 1.00th=[ 7], 5.00th=[ 10], 10.00th=[ 13], 20.00th=[ 21],
| 30.00th=[ 28], 40.00th=[ 37], 50.00th=[ 48], 60.00th=[ 62],
| 70.00th=[ 81], 80.00th=[ 112], 90.00th=[ 194], 95.00th=[ 342],
| 99.00th=[ 567], 99.50th=[ 609], 99.90th=[ 768], 99.95th=[ 844],
| 99.99th=[ 1003]
bw ( KiB/s): min= 64, max= 840, per=100.00%, avg=289.76, stdev=21.45, samples=1843
iops : min= 16, max= 210, avg=72.44, stdev= 5.36, samples=1843
write: IOPS=72, BW=289KiB/s (296kB/s)(34.0MiB/120553msec); 0 zone resets
clat (usec): min=117, max=705504, avg=27815.63, stdev=89996.47
lat (usec): min=117, max=705506, avg=27816.17, stdev=89996.52
clat percentiles (usec):
| 1.00th=[ 145], 5.00th=[ 176], 10.00th=[ 204], 20.00th=[ 241],
| 30.00th=[ 269], 40.00th=[ 310], 50.00th=[ 367], 60.00th=[ 469],
| 70.00th=[ 635], 80.00th=[ 2868], 90.00th=[ 51643], 95.00th=[246416],
| 99.00th=[450888], 99.50th=[509608], 99.90th=[608175], 99.95th=[675283],
| 99.99th=[708838]
bw ( KiB/s): min= 64, max= 1256, per=100.00%, avg=334.59, stdev=30.26, samples=1659
iops : min= 16, max= 314, avg=83.65, stdev= 7.56, samples=1659
lat (usec) : 250=12.11%, 500=19.99%, 750=5.02%, 1000=1.92%
lat (msec) : 2=1.38%, 4=1.04%, 10=3.57%, 20=7.91%, 50=18.27%
lat (msec) : 100=13.80%, 250=8.87%, 500=4.86%, 750=1.21%, 1000=0.05%
cpu : usr=0.02%, sys=0.07%, ctx=17176, majf=0, minf=146
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=8382,8713,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=278KiB/s (285kB/s), 278KiB/s-278KiB/s (285kB/s-285kB/s), io=32.7MiB (34.3MB), run=120553-120553msec
WRITE: bw=289KiB/s (296kB/s), 289KiB/s-289KiB/s (296kB/s-296kB/s), io=34.0MiB (35.7MB), run=120553-120553msec
Disk stats (read/write):
sdd: ios=8405/9278, merge=0/57, ticks=719927/374797, in_queue=1102637, util=97.35%
single hgst 5400rpm 2.5" hdd; ext4; high LBA
fio --filename=f --size=1GB --direct=1 --rw=randrw --bs=4k --runtime=120 --numjobs=8 --time_based --group_reporting --name=fio
fio: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.25
Starting 8 processes
fio: Laying out IO file (1 file / 1024MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=312KiB/s,w=344KiB/s][r=78,w=86 IOPS][eta 00m:00s]
fio: (groupid=0, jobs=8): err= 0: pid=229075: Tue May 2 10:07:52 2023
read: IOPS=69, BW=279KiB/s (286kB/s)(32.8MiB/120084msec)
clat (usec): min=163, max=724292, avg=77702.06, stdev=78177.73
lat (usec): min=163, max=724292, avg=77702.50, stdev=78177.76
clat percentiles (msec):
| 1.00th=[ 8], 5.00th=[ 14], 10.00th=[ 20], 20.00th=[ 30],
| 30.00th=[ 40], 40.00th=[ 49], 50.00th=[ 58], 60.00th=[ 69],
| 70.00th=[ 83], 80.00th=[ 105], 90.00th=[ 150], 95.00th=[ 205],
| 99.00th=[ 481], 99.50th=[ 558], 99.90th=[ 651], 99.95th=[ 667],
| 99.99th=[ 726]
bw ( KiB/s): min= 64, max= 896, per=100.00%, avg=286.96, stdev=18.18, samples=1861
iops : min= 16, max= 224, avg=71.74, stdev= 4.54, samples=1861
write: IOPS=72, BW=291KiB/s (298kB/s)(34.1MiB/120084msec); 0 zone resets
clat (usec): min=125, max=672844, avg=35317.61, stdev=63217.21
lat (usec): min=126, max=672844, avg=35318.26, stdev=63217.24
clat percentiles (usec):
| 1.00th=[ 157], 5.00th=[ 235], 10.00th=[ 281], 20.00th=[ 465],
| 30.00th=[ 922], 40.00th=[ 13960], 50.00th=[ 25297], 60.00th=[ 33817],
| 70.00th=[ 41681], 80.00th=[ 50070], 90.00th=[ 65274], 95.00th=[ 92799],
| 99.00th=[417334], 99.50th=[488637], 99.90th=[624952], 99.95th=[650118],
| 99.99th=[675283]
bw ( KiB/s): min= 64, max= 1080, per=100.00%, avg=309.63, stdev=23.79, samples=1798
iops : min= 16, max= 270, avg=77.41, stdev= 5.95, samples=1798
lat (usec) : 250=3.33%, 500=8.04%, 750=2.77%, 1000=1.38%
lat (msec) : 2=1.09%, 4=0.77%, 10=3.00%, 20=7.95%, 50=33.00%
lat (msec) : 100=25.99%, 250=10.26%, 500=1.80%, 750=0.62%
cpu : usr=0.03%, sys=0.07%, ctx=17206, majf=0, minf=128
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=8384,8739,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=279KiB/s (286kB/s), 279KiB/s-279KiB/s (286kB/s-286kB/s), io=32.8MiB (34.3MB), run=120084-120084msec
WRITE: bw=291KiB/s (298kB/s), 291KiB/s-291KiB/s (298kB/s-298kB/s), io=34.1MiB (35.8MB), run=120084-120084msec
Disk stats (read/write):
sdd: ios=8390/8811, merge=0/27, ticks=651838/334038, in_queue=997115, util=99.76%
samsung 980 + misc sk hynix in md raid 1; both pcie 3.0x4, ~95% fill:
$ fio --filename=f --size=1GB --direct=1 --rw=randrw --bs=4k --runtime=120 --numjobs=8 --time_based --group_reporting --name=fio
fio: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.33
Starting 8 processes
Jobs: 8 (f=8): [m(8)][100.0%][r=136MiB/s,w=138MiB/s][r=34.9k,w=35.2k IOPS][eta 00m:00s]
fio: (groupid=0, jobs=8): err= 0: pid=216387: Thu Feb 16 15:29:55 2023
read: IOPS=35.1k, BW=137MiB/s (144MB/s)(16.1GiB/120002msec)
clat (usec): min=16, max=16876, avg=144.12, stdev=405.88
lat (usec): min=16, max=16876, avg=144.17, stdev=405.88
clat percentiles (usec):
| 1.00th=[ 54], 5.00th=[ 55], 10.00th=[ 56], 20.00th=[ 57],
| 30.00th=[ 57], 40.00th=[ 58], 50.00th=[ 59], 60.00th=[ 61],
| 70.00th=[ 64], 80.00th=[ 80], 90.00th=[ 118], 95.00th=[ 297],
| 99.00th=[ 2147], 99.50th=[ 3064], 99.90th=[ 3916], 99.95th=[ 4948],
| 99.99th=[ 7701]
bw ( KiB/s): min=92240, max=318976, per=100.00%, avg=140304.00, stdev=3728.31, samples=1912
iops : min=23060, max=79744, avg=35076.00, stdev=932.08, samples=1912
write: IOPS=35.1k, BW=137MiB/s (144MB/s)(16.1GiB/120002msec); 0 zone resets
clat (usec): min=14, max=36248, avg=82.76, stdev=562.80
lat (usec): min=14, max=36248, avg=82.89, stdev=562.80
clat percentiles (usec):
| 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 16],
| 30.00th=[ 17], 40.00th=[ 17], 50.00th=[ 17], 60.00th=[ 18],
| 70.00th=[ 18], 80.00th=[ 19], 90.00th=[ 20], 95.00th=[ 23],
| 99.00th=[ 3130], 99.50th=[ 4424], 99.90th=[ 7177], 99.95th=[ 8717],
| 99.99th=[13173]
bw ( KiB/s): min=92112, max=321392, per=100.00%, avg=140380.12, stdev=3765.76, samples=1912
iops : min=23028, max=80348, avg=35095.03, stdev=941.44, samples=1912
lat (usec) : 20=45.18%, 50=3.80%, 100=44.23%, 250=3.08%, 500=0.70%
lat (usec) : 750=0.27%, 1000=0.20%
lat (msec) : 2=1.29%, 4=0.89%, 10=0.34%, 20=0.02%, 50=0.01%
cpu : usr=0.94%, sys=4.18%, ctx=8419043, majf=0, minf=103
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=4208364,4210471,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=137MiB/s (144MB/s), 137MiB/s-137MiB/s (144MB/s-144MB/s), io=16.1GiB (17.2GB), run=120002-120002msec
WRITE: bw=137MiB/s (144MB/s), 137MiB/s-137MiB/s (144MB/s-144MB/s), io=16.1GiB (17.2GB), run=120002-120002msec
Disk stats (read/write):
md0: ios=4204033/4208230, merge=0/0, ticks=588732/338488, in_queue=927432, util=99.92%, aggrios=2104182/4211778, aggrmerge=0/881, aggrticks=294790/189374, aggrin_queue=485135, aggrutil=99.94%
nvme1n1: ios=3785856/4211778, merge=0/882, ticks=237311/49957, in_queue=287397, util=96.65%
nvme0n1: ios=422509/4211779, merge=0/881, ticks=352270/328791, in_queue=682873, util=99.94%
samsung 980 + misc sk hynix in md raid 1; both pcie 3.0x4, ~35% fill:
$ fio --filename=f --size=1GB --direct=1 --rw=randrw --bs=4k --runtime=120 --numjobs=8 --time_based --group_reporting --name=fio
fio: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.33
Starting 8 processes
fio: Laying out IO file (1 file / 1024MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=328MiB/s,w=328MiB/s][r=84.1k,w=84.0k IOPS][eta 00m:00s]
fio: (groupid=0, jobs=8): err= 0: pid=13547: Fri Feb 17 16:10:56 2023
read: IOPS=83.7k, BW=327MiB/s (343MB/s)(38.3GiB/120002msec)
clat (usec): min=16, max=8013, avg=77.29, stdev=107.12
lat (usec): min=16, max=8013, avg=77.34, stdev=107.12
clat percentiles (usec):
| 1.00th=[ 45], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 48],
| 30.00th=[ 52], 40.00th=[ 56], 50.00th=[ 58], 60.00th=[ 60],
| 70.00th=[ 66], 80.00th=[ 85], 90.00th=[ 137], 95.00th=[ 184],
| 99.00th=[ 227], 99.50th=[ 249], 99.90th=[ 1565], 99.95th=[ 1696],
| 99.99th=[ 5735]
bw ( KiB/s): min=240272, max=360152, per=100.00%, avg=334983.26, stdev=3255.80, samples=1912
iops : min=60068, max=90038, avg=83745.82, stdev=813.95, samples=1912
write: IOPS=83.7k, BW=327MiB/s (343MB/s)(38.3GiB/120002msec); 0 zone resets
clat (usec): min=13, max=10709, avg=17.25, stdev=20.45
lat (usec): min=13, max=10709, avg=17.34, stdev=20.45
clat percentiles (usec):
| 1.00th=[ 15], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 16],
| 30.00th=[ 16], 40.00th=[ 17], 50.00th=[ 17], 60.00th=[ 17],
| 70.00th=[ 18], 80.00th=[ 19], 90.00th=[ 20], 95.00th=[ 21],
| 99.00th=[ 25], 99.50th=[ 31], 99.90th=[ 40], 99.95th=[ 51],
| 99.99th=[ 594]
bw ( KiB/s): min=238560, max=366288, per=100.00%, avg=334925.39, stdev=3308.88, samples=1912
iops : min=59640, max=91572, avg=83731.36, stdev=827.22, samples=1912
lat (usec) : 20=46.46%, 50=17.18%, 100=29.13%, 250=6.97%, 500=0.10%
lat (usec) : 750=0.03%, 1000=0.02%
lat (msec) : 2=0.10%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=2.10%, sys=9.77%, ctx=20083001, majf=0, minf=100
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=10042252,10040346,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=327MiB/s (343MB/s), 327MiB/s-327MiB/s (343MB/s-343MB/s), io=38.3GiB (41.1GB), run=120002-120002msec
WRITE: bw=327MiB/s (343MB/s), 327MiB/s-327MiB/s (343MB/s-343MB/s), io=38.3GiB (41.1GB), run=120002-120002msec
Disk stats (read/write):
md0: ios=10033514/10033264, merge=0/0, ticks=735932/129296, in_queue=865284, util=99.95%, aggrios=5021131/10041506, aggrmerge=0/788, aggrticks=368409/118980, aggrin_queue=487523, aggrutil=99.94%
nvme1n1: ios=5037618/10041506, merge=0/788, ticks=340419/121271, in_queue=461834, util=99.94%
nvme0n1: ios=5004645/10041506, merge=0/788, ticks=396400/116690, in_queue=513212, util=99.94%
samsung 980 + misc sk hynix in md raid 0; both pcie 3.0x4, 22% fill:
fio --filename=f --size=1GB --direct=1 --rw=randrw --bs=4k --runtime=120 --numjobs=8 --time_based --group_reporting --name=fio
fio: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.25
Starting 8 processes
fio: Laying out IO file (1 file / 1024MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=402MiB/s,w=399MiB/s][r=103k,w=102k IOPS][eta 00m:00s]]
fio: (groupid=0, jobs=8): err= 0: pid=46981: Mon May 1 10:41:19 2023
read: IOPS=102k, BW=399MiB/s (418MB/s)(46.7GiB/120001msec)
clat (usec): min=14, max=7055, avg=64.35, stdev=80.59
lat (usec): min=14, max=7056, avg=64.38, stdev=80.59
clat percentiles (usec):
| 1.00th=[ 44], 5.00th=[ 44], 10.00th=[ 45], 20.00th=[ 46],
| 30.00th=[ 47], 40.00th=[ 53], 50.00th=[ 55], 60.00th=[ 56],
| 70.00th=[ 57], 80.00th=[ 63], 90.00th=[ 91], 95.00th=[ 149],
| 99.00th=[ 212], 99.50th=[ 223], 99.90th=[ 249], 99.95th=[ 457],
| 99.99th=[ 5604]
bw ( KiB/s): min=388928, max=424200, per=100.00%, avg=408487.10, stdev=873.71, samples=1912
iops : min=97232, max=106050, avg=102121.77, stdev=218.43, samples=1912
write: IOPS=102k, BW=399MiB/s (418MB/s)(46.7GiB/120001msec); 0 zone resets
clat (usec): min=10, max=7361, avg=13.13, stdev=12.85
lat (usec): min=10, max=7361, avg=13.23, stdev=12.85
clat percentiles (nsec):
| 1.00th=[11200], 5.00th=[11456], 10.00th=[11584], 20.00th=[11968],
| 30.00th=[12224], 40.00th=[12480], 50.00th=[12736], 60.00th=[12992],
| 70.00th=[13376], 80.00th=[13888], 90.00th=[14528], 95.00th=[15296],
| 99.00th=[17536], 99.50th=[21376], 99.90th=[32128], 99.95th=[37632],
| 99.99th=[76288]
bw ( KiB/s): min=382704, max=430008, per=100.00%, avg=408297.64, stdev=1115.10, samples=1912
iops : min=95676, max=107502, avg=102074.41, stdev=278.77, samples=1912
lat (usec) : 20=49.71%, 50=18.44%, 100=27.66%, 250=4.15%, 500=0.03%
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
cpu : usr=2.33%, sys=7.39%, ctx=24495111, majf=0, minf=132
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=12250555,12244292,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=399MiB/s (418MB/s), 399MiB/s-399MiB/s (418MB/s-418MB/s), io=46.7GiB (50.2GB), run=120001-120001msec
WRITE: bw=399MiB/s (418MB/s), 399MiB/s-399MiB/s (418MB/s-418MB/s), io=46.7GiB (50.2GB), run=120001-120001msec
Disk stats (read/write):
md0: ios=12236632/12232127, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=6125277/6122772, aggrmerge=0/348, aggrticks=380407/66746, aggrin_queue=447188, aggrutil=99.96%
nvme0n1: ios=6126321/6121605, merge=0/350, ticks=380490/64384, in_queue=444898, util=99.96%
nvme1n1: ios=6124234/6123940, merge=0/346, ticks=380324/69108, in_queue=449479, util=99.73%
/dev/shm; dual channel 2666mhz ddr4, xeon w-1290 50w bios tdp limit
fio --filename=f --size=1GB --direct=0 --rw=randrw --bs=4k --runtime=120 --numjobs=8 --time_based --group_reporting --name=fio
fio: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.25
Starting 8 processes
Jobs: 8 (f=8): [m(8)][100.0%][r=5398MiB/s,w=5405MiB/s][r=1382k,w=1384k IOPS][eta 00m:00s]
fio: (groupid=0, jobs=8): err= 0: pid=48190: Mon May 1 10:50:31 2023
read: IOPS=1381k, BW=5395MiB/s (5657MB/s)(632GiB/120001msec)
clat (nsec): min=304, max=367749, avg=819.88, stdev=480.23
lat (nsec): min=320, max=367766, avg=840.24, stdev=485.38
clat percentiles (nsec):
| 1.00th=[ 548], 5.00th=[ 612], 10.00th=[ 644], 20.00th=[ 684],
| 30.00th=[ 716], 40.00th=[ 748], 50.00th=[ 772], 60.00th=[ 812],
| 70.00th=[ 852], 80.00th=[ 908], 90.00th=[ 1012], 95.00th=[ 1112],
| 99.00th=[ 1384], 99.50th=[ 1592], 99.90th=[ 4256], 99.95th=[17280],
| 99.99th=[19584]
bw ( MiB/s): min= 4941, max= 5568, per=100.00%, avg=5396.70, stdev=12.21, samples=1912
iops : min=1265014, max=1425462, avg=1381555.06, stdev=3124.57, samples=1912
write: IOPS=1381k, BW=5396MiB/s (5658MB/s)(632GiB/120001msec); 0 zone resets
clat (nsec): min=413, max=7148.2k, avg=4435.80, stdev=3709.23
lat (nsec): min=440, max=7148.3k, avg=4467.41, stdev=3710.72
clat percentiles (nsec):
| 1.00th=[ 708], 5.00th=[ 876], 10.00th=[ 2896], 20.00th=[ 3664],
| 30.00th=[ 4016], 40.00th=[ 4256], 50.00th=[ 4512], 60.00th=[ 4768],
| 70.00th=[ 5024], 80.00th=[ 5344], 90.00th=[ 5792], 95.00th=[ 6240],
| 99.00th=[ 7520], 99.50th=[10432], 99.90th=[23936], 99.95th=[25728],
| 99.99th=[33024]
bw ( MiB/s): min= 4944, max= 5562, per=100.00%, avg=5397.67, stdev=11.87, samples=1912
iops : min=1265674, max=1424064, avg=1381803.18, stdev=3039.12, samples=1912
lat (nsec) : 500=0.12%, 750=21.65%, 1000=26.35%
lat (usec) : 2=5.83%, 4=11.00%, 10=34.77%, 20=0.12%, 50=0.16%
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
cpu : usr=11.87%, sys=88.04%, ctx=360012, majf=0, minf=130
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=165738944,165767401,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=5395MiB/s (5657MB/s), 5395MiB/s-5395MiB/s (5657MB/s-5657MB/s), io=632GiB (679GB), run=120001-120001msec
WRITE: bw=5396MiB/s (5658MB/s), 5396MiB/s-5396MiB/s (5658MB/s-5658MB/s), io=632GiB (679GB), run=120001-120001msec
two node ceph, replica 2. one consumer sata ssd osd per node, direct 1gbe crossover between nodes. /dev/rbd0 mounted on desktop with 2gbe to intermediary router.
$ sudo fio --filename=/dev/rbd0 --direct=1 --rw=randrw --bs=4k --runtime=120 --numjobs=8 --time_based --group_reporting --name=fio
fio: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.33
Starting 8 processes
Jobs: 8 (f=8): [m(8)][100.0%][r=152KiB/s,w=176KiB/s][r=38,w=44 IOPS][eta 00m:00s]
fio: (groupid=0, jobs=8): err= 0: pid=86811: Sat Feb 18 20:31:26 2023
read: IOPS=119, BW=477KiB/s (488kB/s)(56.0MiB/120222msec)
clat (usec): min=2, max=21699, avg=718.04, stdev=1723.97
lat (usec): min=2, max=21699, avg=718.33, stdev=1723.96
clat percentiles (usec):
| 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 9],
| 30.00th=[ 13], 40.00th=[ 19], 50.00th=[ 47], 60.00th=[ 375],
| 70.00th=[ 652], 80.00th=[ 930], 90.00th=[ 1369], 95.00th=[ 3490],
| 99.00th=[ 9110], 99.50th=[11338], 99.90th=[16581], 99.95th=[18220],
| 99.99th=[20841]
bw ( KiB/s): min= 64, max= 2728, per=100.00%, avg=536.76, stdev=66.31, samples=1709
iops : min= 16, max= 682, avg=134.19, stdev=16.58, samples=1709
write: IOPS=121, BW=485KiB/s (497kB/s)(56.9MiB/120222msec); 0 zone resets
clat (msec): min=5, max=664, avg=65.21, stdev=85.64
lat (msec): min=5, max=664, avg=65.21, stdev=85.64
clat percentiles (msec):
| 1.00th=[ 9], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 15],
| 30.00th=[ 17], 40.00th=[ 21], 50.00th=[ 46], 60.00th=[ 51],
| 70.00th=[ 54], 80.00th=[ 64], 90.00th=[ 230], 95.00th=[ 296],
| 99.00th=[ 351], 99.50th=[ 376], 99.90th=[ 435], 99.95th=[ 575],
| 99.99th=[ 667]
bw ( KiB/s): min= 64, max= 2240, per=100.00%, avg=486.79, stdev=62.90, samples=1914
iops : min= 16, max= 560, avg=121.70, stdev=15.73, samples=1914
lat (usec) : 4=4.10%, 10=7.56%, 20=9.00%, 50=4.70%, 100=3.24%
lat (usec) : 250=0.06%, 500=2.94%, 750=4.92%, 1000=4.15%
lat (msec) : 2=5.51%, 4=1.26%, 10=2.96%, 20=19.31%, 50=9.52%
lat (msec) : 100=14.28%, 250=2.01%, 500=4.45%, 750=0.04%
cpu : usr=0.02%, sys=0.05%, ctx=34975, majf=0, minf=114
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=14336,14575,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=477KiB/s (488kB/s), 477KiB/s-477KiB/s (488kB/s-488kB/s), io=56.0MiB (58.7MB), run=120222-120222msec
WRITE: bw=485KiB/s (497kB/s), 485KiB/s-485KiB/s (497kB/s-497kB/s), io=56.9MiB (59.7MB), run=120222-120222msec
Disk stats (read/write):
rbd0: ios=14336/14572, merge=0/0, ticks=10027/948912, in_queue=958940, util=99.98%
lxd vm co-located on above ceph cluster:
fio --filename=f --size=1GB --direct=1 --rw=randrw --bs=4k --runtime=120 --numjobs=8 --time_based --group_reporting --name=fio
fio: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.33
Starting 8 processes
fio: Laying out IO file (1 file / 1024MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=500KiB/s,w=480KiB/s][r=125,w=120 IOPS][eta 00m:00s]
fio: (groupid=0, jobs=8): err= 0: pid=1316: Tue Feb 21 07:05:48 2023
read: IOPS=126, BW=505KiB/s (518kB/s)(59.2MiB/120042msec)
clat (usec): min=314, max=105723, avg=6258.89, stdev=8406.28
lat (usec): min=314, max=105724, avg=6259.77, stdev=8406.27
clat percentiles (usec):
| 1.00th=[ 758], 5.00th=[ 996], 10.00th=[ 1221], 20.00th=[ 1614],
| 30.00th=[ 1958], 40.00th=[ 2540], 50.00th=[ 3064], 60.00th=[ 3818],
| 70.00th=[ 4752], 80.00th=[ 9634], 90.00th=[ 17695], 95.00th=[ 22414],
| 99.00th=[ 41157], 99.50th=[ 52691], 99.90th=[ 70779], 99.95th=[ 73925],
| 99.99th=[100140]
bw ( KiB/s): min= 64, max= 1344, per=100.00%, avg=506.33, stdev=28.42, samples=1906
iops : min= 16, max= 336, avg=125.68, stdev= 7.13, samples=1906
write: IOPS=128, BW=513KiB/s (525kB/s)(60.1MiB/120042msec); 0 zone resets
clat (msec): min=6, max=336, avg=56.18, stdev=14.63
lat (msec): min=6, max=336, avg=56.19, stdev=14.63
clat percentiles (msec):
| 1.00th=[ 19], 5.00th=[ 39], 10.00th=[ 43], 20.00th=[ 46],
| 30.00th=[ 50], 40.00th=[ 53], 50.00th=[ 54], 60.00th=[ 58],
| 70.00th=[ 62], 80.00th=[ 65], 90.00th=[ 73], 95.00th=[ 80],
| 99.00th=[ 97], 99.50th=[ 109], 99.90th=[ 144], 99.95th=[ 165],
| 99.99th=[ 305]
bw ( KiB/s): min= 200, max= 920, per=99.85%, avg=512.20, stdev=10.07, samples=1912
iops : min= 50, max= 230, avg=127.15, stdev= 2.57, samples=1912
lat (usec) : 500=0.04%, 750=0.39%, 1000=2.12%
lat (msec) : 2=12.84%, 4=15.52%, 10=9.36%, 20=5.96%, 50=19.50%
lat (msec) : 100=33.88%, 250=0.38%, 500=0.02%
cpu : usr=0.09%, sys=0.27%, ctx=30756, majf=0, minf=96
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=15168,15389,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=505KiB/s (518kB/s), 505KiB/s-505KiB/s (518kB/s-518kB/s), io=59.2MiB (62.1MB), run=120042-120042msec
WRITE: bw=513KiB/s (525kB/s), 513KiB/s-513KiB/s (525kB/s-525kB/s), io=60.1MiB (63.0MB), run=120042-120042msec
Disk stats (read/write):
sda: ios=15151/15424, merge=0/39, ticks=91290/868849, in_queue=963970, util=100.00%
lxd container co-located on above ceph cluster:
fio --filename=f --size=1GB --direct=1 --rw=randrw --bs=4k --runtime=120 --numjobs=8 --time_based --group_reporting --name=fio
fio: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.33
Starting 8 processes
fio: Laying out IO file (1 file / 1024MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=468KiB/s,w=376KiB/s][r=117,w=94 IOPS][eta 00m:00s]
fio: (groupid=0, jobs=8): err= 0: pid=990: Tue Feb 21 07:09:38 2023
read: IOPS=119, BW=477KiB/s (489kB/s)(55.9MiB/120056msec)
clat (usec): min=171, max=617272, avg=6123.32, stdev=12843.83
lat (usec): min=171, max=617273, avg=6123.92, stdev=12843.83
clat percentiles (usec):
| 1.00th=[ 537], 5.00th=[ 660], 10.00th=[ 758], 20.00th=[ 947],
| 30.00th=[ 1270], 40.00th=[ 1991], 50.00th=[ 2737], 60.00th=[ 3392],
| 70.00th=[ 4293], 80.00th=[ 10028], 90.00th=[ 14746], 95.00th=[ 23462],
| 99.00th=[ 45876], 99.50th=[ 58459], 99.90th=[ 90702], 99.95th=[116917],
| 99.99th=[616563]
bw ( KiB/s): min= 64, max= 1248, per=100.00%, avg=481.41, stdev=28.13, samples=1898
iops : min= 16, max= 312, avg=120.35, stdev= 7.03, samples=1898
write: IOPS=121, BW=484KiB/s (496kB/s)(56.8MiB/120056msec); 0 zone resets
clat (msec): min=20, max=736, avg=60.03, stdev=25.68
lat (msec): min=20, max=736, avg=60.03, stdev=25.68
clat percentiles (msec):
| 1.00th=[ 36], 5.00th=[ 41], 10.00th=[ 44], 20.00th=[ 48],
| 30.00th=[ 51], 40.00th=[ 53], 50.00th=[ 55], 60.00th=[ 59],
| 70.00th=[ 63], 80.00th=[ 66], 90.00th=[ 75], 95.00th=[ 87],
| 99.00th=[ 178], 99.50th=[ 211], 99.90th=[ 271], 99.95th=[ 284],
| 99.99th=[ 735]
bw ( KiB/s): min= 120, max= 672, per=100.00%, avg=485.62, stdev=10.27, samples=1909
iops : min= 30, max= 168, avg=121.40, stdev= 2.57, samples=1909
lat (usec) : 250=0.01%, 500=0.22%, 750=4.50%, 1000=6.26%
lat (msec) : 2=8.95%, 4=13.38%, 10=6.22%, 20=6.06%, 50=17.52%
lat (msec) : 100=35.08%, 250=1.69%, 500=0.08%, 750=0.03%
cpu : usr=0.04%, sys=0.11%, ctx=28962, majf=0, minf=101
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=14321,14533,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=477KiB/s (489kB/s), 477KiB/s-477KiB/s (489kB/s-489kB/s), io=55.9MiB (58.7MB), run=120056-120056msec
WRITE: bw=484KiB/s (496kB/s), 484KiB/s-484KiB/s (496kB/s-496kB/s), io=56.8MiB (59.5MB), run=120056-120056msec
Disk stats (read/write):
rbd2: ios=14294/14562, merge=0/32, ticks=85195/872882, in_queue=958077, util=100.00%
pentium 4 with 2x 7200rpm in md raid0
fio --filename=f --size=1GB --direct=1 --rw=randrw --bs=4k --runtime=120 --numjobs=8 --time_based --group_reporting --name=fio
fio: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.25
Starting 8 processes
fio: Laying out IO file (1 file / 1024MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=696KiB/s,w=780KiB/s][r=174,w=195 IOPS][eta 00m:00s]
fio: (groupid=0, jobs=8): err= 0: pid=1010: Sat Feb 18 22:53:26 2023
read: IOPS=178, BW=713KiB/s (730kB/s)(83.6MiB/120058msec)
clat (usec): min=211, max=1153.9k, avg=25935.03, stdev=39358.94
lat (usec): min=212, max=1153.9k, avg=25935.55, stdev=39358.97
clat percentiles (usec):
| 1.00th=[ 1467], 5.00th=[ 3228], 10.00th=[ 4752], 20.00th=[ 7242],
| 30.00th=[ 9241], 40.00th=[ 11731], 50.00th=[ 15008], 60.00th=[ 19006],
| 70.00th=[ 25035], 80.00th=[ 33817], 90.00th=[ 52167], 95.00th=[ 76022],
| 99.00th=[225444], 99.50th=[270533], 99.90th=[367002], 99.95th=[442500],
| 99.99th=[884999]
bw ( KiB/s): min= 72, max= 1424, per=100.00%, avg=717.18, stdev=33.15, samples=1903
iops : min= 18, max= 356, avg=179.19, stdev= 8.28, samples=1903
write: IOPS=181, BW=726KiB/s (743kB/s)(85.1MiB/120058msec); 0 zone resets
clat (usec): min=194, max=1179.5k, avg=18605.54, stdev=38714.65
lat (usec): min=194, max=1179.5k, avg=18606.56, stdev=38714.65
clat percentiles (usec):
| 1.00th=[ 219], 5.00th=[ 265], 10.00th=[ 338],
| 20.00th=[ 619], 30.00th=[ 1418], 40.00th=[ 4948],
| 50.00th=[ 8029], 60.00th=[ 11338], 70.00th=[ 17695],
| 80.00th=[ 26346], 90.00th=[ 42206], 95.00th=[ 63177],
| 99.00th=[ 210764], 99.50th=[ 261096], 99.90th=[ 371196],
| 99.95th=[ 429917], 99.99th=[1098908]
bw ( KiB/s): min= 80, max= 1640, per=100.00%, avg=730.01, stdev=36.28, samples=1904
iops : min= 20, max= 410, avg=182.40, stdev= 9.07, samples=1904
lat (usec) : 250=1.88%, 500=6.14%, 750=4.66%, 1000=1.59%
lat (msec) : 2=2.58%, 4=5.53%, 10=23.01%, 20=22.16%, 50=23.34%
lat (msec) : 100=6.31%, 250=2.16%, 500=0.63%, 1000=0.01%, 2000=0.01%
cpu : usr=0.04%, sys=0.20%, ctx=75378, majf=0, minf=133
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=21390,21782,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=713KiB/s (730kB/s), 713KiB/s-713KiB/s (730kB/s-730kB/s), io=83.6MiB (87.6MB), run=120058-120058msec
WRITE: bw=726KiB/s (743kB/s), 726KiB/s-726KiB/s (743kB/s-743kB/s), io=85.1MiB (89.2MB), run=120058-120058msec
Disk stats (read/write):
md0: ios=21393/22323, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=10698/11104, aggrmerge=0/82, aggrticks=125485/55883, aggrin_queue=186763, aggrutil=88.13%
sdb: ios=10751/11115, merge=0/113, ticks=134999/57446, in_queue=196542, util=88.13%
sda: ios=10645/11093, merge=0/51, ticks=115972/54321, in_queue=176985, util=79.50%
cephfs, kernel client driver, 4 consumer sata ssd osds on four nodes. replica 2. 1gbps all. 1500 mtu client
fio --filename=f --size=1GB --direct=1 --rw=randrw --bs=4k --runtime=120 --numjobs=8 --time_based --group_reporting --name=fio
fio: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.25
Starting 8 processes
fio: Laying out IO file (1 file / 1024MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=1408KiB/s,w=1264KiB/s][r=352,w=316 IOPS][eta 00m:00s]
fio: (groupid=0, jobs=8): err= 0: pid=261316: Tue May 2 16:51:15 2023
read: IOPS=307, BW=1229KiB/s (1258kB/s)(144MiB/120059msec)
clat (usec): min=379, max=713525, avg=3332.56, stdev=6845.26
lat (usec): min=379, max=713525, avg=3332.79, stdev=6845.28
clat percentiles (usec):
| 1.00th=[ 611], 5.00th=[ 758], 10.00th=[ 840], 20.00th=[ 938],
| 30.00th=[ 1037], 40.00th=[ 1172], 50.00th=[ 1385], 60.00th=[ 1713],
| 70.00th=[ 2245], 80.00th=[ 3130], 90.00th=[ 8717], 95.00th=[15533],
| 99.00th=[25035], 99.50th=[32113], 99.90th=[61604], 99.95th=[68682],
| 99.99th=[89654]
bw ( KiB/s): min= 128, max= 3176, per=100.00%, avg=1235.44, stdev=77.18, samples=1908
iops : min= 32, max= 794, avg=308.86, stdev=19.29, samples=1908
write: IOPS=310, BW=1242KiB/s (1272kB/s)(146MiB/120059msec); 0 zone resets
clat (msec): min=4, max=743, avg=22.46, stdev=21.97
lat (msec): min=4, max=743, avg=22.46, stdev=21.97
clat percentiles (msec):
| 1.00th=[ 7], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 10],
| 30.00th=[ 12], 40.00th=[ 13], 50.00th=[ 14], 60.00th=[ 16],
| 70.00th=[ 19], 80.00th=[ 32], 90.00th=[ 59], 95.00th=[ 68],
| 99.00th=[ 84], 99.50th=[ 88], 99.90th=[ 103], 99.95th=[ 112],
| 99.99th=[ 735]
bw ( KiB/s): min= 400, max= 2400, per=100.00%, avg=1249.63, stdev=65.32, samples=1907
iops : min= 100, max= 600, avg=312.41, stdev=16.33, samples=1907
lat (usec) : 500=0.12%, 750=2.11%, 1000=10.82%
lat (msec) : 2=19.90%, 4=9.99%, 10=12.56%, 20=29.24%, 50=7.84%
lat (msec) : 100=7.35%, 250=0.05%, 750=0.01%
cpu : usr=0.06%, sys=0.33%, ctx=147643, majf=0, minf=123
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=36876,37279,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=1229KiB/s (1258kB/s), 1229KiB/s-1229KiB/s (1258kB/s-1258kB/s), io=144MiB (151MB), run=120059-120059msec
WRITE: bw=1242KiB/s (1272kB/s), 1242KiB/s-1242KiB/s (1272kB/s-1272kB/s), io=146MiB (153MB), run=120059-120059msec
cephfs, kernel client driver, 4 consumer sata ssd osds on four nodes. replica 2. 1gbps all. 9000 mtu client
fio --filename=f --size=1GB --direct=1 --rw=randrw --bs=4k --runtime=120 --numjobs=8 --time_based --group_reporting --name=fio
fio: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.25
Starting 8 processes
fio: Laying out IO file (1 file / 1024MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=1152KiB/s,w=1140KiB/s][r=288,w=285 IOPS][eta 00m:00s]
fio: (groupid=0, jobs=8): err= 0: pid=256979: Tue May 2 15:58:37 2023
read: IOPS=309, BW=1237KiB/s (1266kB/s)(145MiB/120060msec)
clat (usec): min=434, max=128951, avg=3445.69, stdev=5943.64
lat (usec): min=435, max=128952, avg=3446.02, stdev=5943.68
clat percentiles (usec):
| 1.00th=[ 676], 5.00th=[ 848], 10.00th=[ 955], 20.00th=[ 1106],
| 30.00th=[ 1237], 40.00th=[ 1385], 50.00th=[ 1565], 60.00th=[ 1860],
| 70.00th=[ 2376], 80.00th=[ 3163], 90.00th=[ 7570], 95.00th=[15926],
| 99.00th=[26608], 99.50th=[35914], 99.90th=[62129], 99.95th=[76022],
| 99.99th=[95945]
bw ( KiB/s): min= 184, max= 2984, per=100.00%, avg=1239.85, stdev=77.29, samples=1913
iops : min= 46, max= 746, avg=309.96, stdev=19.32, samples=1913
write: IOPS=312, BW=1250KiB/s (1280kB/s)(147MiB/120060msec); 0 zone resets
clat (msec): min=5, max=164, avg=22.18, stdev=20.16
lat (msec): min=5, max=164, avg=22.18, stdev=20.16
clat percentiles (msec):
| 1.00th=[ 7], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 10],
| 30.00th=[ 12], 40.00th=[ 13], 50.00th=[ 14], 60.00th=[ 16],
| 70.00th=[ 19], 80.00th=[ 28], 90.00th=[ 58], 95.00th=[ 69],
| 99.00th=[ 87], 99.50th=[ 94], 99.90th=[ 125], 99.95th=[ 138],
| 99.99th=[ 157]
bw ( KiB/s): min= 336, max= 2344, per=100.00%, avg=1252.73, stdev=67.92, samples=1913
iops : min= 84, max= 586, avg=313.18, stdev=16.98, samples=1913
lat (usec) : 500=0.04%, 750=1.04%, 1000=5.30%
lat (msec) : 2=25.15%, 4=11.55%, 10=12.69%, 20=29.33%, 50=7.77%
lat (msec) : 100=6.91%, 250=0.19%
cpu : usr=0.08%, sys=0.37%, ctx=148487, majf=0, minf=114
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=37114,37514,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=1237KiB/s (1266kB/s), 1237KiB/s-1237KiB/s (1266kB/s-1266kB/s), io=145MiB (152MB), run=120060-120060msec
WRITE: bw=1250KiB/s (1280kB/s), 1250KiB/s-1250KiB/s (1280kB/s-1280kB/s), io=147MiB (154MB), run=120060-120060msec
sandisk usb 3 flash drive; ext4
fio --filename=f --size=1GB --direct=1 --rw=randrw --bs=4k --runtime=120 --numjobs=8 --time_based --group_reporting --name=fio
fio: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.25
Starting 8 processes
fio: Laying out IO file (1 file / 1024MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=624KiB/s,w=744KiB/s][r=156,w=186 IOPS][eta 00m:00s]
fio: (groupid=0, jobs=8): err= 0: pid=229903: Tue May 2 10:18:28 2023
read: IOPS=169, BW=679KiB/s (695kB/s)(79.6MiB/120023msec)
clat (usec): min=800, max=324474, avg=21639.12, stdev=23988.54
lat (usec): min=800, max=324474, avg=21639.51, stdev=23988.58
clat percentiles (usec):
| 1.00th=[ 1516], 5.00th=[ 1565], 10.00th=[ 2966], 20.00th=[ 3884],
| 30.00th=[ 5997], 40.00th=[ 11863], 50.00th=[ 17433], 60.00th=[ 21627],
| 70.00th=[ 25035], 80.00th=[ 30016], 90.00th=[ 43779], 95.00th=[ 66847],
| 99.00th=[120062], 99.50th=[141558], 99.90th=[210764], 99.95th=[242222],
| 99.99th=[304088]
bw ( KiB/s): min= 136, max= 1736, per=99.98%, avg=679.90, stdev=31.52, samples=1912
iops : min= 34, max= 434, avg=169.97, stdev= 7.88, samples=1912
write: IOPS=172, BW=689KiB/s (706kB/s)(80.8MiB/120023msec); 0 zone resets
clat (msec): min=2, max=288, avg=25.09, stdev=24.35
lat (msec): min=2, max=288, avg=25.09, stdev=24.35
clat percentiles (msec):
| 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 7],
| 30.00th=[ 12], 40.00th=[ 17], 50.00th=[ 21], 60.00th=[ 24],
| 70.00th=[ 28], 80.00th=[ 34], 90.00th=[ 51], 95.00th=[ 74],
| 99.00th=[ 127], 99.50th=[ 148], 99.90th=[ 201], 99.95th=[ 220],
| 99.99th=[ 253]
bw ( KiB/s): min= 128, max= 1568, per=99.96%, avg=689.78, stdev=29.82, samples=1911
iops : min= 32, max= 392, avg=172.44, stdev= 7.45, samples=1911
lat (usec) : 1000=0.01%
lat (msec) : 2=4.51%, 4=10.24%, 10=15.35%, 20=21.44%, 50=39.24%
lat (msec) : 100=7.25%, 250=1.94%, 500=0.02%
cpu : usr=0.07%, sys=0.29%, ctx=82116, majf=0, minf=122
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=20379,20683,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=679KiB/s (695kB/s), 679KiB/s-679KiB/s (695kB/s-695kB/s), io=79.6MiB (83.5MB), run=120023-120023msec
WRITE: bw=689KiB/s (706kB/s), 689KiB/s-689KiB/s (706kB/s-706kB/s), io=80.8MiB (84.7MB), run=120023-120023msec
Disk stats (read/write):
sda: ios=20363/20822, merge=0/55, ticks=94517/143803, in_queue=238320, util=99.98%
Transcend cf170 32gb
fio --filename=f --size=1GB --direct=1 --rw=randrw --bs=4k --runtime=120 --numjobs=8 --time_based --group_reporting --name=fio
fio: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.28
Starting 8 processes
fio: Laying out IO file (1 file / 1024MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=860KiB/s,w=880KiB/s][r=215,w=220 IOPS][eta 00m:00s]
fio: (groupid=0, jobs=8): err= 0: pid=473: Sun Jul 16 05:16:15 2023
read: IOPS=222, BW=888KiB/s (910kB/s)(104MiB/120019msec)
clat (usec): min=581, max=1332.7k, avg=13366.55, stdev=35758.89
lat (usec): min=582, max=1332.7k, avg=13367.37, stdev=35758.89
clat percentiles (usec):
| 1.00th=[ 816], 5.00th=[ 1139], 10.00th=[ 1237], 20.00th=[ 1975],
| 30.00th=[ 3195], 40.00th=[ 6783], 50.00th=[ 9503], 60.00th=[ 12518],
| 70.00th=[ 15270], 80.00th=[ 18482], 90.00th=[ 22676], 95.00th=[ 25822],
| 99.00th=[ 39060], 99.50th=[267387], 99.90th=[549454], 99.95th=[650118],
| 99.99th=[792724]
bw ( KiB/s): min= 64, max= 1920, per=100.00%, avg=934.69, stdev=51.21, samples=1817
iops : min= 16, max= 480, avg=233.67, stdev=12.80, samples=1817
write: IOPS=225, BW=901KiB/s (923kB/s)(106MiB/120019msec); 0 zone resets
clat (msec): min=2, max=1171, avg=22.31, stdev=49.93
lat (msec): min=2, max=1171, avg=22.31, stdev=49.93
clat percentiles (msec):
| 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 9], 20.00th=[ 10],
| 30.00th=[ 13], 40.00th=[ 14], 50.00th=[ 16], 60.00th=[ 19],
| 70.00th=[ 21], 80.00th=[ 24], 90.00th=[ 29], 95.00th=[ 35],
| 99.00th=[ 253], 99.50th=[ 422], 99.90th=[ 676], 99.95th=[ 802],
| 99.99th=[ 1053]
bw ( KiB/s): min= 64, max= 1576, per=100.00%, avg=936.12, stdev=47.89, samples=1840
iops : min= 16, max= 394, avg=234.03, stdev=11.97, samples=1840
lat (usec) : 750=0.18%, 1000=0.53%
lat (msec) : 2=9.56%, 4=5.49%, 10=20.07%, 20=39.67%, 50=23.17%
lat (msec) : 100=0.24%, 250=0.31%, 500=0.46%, 750=0.27%, 1000=0.05%
lat (msec) : 2000=0.01%
cpu : usr=0.10%, sys=0.33%, ctx=53784, majf=0, minf=127
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=26654,27041,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=888KiB/s (910kB/s), 888KiB/s-888KiB/s (910kB/s-910kB/s), io=104MiB (109MB), run=120019-120019msec
WRITE: bw=901KiB/s (923kB/s), 901KiB/s-901KiB/s (923kB/s-923kB/s), io=106MiB (111MB), run=120019-120019msec
Disk stats (read/write):
sda: ios=26599/27153, merge=0/51, ticks=353237/605028, in_queue=958389, util=99.91%
pcie 3.0x4 nvme on mini-pci 2.0x1 (500MB/s) daughter board
fio --filename=f --size=1GB --direct=1 --rw=randrw --bs=4k --runtime=120 --numjobs=8 --time_based --group_reporting --name=fio
fio: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.33
Starting 8 processes
fio: Laying out IO file (1 file / 1024MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=131MiB/s,w=130MiB/s][r=33.5k,w=33.3k IOPS][eta 00m:00s]
fio: (groupid=0, jobs=8): err= 0: pid=4417: Sat Aug 5 17:57:35 2023
read: IOPS=36.7k, BW=143MiB/s (150MB/s)(16.8GiB/120001msec)
clat (usec): min=51, max=8953, avg=145.89, stdev=163.07
lat (usec): min=51, max=8954, avg=146.25, stdev=163.07
clat percentiles (usec):
| 1.00th=[ 76], 5.00th=[ 78], 10.00th=[ 81], 20.00th=[ 87],
| 30.00th=[ 94], 40.00th=[ 105], 50.00th=[ 121], 60.00th=[ 137],
| 70.00th=[ 153], 80.00th=[ 184], 90.00th=[ 245], 95.00th=[ 277],
| 99.00th=[ 330], 99.50th=[ 363], 99.90th=[ 1827], 99.95th=[ 5407],
| 99.99th=[ 5866]
bw ( KiB/s): min=122464, max=159912, per=100.00%, avg=147002.08, stdev=981.40, samples=1912
iops : min=30616, max=39978, avg=36750.54, stdev=245.35, samples=1912
write: IOPS=36.7k, BW=143MiB/s (150MB/s)(16.8GiB/120001msec); 0 zone resets
clat (usec): min=40, max=2859, avg=64.38, stdev=29.75
lat (usec): min=41, max=2859, avg=64.91, stdev=29.76
clat percentiles (usec):
| 1.00th=[ 42], 5.00th=[ 43], 10.00th=[ 44], 20.00th=[ 48],
| 30.00th=[ 51], 40.00th=[ 55], 50.00th=[ 59], 60.00th=[ 63],
| 70.00th=[ 69], 80.00th=[ 76], 90.00th=[ 94], 95.00th=[ 115],
| 99.00th=[ 143], 99.50th=[ 151], 99.90th=[ 165], 99.95th=[ 172],
| 99.99th=[ 217]
bw ( KiB/s): min=121272, max=163792, per=100.00%, avg=147089.24, stdev=1053.66, samples=1912
iops : min=30318, max=40948, avg=36772.31, stdev=263.41, samples=1912
lat (usec) : 50=13.34%, 100=50.71%, 250=31.41%, 500=4.36%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.12%, 4=0.02%, 10=0.03%
cpu : usr=6.53%, sys=18.84%, ctx=8814440, majf=0, minf=84
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=4405415,4408026,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=143MiB/s (150MB/s), 143MiB/s-143MiB/s (150MB/s-150MB/s), io=16.8GiB (18.0GB), run=120001-120001msec
WRITE: bw=143MiB/s (150MB/s), 143MiB/s-143MiB/s (150MB/s-150MB/s), io=16.8GiB (18.1GB), run=120001-120001msec
Disk stats (read/write):
nvme0n1: ios=4403771/4406473, merge=0/71, ticks=543324/184508, in_queue=727840, util=99.98%
