RAID 10 XFS 的 FIO 随机 128 MB 读取值非常慢

Question

我刚刚安装了两个 Samsung 970 EVO Plus 驱动器，并将最大分区配置为 RAID\xc2\xa010（远 2 布局）。问题是FIO仅报告读取速度为 72\xc2\xa0MB/s。我使用以下命令来构建数组；

\n

mdadm --create --verbose --level=10 --metadata=1.2 --chunk=512 --raid-devices=2 --layout=f2 /dev/md/MyRAID10Array /dev/nvme0n1p3 /dev/nvme1n1p3\n

\n

这应该使我接近引用的读取速度 3500\xc2\xa0MB/s 的两倍。我已使用XFS的默认选项对其进行格式化，

\n

mkfs.xfs /dev/md127\n

\n

磁盘布局如下

\n

fdisk -l /dev/nvme0n1\n

\n

输出：

\n

Disk /dev/nvme0n1: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors\nDisk model: Samsung SSD 970 EVO Plus 1TB\nUnits: sectors of 1 * 512 = 512 bytes\nSector size (logical/physical): 512 bytes / 512 bytes\nI/O size (minimum/optimal): 512 bytes / 512 bytes\nDisklabel type: gpt\nDisk identifier: FF148851-E4A9-41D2-96C6-418280164BC6\n\nDevice            Start        End    Sectors   Size Type\n/dev/nvme0n1p1     2048     821247     819200   400M Linux RAID\n/dev/nvme0n1p2   821248   67930111   67108864    32G Linux swap\n/dev/nvme0n1p3 67930112 1953523711 1885593600 899.1G Linux RAID\n

\n

MDSTAT报告

\n

md127 ：活动 raid10 nvme1n1p3[0] nvme0n1p3 3 \n942664704 块超级 1.2 512K 块 2 远副本 [2/2] [UU]\n位图：0/8 页 [0KB]，65536KB 块

\n

但是当我通过 FIO 运行这个时，

\n

; Random read of 128 MB of data\n\n[random-read]\nrw=randread\nsize=128m\ndirectory=/backup_mount_point\n

\n

我只得到非常糟糕的数字；

\n

random-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1\nfio-3.35\nStarting 1 process\nJobs: 1 (f=1)\nrandom-read: (groupid=0, jobs=1): err= 0: pid=214183: Sun Oct  1 06:28:06 2023\n  read: IOPS=18.2k, BW=71.3MiB/s (74.7MB/s)(128MiB/1796msec)\n    clat (usec): min=14, max=6880, avg=54.09, stdev=53.44\n     lat (usec): min=14, max=6880, avg=54.12, stdev=53.45\n    clat percentiles (usec):\n     |  1.00th=[   51],  5.00th=[   51], 10.00th=[   51], 20.00th=[   52],\n     | 30.00th=[   52], 40.00th=[   53], 50.00th=[   53], 60.00th=[   53],\n     | 70.00th=[   54], 80.00th=[   55], 90.00th=[   57], 95.00th=[   58],\n     | 99.00th=[   75], 99.50th=[   76], 99.90th=[   78], 99.95th=[   80],\n     | 99.99th=[  123]\n   bw (  KiB/s): min=70720, max=74920, per=100.00%, avg=73509.33, stdev=2415.69, samples=3\n   iops        : min=17680, max=18730, avg=18377.33, stdev=603.92, samples=3\n  lat (usec)   : 20=0.03%, 50=0.02%, 100=99.93%, 250=0.02%\n  lat (msec)   : 10=0.01%\n  cpu          : usr=0.39%, sys=9.25%, ctx=32778, majf=0, minf=9\n  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%\n     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%\n     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%\n     issued rwts: total=32768,0,0,0 short=0,0,0,0 dropped=0,0,0,0\n     latency   : target=0, window=0, percentile=100.00%, depth=1\n\nRun status group 0 (all jobs):\n   READ: bw=71.3MiB/s (74.7MB/s), 71.3MiB/s-71.3MiB/s (74.7MB/s-74.7MB/s), io=128MiB (134MB), run=1796-1796msec\n\nDisk stats (read/write):\n    md127: ios=30225/0, merge=0/0, ticks=1407/0, in_queue=1407, util=93.46%, aggrios=16384/0, aggrmerge=0/0, aggrticks=784/0, aggrin_queue=784, aggrutil=93.19%\n  nvme0n1: ios=16384/0, merge=0/0, ticks=768/0, in_queue=768, util=93.04%\n  nvme1n1: ios=16384/0, merge=0/0, ticks=800/0, in_queue=800, util=93.19%\n

\n

如果我转向顺序（rw=read），它会提高到 1803\xc2\xa0MB/s，但仍然不到我预期的三分之一。我在具有 64\xc2\xa0GB RAM 的 AMD Ryzen 系统上运行Arch Linux 。主板是MSI的 X570S Edge Wifi Max。

\n

编辑; 在 Batistuta9 的回答之后，我改变了我的测试并得到了明显更快的结果。这是 FIO 摘要，以防您感兴趣；

\n

random-read: (groupid=0, jobs=1): err= 0: pid=143332: Thu Oct  5 23:23:49 2023\n  read: IOPS=15.0k, BW=58.7MiB/s (61.5MB/s)(20.0GiB/348908msec)\n    clat (usec): min=48, max=2746, avg=66.21, stdev=16.90\n     lat (usec): min=48, max=2746, avg=66.24, stdev=16.90\n    clat percentiles (usec):\n     |  1.00th=[   49],  5.00th=[   50], 10.00th=[   51], 20.00th=[   51],\n     | 30.00th=[   52], 40.00th=[   54], 50.00th=[   69], 60.00th=[   72],\n     | 70.00th=[   74], 80.00th=[   81], 90.00th=[   88], 95.00th=[   91],\n     | 99.00th=[  120], 99.50th=[  130], 99.90th=[  149], 99.95th=[  157],\n     | 99.99th=[  184]\n   bw (  KiB/s): min=57064, max=61112, per=14.93%, avg=60131.86, stdev=374.32, samples=697\n   iops        : min=14266, max=15278, avg=15032.97, stdev=93.58, samples=697\n  lat (usec)   : 50=6.97%, 100=90.22%, 250=2.81%, 500=0.01%, 750=0.01%\n  lat (usec)   : 1000=0.01%\n  lat (msec)   : 2=0.01%, 4=0.01%\n  cpu          : usr=0.74%, sys=5.61%, ctx=5246560, majf=0, minf=10\n  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%\n     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%\n     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%\n     issued rwts: total=5242880,0,0,0 short=0,0,0,0 dropped=0,0,0,0\n     latency   : target=0, window=0, percentile=100.00%, depth=64\nrandom-read: (groupid=0, jobs=1): err= 0: pid=143333: Thu Oct  5 23:23:49 2023\n  read: IOPS=15.2k, BW=59.3MiB/s (62.2MB/s)(20.0GiB/345469msec)\n    clat (usec): min=48, max=2828, avg=65.55, stdev=16.90\n     lat (usec): min=48, max=2828, avg=65.58, stdev=16.90\n    clat percentiles (usec):\n     |  1.00th=[   49],  5.00th=[   50], 10.00th=[   51], 20.00th=[   51],\n     | 30.00th=[   52], 40.00th=[   53], 50.00th=[   60], 60.00th=[   71],\n     | 70.00th=[   73], 80.00th=[   79], 90.00th=[   88], 95.00th=[   90],\n     | 99.00th=[  120], 99.50th=[  130], 99.90th=[  149], 99.95th=[  157],\n     | 99.99th=[  184]\n   bw (  KiB/s): min=59472, max=61528, per=15.08%, avg=60729.74, stdev=344.11, samples=690\n   iops        : min=14868, max=15382, avg=15182.43, stdev=86.02, samples=690\n  lat (usec)   : 50=6.62%, 100=90.63%, 250=2.75%, 500=0.01%, 750=0.01%\n  lat (usec)   : 1000=0.01%\n  lat (msec)   : 2=0.01%, 4=0.01%\n  cpu          : usr=0.76%, sys=5.63%, ctx=5246568, majf=0, minf=9\n  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%\n     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%\n     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%\n     issued rwts: total=5242880,0,0,0 short=0,0,0,0 dropped=0,0,0,0\n     latency   : target=0, window=0, percentile=100.00%, depth=64\nrandom-read: (groupid=0, jobs=1): err= 0: pid=143334: Thu Oct  5 23:23:49 2023\n  read: IOPS=14.3k, BW=55.8MiB/s (58.5MB/s)(20.0GiB/366867msec)\n    clat (usec): min=48, max=2742, avg=69.63, stdev=16.68\n     lat (usec): min=48, max=2742, avg=69.66, stdev=16.69\n    clat percentiles (usec):\n     |  1.00th=[   50],  5.00th=[   50], 10.00th=[   51], 20.00th=[   52],\n     | 30.00th=[   54], 40.00th=[   70], 50.00th=[   72], 60.00th=[   73],\n     | 70.00th=[   75], 80.00th=[   86], 90.00th=[   88], 95.00th=[   92],\n     | 99.00th=[  123], 99.50th=[  133], 99.90th=[  151], 99.95th=[  159],\n     | 99.99th=[  184]\n   bw (  KiB/s): min=55232, max=58272, per=14.20%, avg=57191.21, stdev=322.36, samples=733\n   iops        : min=13808, max=14568, avg=14297.80, stdev=80.58, samples=733\n  lat (usec)   : 50=3.98%, 100=92.88%, 250=3.14%, 500=0.01%, 750=0.01%\n  lat (usec)   : 1000=0.01%\n  lat (msec)   : 2=0.01%, 4=0.01%\n  cpu          : usr=0.74%, sys=5.33%, ctx=5246807, majf=0, minf=9\n  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%\n     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%\n     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%\n     issued rwts: total=5242880,0,0,0 short=0,0,0,0 dropped=0,0,0,0\n     latency   : target=0, window=0, percentile=100.00%, depth=64\nrandom-read: (groupid=0, jobs=1): err= 0: pid=143335: Thu Oct  5 23:23:49 2023\n  read: IOPS=12.6k, BW=49.2MiB/s (51.6MB/s)(20.0GiB/416494msec)\n    clat (usec): min=56, max=2742, avg=79.09, stdev=12.05\n     lat (usec): min=56, max=2742, avg=79.12, stdev=12.05\n    clat percentiles (usec):\n     |  1.00th=[   64],  5.00th=[   70], 10.00th=[   71], 20.00th=[   72],\n     | 30.00th=[   73], 40.00th=[   73], 50.00th=[   74], 60.00th=[   78],\n     | 70.00th=[   86], 80.00th=[   88], 90.00th=[   90], 95.00th=[   95],\n     | 99.00th=[  128], 99.50th=[  137], 99.90th=[  155], 99.95th=[  163],\n     | 99.99th=[  190]\n   bw (  KiB/s): min=48912, max=51584, per=12.51%, avg=50377.18, stdev=326.60, samples=832\n   iops        : min=12228, max=12896, avg=12594.28, stdev=81.63, samples=832\n  lat (usec)   : 100=96.05%, 250=3.95%, 500=0.01%, 750=0.01%, 1000=0.01%\n  lat (msec)   : 2=0.01%, 4=0.01%\n  cpu          : usr=0.64%, sys=4.86%, ctx=5247122, majf=0, minf=10\n  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%\n     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%\n     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%\n     issued rwts: total=5242880,0,0,0 short=0,0,0,0 dropped=0,0,0,0\n     latency   : target=0, window=0, percentile=100.00%, depth=64\nrandom-read: (groupid=0, jobs=1): err= 0: pid=143336: Thu Oct  5 23:23:49 2023\n  read: IOPS=12.6k, BW=49.2MiB/s (51.6MB/s)(20.0GiB/416501msec)\n    clat (usec): min=57, max=2889, avg=79.08, stdev=12.27\n     lat (usec): min=57, max=2889, avg=79.12, stdev=12.27\n    clat percentiles (usec):\n     |  1.00th=[   64],  5.00th=[   70], 10.00th=[   71], 20.00th=[   72],\n     | 30.00th=[   73], 40.00th=[   73], 50.00th=[   74], 60.00th=[   78],\n     | 70.00th=[   86], 80.00th=[   88], 90.00th=[   90], 95.00th=[   95],\n     | 99.00th=[  128], 99.50th=[  137], 99.90th=[  155], 99.95th=[  163],\n     | 99.99th=[  188]\n   bw (  KiB/s): min=48640, max=51608, per=12.51%, avg=50376.18, stdev=314.71, samples=832\n   iops        : min=12160, max=12902, avg=12594.03, stdev=78.66, samples=832\n  lat (usec)   : 100=96.08%, 250=3.92%, 500=0.01%, 750=0.01%, 1000=0.01%\n  lat (msec)   : 2=0.01%, 4=0.01%\n  cpu          : usr=0.68%, sys=4.83%, ctx=5247298, majf=0, minf=10\n  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%\n     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%\n     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%\n     issued rwts: total=5242880,0,0,0 short=0,0,0,0 dropped=0,0,0,0\n     latency   : target=0, window=0, percentile=100.00%, depth=64\nrandom-read: (groupid=0, jobs=1): err= 0: pid=143337: Thu Oct  5 23:23:49 2023\n  read: IOPS=12.6k, BW=49.3MiB/s (51.7MB/s)(20.0GiB/415249msec)\n    clat (usec): min=14, max=3163, avg=78.85, stdev=12.35\n     lat (usec): min=14, max=3163, avg=78.88, stdev=12.35\n    clat percentiles (usec):\n     |  1.00th=[   60],  5.00th=[   70], 10.00th=[   71], 20.00th=[   72],\n     | 30.00th=[   73], 40.00th=[   73], 50.00th=[   74], 60.00th=[   77],\n     | 70.00th=[   86], 80.00th=[   88], 90.00th=[   90], 95.00th=[   95],\n     | 99.00th=[  128], 99.50th=[  137], 99.90th=[  153], 99.95th=[  161],\n     | 99.99th=[  190]\n   bw (  KiB/s): min=49104, max=51799, per=12.54%, avg=50526.68, stdev=317.81, samples=830\n   iops        : min=12276, max=12949, avg=12631.66, stdev=79.43, samples=830\n  lat (usec)   : 20=0.01%, 50=0.04%, 100=96.03%, 250=3.93%, 500=0.01%\n  lat (usec)   : 750=0.01%, 1000=0.01%\n  lat (msec)   : 2=0.01%, 4=0.01%\n  cpu          : usr=0.61%, sys=4.90%, ctx=5246732, majf=0, minf=10\n  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%\n     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%\n     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%\n     issued rwts: total=5242880,0,0,0 short=0,0,0,0 dropped=0,0,0,0\n     latency   : target=0, window=0, percentile=100.00%, depth=64\nrandom-read: (groupid=0, jobs=1): err= 0: pid=143338: Thu Oct  5 23:23:49 2023\n  read: IOPS=12.7k, BW=49.5MiB/s (51.9MB/s)(20.0GiB/413718msec)\n    clat (usec): min=48, max=2834, avg=78.55, stdev=12.65\n     lat (usec): min=48, max=2834, avg=78.59, stdev=12.65\n    clat percentiles (usec):\n     |  1.00th=[   52],  5.00th=[   70], 10.00th=[   71], 20.00th=[   72],\n     | 30.00th=[   72], 40.00th=[   73], 50.00th=[   74], 60.00th=[   77],\n     | 70.00th=[   86], 80.00th=[   88], 90.00th=[   89], 95.00th=[   95],\n     | 99.00th=[  127], 99.50th=[  137], 99.90th=[  153], 99.95th=[  161],\n     | 99.99th=[  190]\n   bw (  KiB/s): min=49776, max=51912, per=12.59%, avg=50713.30, stdev=309.76, samples=827\n   iops        : min=12444, max=12978, avg=12678.31, stdev=77.41, samples=827\n  lat (usec)   : 50=0.20%, 100=95.92%, 250=3.87%, 500=0.01%, 750=0.01%\n  lat (usec)   : 1000=0.01%\n  lat (msec)   : 2=0.01%, 4=0.01%\n  cpu          : usr=0.65%, sys=4.89%, ctx=5246845, majf=0, minf=12\n  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%\n     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%\n     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%\n     issued rwts: total=5242880,0,0,0 short=0,0,0,0 dropped=0,0,0,0\n     latency   : target=0, window=0, percentile=100.00%, depth=64\nrandom-read: (groupid=0, jobs=1): err= 0: pid=143339: Thu Oct  5 23:23:49 2023\n  read: IOPS=13.3k, BW=52.0MiB/s (54.5MB/s)(20.0GiB/394165msec)\n    clat (usec): min=47, max=2904, avg=74.83, stdev=15.26\n     lat (usec): min=47, max=2904, avg=74.86, stdev=15.26\n    clat percentiles (usec):\n     |  1.00th=[   50],  5.00th=[   51], 10.00th=[   53], 20.00th=[   70],\n     | 30.00th=[   71], 40.00th=[   72], 50.00th=[   73], 60.00th=[   75],\n     | 70.00th=[   85], 80.00th=[   87], 90.00th=[   89], 95.00th=[   94],\n     | 99.00th=[  126], 99.50th=[  135], 99.90th=[  153], 99.95th=[  161],\n     | 99.99th=[  188]\n   bw (  KiB/s): min=52096, max=54312, per=13.21%, avg=53229.10, stdev=312.71, samples=788\n   iops        : min=13024, max=13578, avg=13307.26, stdev=78.16, samples=788\n  lat (usec)   : 50=1.65%, 100=94.73%, 250=3.62%, 500=0.01%, 750=0.01%\n  lat (usec)   : 1000=0.01%\n  lat (msec)   : 2=0.01%, 4=0.01%\n  cpu          : usr=0.67%, sys=5.07%, ctx=5246945, majf=0, minf=9\n  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%\n     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%\n     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%\n     issued rwts: total=5242880,0,0,0 short=0,0,0,0 dropped=0,0,0,0\n     latency   : target=0, window=0, percentile=100.00%, depth=64\n\nRun status group 0 (all jobs):\n   READ: bw=393MiB/s (412MB/s), 49.2MiB/s-59.3MiB/s (51.6MB/s-62.2MB/s), io=160GiB (172GB), run=345469-416501msec\n\nDisk stats (read/write):\n    md127: ios=41940854/14, merge=0/0, ticks=2850618/100, in_queue=2850718, util=100.00%, aggrios=20971520/47, aggrmerge=0/2, aggrticks=1428779/139, aggrin_queue=1428932, aggrutil=100.00%\n  nvme0n1: ios=21179809/47, merge=0/2, ticks=1427626/139, in_queue=1427779, util=100.00%\n  nvme1n1: ios=20763231/47, merge=0/2, ticks=1429932/139, in_queue=1430086, util=100.00%\n

\n

Answer 1

您获得了队列深度1的单线程4k 块大小性能基准测试的预期结果。在此处检查驱动器\xe2\x80\x99s 规格 - https://semiconductor.samsung.com/consumer-storage/internal-ssd/970evoplus/。对于此类模式，结果应高达 19000 IOPS（您获得了 18200 IOPS）。\n在 fio 基准文件（随机读取）中添加以下参数并重新运行测试：

\n

numjobs=8\niodepth=64\ndirect=1\nioengine=libaio\n

\n

I\xe2\x80\x99d 还建议将基准测试文件的大小从 128MB 增加到至少 20GB。

\n