测试环境：
CPU:Kunpeng 920 8Core
MEM:16G
Storage:200G
OS:openEuler 20.03 (LTS-SP3)
1 测试磁盘性能工具——Fio
1.1 编译安装——Fio
安装Fio环境
yum install libaio-devel -y
通过yum源直接安装
yum list *fio*
运行以上指令，回显如下即可使用yum安装。
[root@controller nmon]# yum list *fio*
Last metadata expiration check: 0:01:17 ago on Tue 03 Jan 2023 10:31:48 AM CST.
Installed Packages
fio.aarch64 3.7-7.oe1 @OS
fio-help.aarch64 3.7-7.oe1 @OS
Available Packages
fio.src 3.7-7.oe1 source
fio-debuginfo.aarch64 3.7-7.oe1 debuginfo
fio-debugsource.aarch64 3.7-7.oe1 debuginfo
源码编译安装
下载源代码：
wget https://brick.kernel.dk/snaps/fio-3.7.tar.gz
解压并编译：
tar -zxvf fio-3.7.tar.gz
cd fio-3.7
make
make install
1.2 验证与运行——Fio
执行如下命令，查看Fio版本
fio --version
返回内容如下所示，表示安装已经完成：
[root@controller ~]# fio --version
fio-3.7
运行
参数说明：
filename=/dev/vdb 测试文件名称，通常选择需要测试的盘的data目录
direct=1 测试过程绕过机器自带的buffer，使测试结果更真实
rw=read 顺序读
rw=write 顺序写
rw=readwrite 顺序混合读写
rw=randwrite 随机写
rw=randread 随机读
rw=randrw 随机混合读写
bs=16k 单次io的块文件大小为16k
bsrange=512-2048 同上，提定数据块的大小范围
size=1g 本次的测试文件大小为1g，以每次4k的io进行测试
numjobs=30 本次的测试线程为30
runtime=1000 测试时间为1000秒，如果不写则一直将1g文件分4k每次写完为止
ioengine=psync io引擎使用pync方式
group_reporting 关于显示结果的，汇总每个进程的信息
lockmem=1g 只使用1g内存进行测试
zero_buffers 用0初始化系统buffer
nrfiles=8 每个进程生成文件的数量
rwmixwrite=30 在混合读写的模式下，写占30%
1）顺序读示例
fio -filename=/dev/vdb -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=4k -size=1G -numjobs=4 -runtime=1000 -group_reporting -name=mytest
返回结果：
[root@controller ~]# fio -filename=/dev/vdb -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=4k -size=1G -numjobs=4 -runtime=1000 -group_reporting -name=mytest
mytest: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 4 threads
Jobs: 1 (f=1): [R(1),_(3)][100.0%][r=21.4MiB/s,w=0KiB/s][r=5475,w=0 IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=4): err= 0: pid=2830310: Tue Jan 3 11:12:58 2023
read: IOPS=8562, BW=33.4MiB/s (35.1MB/s)(4096MiB/122455msec)
clat (usec): min=200, max=80336, avg=463.72, stdev=513.63
lat (usec): min=200, max=80336, avg=463.84, stdev=513.64
clat percentiles (usec):
| 1.00th=[ 265], 5.00th=[ 285], 10.00th=[ 302], 20.00th=[ 318],
| 30.00th=[ 334], 40.00th=[ 351], 50.00th=[ 371], 60.00th=[ 392],
| 70.00th=[ 420], 80.00th=[ 465], 90.00th=[ 586], 95.00th=[ 848],
| 99.00th=[ 2343], 99.50th=[ 3032], 99.90th=[ 6456], 99.95th=[ 8717],
| 99.99th=[14091]
bw ( KiB/s): min= 4216, max=10944, per=25.12%, avg=8605.29, stdev=1153.54, samples=973
iops : min= 1054, max= 2736, avg=2151.32, stdev=288.39, samples=973
lat (usec) : 250=0.29%, 500=84.39%, 750=9.11%, 1000=2.27%
lat (msec) : 2=2.52%, 4=1.14%, 10=0.25%, 20=0.03%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=0.36%, sys=0.72%, ctx=1048596, majf=0, minf=4
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1048576,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=33.4MiB/s (35.1MB/s), 33.4MiB/s-33.4MiB/s (35.1MB/s-35.1MB/s), io=4096MiB (4295MB), run=122455-122455msec
Disk stats (read/write):
vdb: ios=1034729/436, merge=10932/4, ticks=473564/120, in_queue=15252, util=99.89%
2）随机混合读写示例
fio -filename=/dev/vdb -direct=1 -iodepth 1 -thread -rw=randrw -ioengine=psync -bs=4k -size=1G -numjobs=30 -runtime=1000 -group_reporting -name=mytest
返回结果：
[root@controller ~]# fio -filename=/dev/vdb -direct=1 -iodepth 1 -thread -rw=randrw -ioengine=psync -bs=4k -size=1G -numjobs=30 -runtime=1000 -group_reporting -name=mytest
mytest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 30 threads
Jobs: 6 (f=6): [m(2),_(1),m(1),_(2),m(1),_(4),E(1),_(1),m(1),_(6),m(1),_(9)][99.8%][r=16.7MiB/s,w=17.1MiB/s][r=4282,w=4372 IOPS][eta 00m:01s]
mytest: (groupid=0, jobs=30): err= 0: pid=2834861: Tue Jan 3 11:32:59 2023
read: IOPS=7914, BW=30.9MiB/s (32.4MB/s)(14.0GiB/496774msec)
clat (usec): min=240, max=215504, avg=1785.65, stdev=1555.16
lat (usec): min=241, max=215504, avg=1785.77, stdev=1555.17
clat percentiles (usec):
| 1.00th=[ 433], 5.00th=[ 807], 10.00th=[ 906], 20.00th=[ 1020],
| 30.00th=[ 1106], 40.00th=[ 1221], 50.00th=[ 1336], 60.00th=[ 1483],
| 70.00th=[ 1729], 80.00th=[ 2147], 90.00th=[ 3097], 95.00th=[ 4228],
| 99.00th=[ 8225], 99.50th=[10814], 99.90th=[16712], 99.95th=[18744],
| 99.99th=[24511]
bw ( KiB/s): min= 360, max= 1856, per=3.34%, avg=1057.75, stdev=204.65, samples=29715
iops : min= 90, max= 464, avg=264.42, stdev=51.17, samples=29715
write: IOPS=7916, BW=30.9MiB/s (32.4MB/s)(15.0GiB/496774msec)
clat (usec): min=546, max=215138, avg=1991.94, stdev=1806.19
lat (usec): min=546, max=215138, avg=1992.15, stdev=1806.19
clat percentiles (usec):
| 1.00th=[ 775], 5.00th=[ 881], 10.00th=[ 947], 20.00th=[ 1057],
| 30.00th=[ 1156], 40.00th=[ 1270], 50.00th=[ 1418], 60.00th=[ 1614],
| 70.00th=[ 1926], 80.00th=[ 2507], 90.00th=[ 3687], 95.00th=[ 5014],
| 99.00th=[ 9110], 99.50th=[11207], 99.90th=[16057], 99.95th=[18482],
| 99.99th=[25560]
bw ( KiB/s): min= 352, max= 1736, per=3.34%, avg=1057.96, stdev=204.19, samples=29715
iops : min= 88, max= 434, avg=264.48, stdev=51.05, samples=29715
lat (usec) : 250=0.01%, 500=0.89%, 750=1.26%, 1000=14.38%
lat (msec) : 2=57.89%, 4=18.56%, 10=6.35%, 20=0.65%, 50=0.03%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=0.09%, sys=0.25%, ctx=7864440, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3931786,3932534,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=30.9MiB/s (32.4MB/s), 30.9MiB/s-30.9MiB/s (32.4MB/s-32.4MB/s), io=14.0GiB (16.1GB), run=496774-496774msec
WRITE: bw=30.9MiB/s (32.4MB/s), 30.9MiB/s-30.9MiB/s (32.4MB/s-32.4MB/s), io=15.0GiB (16.1GB), run=496774-496774msec
Disk stats (read/write):
vdb: ios=3931755/3934156, merge=4/20, ticks=6985720/7797261, in_queue=2774632, util=100.00%
结果说明：
bw：磁盘的吞吐量，顺序读写考察的重点，类似于下载速度。
iops：磁盘的每秒读写次数，随机读写考察的重点。
util：磁盘利用率。
1.3 其他——Fio
Fio文档
https://fio.readthedocs.io/en/latest/fio_doc.html
Fio源码仓库链接
https://brick.kernel.dk/snaps/
详细参数
fio [options] [job options]
--debug=options Enable debug logging. May be one/more of:
process,file,io,mem,blktrace,verify,random,parse,
diskutil,job,mutex,profile,time,net,rate,compress,
steadystate,helperthread
--parse-only Parse options only, don't start any IO
--output Write output to file
--bandwidth-log Generate aggregate bandwidth logs
--minimal Minimal (terse) output
--output-format=type Output format (terse,json,json+,normal)
--terse-version=type Set terse version output format (default 3, or 2 or 4)
--version Print version info and exit
--help Print this page
--cpuclock-test Perform test/validation of CPU clock
--crctest=[type] Test speed of checksum functions
--cmdhelp=cmd Print command help, "all" for all of them
--enghelp=engine Print ioengine help, or list available ioengines
--enghelp=engine,cmd Print help for an ioengine cmd
--showcmd Turn a job file into command line options
--eta=when When ETA estimate should be printed
May be "always", "never" or "auto"
--eta-newline=time Force a new line for every 'time' period passed
--status-interval=t Force full status dump every 't' period passed
--readonly Turn on safety read-only checks, preventing writes
--section=name Only run specified section in job file, multiple sections can be specified
--alloc-size=kb Set smalloc pool to this size in kb (def 16384)
--warnings-fatal Fio parser warnings are fatal
--max-jobs=nr Maximum number of threads/processes to support
--server=args Start a backend fio server
--daemonize=pidfile Background fio server, write pid to file
--client=hostname Talk to remote backend(s) fio server at hostname
--remote-config=file Tell fio server to load this local job file
--idle-prof=option Report cpu idleness on a system or percpu basis
(option=system,percpu) or run unit work
calibration only (option=calibrate)
--inflate-log=log Inflate and output compressed log
--trigger-file=file Execute trigger cmd when file exists
--trigger-timeout=t Execute trigger at this time
--trigger=cmd Set this command as local trigger
--trigger-remote=cmd Set this command as remote trigger
--aux-path=path Use this path for fio state generated files