磁盘性能优化-尝试 PCI-e Passthrough RAID 控制器

背景

https://c.cra.moe/t/topic/24/3

在更改磁盘控制器改为改为 VirtIO SCSI single 后,部分软件同步时仍会出现[generator] io timeout after 120 seconds -- exiting的报错。同时我们还发现通过pve虚拟的磁盘的 Sector size (主要是Physical block size和Host的不一样,Host的Block size是4096bytes,虚拟化之后的磁盘变成了512bytes,PVE的qemu调整这个值之后重新挂载磁盘并不起效)。

尝试修复

根据PVE的文档设置PCI Passthrough

https://pve.proxmox.com/wiki/PCI(e)_Passthrough

需要注意的是,当虚拟机的内存较大时(按照文章的说法是96G,镜像站虚拟机的内存为350G),需要把对应虚拟机的qemu配置里加上一行 hugepages: 2 (参考下方链接)

https://wiki.debian.org/Hugepages

配置完之后的qemu configuration如下:

agent: 1
hugepages: 2
bios: ovmf
bootdisk: scsi0
cores: 36
cpu: host,flags=+pcid;+spec-ctrl;+aes
efidisk0: local-lvm:vm-100-disk-1,size=4M
hostpci0: 58:00.0,pcie=1
machine: q35
memory: 363520
name: mirrors-debian
net0: virtio=XX:XX:XX:XX:XX:XX,bridge=vmbr0,firewall=1
numa: 1
ostype: l26
scsi0: local-lvm:vm-100-disk-0,cache=writethrough,size=XG
scsihw: virtio-scsi-single
smbios1: uuid=X
sockets: 2
vmgenid: X

重启后即可在虚拟机内看到RAID控制器,物理磁盘和LVM,之后挂载即可。

物理磁盘:

Disk /dev/sdb: 72.8 TiB, 80009997844480 bytes, 156269527040 sectors
Disk model: RAID 930-8e 4GB 
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 262144 bytes / 1048576 bytes


Disk /dev/sdc: 72.8 TiB, 80009997844480 bytes, 156269527040 sectors
Disk model: RAID 930-8e 4GB 
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 262144 bytes / 1048576 bytes

LVM:

Disk /dev/mapper/data-mirrors: 40 TiB, 43980465111040 bytes, 85899345920 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 262144 bytes / 1048576 bytes


Disk /dev/mapper/data-backup: 105.6 TiB, 116039522189312 bytes, 226639691776 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 262144 bytes / 1048576 bytes

性能改善

如上图,目前磁盘的持续读/写速度约为100M/s。

1 个赞