Author: Glori

在准备正式启用Proxmox之前，需要对ZFS, Raid-1，备份等操作作验证操作时，可以使用Virtualbox工具熟悉与测试。

为了简化设置， 所有的网卡都可以在VirtbualBox设置为”Host Only Adapter”。以下是Virtualbox的网络连通性表（源自Virtualbox Manual）：

如果是单PVE+单PBS(Proxmox Backup Server)的测试场景，VM1<->VM2在Host-only模式下完全够用。

Pre-conditions:

/dev/sda -- existing good disk
/dev/sdb -- exiting bad disk to be replaced
/dev/sdc -- new disk

Reference

https://hosting-tutorials.co.uk/tutorials/linux/replacing-a-zfs-root-disk-with-proxmox
https://pve.proxmox.com/wiki/Host_Bootloader#sysboot_proxmox_boot_setup
https://pve.proxmox.com/wiki/ZFS_on_Linux#_installation_as_root_file_system

Steps

Replace disk on root zfs raid-1

zpool offline rpool /dev/disk/by-id/ata-VBOX_HARDDISK_VB8b4872d1-c28eed93 # Offline the bad disk
fdisk -l # Check new disk is presenting
sgdisk /dev/sda -R /dev/sdc # Copy partition table from sda to sdc
sgdisk -G /dev/sdc	# Same - Randomize the disk guids  #>sgdisk --randomize-guids /dev/sdc	# Randomize the disk guids
#grub-install /dev/sdc --NOT work on new Proxmox version, check following new command(not verified)
#check partitions of sdc first -- there're 3 partitions now
proxmox-boot-tool format /dev/sdc2
proxmox-boot-tool init /dev/sdc2 # to be verified
#proxmox-boot-tool init /dev/sdc2 grub # may be work on old version of proxmox
proxmox-boot-tool refresh # add/update new UUID to /etc/kernel/proxmox-boot-uuids
proxmox-boot-tool clean # clean unused UUID in /etc/kernel/proxmox-boot-uuids
reboot	#make sure system can boot normally
# Resize disk(parted) at this point
zpool replace -f rpool /dev/disk/by-id/ata-VBOX_HARDDISK_VB8b4872d1-c28eed93 /dev/disk/by-id/ata-VBOX_HARDDISK_VBdb5f453c-7a4145b7-part3 # replace bad disk with new disk's partition 3, DO NOT rebook before resilvering was done.

# Reference:
# Resize disk on root zfs raid-1
# 1st Try
zpool offline rpool ata-VBOX_HARDDISK_VB8b4872d1-c28eed93
fdisk /dev/sdc
	d
	3		#delete /dev/sdc3 partition
	n
	3
	(Enter)
	(Enter)	#create new partition /dev/sdc3 with all rest size
	t
	157		#change sdc3 partition type to "Solaris /usr & Apple ZFS"
	p
	w
zpool online rpool ata-VBOX_HARDDISK_VB8b4872d1-c28eed93
#wait until reslizer done and do the same on another raid-1 disk.
# 2nd Try
# Maybe this is the correct way to resize the disk:
# https://blog.doussan.info/posts/how-to-expand-a-zfs-pool-partitions/
parted /dev/sda
	print
	resizepart 3 100%
	print
	q

Notes:
1. Feel like bootable ZFS raid-1 (zpool) should contains 3 partitions: sdc1 sdc2 is used for boot, sdc3 is actually data partition.
2. Additional commands to perform for partition table copy and bootable partition setup
3. There's no easy way to replace the bootable raid-1/mirror or on GUI -- maybe I can try another open source solution(OMV) to see if that's better?

Note or Summary

1. Feel like bootable ZFS raid-1 (zpool) should contain 3 partitions: such as sdc1 sdc2 is used for boot, sdc3 is actually data partition.
2. Additional commands to perform for partition table copy and bootable partition setup
3. There’s no easy way to replace the bootable raid-1/mirror or on GUI — maybe I can try another open source solution(OMV) to see if that’s better?