The main Proxmox Backup Server (PBS) installed in our office was installed with default settings and with a single mirrored pool. It was supposed to be temporary, but had been used longer than we had planned for. You know how that goes, right?
It was long overdue for us to replace the installation with one that has separate OS and data pools. After perusing the internet, we found nobody explaining how this could be done, and the suggestions that we saw were to simply reinstall and re-run a backup job to the new drive. Since we wanted to retain the configuration, the backup history and the connection to the remote PBS server that syncs with this one, we didn’t want to accept this answer. It seemed it would be more work to reconfigure all of these connections than to figure out how to migrate the configuration and data. We didn’t take notes during the process, but I wanted to document what we did as soon as it was finished to have the best chance of remembering it accurately. This is that “brain dump” and we hope it helps someone in the future.
NOTE: This process worked for us, but it might not work in your situation. What’s worse, we may have forgotten an important step that when omitted, causes you catastrophic data loss. With that said, if you choose to follow along, you are choosing to take an educated and calculated risk. Proceed at your own peril.
First, check ‘zpool status’ and ‘zfs status’ to be sure all is good. We only had “rpool” as a pool and the dataset named “backup1” was part of the same pool which also housed the OS, config, etc. We needed to replace the drives, as one had failed, the mirror was degraded and I didn’t trust the second drive.
This is what it might look like:
root@pbs:~# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
rpool 1.82T 1001G 848G - - 11% 48% 1.00x ONLINE -
root@pbs:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 743G 573G 140K /rpool
rpool/ROOT 3.22G 328G 140K /rpool/ROOT
rpool/ROOT/pbs-1 3.22G 328G 3.22G /
rpool/backup1 740G 328G 730G /rpool/backup1Take note of what the pools are named and be sure you are confident in your accuracy.
To begin the migration, we plugged in new drives, then use the GUI under Administration –> Storage/Disks –> ZFS to create a new mirror. I checked off the option to add a new datastore and called it “pbspool1”.
Then, I removed the datastore from the GUI, (silly, I know) and went to the commandline. We likely could have proceeded without the checkmark and saved a step, but this is what was done.
zfs create pbspool1/backup1Once we had this, I went back to the GUI and added the datastore back, manually typing in the path: /mnt/datastore/pbspool1/backup1, and named it “newbackup1.”
The first copy was done with the system online. It won’t create a clean, consistent backup since files are changing while the sync is happening, but the majority of the data will be copied without down-time and we can catch it up later. We ran the following:
rsync -av --progress --delete /rpool/backup1/ /mnt/datastore/pbspool1/backup1/(BE CAREFUL WITH –delete !!! Use -n for a dry-run to test and be sure that you had things correct! Reversing the directory paths can lead to losing all data!)
When the first sync was complete, we checked the GUI and viewed the “Content” of “newbackup1” and it listed all of the backups that I had on “backup1.”
NOTE: We may have needed to set permissions here, but I don’t recall the exact order. If the backups dont’ show up properly, you can look for the “chown” command below. Run it, restart the Proxmox services and check again.
I then stopped the services on PBS:
service proxmox-backup stop
service proxmox-backup-proxy stopLeaving the services down, we ran rsync again, to completion to catch up the new dataset with any changes that occurred during and after the initial sync.
rsync -av --progress --delete /rpool/backup1/ /mnt/datastore/pbspool1/backup1/Next, we had to set permissions:
chown -R backup:backup /mnt/datastore/pbspool1/backup1Once that was complete, we edited the datastore config in /etc/proxmox-backup/datastore.cfg while the proxmox serves were stopped. The datastores were renamed so “backup1” became “oldbackup1” and “newbackup1” became “backup1.”
NOTE: Re-using the exact same name allows all jobs and remote connections to work on the new dataset without reconfiguration of the backup and sync jobs!
datastore: oldbackup1
comment
gc-schedule daily
notify gc=error,sync=error,verify=error
path /rpool/backup1
datastore: backup1
comment
gc-schedule daily
notify gc=error,sync=error,verify=error
path /mnt/datastore/pbspool1/backup1
We then started the Proxmox services:
service proxmox-backup start
service proxmox-backup-proxy startGoing back to the GUI to verify, we saw all the backups under “backup1” which showed its path as “/mnt/datastore/pbspool1/backup1/” so I was happy that it was now using the backup data from the new mirror.
At this point, we removed “oldbackup1” with the GUI under “Options.”
We chose to stop and let it run overnight for some extra assurance. We checked in the morning to be sure the jobs worked, new backups showed up, etc. We also checked to be sure we could access the backups from PVE and that the remote PBS server could still sync with this one. If there had been an issue, we’d know what part of the process to investigate. If we had completed everything below and then found a problem, we could be in a situation to wonder if it was the OS, configuration or permissions and it may have been harder to nail down the root cause and appropriate fix.
All worked to my satisfaction overnight and the next day, we replaced the OS drives as follows:
Since the ZFS pool we had was much larger than the 2 new disks we wanted to place the OS on, we couldn’t use ZFS commands to replace the drives in a mirror and continue on without re-installation. So, I researched how to transfer the config to a new box, new OS drive, etc. The search results were not quite what we had hoped for, but I had enough information to allow me to proceed and “fill in the blanks” with some educated guesses. Spoiler: It worked.
Since we were planning to install a fresh copy of PBS onto new drives and use the same configuration, we wanted to be sure that the config we copied over was compatible with the version we were going to be using.
To that end, we upgraded PBS to the latest version, installed all updates and rebooted. Then, as a precaution to avoid being locked out, we disabled 2FA on the main admin account. I saw some warning on this, and we wanted to avoid the possible issue.
To get the configuration backed up, we used rsync over SSH from my desktop PC, which uses Linux Mint, a Debian / Ubuntu variant. To make this work, we first needed to install rsync on PBS:
apt-get install rsync…and stop the Proxmox services:
service proxmox-backup stop
service proxmox-backup-proxy stopThen, from my desktop I ran:
rsync -avR --delete root@192.168.123.101:/etc/proxmox-backup ./proxmox-backup/
rsync -avR --delete root@192.168.123.101:/var/lib/proxmox-backup ./proxmox-backup/
rsync -avR --delete root@192.168.123.101:/etc/ssh ./proxmox-backup/
rsync -avR --delete root@192.168.123.101:/var/log/proxmox-backup/tasks ./proxmox-backup/192.168.123.101 should be changed to the IP address of your PBS server.
NOTE: Syncing the logs took the most time for us
Since very little time had passed since we updated PBS, we were satisfied that the likelihood of installing the exact same version was high and therefore the config files should be completely compatible with the new installation.
We unplugged the existing OS drives, connected the new drives and installed a fresh copy of PBS. After checking for updates on this new server, we stopped the services:
service proxmox-backup stop
service proxmox-backup-proxy stopThen, from my desktop I ran:
#This first line is optional. I did it to keep the same
#rsync -av --delete --progress ./proxmox-backup/etc/ssh/ root@192.168.123.101:/etc/ssh/
rsync -av --delete --progress ./proxmox-backup/etc/proxmox-backup/ root@192.168.123.101:/etc/proxmox-backup/
rsync -av --delete --progress ./proxmox-backup/var/lib/proxmox-backup/ root@192.168.123.101:/var/lib/proxmox-backup/
rsync -av --delete --progress ./proxmox-backup/var/log/proxmox-backup/tasks/ root@192.168.123.101:/var/log/proxmox-backup/tasks/NOTE: I ran the ssh sync then restarted the SSH server to make the new server use the same keys and appear as the old server. You probably won’t need this, but I wanted the same identity to be used.
Before starting any services, we set permissions:
chown -R backup:backup /var/log/proxmox-backup/tasks/
chown -R backup:backup /var/lib/proxmox-backup/
chown backup:backup /etc/proxmox-backup
chown backup:backup /etc/proxmox-backup/*
chown root:root /etc/proxmox-backup/authkey.key
chown root:root /etc/proxmox-backup/tfa.json
chown backup:backup /etc/proxmox-backup/.*In /etc/proxmox-backup I ran many commands (pulled from bash history). In hindsight, we’re not sure which were necessary, but again, it worked.
chown backup:backup /etc/proxmox-backup
chown root:root /etc/proxmox-backup/authkey.key
chown root:root /etc/proxmox-backup/tfa.json
chown backup:backup /etc/proxmox-backup/.*
chown root /etc/proxmox-backup/authkey.pub
chown root /etc/proxmox-backup/csrf.key
chown root /etc/proxmox-backup/datastore.cfg
chown root /etc/proxmox-backup/node.cfg
chown root /etc/proxmox-backup/proxy.key
chown root /etc/proxmox-backup/proxy.pem
chown root /etc/proxmox-backup/prune.cfg
chown root /etc/proxmox-backup/remote.cfg
chown root /etc/proxmox-backup/sync.cfg
chown root /etc/proxmox-backup/tfa.json.lock
chown root /etc/proxmox-backup/traffic-control.cfg
chown root /etc/proxmox-backup/user.cfg
chown root /etc/proxmox-backup/verification.cfgThis is what it looks like now:
root@pbs:~# ls -l /etc/proxmox-backup/
total 55
-rw-r----- 1 backup backup 104 May 28 15:32 acl.cfg
-rw------- 1 root root 3243 Feb 3 14:25 authkey.key
-rw-r----- 1 root backup 800 Feb 3 14:25 authkey.pub
-rw-r----- 1 root backup 1675 Feb 3 14:25 csrf.key
-rw-r----- 1 root backup 118 May 29 11:20 datastore.cfg
-rw-r----- 1 root backup 3243 Feb 3 14:25 proxy.key
-rw-r----- 1 root backup 2147 Feb 3 14:25 proxy.pem
-rw-r----- 1 root backup 228 May 28 15:34 prune.cfg
-rw------- 1 backup backup 158 Feb 8 14:58 shadow.json
-rw------- 1 root root 298 May 28 15:14 tfa.json
-rw-rw---- 1 root backup 0 Feb 3 14:26 tfa.json.lock
-rw-r----- 1 root backup 137 Mar 23 12:37 traffic-control.cfg
-rw-r----- 1 root backup 177 Feb 8 15:01 user.cfg
-rw-r----- 1 root backup 103 Feb 14 09:34 verification.cfgOnce the permissions were set, we started the Proxmox services:
service proxmox-backup start
service proxmox-backup-proxy startOnce this was complete, we checked the GUI to be sure that we could log in, view the backups, view the backups from PVE, etc. All is working, so we’re happy with the results.
As the last step, we re-enabled 2FA. Don’t forget to return your environment to its secure state!
We hope this helps you on your journey. Again, if you choose to use these notes, it’s at your own risk. If you want more assurance than that, feel free to reach out for a consultation on your project. We’d be happy to oversee the environment changes to ensure a good outcome for you.