Tagged: proxmox

1 post

  1. Proxmox upgrades

    · 4 min read #proxmox #cluster #upgrading

    I run a 3-node Proxmox cluster homelab.

    It's something I've put off for many years, but in 2025 I decided it was time to start.

    I committed myself to a number of rules for this project:

    • Prioritise Reduce, Reuse, Recycle principles
    • No individual item to cost >£150
    • Must provide genuine benefits to my lifestyle, not just entertainment/learning

    I'll cover some of those benefits in future posts.

    The cluster:

    1. ~£50 Optiplex micro 7040 - i5-6500T (4C/4T), 16GB RAM, 256GB SSD
    2. ~£90 Optiplex micro 7060 - i5-8500T (6C/6T), 32GB RAM, 256GB NVMe, 120GB SSD
    3. ~£130 Optiplex micro 3090 - i3-10105T (4C/8T), 16GB RAM, 256GB NVMe, 256GB SSD

    A few RAM upgrades have already been added (included in specs above), costing between ~£20-£50.

    So. The disk drives

    Nodes 2 and 3 I had already propped up with ye olde SSD drives I had lying around. The backup service had started to fail due to lack of space, so upgrading hard disks bubbled up my backlog at quite a pace.

    After a few days of crawling eBay, I found a seller with 3 Samsung EVO 850 500Gb drives for £104. Each rated at 95% health or above. This felt like the perfect time to give all 3 nodes a standard amount of space, and a few days later they arrived in crisp plastic cases.

    I've replaced/upgraded hard drives for computers countless times, and didn't expect this migration to be much of a challenge. Oh the naivety!

    First up - Node 1. Before I even began the upgrade I remembered it was missing an NVMe drive. And I lacked a usb-to-SATA adaptor*. Node 1 postponed.

    Node 2. I migrated everything off the old SSD and installed the replacement. It didn't boot. My cluster sits under my TV, and I was seeing nothing on the screen**. I could ping the box but no other services worked. I returned the original disk. Perfect boot. Root cause? - I had previously, whimsically assigned /dev/sda1 in fstab. For those familiar, this means during boot the OS was trying to load the old drive, failing, and crashing to emergency boot - with no SSH.

    /dev/sda1   /mnt/ssd_backup   ext4   defaults   0 2

    The fix for this is to tie the mount by the UUID, and setting the nofail flag.

    UUID=4f3a9c1e-2b7d-4c55-a1aa-0e9c8b6d2f01  /mnt/ssd_backup  ext4  defaults,nofail  0 2

    But why was nothing showing on the screen?

    Initially, the TV takes a age because it doesn't cope with the vintage resolution used during POST. Resolutions can change during boot, as more sophisticated drivers come online. This causes a TVs HDMI handshake to renegotiate.

    The GPU was pointing to a VM on the box - specifically Immich. This is so Immich can leverage its capabilities for its AI processes. Once boot goes past grub all video display is gone. You can see how the small window I might get some signal could be missed.

    Node 3. No surprises here. It was setup the same as node 2. Same daft fstab setup. No Immich this time, but the GPU was assigned to another VM for some experiments I had been doing.

    What I expected to take 2-3 hours ended up taking twice as long. In the end, no data was lost. Every container was neatly moved and returned to where it belongs. All Proxmox hosts got package updates and upgrades, backups were re-engineered with more space, alerting for low disk space, and another 9 or 10 safety checks.

    With lessons learnt, logged, and a new found appreciation for the Linux boot process - I am once again ready to break things.

    * I've since ordered a usb-to-SATA adaptor and NVMe for node 1

    ** Also ordered a 7" portable screen

← All tags · All posts