First off, I know ultimately I’m the only person who can decide if it’s worth it. But I was hoping for some input from your collective experience.

I have a server I built currently running Ubuntu 22.04. I’m using KVM/qemu to host VMs and have recently started exploring the exciting world of Docker, with a VM dedicated to Portainer. I manage the VMs with a mix of virt-manager via xRDP, cli tools, and (if I’m feeling extra lazy) Cockpit. Disks are spindles currently in software Raid 10 (md), and I use LVM to assign volumes to the KVM VMs. Backups are via a script I wrote to snapshot the LVM volume and back it up to B2 via restic.

It all works. Rather smoothly except when it doesn’t 😀.

I’ve been planning an HD upgrade and was considering using that as an excuse to start over. My thoughts are to either install Debian and continue with my status quo, or to give Proxmox a try. I’ve been reading alot of positive comments about it here and I have longed for one unified web interface to manage my VMs.

My main concerns are:

  1. Backups. I want to be able to backup to B2 but from what I’ve read I don’t see a way to do that. I don’t mean backup to a local repository and then sync that to B2. I’m talking direct to B2.
  2. Performance. People rave about ZFS, but I have no experience. Will I get at least equivalent performance out of ZFS and how much RAM will that cost me? Do I even need ZFS or can I just continue to store VMs the way I do today?

Having never used Proxmox to compare I’m really on the fence about this one. I’d appreciate any input. Thanks.

  • Shertson@lemmy.world
    link
    fedilink
    English
    arrow-up
    11
    ·
    1 year ago

    I’ll add my voice to the chorus and recommend Proxmox. I’ve never tried xcp-ng; it looks nice and I’m interested, but Proxmox has worked well for me.

  • TrenchcoatFullOfBats@belfry.rip
    link
    fedilink
    English
    arrow-up
    7
    arrow-down
    1
    ·
    1 year ago

    Another vote for Proxmox.

    Backups: Proxmox Backup Server (yes, it can run in a Proxmox VM) is pretty great. You can use something like Duplicati to backup the PBS datastore to B2.

    Performance: You can use ZFS in Proxmox, or not. ZFS gets you things like snapshots and raidz, but you will want to make sure you have a good amount of RAM available and that you leave about 20% of available drive space free. This is a good resource on ZFS in Proxmox.

    Performance-wise, I have clusters with drives running ZFS and EXT4, and I haven’t really noticed much of a difference. But I’m also running low-powered SFF servers, so I’m not doing anything that requires a lot of heavy duty work.

      • TrenchcoatFullOfBats@belfry.rip
        link
        fedilink
        English
        arrow-up
        1
        ·
        1 year ago

        I would say it’s at the “bottom” of the stack - Debian is the base layer, then Proxmox, then your VMs.

        Clustering just lets the different nodes share resources (more options with ZFS) and allows management of all nodes in the cluster from the same GUI.

  • hockeyhippie@lemm.ee
    link
    fedilink
    English
    arrow-up
    4
    ·
    1 year ago

    I run Proxmox in a cluster and TrueNAS in a VM on one of the nodes. It’s been really convenient. My nodes run a mix of LXC containers for different things + Docker or regular VM’s for other software.

      • TrenchcoatFullOfBats@belfry.rip
        link
        fedilink
        English
        arrow-up
        1
        ·
        1 year ago

        I haven’t done it myself, but I have looked into the process in the past. I believe you do it just like paying any drive through to any Proxmox VM.

        It’s fairly simple - you can either pass the entire drive itself through to the VM, or if you have a controller card the drive is attached to, you can pass that entire PCIe device through to the VM and the drive will just “come with it”.

  • LKummer@lemmy.world
    link
    fedilink
    English
    arrow-up
    4
    ·
    1 year ago

    Proxmox wont make backups to B2 easier, but since it is basically a web interface and API for Debian and KVM/QEMU you might be able to use your current backup strategy with very little modification.

    As for ZFS, you can expect to use about a GB of RAM for each TB in a ZFS pool. I (only) run 2x 4TB drives in ZFS mirror and it results in about 4-5 GB of RAM overhead.

    Another point you might want to consider is automation and the ability to use infrastructure as code. You can use the Proxmox Packer builder and Terraform provider to automate building machine images and cloning virtual machines. If you’re into the learning experience it’s definitely a consideration. I went from backing up entire VM disks to backing up only application data, making it faster and cheaper. It also enabled a lot of automated testing. For a homelab it’s a bit much, the learning experience is the biggest part. It’s an entire rabbit hole.

    If you want to see how the automation looks like, check out my example infrastructure repo and the matching tutorial. Also check out my Alpine machine image repo which includes automated tests for image cloning, disk resizing and a CI pipeline.

    • 𝓢𝓮𝓮𝓙𝓪𝔂𝓔𝓶𝓶OP
      link
      fedilink
      English
      arrow-up
      1
      ·
      1 year ago

      Proxmox wont make backups to B2 easier, but since it is basically a web interface and API for Debian and KVM/QEMU you might be able to use your current backup strategy with very little modification.

      I found this which leads me to believe I may be able to pipe zfs send to restic to replicate my current disk backup strategy. Presumably I could fire up a VM and build a zfs storage pool in it to test that theory out.

      As for ZFS, you can expect to use about a GB of RAM for each TB in a ZFS pool. I (only) run 2x 4TB drives in ZFS mirror and it results in about 4-5 GB of RAM overhead.

      So if I were to put 4x4TB in a RAID10 equivalent pool I’d be looking at ~ 8GB not 16, whew.

      For a homelab it’s a bit much, the learning experience is the biggest part. It’s an entire rabbit hole.

      The rabbit hole is where all the fun is. Templating was something I never really got around to in my current setup. I do have an ansible playbook and set of roles that will take a brand new Ubuntu VM and configure it just how I like it.

      Thanks for all the info. I’ll be sure to check out your repo.

      • 𝓢𝓮𝓮𝓙𝓪𝔂𝓔𝓶𝓶OP
        link
        fedilink
        English
        arrow-up
        1
        ·
        1 year ago

        I found this which leads me to believe I may be able to pipe zfs send to restic to replicate my current disk backup strategy. Presumably I could fire up a VM and build a zfs storage pool in it to test that theory out.

        Replying to myself but I think this is a square peg, round hole, situation.

        If I’m starting over with proxmox I likely need to rethink my entire backup strategy.

  • RotaryKeyboard@lemmy.ninja
    link
    fedilink
    English
    arrow-up
    4
    arrow-down
    1
    ·
    1 year ago

    Please give Proxmox a try! It was such a huge quality of life improvement when I migrated to it. I can’t speak to your backup needs or to the performance of ZFS, since I don’t use either of those. I just think that Proxmox took a lot of the pain out of my homelab management experience without taking away my capabilities to customize it. Highly recommend!

  • gabriele97@lemmy.g97.top
    link
    fedilink
    English
    arrow-up
    2
    ·
    1 year ago

    I started with proxmox and I’ll continue to use it because it’s very nice to use. As backup I use an rclone mount that is shared via NFS (everything inside a container) and I set that NFS share as a backup storage in proxmox. I think it is a bit convoluted but works fine enough for now.

  • Matthias Liffers@social.tthi.as
    link
    fedilink
    arrow-up
    1
    ·
    1 year ago

    I just swapped from Ubuntu to Debian but I don’t use VMs - only containers. I back my files up directly to B2 using autorestic, also running in a container that is scheduled by… another container (chadburn).

    No need for any VMs in my house. I honestly can’t see the point of them when containers exist.

    • 𝓢𝓮𝓮𝓙𝓪𝔂𝓔𝓶𝓶OP
      link
      fedilink
      arrow-up
      2
      ·
      1 year ago

      Eh, to each their own. In fairness, some iteration of my current setup has existed for many years and I’ve only just get my feet wet with containers in the last month.

    • TrenchcoatFullOfBats@belfry.rip
      link
      fedilink
      arrow-up
      1
      ·
      1 year ago

      Just an FYI to OP: If you’re looking to run docker containers, you should know that Proxmox specifically does NOT support running docker in an LXC, as there is a very good chance that stuff will break when you upgrade. You should really only run docker containers in VMs with Proxmox.

      Proxmox Staff:

      Just for completeness sake - We don’t recommend running docker inside of a container (precisely because it causes issues upon upgrades of Kernel, LXC, Storage packages) - I would install docker inside of a Qemu VM as this has fewer interaction with the host system and is known to run far more stable.

  • TCB13@lemmy.world
    link
    fedilink
    English
    arrow-up
    6
    arrow-down
    6
    ·
    edit-2
    1 year ago

    You most likely don’t need Proxmox and its pseudo-open-source bullshit. My suggestion is to simply with with Debian 12 + LXD/LXC, it runs VMs and containers very well.

      • TrenchcoatFullOfBats@belfry.rip
        link
        fedilink
        English
        arrow-up
        5
        ·
        1 year ago

        As far as I’m aware, everything in Proxmox is open source.

        I think some people get annoyed by the Red Hat style paid support model, though. There is a separate repo for paying customers, but the non-subscription repo is just fine, and the official forums are a great place to get support, including from Proxmox employees.

      • TCB13@lemmy.world
        link
        fedilink
        English
        arrow-up
        2
        ·
        1 year ago

        As said, they’ve separate repositories, annoying messages asking you for a license all the time etc. At some point you’ll find out that their solution doesn’t offer anything particular of value that you can’t get with other less company dependent solutions like I described before. You may explore the LXD native GUI… or heck even Cockpit or Webmin might be decent options.