Caravan (Part 1) - A New Plan For A New Place

My partner and I are looking for a new place to be.  Being the computing-dependent nerd that I am, I want to be able to have all the services I depend on in our current place available in the new place as soon as there's power and Internet.  To that end, I decided what I ought to do is build a road-ready rack with a firewall and a few computers in it.  Once we close on the new place and get utilities turned on, I can pop the rack in there, and be up and running in an hour or 2.

Some background -  About a year ago, I had switched from all my "vital services" being on VMs on a Proxmox cluster to hosting them on a couple RasPi 5s.  I consolidated a lot.  Where before I had separate VMs for my RMVOD, Icecast2, Desktop, food inventory web app, file services, and so on, I decided I could do a lot more with a lot less CPU and memory overhead if I just had it all running on one 16GB RasPi 5.  I have a second RP5 that is the Jenkins host and hosts backup media.  Actually there's a third RP5 that acts as a bastion server and rack console.  This resulted in using a lot less power and generating a lot less heat.  The file services are facilitated by 4 x 8TB disks configured in a RAID5 arrangement in a USB3 enclosure, viz:

 

Can be got here: https://www.amazon.com/dp/B0DD3GZSGJ 

 That USB3 4-bay HDD enclosure hangs off the one main 16GB RasPi 5, and holds my audio and video stashes, my personal file share, and various sundry other bits and bobs.  Other than that, and USB connections to printers and backup media, basically my whole "infrastructure" is handled by 3 Raspberry Pi 5s and a tiny Netgate box doing the routing. 

  This transition has worked out well.  My plan is to more or less replicate this success but in a portable package.

 Enter Caravan

 The name "caravan" was just meant as a nod to portability and the kind of "home away from home" self-contained nature of a camping trailer or RV.  I settled on a 6-space rack enclosure, such that it could handle a switch, a shelf for a Netgate 1100, 2U for a RasPi enclosure, and a PDU.

I ended up getting this enclosure:

 

Can be got here:  https://www.amazon.com/dp/B0CZ41ZHD6

 It comes with screws for the rails (it takes 10-32 screws, but prefers them short), so that's handy, but there's a few shortcuts taken by the manufacturer that might cause frustration.  The plastic stinks -- it off-gasses... It claims to be HDPE... and maybe it is, but I've never come across HDPE stinking like this.  Screws holding the rack rails to the enclosure are poorly placed.  Things like that.  On the bright side it's cheap, and I'm not going on tour with Aerosmith (or whoever the kids like today), so extreme toughness and/or ATA-certification is not important for me in this application.  Surviving an 8-hour car ride is the thing. 

 The switch I settled on is a simple TP-link 24-port unmanaged gigabit switch (TL-SG1024S). It fits in 1U, comes with the rack ears and is under a hundred bucks. 

 Can be got here: https://www.amazon.com/dp/B0779R9LJ3 

 The PDU is kind of an odd choice.  I went with one from Pyle.  Yeah, Pyle.  Normally, Tripp-Lite has been my go-to for rack PDUs in my personal/home stuff.  They're pretty bulletproof, and I've found them reliable.  In this instance, tho, I was anticipating having a bunch of wall-wart power supplies (because no one has figured out how to do a rack-mount muli-Pi power supply yet, apparently) and 1U PDUs with traditional outlets are just not happy with that.  This Pyle unit has all pigtails in the back.  It really is a thing of beauty.  Viz:

 

Can be got here: https://www.amazon.com/dp/B072RC5NZQ 

 The big deal, tho, is the enclosure/mount for the Raspberry Pis.  I wanted something that would accommodate the PCIe SSD capability offered by the Pi 5.  I also wanted to be able to stuff it with RasPis, because... who knows what kind of goofy shenanigans I might get up to?  I went with a GeekPi unit which stands the RasPis on their side, and has a daughter board for each of the 10 "caddies" which uses pogo pins for power delivery and some GPIO connections, and connects to the PCIe connector on the Pi for M.2 SSD support via flat-flex.  This enclosure holds up to 10 RasPi 5s.  My only big gripe with this thing is that the screws for the caddies aren't captive.  The daughter boards seem to work great, and packing up to 40 cores of Raspberry Pi goodness into 2U is pretty sweet.  The daughter boards move the power connection to the front panel, so easy power on/off is right there.

 

 Can be got here: https://www.amazon.com/dp/B0DQL2N6V9

I have the rack assembled and 7 of the RasPi slots occupied.  Now I'm building out services. I've currently got 1 16GB, 2 8GB and 4 4GB RasPis mounted.  One of the 8GB units has a 1TB SSD setup as a MariaDB server, with the OS on a MicroSD, and the data on the SSD.  The other 8GB unit also has a 1TB SSD, but the OS is installed on it.  This one is serving as a Desktop machine with xrdp installed.  I have one of the 4GB RasPis serving as a bastion server and rack console, and the other 3 4GB units have not been assigned, although I expect one of them will be a file & print server (and maybe identity) and one will likely be media streaming.  I have designs for setting up local email, chat, groupware, an Asterisk PBX, and a few other services.  The 16GB unit will host most of the dynamic web stuff (minus database and file needs).  I haven't worked it all out yet, and I have some room to grown in the GeekPi enclosure.

More to come

There's plenty more to do and a lot more decisions to take.  I'll update as this progresses, but I thought it might be nice to offer some of the stuff I've done so far in case anybody's considering a similar project.

I think that's all for now. 

 

 

Proxmox Migration - Progress Report #1

I have procured parts to build a "modern" Proxmox host, including an ATX MoBo, 12C/24T AMD Ryzen 7900X, 128GB RAM, a 2TB SSD, and 4 x 1GB NIC.  I've pre-built the new host (Joanna, named after Roz mother from Frasier ...and, yes, I've set up that many machines that I'm going that deep into the bench of Frasier characters) on a "test bench" frame I bought.

The parts... minus the NIC and SSD.  Also ended up
using the cooler that came with the CPU.

I have done some storage experiments, and arrived at the conclusion that using the existing 18TB disks in a XFS RAID5 will be the way to go.  That means I will need to wipe the disks and re-format them when I do the OS Reinstall on Jerome.  For continuity of service, that means I need to port those files off to somewhere else.  I have decided to light up Floyd, my old 20-bay storage server, which has been cold iron for a little over a year (that's when I migrated storage to the disks onboard Jerome).  Floyd is up and operating as expected.  I have migrated my video library back over there, and now I have to migrate my other file shares over there.  Then I need to repoint all my share mounts.  That'll be fun.  This is in preparation for the OS install on Jerome.

Of course, I'll also need to migrate all the VMs currently running on Jerome off to Joanna.  In previous migration attempts, I have not forced the MAC address of the copy to match the MAC of the original.  For these remaining VMs, I really want to make sure the copies on Joanna look just like the originals did on Jerome.  Hopefully I can coordinate everything and set the MACs, and they just come up.

I've done some networking experiments with the new NIC, and managed to get 2 of the 4 interfaces set up, which is the minimum I need to set-up my replacement PFsense VM.  I plan on the other two being "room for expansion".  I've decided to go with a replacement rather than a migration, as I want to be able to "flip the switch" on my other machines -- from VirtualBox-based VM to Proxmox-based VM -- without any down time.  Once Jerome is back up running Proxmox, I'll just set up another PFsense VM there.  ...but I'll be able to take my time and make sure all the network interfaces are set up and working as I expect.

I had hoped that I could setup a "storage" VM on Joanna, and mount the shares from Floyd there, and re-share them via NFS, which, I'd hoped, would eliminate the need to re-write all the /etc/fstab files on all the machines on the network that use the shares twice.  It looks like that's not going to work.  However, I did figure out how I'm going to do the big disk array once Jerome is running Proxmox, and I will be using a storage VM to share those filesystems... I may even try out TrueNAS.  Who knows.  At a minimum, I can just use Debian, as my needs are pretty contained.

Ultimately, my plan is for Jerome to be the "prime" VM host, where all my "full time" VMs (storage, firewall, desktop, streaming servers) will run, and Joanna will be my "secondary" VM host, where "lab machines" and other non-full time VMs will live.  Each VM host will have its own local storage, but the "Big Store" will live on Jerome for the foreseeable future.  Floyd (named, incidentally, for Mayberry's favorite barber -- I used to name my machines after Andy Griffith Show characters) will go back to cold iron once the migration on Jerome is complete.

Today, I plan to move the guts of Joanna into the case I'd originally bought for Zora.  Then, I'll migrate some (all?) the non-PFsense VMs to Joanna.  Maybe I can also set up a PFsense VM on Joanna.  We'll see.  It looks like it'll be a full, rich day.

Migrating from VirtualBox to Proxmox - A beginning

A long time ago, now (2019, as it turns out), I set up a super-duper AMD Threadripper machine with 16 cores and 32 threads, and 128GB of RAM to be a Virtual Machine host for my home lab. Since then, it has evolved into the host for my PFsense router, my "desktop" machine, and a bunch of other VMs, as well as my storage server. At the time, I set it up with a regular Ubuntu install with VirtualBox running on it. All these VMs are being run on VirtualBox. 

I know. 

Well, time-and-a-half went by, and I became aware of Proxmox. I never really had spare hardware with enough grunt to really try it out, so I kinda let it go. Then, my employment relationship with that online furniture retailer (as well as those of about 500 other folks) suddenly ended. The machine I'd put together as a "BYO" desktop for work was suddenly out of a job as well. That was a pretty beefy machine... AMD Ryzen 7, 8C16T with 32GB of RAM... but it had a sort of narcolepsy problem. it would just... for no apparent cause... stop. It didn't seem to be thermally-related... not the power supply. I have no explanation... it just farts out... but I can usually get a good 6-8 hours out of it, sometimes a lot more. ...so it finally dawned on me that this old beast might be a good Proxmox test mule.   That was a couple months ago.  I decided to give it a go, and I like it. I like it enough that I want to migrate my "production" VM host to Proxmox. 

Well. 

I'd hoped to migrate the VMs, one by one, to Proxmox on a different host, and then do an OS re-install on Jerome (the Threadripper box - named after the "good family man" to whom, Frasier and Niles owed a "favor") with Proxmox, and migrate some or all of the VMs back over. 

The old BYO machine (named Zora, after Fraiser's "excitable" Greek aunt... I have a lot of machines named after characters from Frasier) is based on a Mini-ITX motherboard, so it has only one PCIe slot, and that's taken up by a video card. I need more Ethernet ports to run my PFsense router... and, of course, there's the "oops I forgot how to computer" problem. So, I decided to build a new machine, with a full ATX motherboard, and slots... and I managed to find an AMD Ryzen 7 CPU with 12 cores and 24 threads AND a built-in GPU... so that's exciting. Not QUITE as heavy as Jerome, but pretty good... and 128GB of RAM. So, I can put a 4 x 1GB Ethernet board in there, and run my PFsense router, and the board has 4 SATA ports, so I can figure out how to do my storage. 

Ah, yes, storage. Jerome has three 18TB drives in a RAID5 array, and those are run by the host OS. I'm thinking that under Proxmox, I will have the raw drives forwarded up to one of the VMs that will serve as the storage server... but I'm not entirely sure how that will work. I haven't really dug into the docs yet. I know it's been done in other KVM environments, so... I'm confident it should be doable. When I get the new machine up and running I can try it out. It would be lovely to just be able to take the disks as they are, and plug them in on a fresh Proxmox install, build a fileshare VM and just... ta-da! We'll see. 

So, the parts for the new Proxmox host are supposed to show up today. Hopefully, everything goes together without a hitch, and I can begin properly experimenting. 

Updates when they're warranted.