Proxmox Migration - Progress Report #2

Joanna - the new Proxmox host - is up and running, and all but two VMs have been migrated, and demonstrated to work properly.  I've created a new PFsense VM on joanna, and configured and tested it.  This will operate in place of the one still running on Jerome - the old VM host based on VirtualBox.  The "old" PFsense VM will not be preseved.  A new one will be created once jerome has been setup to run Proxmox.

The other VM still running on Jerome is my SO's desktop VM (running - shudder - MS Windows).  The one dificulty I expect may happen when migrating is the whole "you've replaced too much hardware" alarm going off and having to re-register MS Windows 10.  On the plus side, it's retail Win 10 Pro.  As such, my understanding is that you can re-home the license, unlike OEM versions.  The possible downside is Win10 is approaching EOL (October of next year) and MS would really rather you use Win11 because telemetry.  We'll see.  I'm confident in my process for migrating VMs from VirtualBox to Proxmox.  I'm less confident in Microsoft.

Floyd - the old storage host - is still plugging along.  All shares have been moved off the storage on jerome, and all clients are re-pointed to floyd.  That's good.

I'm planning to migrate routing from the PFsense VM on jerome to the one on joanna today, and shutting down the one on jerome.  This coming Saturday, I'm planning to migrate the last VM - the Win10 one - off jerome, and lights-out the box for the last time as an Ubuntu-based VM host running VirtualBox.  Then, hopefully, I can take my time getting jerome stood up as a properly-configured Proxmox host.

Proxmox Migration - Progress Report #1

I have procured parts to build a "modern" Proxmox host, including an ATX MoBo, 12C/24T AMD Ryzen 7900X, 128GB RAM, a 2TB SSD, and 4 x 1GB NIC.  I've pre-built the new host (Joanna, named after Roz mother from Frasier ...and, yes, I've set up that many machines that I'm going that deep into the bench of Frasier characters) on a "test bench" frame I bought.

The parts... minus the NIC and SSD.  Also ended up
using the cooler that came with the CPU.

I have done some storage experiments, and arrived at the conclusion that using the existing 18TB disks in a XFS RAID5 will be the way to go.  That means I will need to wipe the disks and re-format them when I do the OS Reinstall on Jerome.  For continuity of service, that means I need to port those files off to somewhere else.  I have decided to light up Floyd, my old 20-bay storage server, which has been cold iron for a little over a year (that's when I migrated storage to the disks onboard Jerome).  Floyd is up and operating as expected.  I have migrated my video library back over there, and now I have to migrate my other file shares over there.  Then I need to repoint all my share mounts.  That'll be fun.  This is in preparation for the OS install on Jerome.

Of course, I'll also need to migrate all the VMs currently running on Jerome off to Joanna.  In previous migration attempts, I have not forced the MAC address of the copy to match the MAC of the original.  For these remaining VMs, I really want to make sure the copies on Joanna look just like the originals did on Jerome.  Hopefully I can coordinate everything and set the MACs, and they just come up.

I've done some networking experiments with the new NIC, and managed to get 2 of the 4 interfaces set up, which is the minimum I need to set-up my replacement PFsense VM.  I plan on the other two being "room for expansion".  I've decided to go with a replacement rather than a migration, as I want to be able to "flip the switch" on my other machines -- from VirtualBox-based VM to Proxmox-based VM -- without any down time.  Once Jerome is back up running Proxmox, I'll just set up another PFsense VM there.  ...but I'll be able to take my time and make sure all the network interfaces are set up and working as I expect.

I had hoped that I could setup a "storage" VM on Joanna, and mount the shares from Floyd there, and re-share them via NFS, which, I'd hoped, would eliminate the need to re-write all the /etc/fstab files on all the machines on the network that use the shares twice.  It looks like that's not going to work.  However, I did figure out how I'm going to do the big disk array once Jerome is running Proxmox, and I will be using a storage VM to share those filesystems... I may even try out TrueNAS.  Who knows.  At a minimum, I can just use Debian, as my needs are pretty contained.

Ultimately, my plan is for Jerome to be the "prime" VM host, where all my "full time" VMs (storage, firewall, desktop, streaming servers) will run, and Joanna will be my "secondary" VM host, where "lab machines" and other non-full time VMs will live.  Each VM host will have its own local storage, but the "Big Store" will live on Jerome for the foreseeable future.  Floyd (named, incidentally, for Mayberry's favorite barber -- I used to name my machines after Andy Griffith Show characters) will go back to cold iron once the migration on Jerome is complete.

Today, I plan to move the guts of Joanna into the case I'd originally bought for Zora.  Then, I'll migrate some (all?) the non-PFsense VMs to Joanna.  Maybe I can also set up a PFsense VM on Joanna.  We'll see.  It looks like it'll be a full, rich day.

Migrating from VirtualBox to Proxmox - A beginning

A long time ago, now (2019, as it turns out), I set up a super-duper AMD Threadripper machine with 16 cores and 32 threads, and 128GB of RAM to be a Virtual Machine host for my home lab. Since then, it has evolved into the host for my PFsense router, my "desktop" machine, and a bunch of other VMs, as well as my storage server. At the time, I set it up with a regular Ubuntu install with VirtualBox running on it. All these VMs are being run on VirtualBox. 

I know. 

Well, time-and-a-half went by, and I became aware of Proxmox. I never really had spare hardware with enough grunt to really try it out, so I kinda let it go. Then, my employment relationship with that online furniture retailer (as well as those of about 500 other folks) suddenly ended. The machine I'd put together as a "BYO" desktop for work was suddenly out of a job as well. That was a pretty beefy machine... AMD Ryzen 7, 8C16T with 32GB of RAM... but it had a sort of narcolepsy problem. it would just... for no apparent cause... stop. It didn't seem to be thermally-related... not the power supply. I have no explanation... it just farts out... but I can usually get a good 6-8 hours out of it, sometimes a lot more. ...so it finally dawned on me that this old beast might be a good Proxmox test mule.   That was a couple months ago.  I decided to give it a go, and I like it. I like it enough that I want to migrate my "production" VM host to Proxmox. 

Well. 

I'd hoped to migrate the VMs, one by one, to Proxmox on a different host, and then do an OS re-install on Jerome (the Threadripper box - named after the "good family man" to whom, Frasier and Niles owed a "favor") with Proxmox, and migrate some or all of the VMs back over. 

The old BYO machine (named Zora, after Fraiser's "excitable" Greek aunt... I have a lot of machines named after characters from Frasier) is based on a Mini-ITX motherboard, so it has only one PCIe slot, and that's taken up by a video card. I need more Ethernet ports to run my PFsense router... and, of course, there's the "oops I forgot how to computer" problem. So, I decided to build a new machine, with a full ATX motherboard, and slots... and I managed to find an AMD Ryzen 7 CPU with 12 cores and 24 threads AND a built-in GPU... so that's exciting. Not QUITE as heavy as Jerome, but pretty good... and 128GB of RAM. So, I can put a 4 x 1GB Ethernet board in there, and run my PFsense router, and the board has 4 SATA ports, so I can figure out how to do my storage. 

Ah, yes, storage. Jerome has three 18TB drives in a RAID5 array, and those are run by the host OS. I'm thinking that under Proxmox, I will have the raw drives forwarded up to one of the VMs that will serve as the storage server... but I'm not entirely sure how that will work. I haven't really dug into the docs yet. I know it's been done in other KVM environments, so... I'm confident it should be doable. When I get the new machine up and running I can try it out. It would be lovely to just be able to take the disks as they are, and plug them in on a fresh Proxmox install, build a fileshare VM and just... ta-da! We'll see. 

So, the parts for the new Proxmox host are supposed to show up today. Hopefully, everything goes together without a hitch, and I can begin properly experimenting. 

Updates when they're warranted.

RMVOD v0.9.2 is HERE! It's a BIG DEAL!

New Features:

 0) User Sessions.  You have the option to log in at the browser and track your viewing across multiple devices, such that your recommendations will be consistent across devices, and your tv series viewing history will be aggregated.  User sessions also enables you to bring your settings with you, like playback speed, watch next episode, and resume playback.  In addition to your settings, watch data like "current artifact" and "time offset" are also captured in your session, so you can switch devices easily in mid playback.

1) Recent Episodes. Thanks to the existence of User Sessions, it is now possible to keep track of what the latest episode is that you've watched for a TV Series, and either re-watch or watch the next one with just one click.

Bug Fixes and Minor Tweaks:

Posters now fetched from OMDbAPI - The previous source of poster images became unusable.  RMVOD has been shifted to take advantage of the service on OMDbAPI 

Added "stats" - The "Settings" tab has a clickable link which fetches the current "statistics" for the server, including the current count of movies, tv episodes and tv series, as well as viewing statistics for the top 10 tags among tv series and movies, and the top 10 titles for each tag.

OMDbAPI "Series Fetch" bug fix - Fetching series info from OMDbAPI had a bug, which has been fixed.

Added Playback Speed option to "Settings" - Playback Speed can be set as a option in the Settings tab.

Fix title set on play. - There was an issue with setting the page title when playback starts.

Fix Recs Poster Image thing. - There was an issue in the recommendations artifact details wherein the artifact details were not getting the "oops no poster" when there was no poster for the artifact.  

Reworked statistics to have the period delivered by the client, rather than hard-coded in the API.  

Download the latest stable code here:

https://github.com/taobear68/rmvod/releases/tag/v0.9.2

RMVOD v0.9.1 is HERE! It's a DOOZIE!

 Hi folks!

So, I've created a new tag on the RIBBBITmedia VideoOnDemand code repo, and this one undoes the whole "feature complete for release 1.0" thing.  Many new features.  Oops!

In this tag you'll find several new features.  Notably:

0) Config file.  I'd planned for some time to implement this, and this seemed a really good time to do it.

1) Recommendations (Recs).  This is a new tab in the RMVOD screen, which provides a "carousel-like" interface for browsing artifacts based on connections to viewing trends on the server.  The Recs interface shows artifact details in a floating box when the artifact's image is clicked.  For TV Series artifacts, options are presented to start playing the series from the beginning or to select a season, and browse episodes to select one to play.  Recommendations are cached on the server and expire after a period of days set in the API's config file.  There is a "Refresh Recs" link on the Settings tab to force a refresh of the recommendations.  Recommendations functionality requires a new table in the database.  See README.txt in the root of the code tree for details on upgrading an existing system to support this feature.  For systems where the DB is hosted on the same box as the API, and the box is really-low resource... like a RasPi 2 something even less capable... Recommendations should be disabled in the API's config file.

2) Quick Search Links.  This is a new set of links on the "List/Search" tab which present the lists of artifacts from the Recs tab in the familiar list format.  This also depends on Recs being enabled on the server.

3) "Last Search" display.  Now, when you execute a search, the "search factors" will be displayed below the artifact list to remind you of what you searched for, whether the search is single factor or multi-factor.

4) "Play fullscreen" option.  This is a checkbox on the Settings tab which asks the player to initiate playback in full-screen mode.  This currently only works for playback initiated by the user clicking the artifact to initiate playback.  When playing a series, the second and later episodes played will not be in full-screen.  This is a known issue related to a security feature in popular browsers.  I'm looking for a work around, but for now, it eludes me.

5) "Populate Episode Details" link. In the Artifact Details box for a TV Series, at the bottom, to the right of the "Edit" and "Associate Episodes" links, there is now a "Populate Episode Details" link which initiates an operation on the server to query IMDbAPI for the details relating to the "series" artifact and each of the episode artifacts.  This requires that the series artifact already has episodes associated with it, and has its "imdbid" value set appropriately.  The use of this feature requires a "key" for the API which can be obtained by visiting here https://www.omdbapi.com/ and setting the key value in the API's config file.

Download the latest "stable" code here:

https://github.com/taobear68/rmvod/releases/tag/v0.9.1

RMVOD, upstream data, and new features

 I don't know if this is permanent or not... but there definitely seems to be a *period* of development arriving, where the big, exciting features of which users would be readily aware are fading out, and it's more... stuff administrators would appreciate.  

I was recently in contact with Brian who maintains the OMDb API (https://www.omdbapi.com/) which allows weirdos like me to have a source of "Movie Database" content available via a simple API.  I've supported him (nominally) on Patreon for a while... you can access the api for free, but you're throttled to some number of requests per day, but if you sign up for his Patron, you can get many more requests per day.  Recently my script that I'd written to scrape "Movie Database" data from the Internet (not using Brian's API) has been throwing errors, and I think it's because the "Movie Database" I'd been hitting for TV Series Episode data didn't like getting scraped by a bot... so, they started sending me 403s.  

That's when I reached out to Brian and asked him if there was a way to get series episode data from his API... and almost immediately, he was like, "You mean *this*?" with a link to  page outlining an as-yet unpublished endpoint on the API which returns the Episodes for a Season of a TV Series.  Well... I was over the moon... I said, yes, that appears to be just the thing.  Then I tried it out, and discovered there appeared to be a bug in the way it linked episodes to the "series" entity.  I tried it out a couple different ways, made some notes, and told Brian I think I found a bug.  He dove right in and within a few days, the "unpublished" series data endpoint was working properly.

So, now, I have a "one button" solution in the RMVOD UI for populating all the data for the Episodes of a TV Series.  The thing is, this feature is not something "normal viewers" are going to care about or use.  From the standpoint of an administrator... the person who has to maintain a local instance of RMVOD, this is a boon.  To someone only concerned with "What is there to watch", this feature doesn't matter.  I think this is the character of features that will be coming down the pipe for the foreseeable future.

There will be a new "tag" coming soon, which will include the feature outlined above.  If you have an existing set-up, the new features will require a DB update (adding a table), which is not automated.  If you're incomfortable running SQL from the command line and you don't have an instance of RMVOD set up, then you might want to wait, as a fresh install will have the new table.  

The current tag/release is here:  https://github.com/taobear68/rmvod/releases/tag/v0.9.0 if you want to give it a try.

RMVOD v0.8.0 is here!

RMVOD v0.8.0 is here!

RIBBBIT media VideoOnDemand is now at v0.8.0, and has been moved to its own GitHub Repo

Do you want to try it out?  You can download it here.

If you don't know what this is all about here are some previous blog posts about it here and here.

...and if you don't have the patience and/or time to read those now, here's a quick recap:

I wanted to be able to have a browser-based video-on-demand system with some search functionality  -- something where I wasn't dependent on wacky changes and release cycles, and having to take onboard features I neither want nor need just to get the bug fixes.  I wasn't interested in sharing my library in or out.  I just wanted to be able to watch my videos on my home network and not have to install an app or be running a particular OS.  

...et voila!  rmvod is here!

The targeted feature set for v1.0 is in place.  Bug fixes and minor tweaks remain.

The  install script should work for Debian, Raspberry Pi OS, and Ubuntu Server.  Other Debian-based distros should at least be close to working if their package names line up.