Reconstuction and Planning
My home servers need some attention and this is my thought process.
The plex box
I’ve noticed that my plex server has been running low on ram after deciding to deploy radarr and sonarr to help organize things. Between a small handful of docker apps, plex, and a 4GB transcoding ramdisk; things feel a little tight. I had to add some extra swap space to avoid crashing and wind up using almost 4GB of the 5ish GB it has total; I added a 4GB swap on top of the 1gig or so the ubuntu installer added or maybe I did. idk.
The system has about 7.67GB of usable ram. I might be able to tweak the iGPU vram in the bios when I can bothered to reboot it to do so.
I’ve noticed when a 1080p video is playing, the ramdisk approached 75% full for one show. That alone tells me it could use more ram.
I need to find some reasonably priced pair of 8GB DDR3 Low Density RAM.
I also did some tweaking and ram usage seems more reasonable but I’m not sure if htop is measuring the ram usage from docker properly. Atm, plex in docker is using like 1.6GB but htop says 2.92GB ram, 1GB swap. More research required.
The lab server
I have a 2.2TB RAW ZFS volume across 4x 600GB SAS 10K Disks setup as RAIDZ1 and research tells me that 4 disk in RAIDZ1 is not a ideal setup. Appearently z1 likes odd number disks. Maybe I’ll buy another one or two and rebuild the array. add the 5th one for z1 or 2 disks for z2. Either way, I should gain some space while I’m at it. :)
In conjunction to this, I’ve learned that 32GB of the 64GB ram has gone to ZFS in the form of ARC or some sort of ram cache. Which is neat but I didn’t expect half the ram to go to ZFS when reading forums posts suggest it generally wants 1GB per TB of space. I guess it’s that plus the cache and by default on proxmox it must only be set to consume half the avaliable ram because another post or that same one suggested that it can potentially consume almost all of the ram bar 1GB for the system. Oof. ZFS sounds cool but can be costly it seems. Gonna keep using it for a bit but it’s tempting me into buying more ram to increase the cache. I’m thinking an extra 64GB in the form of 4x16GB DDR3 ECC sticks ought to do.
I’m also expecting some neat little pcie enterprise class ssds in the mail soon. I’m thinking of trying one as ZFS cache and the other a lvm volume and delete default lvm volume (local-lvm, I think) from the boot disk. It’s a lab box. I’m experimenting with what I can do on a single box.
One day I wanna play with clusters but that requires one or two more machines, one more compute and a storage box to share data between them. And my parents don’t like that idea so much. Something about power consumption but they could always run a dedicated power line into my room or setup a building outside to do this in. :/
Update to the server - Again. :) (August 13, 2020)
So like a couple months ago I ran across a ad on reddit for some 900GB drives for like $25 each and I bought 4. 3 arrived functional but useless as they refused to write any data, couldn’t even partition them. So I waited a bit to think on it and decided to grab a H200 in IT Mode off ebay for $45 to allow direct access to all the hard drives. After testing the 3 bad drives to try and get them to do anything only to fail, I reached out the reddit seller and he agreed to send 3 replacements. Win. Dunno what to do with the bad ones atm.
Replacements came in, installed, and now all 8 drives are in a big ZFS pool. 4 mirror sets of 2 drives each giving me a total of 2.62TB of storage. Idk how optimal it is or if it’s even a good idea but at least there’s redundency.
I’m using the pcie ssds to run the VMs and Containers with LVM Thin Privisioning and the hard drives for backups, larger vm storage, and misc data.
I also added some spare dimms for an extra 8GB of ram. Not that it needs it but ehh. It does throw some warnings on POST but otherwise seems to work.
Also found out the H200 doesn’t work in the storage slot with it’s current firmware. Not sure how to modify it to work in the storage slot atm while keeping it in IT mode.
I’m also experimenting with grafana and figuring out how and what sort of data I want to feed into it. I’m a little confused on how to make dashboards and planning my own layout so I’m using premade ones and piping data into it.
Adding radarr and sonarr prompted me to reorganize my shows on the video disks so they’re not so spread out. Entire shows on one disk, not seasons here and there. That way sonarr can see all I have and help me find what I’m missing. As well as ingesting new content from a download folder and putting them whereever the show happens to live. These programs live in docker.
Grafana and the few data collection apps. Varken, and something else. Using influxdb to store and retrieve data. Still unsure how the db doesn’t just grow as data comes in. Best way to learn is to watch it in action I guess.
At least I can add a few buzz words to my resume soon.
Also have proxmox setup to feed data into grafana for funsies. And data from pfsense… also for funsies. :)
Stay safe my fellow nerds. :)