So, just for the sake of it I’ve been trying to get my lab to be HA -or as HA as a small homelab can be-
My current set up is a follows:
3 proxmox servers with some Debian VMs, the VMs run docker swarm
A NAS, with Truenas
ISProuter -> OpenWRTRouter -> VM [Port fowards 80/443]
This works like a charm when I am in my LAN, but when I access from outside, if the VM that has 80/443 port forwarded to is down (which it never is) I’d loose connectivity.
I have now idea how to solve this little problem in a efficient way, maybe a reverse proxy running on my OpenWRT? (Which’d only move the point of failure to my router, but if my router goes down is gameover already anyways) has anyone attempted this?
Any opinions/ideas?
You’re discovering that there’s ALWAYS a single point of failure. Even if every service is fault tolerant, you likely have a single network or power infrastructure. So, you have to figure out what you’re willing to tolerate. You could look into CARP or keepalived to make your reverse proxy more resilient. It’s probably overkill for a homeland, but could be a useful learning exercise.
Our new dog chewed up the Ethernet cable from my modem to my router while I was at work (well, commuting to) the other day. She found the only exposed 6 inches of it and went to town. Everything runs through the router. I had also just re-done some music library file structures and reset my downloaded songs right before leaving, assuming it would queue up and fill up the cache as I went about my day. Something I hadn’t done for over two years, but I wanted a music library so we could put calming music on for the pup that wouldn’t end up in my carefully curated library.
I have my music app set to pre-cache 10 songs, and ended up with 12 songs downloaded, so somewhere around 5-10 minutes after I started playing music on my commute was when the tasty cable was discovered. That was an excruciating day, listening to the same 12 songs over and over again.
Lesson learned about single points of failure in a new way. The worst part was I got a message about it from my fiancé when I got to work, so I knew what happened and there was nothing I could do about it. I just got to look at the world’s strongest firewall all day long.
Ugh! I had some of the same issues a while ago with a Jack Russell I adopted. Cool dog, high octane energy, eager to learn new things. Since he was teething, everything became a chew toy regardless of the mountain of chew toys I had already provided. USB cables, Ethernet cables, power cords, I’ve replaced a bunch. Thing about a Jack Russell is you can teach them anything and they are eager to learn and please, however, if they pick up a bad habit, it’s hard to break them of that. He doesn’t chew anything any more, but there for a stint, he was hell on wheels.
I remember a TV station I worked at, that had a lot of good redundancies with 3 redundant UPSs that could keep a bunch of equipment on air until the big generator took over, one day had the UPS controller die and took all 3 UPSs out. I think it took the engineers a couple days to get everything back up and running.
Def went into the rabbit hole without any idea how many of these single points I’d need to address, and the more I mitigate the more I find. Like you said, this is very much overkill, I am just doing it to learn and have some good old homelab fun before we are all forced to rent “cloud” PCs
Thanks for the suggestions, I’ll look into those!