All of them

And you can even export it there.

The problem is that I want failover to work if a site goes offline, this happens quite a bit with private ISP where I live and instead of waiting for the connection to be restored my idea was that kubernetes would see the failed node and replace it.

Most data will be transfered locally (with node affinity) and only on failure would the pods spread out. The problem that remained in this was storage which is why I’m here looking for options.

Thanks for the info!

I’ll try Rook-Ceph, Ceph has been recommended quite a lot now, but my nvme drives sadly don’t have PLP. Afaict that should still work because not all nodes will face power loss at the same time.

I’d rather start with the hardware I have and upgrade as necessary, backups are always running for emergency cases and I can’t afford to replace all hard drives.

I’ll join Home Operations and see what infos I can find

It’s fine if the bottleneck is upload/download speed, there’s no easy way around that.
The other problems like high latency or using more bandwith than is required are more my fear. Maybe local read cache or stuff like that can be a solution too but that’s why I’m asking for what is in use and what works vs what is better reserved for dedicated networks.

Ceph (and longhorn) want “10 Gbps network bandwidth between nodes” while I’ll have around 1gbit between nodes, or even lower.

What’s your experience with Garage?

I heard that ceph lives and dies with the network hardware. Is a slow internet connection even usable when the docs want 10 gbit/s networking between nodes?

They both support k8s, juicefs with either just a hostpath (not what i’d use) or the JuiceFS CSI Driver. Linstore has an operator which uses drbd and provides it too.

If you know of storage classes which are useful for this deployment (or just ones you want to talk about in general) then go on. From what I’m seeing in this thread I’ll probably have to deploy all options that seem reasonable and test them myself anyways.

Well, if that is the case then I will have to try them all but I’m hoping at least general behaviour would be similar to others so that I can start with a good option.

I want the failover to work in case of internet or power outage, not local cluster node failure. Multiple clusters would make configuration and failover across locations difficult or am I wrong?

I mean storage backends as in the provisioner, I will use local storage on the nodes with either lvm or just storage on a filesystem.

I already set up a cluster and tried linstore, I’m searching for experiences with the options because I don’t want to test them all.

I currently manage all the servers with a NixOS repository but am looking for better failover.

I’m talking about software RAID, for example btrfs.

Most RAID levels are for redundancy, not speed and software RAID doesn’t need drives of the same size.

if one drive dies it’s got a copy on another drive

Do you want JBOD or a RAID? Cause dealing with disk failure is a feature of RAID

When making an application instead of coding for one platform you have to code for 5 and also convince Apple and Google to accept your app (Nextcloud is really feeling this one).

Meanwhile HTML + JavaScript works on most smart fridges.

Tldr:

Rootful podman with podman run --userns=auto is more secure than one rootless host user running many pods, because those pods could (theoretically) attack each other.
though you still have the possibility of an exploit in the image pull

Rootless podman running one pod (as in service including database and so on) per host user with different subuid Ranges is the most secure, but you have to actually set that up which can be a lot of work depending on distribution.

^{We’re a smaller chamber but a lot more echo.}

Somethign I haven’t seen mentioned yet is clevis and tang, basically if you have more than one server then they can unlock each other and if they’re spatially separated then it is very unlikely they get stolen at the same time.

Though you have to make sure it stops working when a server get stolen, using a mesh VPN works just as well after the server is stolen so either use public IPS and a VPN or use a hidden raspberry pi that is unlikely to be stolen or make the other server stop tang after the first one is stolen.

Luckely we’re not relying on emails for security relevant and or private information, right?

The emails are unencrypted, emails in transit are in transit between the e-mail servers and relays and use secure tls channels.
They are only encrypted from your phone/notebook/browser to the server, then when send they will be encrypted till the next server.

Every server/relay first decrypts everything send to it, because it has to due to the TLS terminating at each server.

See also your source:

Transport Encryption: This form of encryption is used to secure your emails while they are transmitted over the internet. Most of today’s email services, including Gmail, employ transport layer security (TLS) to protect emails in transit. While it encrypts emails between servers, it doesn’t protect the content once it reaches the recipient’s inbox.¹

In practical terms, Your e-mail server, your e-mail servers relay (if it has any) and your recipients relay server/server can all read your email unless

End-to-End Encryption (E2EE): E2EE takes encryption a step further. It ensures that only the sender and the recipient can decrypt and read the emails. Even the email service provider cannot access the contents of the email. E2EE is typically achieved through third-party encryption tools or services.¹

Which takes active effort from both the sender and the recipient to make work - it’s almost only possible with people you know and little else.

¹ https://umatechnology.org/gmails-new-encryption-can-make-email-safer-heres-why-you-should-use-it/