test Rook / Ceph for better HA #85
Labels
No labels
bug
cleanup
duplicate
enhancement
forgefriends
help wanted
hetzner
invalid
label workflow
need more info
question
refactor
static-site
sync
wontfix
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
infrastructure/k8s-cluster#85
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Currently we use local / nfs storage, which is replicated bewteen hetzner05 and hetzner06.
There are manual steps to move that to a different node.
We should try Rook1 / Ceph on our three node cluster and check if it has enough performance and resilience.
https://rook.io ↩︎
we can use
/dev/nvme1n1p5as storage volume, it's only used on hetzner06 for backups, which can be moved tohetzner03:/dev/nvme0n1(which is totally unused)Another alternate is to use a three node
glusterfsonxfs. We use it at work and it's pretty stable.This would also allow to fail a node without much interruption.
All nodes would be equally and we can use all nodes for scheduling.
I agree it should be tested. But testing anything on hardware that is in production does not strike me as prudent.
@earl-warren wrote in https://invisible.forgejo.org/infrastructure/k8s-cluster/issues/662#issuecomment-5829:
the good thing is that we can test it without interrupting running services. we can simply use a copy of forgejo-code to test performance
I'll be much more comfortable if we've such a replicated system, because a downtime of a node will only cause a few seconds of service outages (after reacting of an admin of cause)
@viceice wrote in https://invisible.forgejo.org/infrastructure/k8s-cluster/issues/662#issuecomment-5831:
What worries me most is what needs to be installed on the existing machines and the potential impact on the cluster. It involves kernel modules, new services and they all require a specific network setup.
Me too. As much as I like the robustness and reliability of the current system, the need for manual intervention when storage fails makes it more difficult to deal with in times of crisis.
Here is an idea.
Three new machines are rented to prepare for this with zero risk for the current cluster. They can be setup as three entirely new nodes. And use an entirely different storage network (i.e. not the current VLAN dedicated to DRBD/NFS, two new ones for the client side and backend side). It will need different firewall rules too. The other VLAN dedicated to k8s will be shared between all nodes. This is a level of compartmentalization that sounds very secure, even during the early stages of experimentation.
When these new nodes are operating in a way that feels reliable and fast enough (can't be slower than NFS over DRBD over two countries, that's a given 😁), deployments can be migrated to it, gradually. With the option to move the deployment back to the other nodes should something go sideways for some reason.
Assuming this can be completed over three months calendar time, the cost would be €450 and an estimate of time in the range of 100h / ten days and 200h / twenty days of work. You work a lot faster than I do so maybe less if you are the one doing most of the work. In any case I don't believe this is something that can fit in less than 50h / five days of work between installation, documentation, experimentation, mistakes, getting used to new problems, bumping into dead ends, monitoring, stabilizing, finalizing, decommissioning the old nodes. So even in the most optimistic case at a 60€ / hour rate that's 3,000€ of human labor and more likely around 6,000€ altogether.
I can cover the costs of these additional nodes for the period. However I cannot spend the time to lead this effort: it is too much time given my availability and what I am set to do before I go on sabbatical 1 January 2026. But I can certainly be in the passenger seat.
If you are willing to work on that sooner rather than wait 2027, I think choosing the technology stack you are most familiar with (glusterfs) is the most sensible choice. It makes a world of difference if you are familiar with how that can be setup and repaired.
rook / ceph would not be installed on the host. they would run as pods1 inside the cluster, so not much headaches to think about 😉
glusterfs would need some system packages2. the daemons are running in userspace, so no kernel issues. It's primary an overlay filesystem over multiple normal ext4 / xfs file systems. So i also see no big deals to care.
We can of cause build the glusterfs in 3 new servers and then add those as new k3s nodes when they're running.
We can mask them with custom taints, and transfer some workloads to validate.
Benefit of extending: all manual k8s secrets are already there.
When all runs fine, then we can simple move the floating ip's and decomission the old servers.
https://rook.io/docs/rook/latest-release/Helm-Charts/operator-chart ↩︎
https://packages.debian.org/bookworm/glusterfs-server ↩︎
if we fully separate them, then we (me) should do #242 first
@viceice wrote in https://invisible.forgejo.org/infrastructure/k8s-cluster/issues/662#issuecomment-5850:
Interesting! I did not know it was even possible. In that case my concerns do not apply.
The way I understand it, this will still need to load the ceph/rbd kernel module to mount the ceph block device that a Ceph backed pod will use. However, given that those drivers have been stable and in the kernel for the past ten years or so, I think it is not too much of a risk.
I feel very good about this 😁
P.S. Codeberg also has some Ceph experience which may help figure out the gory details.
The partitions currently mounted on /srv can be emptied and used for Ceph. Upgrade backups can be sent to hetzner01 which has a lot of free space and has a connection fast enough that it does not make a lot of difference compared to local storage. In the context of recovering files that is.
@earl-warren wrote in https://invisible.forgejo.org/infrastructure/k8s-cluster/issues/662#issuecomment-5855:
also the second disk on hetzer03 is also not used.
if ceph works fine, then i plan to use the second disk of each server as additional storage for hopefully better performance.
hetzner05
hetzner06
hetzner03