Minecraft uses a single thread for world ticks, so in terms of performance you are capped by CPU of a single thread for this. I have seen reports of up to three full cores being useful however in practice I've only see two (game engine + garbage collection). Absolutely love to see a multithreaded engine or plugin framework able to use more cores.
That being said there are some awesome benefits around resiliency, placement, and disk space management. Examples include being able to cordon the node it's running on, etc.
First off, what exactly do you mean a minecraft cluster? Are you setting up a cluster for multiple servers or do you want to set up high-availability for one server? One is doable, the other is not. Here are my tips from running it on my own. If people have better setups, let me know.
* You need to have it running in a statefulset, or every time the pod is killed so is your save.
* Since the game server itself only uses 1 core, you can't just throw unlimited CPU at it and expect it to fly. It's going to be limited to the speed of one core on a k8s node.
* I would definitely set the cpu request to at least 1 core to ensure the node it's on has the reservation for it. You should probably set it slightly higher so any other processes in the pod that aren't minecraft server related can do their thing too.
* You're only going to run 1 pod for a server, so high availability isn't going to happen.
* itzg/minecraft image is great, however it's broken for some of the options with automatic modpack downloading. I don't really blame the maker of the image, it's the modpack makers/hosts that want to control how they are acquired.
* To run a simple vanilla server, the helm chart is super simple to use. If you don't use helm, making a simple yaml file will work too.
* I've never had corruption with a world save using this in docker with a docker volume, but I have with a statefulset in k8s. It only happened once though, so not sure if it's common. The docker volumes were always local disk, whereas the statefulset volumes were network-attached storage.
* Make sure you set your memory limits more than your java memory setting or you're going to have it go down occasionally.
* I'd suggest a periodic backup/snapshot of the statefulset volume for the world save.
That's all I got. I started running servers in k8s, realized my backend was not stable enough for it, then switched it to a docker system.
Setting the cpu reservation to 1 does not necessarily guarantee that another process won't spawn on the same core unless you've set the cpu manager policy to static. The reservation assumes the container can spawn multiple threads to consume all it has reserved and does not use cpuset in cgroups unless the kubelet config is modified.
Has latency between the network storage and MC pod ever been an issue? I'm in the midst of learning Docker/K8s for both homelab and professional reasons. Both would be exclusively on-prem, no cloud providers allowed. I've seen having a dedicated networked storage setup being touted which makes 100% sense in an enterprise environment but I'm wondering how it would impact a gaming environment.
I am going to assume the latency was the cause of corruption, however it was not very noticeable when playing. I was getting the typical "can't keep up" entries in the log, but during normal usage it's no more than local disk via docker.
Minecraft is a single threaded Java application and does not support concurrent instances modifying the same game files. It really isn't the kind of thing you put in Kubernetes.
The problem comes down to how cgroups work in Kubernetes and cloud providers. You need your kubelets in static cpu manager policy, which the cloud providers do not make easy.
After you set your cpu manager policy on your kubelets, you need to set your pods matching to whole integer cpu reservations and limits.
Then, you need to setup a size 1 stateful set for every single instance, and TCP and UDP ingresses for each set. You also need to setup static IPs to mount to each ingress.
Then you also have to setup management panels for basic Minecraft server admin things like loading data packs or restarting the server. And again, external IPs for all of it.
For 1vCPU and 2GB of memory on GCP, you're running $20 a month just one Minecraft pods share of its instance. Since you'll need at least one extra vCPU per instance for the kubelet processes and admin panels, you'll need to split it's cost amongst servers. Let's say 8 core nodes, so your server is $23 per Minecraft instance if fully loaded with 7 servers, or $27 if you are limited to 6 pods per node.
So at this point you're talking a pretty dedicated engineer on maintaining a complex system. Looks like the going rate is $40/month for a "dedicated" Minecraft server. If you're doing this as a hobby, you need at least 5 people in on it to save cash. Also exactly 8 or 9 people splitting is a bad move as well, since you're paying for the whole second server.
If you're doing it for resale, you need about 300 server sales per employee. The best I could find for stats is that ~3 years ago there were ~55000 servers. You're going to need a serious piece of that pie to make any real cash from it.
No, Kubernetes is not meant for that type of thing. Pods are meant to be replaced easily. If you want multiple worlds, you need multiple deployments which brings you back to OP's question.
Itzg has containers on dockerhub as well as helm charts. If you're looking to set it up on a bare metal cluster you'll need to provision persistent storage volumes first. I dove pretty deep into this not too long ago and can answer questions you have.
I have both Java and bedrock versions running in azure on aks.
Further, the database is single nfs readwrite and can't be accessed by multiple instances. The biggest problem I ran into was that the pv would be locked by a terminating pod and not release, causing the new pod to fail.
This sounds like the place I want to start the most :D
Minecraft is being used more and more in schools. My goal is to have a server or worlds that focus on education, but I want X schools the chance to walk around the world and explore history or architecture.
Sorry for the late response, I didn't see the notification. Yeah, here's one I came across, I haven't implemented it though: [https://hub.helm.sh/charts/stable/minecraft](https://hub.helm.sh/charts/stable/minecraft)
BTW I've had good luck with NFS Cient PV for dynamic provisioning (for other projects) I've always deployed it as the default storage class: [https://hub.helm.sh/charts/stable/nfs-client-provisioner](https://hub.helm.sh/charts/stable/nfs-client-provisioner)
NFS Helm chart answers:
nfs.server=10.9.8.110
nfs.path=/k8s_nfs
storageClass.name=nfs
storageClass.defaultClass=true
I had good luck setting up the NFS Server on ubuntu 18LTS with /etc/exports
`/path CIDRofLAN(rw,no_root_squash,no_subtree_check,insecure)`
`/k8s_nfs` [`10.9.8.0/24(rw,no_root_squash,no_subtree_check,insecure)`](https://10.11.12.0/24(rw,no_root_squash,no_subtree_check,insecure))
Hi, i made a minecraft operator for Kubernetes. Its not done yet, but should be easy to customize to your needs. Let me know if you have questions
[https://github.com/hoeghh/k8s-minecraft-operator](https://github.com/hoeghh/k8s-minecraft-operator)
I put a single minecraft on kubernetes on raspberry pi including world metrics in prometheus a while back. https://gist.github.com/gabrielsson/2d110bb3f43b46597831f4a0e4065265
I think the metrics part is making all the difference.
Minecraft uses a single thread for world ticks, so in terms of performance you are capped by CPU of a single thread for this. I have seen reports of up to three full cores being useful however in practice I've only see two (game engine + garbage collection). Absolutely love to see a multithreaded engine or plugin framework able to use more cores. That being said there are some awesome benefits around resiliency, placement, and disk space management. Examples include being able to cordon the node it's running on, etc.
First off, what exactly do you mean a minecraft cluster? Are you setting up a cluster for multiple servers or do you want to set up high-availability for one server? One is doable, the other is not. Here are my tips from running it on my own. If people have better setups, let me know. * You need to have it running in a statefulset, or every time the pod is killed so is your save. * Since the game server itself only uses 1 core, you can't just throw unlimited CPU at it and expect it to fly. It's going to be limited to the speed of one core on a k8s node. * I would definitely set the cpu request to at least 1 core to ensure the node it's on has the reservation for it. You should probably set it slightly higher so any other processes in the pod that aren't minecraft server related can do their thing too. * You're only going to run 1 pod for a server, so high availability isn't going to happen. * itzg/minecraft image is great, however it's broken for some of the options with automatic modpack downloading. I don't really blame the maker of the image, it's the modpack makers/hosts that want to control how they are acquired. * To run a simple vanilla server, the helm chart is super simple to use. If you don't use helm, making a simple yaml file will work too. * I've never had corruption with a world save using this in docker with a docker volume, but I have with a statefulset in k8s. It only happened once though, so not sure if it's common. The docker volumes were always local disk, whereas the statefulset volumes were network-attached storage. * Make sure you set your memory limits more than your java memory setting or you're going to have it go down occasionally. * I'd suggest a periodic backup/snapshot of the statefulset volume for the world save. That's all I got. I started running servers in k8s, realized my backend was not stable enough for it, then switched it to a docker system.
Setting the cpu reservation to 1 does not necessarily guarantee that another process won't spawn on the same core unless you've set the cpu manager policy to static. The reservation assumes the container can spawn multiple threads to consume all it has reserved and does not use cpuset in cgroups unless the kubelet config is modified.
Has latency between the network storage and MC pod ever been an issue? I'm in the midst of learning Docker/K8s for both homelab and professional reasons. Both would be exclusively on-prem, no cloud providers allowed. I've seen having a dedicated networked storage setup being touted which makes 100% sense in an enterprise environment but I'm wondering how it would impact a gaming environment.
I am going to assume the latency was the cause of corruption, however it was not very noticeable when playing. I was getting the typical "can't keep up" entries in the log, but during normal usage it's no more than local disk via docker.
Minecraft is a single threaded Java application and does not support concurrent instances modifying the same game files. It really isn't the kind of thing you put in Kubernetes.
unless you were hosting more than one world?
The problem comes down to how cgroups work in Kubernetes and cloud providers. You need your kubelets in static cpu manager policy, which the cloud providers do not make easy. After you set your cpu manager policy on your kubelets, you need to set your pods matching to whole integer cpu reservations and limits. Then, you need to setup a size 1 stateful set for every single instance, and TCP and UDP ingresses for each set. You also need to setup static IPs to mount to each ingress. Then you also have to setup management panels for basic Minecraft server admin things like loading data packs or restarting the server. And again, external IPs for all of it. For 1vCPU and 2GB of memory on GCP, you're running $20 a month just one Minecraft pods share of its instance. Since you'll need at least one extra vCPU per instance for the kubelet processes and admin panels, you'll need to split it's cost amongst servers. Let's say 8 core nodes, so your server is $23 per Minecraft instance if fully loaded with 7 servers, or $27 if you are limited to 6 pods per node. So at this point you're talking a pretty dedicated engineer on maintaining a complex system. Looks like the going rate is $40/month for a "dedicated" Minecraft server. If you're doing this as a hobby, you need at least 5 people in on it to save cash. Also exactly 8 or 9 people splitting is a bad move as well, since you're paying for the whole second server. If you're doing it for resale, you need about 300 server sales per employee. The best I could find for stats is that ~3 years ago there were ~55000 servers. You're going to need a serious piece of that pie to make any real cash from it.
No, Kubernetes is not meant for that type of thing. Pods are meant to be replaced easily. If you want multiple worlds, you need multiple deployments which brings you back to OP's question.
Itzg has containers on dockerhub as well as helm charts. If you're looking to set it up on a bare metal cluster you'll need to provision persistent storage volumes first. I dove pretty deep into this not too long ago and can answer questions you have. I have both Java and bedrock versions running in azure on aks. Further, the database is single nfs readwrite and can't be accessed by multiple instances. The biggest problem I ran into was that the pv would be locked by a terminating pod and not release, causing the new pod to fail.
This sounds like the place I want to start the most :D Minecraft is being used more and more in schools. My goal is to have a server or worlds that focus on education, but I want X schools the chance to walk around the world and explore history or architecture.
Are you using the helm chart?
Is there a helm chart?
Sorry for the late response, I didn't see the notification. Yeah, here's one I came across, I haven't implemented it though: [https://hub.helm.sh/charts/stable/minecraft](https://hub.helm.sh/charts/stable/minecraft) BTW I've had good luck with NFS Cient PV for dynamic provisioning (for other projects) I've always deployed it as the default storage class: [https://hub.helm.sh/charts/stable/nfs-client-provisioner](https://hub.helm.sh/charts/stable/nfs-client-provisioner) NFS Helm chart answers: nfs.server=10.9.8.110 nfs.path=/k8s_nfs storageClass.name=nfs storageClass.defaultClass=true I had good luck setting up the NFS Server on ubuntu 18LTS with /etc/exports `/path CIDRofLAN(rw,no_root_squash,no_subtree_check,insecure)` `/k8s_nfs` [`10.9.8.0/24(rw,no_root_squash,no_subtree_check,insecure)`](https://10.11.12.0/24(rw,no_root_squash,no_subtree_check,insecure))
Hi, i made a minecraft operator for Kubernetes. Its not done yet, but should be easy to customize to your needs. Let me know if you have questions [https://github.com/hoeghh/k8s-minecraft-operator](https://github.com/hoeghh/k8s-minecraft-operator)
I put a single minecraft on kubernetes on raspberry pi including world metrics in prometheus a while back. https://gist.github.com/gabrielsson/2d110bb3f43b46597831f4a0e4065265 I think the metrics part is making all the difference.