I got tired of cloud pricing surprises, so I built my own Kubernetes platform

Reason

I have been using different cloud providers along the years and each and every one of them comes with its own frustration. Either the documentation is not clear, end up paying more than a billing calculator would show. Lots of hidden configurations that can take you down the rabbit hole quite fast. I also had people asking what platform do I recommend to run a kubernetes cluster and I was always struggling suggesting one against the other because there was a catch on every provider. Therefore I decided to build my own hosted kubernetes service called Aether Platform. Aether is built on Hetzner infrastructure since they do not offer managed kubernetes service themselves — Aether is the managed control plane, Ceph/NVMe storage, and networking layer on top.

In this short post I will write down a few reasons why I did choose to go on this path.

DISK
First of all if you need an application that have extensive reads/writes to the disk some of the cloud providers are capping the IOPS to certain amount of disk size. If you need to have higher IOPS you only have one option and that is over provision which also end up costing you a lot more than you wanted to pay.
I chose to build a ceph cluster based on NVME drives and have no limitations on the cluster. On Aether the benchmark that we have run show 1066MiB/s for block storage and 1060MiB/s for the shared storage on Sequential write, 1M qd16 test. For more details feel free to read about it here.

RESOURCE ALLOCATION
Then whenever I needed to choose an instance type I was presented with tens of options some of them better others worse. I always had to go to a certain instance type instead of being able to choose the exact resource amount that I need. Sometimes I needed to go with 32GB RAM when I only needed a few megabytes above the 16G RAM which also increased the per cluster costs because double the ram costs double the money.
In Aether I let customers choose whatever amount of resources it needs without forcing them to deploy a certain package. Of course I have a very limited amount of packages with preconfigured resources that are hugely discounted to get the customers going fast and easy but after that they can create a new node pool with whatever resource allocation they like. Resource cost is transparently written in the webpage and there are no hidden costs whatsoever.

MULTIZONE COST
Most of the cases if I wanted a highly available cluster option I needed to go multi zone option which then if not configured the zone specific traffic it added a hidden inter zone transfer cost which really hit me the wrong way. Also when I configured the loadbalancer to be zone aware I realized that I cannot really do that because if I set preference to the same zone on the backend then the loadbalancer did not sent the traffic to pods located on different zones when the pod on the same zone was not reachable for certain reasons.
In Aether we don't have multi-zone but we have multi region supported. Currently we do not charge any transfer costs - only the loadbalancer price which is €8/month. I hope it will stay this way however it is not entirely dependent on us as the upstream provider (Hetzner) might introduce a cost which will translate into a cost on our side as well.

CONTAINER REGISTRY

It happened to me that I created a container registry and started using it to realize later that months that the cloud provider charged us lots for transfer costs because every time an update for an application or scaling happened triggered by the autoscaler the application pulled the image from the container registry via its public endpoint which was also applying transfer costs.
In Aether you pay for the storage you keep €0.09/GB/month - not for pulling images, no matter how many times the autoscaler pulls the image.