From GCP to Kubernetes
BACKThe challenge
At TrueAbility we provision ephemeral test environments with two or more virtual machines used by candidates to assess their skills in order to get a certificate. Our customers include world-renowned names, such as Google, Elastic, HashiCorp, VMWare and more.
During 2023/2024, we completed the installation of several bare metal Kubernetes clusters worldwide, while simultaneously developing our own Heroku-like Platform-as-a-Service (PaaS) solution, called Build.io. We decided to migrate the provisioning of the VMs there with the aim of reducing our GCP bill to zero. I was tasked with leading the migration project.
The path to the goal
Initial investigation of available tools potentially viable to complete the task such as kubeclient and k8s-ruby. Having encountered some compatibility issues with the Kubernetes version we were using and how Build.io works, we decided to build our own.
Outlined the YAML definition for various resources to rebuild a POC of what TrueAbility does on GCP: virtualmachine,loadbalancer, networkpolicy, virtualmachinesnapshot,virtualmachinerestore, are the key resources we needed to define.
Instead of using images, converted existing images to docker images and pushed them to our internal registry. Paired with the datavolume resource type, it allowed us to have an easier way to handle image updates.
Establishing a direct communication channel with the KubeVirt dev team to shed light on some of the initial unknowns. Shoutout to their amazing support!
Building a Ruby service to communicate with the k8s API control plane to dynamically generate all required resources on demand.
Successfully reduced our GCP bill by approximately $12,000 per month, achieving our zero-cost target.