From GCP to Kubernetes
BACKThe challenge
At TrueAbility we provision ephemeral test environments with two or more virtual machines used by candiates to assess their skills in order to get a certificate. Our customers include world-renowned names, such as Google, Elastic, HashiCorp, VMWare and more.
During 2023/2024, we completed the installation of several self-managed Kubernetes clusters worldwide. We decided to migrate the provisioning of the test environment VMs to our new clusters, with the aim of reducing our GCP bill to zero. I was tasked with leading the migration project.
THE PATH TO THE GOAL
Trying to avoid reinventing the wheel: initial investigation of available tools potentially viable to complete the task — none of them was a good fit
Estabilishing a direct communication channel with the KubeVirt dev team (email and Slack) to shed light on some of the initial unknowns
Setting up a sensible local dev environment to test provisioning on k8s, achieved by redirecting the staging domain by leveraging ngrok endpoints and edges — no local dev environment to test provisioning was available before this; in terms of dev speed, this was a game changer!
Converting GCP resources to the k8s equivalents (VPCs, additional disks, etc)
Tweaking original GCP VM golden images to work on k8s
Building a Ruby on Rails service to communicate with the k8s API control plane to dynamically generate all required resources on demand
Implementing VM imaging with VirtualMachineSnapshot and restore with VirtualMachineRestore to be able to restart old test environments when needed
After extensive testing, careful migration of live test environments provisioning to k8s
In a time span of 3 months virtually zeroing our GCP bill 🏁