Introduction
The hypervisor is the secret sauce to running virtual machines. If you’ve ever created an EC2 instance or a Droplet, you’ve leveraged the fruits of a hypervisor. I’ve been wanting to learn about this part of the cloud stack for a while and I finally have a project that enables me to do so. I have some pretty large servers where can take advantage of virtualization to horizontally leverage all of their available resources. The lightweight nature of the kind of virtualization enabled by KVM and QEMU on Linux is what makes VMs and attractive solution to utilizing resources on large servers.
Outline
- Motivation
- Server Specifications
- Kubernetes
- Why bare metal kubernetes is inefficient
- Solution
- Local laptop
- Packer
- virsh
- networking
- kubernetes
Images
I was lucky enough to find chef/bento which has a wealth of Packer build scripts available for use. I had to modify one to get Ubuntu 22.04 (Jammy Jellyfish), but that seemed to be relatively straightforward. Unfortunately, I think that there has been some drift in that repo since when I forked it. That will make merging upstream probably more trouble than it is worth for my project. In future projects, I will take a look at the new repo, it seems like it still includes files for new distributions.
Using my current version, I could build an image with the following command:
export name="kubernetes_worker_001";
PACKER_LOG=1 packer build \
-only=qemu \
-var "name=$name" \
-var "hostname=$name" \
bento/packer_templates/ubuntu/ubuntu-22.04-amd64.json
This produces an image which can be installed. The image is a one-off, you need to build additional images if you want more VMs. And then it can be installed with:
export name="kubernetes_worker_001";
sudo virt-install \
--connect qemu:///system \
--name $name \
--network network=default \
--disk path=/home/ansible/builds/packer-$name/ubuntu-22.04-amd64,device=disk,bus=virtio,format=qcow2 \
--ram 8192 \
--vcpus 2 \
--os-variant ubuntu22.04 \
--sound none \
--rng /dev/urandom \
--virt-type kvm \
--import \
--wait 0
Note: This is pretty strictly tied to the version of Bento I have.
At this point, that’s a computer. You can start, stop, and get a console to it. This is somewhat different to the “golden image” you would use in a cloud environment. The cloud provider will keep that image for you and when you want a new instance, copy it on creation. Here, you are left with an image after build that is runnable, but that could be copied to a new location for other instances. The difference is subtle but important in that if you boot up that image after it is built, it is now different. I made the mistake of trying to install an image several times not realizing that they’re their own computers now.