{"id":20993,"date":"2024-01-24T07:43:04","date_gmt":"2024-01-24T12:43:04","guid":{"rendered":"https:\/\/qxf2.com\/blog\/?p=20993"},"modified":"2024-01-24T07:43:04","modified_gmt":"2024-01-24T12:43:04","slug":"build-a-microvm-using-firecracker","status":"publish","type":"post","link":"https:\/\/qxf2.com\/blog\/build-a-microvm-using-firecracker\/","title":{"rendered":"Build a microVM using Firecracker"},"content":{"rendered":"

This post will cover how to create a custom microVM using Firecracker<\/a>. We will add a simple Flask app to the microVM. By doing so, you will learn how to install custom packages to your own microVM. This post assumes that you have used Firecracker and tried out a basic example already.<\/p>\n

Why this post?<\/h3>\n

AWS Lambda<\/a> stands out as one of our preferred AWS services at Qxf2 Services<\/a>. We use it to develop event-driven applications for internal use, such as sending reminders on our team channel for upcoming holidays, conducting weekly surveys to monitor our technological exposure, and checking for Personal Time Off (PTO) messages on a channel, etc. While we have tested\/explored a few aspects of Lambdas, we had been curious about how serverless environment for AWS Lambda was built.
\nFirecracker<\/a> is a open source virtualization technology built by AWS to optimize running Lambdas, it is a virtual machine monitor that uses Linux Kernel-based Virtual Machine(KVM) to create and manage microVMs.
\nWe decided to explore Firecracker by building a microVM, when we set out on our task, we referred multiple posts to build the microVM, we hit a few issues when configuring network interface for the VM, while debugging we found out the issues were caused by the different commands we were using from different posts. We later identified an useful resource<\/a> in the Firecracker repo and realised with a few changes to a couple of scripts we will be able to create a microVM to suit our purpose in no time, this post is a documentation of how we went about it.<\/p>\n

\n

Creating our custom microVM<\/h3>\n

In this section, we will add our custom Flask application into the microVM. The app chosen is cars-api<\/a>. There is nothing particularly special about the application itself. We are simply using it to illustrate how to:
\n1. Create a custom ext4 filesystem
\n2. Build a kernel
\nto add your own applications into a microVM and then run it using Firecracker.<\/p>\n

The microVM recipe:<\/h5>\n

We used the recipe provided by Firecracker<\/a> to create a microVM, the scripts<\/a> that form the recipe simplifies building a microVM, in a brief summary: the resources\/rebuild.sh<\/a> script copies the contents of the overlay<\/a> to a temporary directory then lays out the filesystem from public.ecr.aws\/ubuntu\/ubuntu:jammy<\/strong> docker image and calls the resources\/chroot.sh<\/a> script, the chroot.sh script then copies the contents of the temporary directory to the filesystem and installs a few packages on it and cedes control back to the resources\/rebuild.sh<\/a> script which then copies the filesystem to the temporary directory and builds an ext4 filesystem image from that directory and then finally builds Linux kernels from source based on these configs<\/a>.
\n\"How<\/a>
\nNote:<\/strong>If you wish to create a microVM from the ground up instead of using a recipe, refer the instructions in Creating Custom rootfs and kernel Images<\/a>.<\/p>\n

Modifications to the recipe to build our customVM:<\/h5>\n
a. Adding packages required:<\/h6>\n

Firecracker is able to boot the microVM really fast by including only packages that are absolutely necessary to run the VM, to run the cars-api service we needed python3.10-venv<\/strong> & python3-pip<\/strong> & git<\/strong> packages installed on the VM.
\nWe modified the resources\/chroot.sh<\/a> file to include these packages<\/p>\n

\r\npackages=\"udev systemd-sysv openssh-server iproute2 curl socat python3-minimal python3.10-venv python3-pip git iperf3 iputils-ping fio kmod tmux hwloc-nox vim-tiny trace-cmd linuxptp\"\r\n<\/pre>\n
b. Creating a Python3.10 virtualenv:<\/h6>\n
We modified the resources\/chroot.sh<\/a> file to add steps to create a Python3.10 virtualenv<\/p>\n
\r\n# Create a venv\r\nmkdir -p \/home\/venv\r\npython3 -m venv \/home\/venv\/cars_api\r\n<\/pre>\n
c. Cloning the cars-api repo and installing the Python packages from requirements file on to the virtualenv:<\/h6>\n
We modified the resources\/chroot.sh<\/a> file again to add steps to create a new projects directory and clone the Test automation repo inside it<\/p>\n
\r\n# Clone cars API repo\r\nmkdir -p \/home\/projects\r\ngit clone https:\/\/github.com\/qxf2\/cars-api.git \/home\/projects\/cars-api\r\nsource \/home\/venv\/cars_api\/bin\/activate\r\npython -m pip install -r \/home\/projects\/cars-api\/requirements.txt\r\ndeactivate\r\n<\/pre>\n
d. Adding cars-api as a systemd service:<\/h6>\n
We added a cars-api<\/a> systemd service file:<\/p>\n
\r\n[Unit]\r\nDescription=A Cars API service\r\nAfter=local-fs.target\r\n\r\n[Service]\r\nExecStart=\/usr\/local\/bin\/cars_api.sh\r\nStandardOutput=inherit\r\nKillMode=process\r\nRestart=always\r\nRestartSec=3\r\n\r\n[Install]\r\nWantedBy=multi-user.target\r\n<\/pre>\n
and its corresponding script – cars_api.sh<\/a>:<\/p>\n
\r\n#!\/bin\/bash\r\n\r\nsource \/home\/venv\/cars_api\/bin\/activate\r\npython \/home\/projects\/cars-api\/cars_app.py\r\n<\/pre>\n
to the overlay<\/a> directory.<\/p>\n
e. Setting up internet access:<\/h6>\n
The fcnet<\/a> systemd service executes fcnet-setup.sh<\/a> script to assign the IP address – 172.16.0.2<\/strong> to the microVM based on the MAC address – 06:00:AC:10:00:02<\/strong>. The MAC address is set when we start the microVM. We modified the fcnet-setup.sh<\/a> script to set gateway and nameserver for the microVM.<\/p>\n
\r\n# Add default gateway\r\nip route add default via 172.16.0.1 dev eth0\r\n# Set nameserver\r\necho \"nameserver 8.8.8.8\" > \/etc\/resolv.conf\r\n<\/pre>\n
You can find the updated script here – fcnet-setup.sh<\/a><\/p>\n
f. Increasing the size of the microVM to accomadate the new changes:<\/h6>\n
The default size of the microVM created using the recipe is 300M<\/strong> to accommodate the new packages installed we modified the resources\/rebuild.sh<\/a> file to increase the size of the VM, increase the size to 400M<\/strong><\/p>\n
\r\n# Default size for the resulting rootfs image is 300M, increase the size to 400M\r\nlocal SIZE=${3:-400M}\r\n<\/pre>\n
Building the filesystem & kernel<\/h5>\n
To build the custom filesystem capable of running the cars-api at startup & kernel we ran the resources\/rebuild.sh<\/a> script.<\/p>\n
\r\nxxxxx@yyyyyyy:\/home\/xxxxx\/projects\/firecracker# .\/resources\/rebuild.sh\r\n...\r\n...\r\n[4.0K]   \/home\/xxxx\/projects\/firecracker\/resources\/aarch64\r\n\u251c\u2500\u2500 [1.8M]  initramfs.cpio\r\n\u251c\u2500\u2500 [400M]  ubuntu-22.04.ext4\r\n\u251c\u2500\u2500 [2.5K]  ubuntu-22.04.id_rsa\r\n\u251c\u2500\u2500 [5.7K]  ubuntu-22.04.manifest\r\n\u251c\u2500\u2500 [ 60M]  ubuntu-22.04.squashfs\r\n\u251c\u2500\u2500 [6.0M]  vmlinux-4.14.336\r\n\u251c\u2500\u2500 [ 47K]  vmlinux-4.14.336.config\r\n\u251c\u2500\u2500 [ 16M]  vmlinux-5.10.208\r\n\u251c\u2500\u2500 [ 56K]  vmlinux-5.10.208.config\r\n\u251c\u2500\u2500 [ 16M]  vmlinux-6.1.74\r\n\u2514\u2500\u2500 [ 62K]  vmlinux-6.1.74.config\r\n\r\n1 directory, 11 files\r\n<\/pre>\n
The resources\/rebuild.sh<\/a> script took ~2.5 hours to complete, it built a ubuntu 22.04 ext4 filesystem & kernels of three different versions by default, to prevent building redundant kernels(we need only one) and to reduce the time it takes for the build script to complete we modified the script to support CLI params to build filesystem or kernel on demand.<\/p>\n
\r\nxxxxx@yyyyyyy:\/home\/xxxxx\/projects\/firecracker# .\/resources\/rebuild.sh -h\r\n\tUsage:\r\n\t\th : Help\r\n\t\tf : Build Ubuntu-22.04 filesystem\r\n\t\tk  : Build kernel of version , supports only 4.14, 5.10, 6.1 for now\r\nxxxxx@yyyyyyy:\/home\/xxxxx\/projects\/firecracker# .\/resources\/rebuild.sh -f -k 6.1\r\n...\r\n...\r\n[4.0K]  \/home\/xxxx\/projects\/firecracker\/resources\/aarch64\r\n\u251c\u2500\u2500 [1.8M]  initramfs.cpio\r\n\u251c\u2500\u2500 [400M]  ubuntu-22.04.ext4\r\n\u251c\u2500\u2500 [2.5K]  ubuntu-22.04.id_rsa\r\n\u251c\u2500\u2500 [6.4K]  ubuntu-22.04.manifest\r\n\u251c\u2500\u2500 [ 87M]  ubuntu-22.04.squashfs\r\n\u251c\u2500\u2500 [ 16M]  vmlinux-6.1.74\r\n\u2514\u2500\u2500 [ 62K]  vmlinux-6.1.74.config\r\n\r\n1 directory, 7 files\r\n+07:51:26 getopts :hfk: option\r\n<\/pre>\n
You can find the updated script here resources\/rebuild.sh that support CLI params<\/a><\/p>\n
\n
How to run the new microVM<\/h3>\n
Running a microVM entails these 2 steps:
\n1. Setting up network interface
\n2. Staring microVM using config file
\nNote:<\/strong> This section assumes you already have Firecracker binary in your system path<\/p>\n
1. Setting up network interface on the host:<\/h5>\n
Firecracker supports only TUN\/TAP<\/a> network, we created a tap device to route traffic from\/to the microVM. A tap<\/a> is a virtual network interface(like an ethernet card), it exposes file descriptor to an application to send\/receive packets, we followed the steps here network-setup<\/a> to create a tap device tap0<\/code> and together with a few iptable rules<\/code> we were able to route the packets written to tap0 to eth0 – the primary network interface on the host.
\nWe had created a network script to help us setup the tap device before running the microVM every time. This is how the script looks:<\/p>\n
\r\n#!\/bin\/bash\r\n# https:\/\/github.com\/firecracker-microvm\/firecracker\/blob\/main\/docs\/getting-started.md\r\n\r\nset -eux\r\n\r\nTAP_DEV=\"tap0\"\r\nTAP_IP=\"172.16.0.1\"\r\nMASK_SHORT=\"\/30\"\r\n\r\n# Setup network interface\r\nsudo ip link del \"$TAP_DEV\" 2> \/dev\/null || true\r\nsudo ip tuntap add dev \"$TAP_DEV\" mode tap\r\nsudo ip addr add \"${TAP_IP}${MASK_SHORT}\" dev \"$TAP_DEV\"\r\nsudo ip link set dev \"$TAP_DEV\" up\r\n\r\n# Enable ip forwarding\r\nsudo sh -c \"echo 1 > \/proc\/sys\/net\/ipv4\/ip_forward\"\r\n\r\n# Set up microVM internet access\r\nsudo iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE || true\r\nsudo iptables -D FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT \\\r\n    || true\r\nsudo iptables -D FORWARD -i tap0 -o eth0 -j ACCEPT || true\r\nsudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE\r\nsudo iptables -I FORWARD 1 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT\r\nsudo iptables -I FORWARD 1 -i tap0 -o eth0 -j ACCEPT\r\n<\/pre>\n
You can find this script here – setup_tap.sh<\/a>
\nThis scripts creates a tap0<\/strong> interface on the host and assigns 172.16.0.1<\/strong> IP address to that device.<\/p>\n
2. Starting microVM using config file:<\/h5>\n
With the ubuntu ext4 filesystem and a kernel at our disposal, the next step is to start the microVM. There are couple of ways to start a microVM:
\na. run the firecracker binary & create a unix socket and then run subsequent API requests to set VM configuration values
\nb. run the firecracker binary with a JSON with VM configuration values<\/p>\n
We used the JSON config file approach, we set the rootfs – ubuntu-22.04.ext4<\/strong> we created, network interface – eth0<\/strong>, MAC – 06:00:AC:10:00:02<\/strong> with a few other values in the JSON file.
\nthis is the JSON configuration file we had used:<\/p>\n
\r\n{\r\n  \"boot-source\": {\r\n    \"kernel_image_path\": \"vmlinux-6.1.72\",\r\n    \"boot_args\": \"ro console=ttyS0 noapic reboot=k panic=1 pci=off\"\r\n  },\r\n  \"drives\": [\r\n    {\r\n      \"drive_id\": \"rootfs\",\r\n      \"path_on_host\": \"ubuntu-22.04.ext4\",\r\n      \"is_root_device\": true,\r\n      \"is_read_only\": false\r\n    }\r\n  ],\r\n  \"network-interfaces\": [\r\n      {\r\n          \"iface_id\": \"eth0\",\r\n          \"guest_mac\": \"06:00:AC:10:00:02\",\r\n          \"host_dev_name\": \"tap0\"\r\n      }\r\n  ],\r\n  \"machine-config\": {\r\n    \"vcpu_count\": 2,\r\n    \"mem_size_mib\": 1024\r\n  }\r\n}\r\n<\/pre>\n
You can find the config file here – vmconfig.json<\/a>
\nWith the filesystem – ubuntu-22.04.ext4<\/strong> and kernel – vmlinux-6.1.72<\/strong> images in the same directory as the vmconfig.json<\/strong>, the following command starts the microVM:<\/p>\n
\r\nfirecracker --config-file vmconfig.json\r\n<\/pre>\n
We waited a couple of secs to have a functional VM on our hands.<\/p>\n
\r\nUbuntu 22.04.3 LTS ubuntu-fc-uvm ttyS0\r\n\r\nubuntu-fc-uvm login: root (automatic login)\r\n\r\nWelcome to Ubuntu 22.04.3 LTS (GNU\/Linux 5.10.208 aarch64)\r\n\r\n * Documentation:  https:\/\/help.ubuntu.com\r\n * Management:     https:\/\/landscape.canonical.com\r\n * Support:        https:\/\/ubuntu.com\/advantage\r\n\r\nThis system has been minimized by removing packages and content that are\r\nnot required on a system that users do not log into.\r\n<\/pre>\n
We were able to validate that the cars_api.service was running inside the microVM:<\/p>\n
\r\nroot@ubuntu-fc-uvm:~# journalctl -u cars_api -f\r\nJan 22 07:03:59 ubuntu-fc-uvm systemd[1]: Started A Cars API service.\r\nJan 22 07:04:02 ubuntu-fc-uvm cars_api.sh[537]:  * Serving Flask app 'cars_app'\r\nJan 22 07:04:02 ubuntu-fc-uvm cars_api.sh[537]:  * Debug mode: off\r\nJan 22 07:04:02 ubuntu-fc-uvm cars_api.sh[537]: WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.\r\nJan 22 07:04:02 ubuntu-fc-uvm cars_api.sh[537]:  * Running on all addresses (0.0.0.0)\r\nJan 22 07:04:02 ubuntu-fc-uvm cars_api.sh[537]:  * Running on http:\/\/127.0.0.1:5000\r\nJan 22 07:04:02 ubuntu-fc-uvm cars_api.sh[537]:  * Running on http:\/\/127.0.0.1:5000\r\nJan 22 07:04:02 ubuntu-fc-uvm cars_api.sh[537]: Press CTRL+C to quit\r\n<\/pre>\n
We were able to fire API queries against the cars_api<\/strong> service running on localhost<\/strong> in port 5000<\/strong> inside the microVM from the host:<\/p>\n
\r\nxxxxx@yyyyyyy:\/home\/xxxxx\/projects# curl -I http:\/\/172.16.0.2:5000\r\nHTTP\/1.1 200 OK\r\nServer: Werkzeug\/3.0.1 Python\/3.10.12\r\nDate: Mon, 22 Jan 2024 07:09:02 GMT\r\nContent-Type: text\/html; charset=utf-8\r\nContent-Length: 10625\r\nConnection: close\r\n<\/pre>\n
To exit the VM run reboot<\/strong> inside it.<\/p>\n
\r\nroot@ubuntu-fc-uvm:~# reboot\r\n[  OK  ] Removed slice Slice \/system\/modprobe.\r\n[  OK  ] Stopped target Graphical Interface.\r\n[  OK  ] Stopped target Multi-User System.\r\n.......\r\n[  751.225980] reboot: Restarting system\r\n2024-01-22T07:16:28.274015407 [anonymous-instance:fc_vcpu 0] Received KVM_SYSTEM_EVENT: type: 2, event: 0\r\n2024-01-22T07:16:28.274107593 [anonymous-instance:main] Vmm is stopping.\r\n2024-01-22T07:16:28.275004801 [anonymous-instance:main] Vmm is stopping.\r\n2024-01-22T07:16:28.285907182 [anonymous-instance:main] RunWithApiError error: MicroVMStopped without an error: Ok\r\nError: RunWithApi(MicroVMStoppedWithoutError(Ok))\r\n<\/pre>\n
\n
How to re-run the microVM<\/h3>\n
To re-run the microVM, delete the unix socket file – \/run\/firecracker.socket<\/strong> the Firecracker binary created last time the microVM was started using the config file<\/p>\n
\r\nrm \/run\/firecracker.socket\r\n<\/pre>\n
Now to start the microVM against, use the same command as last time:<\/p>\n
\r\nfirecracker --config-file vmconfig.json\r\n<\/pre>\n
\n
There you go, a functional micoVM created using the recipe in minutes.<\/p>\n
\n
Hire technical testers from Qxf2<\/h3>\n
This post was written by an experienced tester. Surprised? Qxf2 hires technical testers that continually evolve and keep themselves up to date. Our testers go well beyond traditional “manual” or “automation” testing. We dive deep into things and gel well with engineering teams. If you are looking to hire technical testers and get a fresh perspective on testing, contact us<\/a>.<\/p>\n
\n","protected":false},"excerpt":{"rendered":"
This post will cover how to create a custom microVM using Firecracker. We will add a simple Flask app to the microVM. By doing so, you will learn how to install custom packages to your own microVM. This post assumes that you have used Firecracker and tried out a basic example already. Why this post? AWS Lambda stands out as […]<\/p>\n","protected":false},"author":9,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[407,406],"tags":[],"class_list":["post-20993","post","type-post","status-publish","format-standard","hentry","category-firecracker","category-microvm"],"_links":{"self":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts\/20993","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/users\/9"}],"replies":[{"embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/comments?post=20993"}],"version-history":[{"count":104,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts\/20993\/revisions"}],"predecessor-version":[{"id":21868,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts\/20993\/revisions\/21868"}],"wp:attachment":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/media?parent=20993"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/categories?post=20993"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/tags?post=20993"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}