devctr-image.md

Publishing a New Container Image

About the Container Image

Firecracker uses a Docker container to standardize the build process. This also fixes the build tools and dependencies to specific versions. Every once in a while, something needs to be updated. To do this, a new container image needs to be built locally, then published to the AWS ECR registry. The Firecracker CI suite must also be updated to use the new image.

Prerequisites

1. Access to the fcuvm ECR repository.

2. The docker package installed locally. You should already have this if you've ever built Firecracker from source.

3. Access to both an x86_64 and aarch64 machines to build the container images.

Steps

x86_64

1. Login to the Docker organization in a shell. Make sure that your account has access to the repository:

   aws ecr-public get-login-password --region us-east-1 \
   | docker login --username AWS --password-stdin public.ecr.aws
   

2. Navigate to the Firecracker directory. Verify that you have the latest container image locally.

   docker images
   REPOSITORY                         TAG     IMAGE ID        CREATED         SIZE
   public.ecr.aws/firecracker/fcuvm   v26     8d00deb17f7a    2 weeks ago     2.41GB
   

3. Make your necessary changes, if any, to the Dockerfile(s). There's one for each supported architecture in the Firecracker source tree.

4. Commit the changes, if any.

5. Build a new container image with the updated Dockerfile.

   a: Additionally also checks for any outdated python packages and tries to update them. This makes sure that python packages versions are up to date with latest versions.

    tools/devtool build_devctr
   

   b: Builds a container image but skips performing updates of python packages. The container image will use the locked versions of python packages.

    tools/devtool build_devctr --no-python-package-update
   

6. Verify that the new image exists.

   docker images
   REPOSITORY                         TAG       IMAGE ID         CREATED       SIZE
   public.ecr.aws/firecracker/fcuvm   latest    1f9852368efb     2 weeks ago   2.36GB
   public.ecr.aws/firecracker/fcuvm   v26       8d00deb17f7a     2 weeks ago   2.41GB
   

7. Tag the new image with the next available version and the architecture you're on.

   docker tag 1f9852368efb public.ecr.aws/firecracker/fcuvm:v26_x86_64
   
   docker images
   REPOSITORY                         TAG          IMAGE ID       CREATED
   public.ecr.aws/firecracker/fcuvm   latest       1f9852368efb   1 week ago
   public.ecr.aws/firecracker/fcuvm   v27_x86_64   1f9852368efb   1 week ago
   public.ecr.aws/firecracker/fcuvm   v26          8d00deb17f7a   2 weeks ago
   

8. Push the image.

   docker push public.ecr.aws/firecracker/fcuvm:v27_x86_64
   

aarch64

Login to the aarch64 build machine.

Steps 1-4 are identical across architectures, change x86_64 to aarch64.

Then continue with the above steps:

1. Build a new container image with the updated Dockerfile.

   tools/devtool build_devctr
   

2. Verify that the new image exists.

   docker images
   REPOSITORY                         TAG        IMAGE ID            CREATED
   public.ecr.aws/firecracker/fcuvm   latest     1f9852368efb        2 minutes ago
   public.ecr.aws/firecracker/fcuvm   v26        8d00deb17f7a        2 weeks ago
   

3. Tag the new image with the next available version and the architecture you're on.

   docker tag 1f9852368efb public.ecr.aws/firecracker/fcuvm:v26_aarch64
   
   docker images
   REPOSITORY                         TAG            IMAGE ID
   public.ecr.aws/firecracker/fcuvm   latest         1f9852368efb
   public.ecr.aws/firecracker/fcuvm   v27_aarch64    1f9852368efb
   public.ecr.aws/firecracker/fcuvm   v26            8d00deb17f7a
   

4. Push the image.

   docker push public.ecr.aws/firecracker/fcuvm:v27_aarch64
   

5. Create a manifest to point the latest container version to each specialized image, per architecture.

   docker manifest create public.ecr.aws/firecracker/fcuvm:v27 \
       public.ecr.aws/firecracker/fcuvm:v27_x86_64 public.ecr.aws/firecracker/fcuvm:v27_aarch64
   
   docker manifest push public.ecr.aws/firecracker/fcuvm:v27
   

6. Update the image tag in the devtool script. Commit and push the change.

   PREV_TAG=v26
   CURR_TAG=v27
   sed -i "s%DEVCTR_IMAGE_TAG=\"$PREV_TAG\"%DEVCTR_IMAGE_TAG=\"$CURR_TAG\"%" tools/devtool
   

Troubleshooting

Check out the rust-vmm-container readme for additional troubleshooting steps and guidelines.

I can't push the manifest

docker manifest is only supported when experimental cli features are enabled

See this article for explanations and fix.

How to test the image after pushing it to the Docker registry

Either fetch and run it locally on another machine than the one you used to build it, or clean up any artifacts from the build machine and fetch.

docker system prune -a

docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE

tools/devtool shell
[Firecracker devtool] About to pull docker image public.ecr.aws/firecracker/fcuvm:v15
[Firecracker devtool] Continue?

I don't have access to the AWS ECR registry

docker push public.ecr.aws/firecracker/fcuvm:v27
The push refers to repository [public.ecr.aws/firecracker/fcuvm]
e2b5ee0c4e6b: Preparing
0fbb5fd5f156: Preparing
...
a1aa3da2a80a: Waiting
denied: requested access to the resource is denied

Only a Firecracker maintainer can update the container image. If you are one, ask a member of the team to add you to the AWS ECR repository and retry.

I pushed the wrong tag

Tags can be deleted from the AWS ECR interface.

Also, pushing the same tag twice will overwrite the initial content.

I did everything right and nothing works anymore

If you see unrelated Python errors, it's likely because the dev container pulls Python 3 at build time. Python 3 means different minor versions on different platforms, and is not backwards compatible. So it's entirely possible that docker build has pulled in unwanted Python dependencies.

To include only your changes, an alternative to the method described above is to make the changes inside the container, instead of in the Dockerfile.

Let's say you want to update cargo-audit (random example).

1. Enter the container as root.

   tools/devtool shell -p
   

2. Make the changes locally. Do not exit the container.

   cargo install cargo-audit --force
   

3. Find your running container.

   docker ps
   CONTAINER ID        IMAGE               COMMAND             CREATED
   e9f0487fdcb9        fcuvm:v14       "bash"              53 seconds ago
   

4. Commit the modified container to a new image. Use the container ID.

   docker commit e9f0487fdcb9 fcuvm:v15_x86_64
   

   docker image ls
   REPOSITORY      TAG                 IMAGE ID            CREATED
   fcuvm           v15_x86_64          514581e654a6        18 seconds ago
   fcuvm           v14                 c8581789ead3        2 months ago
   

5. Repeat for aarch64.

6. Create and push the manifest.