Building multiarchitecture aware containers

Multi-architecture is not something that most people think about - everyone assumes that the only architecture that counts is x86_64, but that's not the case!

In my job at Red Hat I care about making sure that software running on ppc64le, arm64 and s390x behaves just the same as it does on x86_64, which brings me to containers.  Did you know that container images are architecture specific?

It makes sense, right?  Containers contain software, and often software is architecture specific.  When you pull down a container, you never specify an architecture to use.  So how does that work?  And more importantly to us developers, how do I ensure my containers are multiarch-aware and "just work" no matter which platform we are running on?

For the purposes of this post, I'm going to be using podman and quay.io, but you could just as easily use docker and dockerhub.  I much prefer the serviceless design of podman, but that's up to you.

[Aside: From here on in I'm going to use the term amd64 in lieu of x86_64.  You can read more about that here.  Likewise you might be wondering why we use arm64 instead of aarch64, you can read about that too.  And if that's not enough, here's a nice link on golang architectures]

A simple application we want to containerise

To begin with, let's build a simple Flask python application.  Something that is super trivial, but can demonstrate software running in a container.  To that end, I present you moo-chop.  It just prints a hello world message in a random colour, like this:

We want to deploy this application via a container on any of {arm64, amd64, ppc64le, s390x}, so how do we do it?

Setting up a container registry

Next we need to create an account on a container registry, and create a repository.  Pretty trivial for the likes of you I'm sure.  I've done this over on quay.io, which you can see here: https://quay.io/repository/mrdaredhat/moo-chop

Setting up our environment

Just for readability, I'll define some environment variable here that you'll need to use on each architecture build host and where you construct your manifest. 

# One time setup

$ ARCH=ppc64le # or amd64, arm64 or s390x

$ QUSER=#your quay.io username

$ QPASS=#your quay.io password

$ PROJECT=moo-chop

$ GITREPO=https://github.com/mrda/$PROJECT.git

Building our containers on each architecture, and pushing them to quay.io

Now we have source code we want to build, and have access to a container repository, we want to build containers for each architecture and push these to the container repository.

The process we follow is exactly the same for each architecture, so I'll only show the steps once.

# Repeat for each architecture you want to support

$ ssh <build-host-for-this-architecture>

# Paste in your environment variables here

$ mkdir -p src

$ cd src

$ git clone $GITREPO

$ cd $PROJECT

# Build your software.  In our case, there's no compiling needed

# as we're only distributing python code.  But if this were C or Go

# or something else, this would be your build step.

$ podman build -t quay.io/$QUSER/$PROJECT:$ARCH --arch $ARCH .

$ podman login quay.io --username $QUSER --password $QPASS

$ podman push quay.io/$QUSER/$PROJECT:$ARCH

^D

And that's one architecture down.  Rinse, lather, repeat for all architectures that you want to build for.

Checking your architecture specific containers on quay.io

The podman push commands that you did in the step above pushed your built containers into the container registry.  You should verify they are all there as expected.  In my case, I can do this by visiting https://quay.io/repository/mrdaredhat/moo-chop?tab=tags and seeing the tag for each container build that is now available.

Building a multiarchecture manifest

We now have built containers for each architecture, and uploaded them to our favourite container registry.  We now need to build a manifest so that the right image is automatically discoverable when we request the base container from the registry.  We do this by building a manifest that links the container images to each architecture.

$ podman login quay.io --username $QUSER --password $QPASS

$ podman manifest create quay.io/$QUSER/$PROJECT:latest

$ podman manifest add quay.io/$QUSER/$PROJECT:latest --arch s390x docker://quay.io/$QUSER/$PROJECT:s390x

$ podman manifest add quay.io/$QUSER/$PROJECT:latest --arch ppc64le docker://quay.io/$QUSER/$PROJECT:ppc64le

$ podman manifest add quay.io/$QUSER/$PROJECT:latest --arch amd64 docker://quay.io/$QUSER/$PROJECT:amd64

# Push the manifest up to quay.io

$ podman manifest push quay.io/$QUSER/$PROJECT:latest docker://quay.io/$QUSER/$PROJECT

Testing it out to see that it all works

Let's examine the manifest to make sure it's multiarchitecture-aware.

$ podman manifest inspect docker://quay.io/$QUSER/$PROJECT:latest | jq '.manifests[]  | .digest, .platform'

"sha256:e9aea7d03e2d6f77aa79ffb395058d68778c72f52cf4264472a86978a6e9d470"

{

  "architecture": "s390x",

  "os": "linux"

}

"sha256:fd1e3f1a05e8c5df91725760241edf8d676c76da7a451457796f41f6e9ea7940"

{

  "architecture": "ppc64le",

  "os": "linux"

}

"sha256:60b2cbbc4fe9becb95c9d27b89b966b12d7fa8029d29928c900651a09abd6a3b"

{

  "architecture": "amd64",

  "os": "linux"

}

Let's try pulling down and starting the container (note: we aren't specifying the architecture in the podman run command.  It determines the correct container image based on architecture and pulls and runs the correct one)

$ podman run --rm -it -p 5000:5000 quay.io/$QUSER/$PROJECT

And that works on any of arm64, amd64, ppc64le, s390x!