In this blog post I’m going to talk through my recent experiences as I attempted to ditch Docker Desktop — the licensing changes that come into effect at the end of January being the primary motivator.
Without going into any detail about it, let’s just say I’m not a fan of taking something that you’ve made freely available previously and deciding that you now want to charge for it!
In the end I tried t̶h̶r̶e̶e̶ five options for Mac — landing on one as my preference as it covered both the need to run on the newer Apple Silicon and allow mounting of volumes on the host OS, which is something I do fairly frequently (mostly to shorten the feedback loop when testing changes that run on an image intended to run in CI).
Disclaimer: Most of the steps detailed below were found through following other fantastic blog posts I found out there 👏. These are of course noted wherever I’ve used them, with a few tweaks of my own I’ve made on top of these excellent guides. Hopefully having these options together in one blog post is somewhat helpful in choosing between them too!
TL:DR
Updated 17/12/2022: Several months back I made the switch from Rancher Desktop to colima and haven’t looked back. It feels more streamlined and performant with some additional useful configuration options, as well as just less noisy. I’m now updating this post to reflect that colima is now my recommendation — although Rancher Desktop remains a perfectly viable choice (particularly if you prefer a bit of GUI action to configure things!).
The remainder of the article is as it was — charting the various options I tried, but with an elaboration on colima and its benefits below, as the most recent option I’ve switched to.
Sidebar: I also had to solve this problem for my Windows 10 + WSL (Ubuntu) setup. If you’d like to know more about that, I’ve covered it in a similar blog post on my own website — you can read it here.
Recommended Option — Colima
The installation is incredibly straight-forward, just brew install colima . You need the docker CLI installed too (brew install docker) if you don’t have it already also.
Following installation, you then issue colima start when you want to start the daemon, and after that completes, you should find that docker commands work as normal. The first time you do this is a little slower due to downloading the image and configuring it, but following that only takes a few seconds on my machine. You can of course use colima stop to shut it down to save resources on your machine if desired between docker sessions.
I’ve included some further tips n tricks for running it with better compatibility with third party tooling, as well as exploring some of its other options in my own blog site. This link should take you directly to the relevant bit: https://alexos.dev/2022/01/02/docker-desktop-alternatives-for-m1-mac/#recommended-option---colima
Alternatively, see their source on Github for additional configuration options as needed: https://github.com/abiosoft/colima.
Option 1: Docker + Hyperkit + Minikube
I started here. This is a simple near “drop-in” replacement for Docker Desktop, but does not work on M1 Macs. I used this on my older Macbook for a little while before replacing it with Rancher Desktop. It’s fully docker compliant, if there is such a thing.
The instructions that follow are heavily based on this excellent blog post, which has some additional advice, especially if you want to get more out of the local Kubernetes cluster:
# pre-req: full install of XCode needed - just the CLI isn't enough# this steps fails on Apple Silicon: https://github.com/moby/hyperkit/issues/310
brew install hyperkitbrew install docker # don't use --cask
brew install minikube# bring up the daemon - which errors on Apple Silicon
minikube start --driver=hyperkit --keep-context# tells docker in your *current shell* to use minikube's daemon
eval $(minikube docker-env)
I have the minikube start command set up in a start-docker.sh script I can run when needed, and the minikube docker-env in my shell startup (.zshrc, in my case).
As you can see, pretty straight-forward standard brew installation stuff — plus a couple of commands to run before you try and do docker things (I personally never had Docker running all the time on startup anyway, as it was such a battery drain). As it’s still just the same docker CLI, the credentials helper to connect to a private registry also works fine out-the-box.
However, volume mounts from the host did not … but thankfully the blog post I linked above has captured the solution for this. You can minikube mount to spin up a process to mount your local directory into the minikube VM:
minikube mount your-local-directory/:/build >/dev/null 2>&1 &docker run -v /build:/build --rm -it eu.gcr.io/my-private-registry-project/alex-ubuntu:latest
In my opinion, the advantages of this option — and why I kept it as the setup on my older Macbook — are:
- You’re still using the docker CLI, so very good from a compatibility point of view
- There’s minimal extra config/scripts needed — almost a drop-in replacement
- It’s great if you want to do Kubernetes development locally and liked that feature in Docker Desktop
Downsides:
- Doesn’t currently support Apple Silicon (or at least, using the hyperkit driver does not)
- Volume mounts aren’t seamless (although pretty simple tbh)
However, I also needed an option that worked with Apple Silicon. My first attempt was with Podman …
Option 2: Podman
After realising that hyperkit didn’t work on M1, this was the next option I tried. I’d heard good things. It mostly worked fine but, as mentioned earlier, for me the crucial issue was the lack of ability to mount volumes from the host OS. I use this option a lot.
That said, if that’s not important to you or they fix it subsequently, I’ve included the steps below. these were cobbled together from the Podman installation guide itself plus this great blog post — although I didn’t need most of the complexity involved here (I’m guessing it has been fixed since).
From their install guide — things are nice and simple:
brew install podman
podman machine initpodman machine start # bring up the daemon
Your docker equivalents should then work as intended:
# builds a Dockerfile containing a basic nginx image
podman build -t podman-test -f Dockerfile .# run it, exposing port
podman run -d -p 8080:80 podman-test# see the appropriate output from nginx
curl http://localhost:8080/
Other similar docker commands I tend to use also seem present:
podman ps
podman stop <id>
podman exec -i -t <id> /bin/bash
# a subtlety - requires -i -t rather than allowing -itHowever, as mentioned earlier this crucially does not work:
# /build is empty 😞
podman run --rm -it -p 8080:80 -v $(pwd):/build podman-testI looked around this topic a bit and there are some suggested workarounds, such as this one to mount the directory onto the podman VM first. But these look quite hasslesome (caveat: I didn’t try very hard 😉)
I therefore backed away at this point as I had another option to try first … Enter lima + nerdctl …
Option 3: Lima + nerdctl
This option ticked all the boxes for me and I ran with it for a little while without issue, although with more setup needed than the minikube option. I’m comfortable with that though. I like this because it: a) distances me from Docker Inc. changes to licensing in the future (and a little bit on principle, not gonna lie!), and b) puts me closer to the OCI runtime of our Production Kubernetes clusters (they’re GKE, which just run containerd by default now).
I followed the great guide in this blog post, which basically boils down to:
brew install lima
limactl start default # accepted the defaults to setup the VMYour docker equivalents then look like this (which can of course be aliased):
# Dockerfile containing a basic nginx image
lima nerdctl build -t lima-test -f Dockerfile .lima nerdctl run -d -p 8080:80 lima-test curl http://localhost:8080/
Other similar docker commands also seem fine, just like podman:
lima nerdctl ps
lima nerdctl stop <id>
lima nerdctl exec -it <id> /bin/bashCrucially, this worked too without any special config or setup needed:
lima nerdctl run --rm -it -p 8080:80 -v $(pwd):/build --entrypoint=/bin/bash lima-testThat said, there were a couple of other steps I needed to go through to deal with my other requirements.
Because it’s not Docker, the existing credentials helper I had setup to connect to e.g. Google Container Registry did not automatically work. Instead, these credentials need to be readily available on the intermediary lima VM, rather than the host. To solve this, I opted to jump onto the VM and install gcloud, login as I normally would, then ensure those credentials were available to the root user.
To do this, we start with limactl shell default which should get you a shell prompt on your default lima VM. We then download and unpack the GCloud SDK:
# update with your preferred gcloud version
wget --no-verbose -O /tmp/google-cloud-sdk.tar.gz \
https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-367.0.0-linux-x86_64.tar.gz && \
sudo tar -C /opt --keep-old-files -xz -f /tmp/google-cloud-sdk.tar.gz && \
sudo chown $(whoami):$(whoami) /opt && \
sudo chown -R $(whoami):$(whoami) /opt/google-cloud-sdk && \
rm -f /tmp/google-cloud-sdk.tar.gzWe then ensure the required binaries are in the path, and login:
sudo ln -s /opt/google-cloud-sdk/bin/gcloud /usr/bin/gcloud
sudo ln -s /opt/google-cloud-sdk/bin/docker-credential-gcloud /usr/bin/docker-credential-gcloudgcloud auth login # login as normal
gcloud auth configure-docker # will warn about docker path, can ignore
Unfortunately, we’re not quite there yet — but nearly! Back on the host machine, spinning up my Ubuntu docker image was met with an error message I’ve seen a few times before on Apple Silicon: standard_init_linux.go:228: exec user process caused: exec format error. We need to do a bit of work to give QEMU (the hypervisor behind the scenes) the option to execute non-native images.
Thankfully the nerdctl docs point you in the right direction on this one, via this super-useful emulator. We therefore do the following:
limactl shell default # onto the VM again
sudo systemctl start containerdsudo nerdctl run --privileged --rm tonistiigi/binfmt --install all
… et voila! Our ubuntu image built on amd64 in a private container registry with a local host volume mount works without issue 🎉:
lima nerdctl run -v $(pwd):/build --rm -it eu.gcr.io/my-private-gcr-project/alex-ubuntu:latestA small note: if you need that local directory to be writable by the container, you need to edit the file
~/.lima/default/lima.yamlon your host. There’s amounts:section where you can choose to make your home directory and everything in it writable (dodgy if you run untrusted containers!), or add a block to a separate mount path, similar to the one already there for /tmp (the option I took!).
All that’s left is to add the limactl start default to your startup script and alias lima nerdctl to something — you can even alias this to docker if you wish (although I personally prefer to be more explicit — I chose to alias it to nerd 🤘).
In my opinion, the advantages of this option are:
- It works on Apple Silicon! 🎉
- It handles volume mounts seamlessly.
- It distances you from docker itself and any further licensing fun and games down the line.
- For me, it’s closer to my Production container stack.
The downsides:
- It’s not actually docker — so there’s a risk of hitting compatibility issues in comparison to e.g. how things are behaving in CI, perhaps.
- It’s a little bit of a faff to get working with a private container registry in particular.
- It seems to take a bit longer to startup than Minikube / Docker Desktop. This is pretty anecdotal though, and not particularly impactful to me day-to-day.
Option 4 — Rancher Desktop
Updated 14/01/2022: So we’re done right? Lima + nerdctl does the trick? As mentioned back at the top — whilst that was my preferred option for a while, I’ve recently discovered Rancher Desktop, and it’s latest version (0.7+) introduced Apple Silicon support.
This used to be the option I used on both my MacOS machines for quite a while, before switching to colima as described near the top of this article. Rancher remains a perfectly fine choice, with good compatibility.
Why did I use this option over the rest?
- ☑️ Its installation process is very simple. I love a good bit of command line fiddling, but if it’s not necessary and the software keeps itself patched, then it doesn’t have to be.
- ☑️ It works with all my device types — M1 Mac, Intel Mac, Windows 10.
- ☑️ It’s fully compatible with the docker CLI. This is good news for existing build scripts (e.g. if you need to run these or your tests locally, and they depend on being able to call
dockerdirectly — pretty common, and avoids surprises if you aliasednerdctl). - ☑️ Volume mounts to the host OS work without any special configuration or adjustments to scripts / compose files.
- ☑️ It has a local Kubernetes environment, for the odd occasion I find that handy to have.
The installation process is extremely simple with only a few choices to make (it does most of it behind the scenes — and makes use of lima to do it on Mac). You will be prompted to elevate your access a few times, which is understandable given what you’re setting up here.
I opted to use the recommended stable Kubernetes release and the dockerd/moby engine — but I really like that it offered me the choice of containerd (nerdctl) in case that becomes handy in the future (and I understand from the docs that they can coexist too).
I did hit a couple of small issues that were local to my device, and probably a result of my various experiments on this topic! Here’s a couple of quick bits of troubleshooting advice if you need it:
- As a fairly new product, its user experience when things don’t work is not stellar. Logs can be found on Mac here:
~/Library/Logs/rancher-desktop/and provided me with the clues I needed. - It is particular about permissions. For example, the
/optdirectory on my Mac was not owned byroot— the logs above clued me in to this need, as I was just faced with anError Starting Kubernetes — limactl exited with code 1error through the UI. - You must run it from the
/Applicationson your Mac. Copy it over if you didn’t pick the .dmg install option. - You must use versions >0.7.0 for it to work on Apple Silicon.
- Aside: On Windows it needs to reboot a few times. Just roll with it 😏
Summary
I hope you found this article useful — I’ve presented a range of options to show my thought process, and am recommending colima (https://github.com/abiosoft/colima) as the pick, with Rancher Desktop (https://rancherdesktop.io/) as a viable alternative.
Hopefully you found this run through the steps useful for your particular setup. As always with these things — and indeed in my own experience following the existing advice out there — it may not work flawlessly on your kit.
If you find any issues, do let me know via the comments — I’d be interested to keep this post up to date with any additional advice over time too!
