Kubernetes 1.31 || Testing the Image Volume mount feature using Minikube

With Kubernetes new version 1.31 (https://kubernetes.io/blog/2024/08/13/kubernetes-v1-31-release/) there are so many features releases for which are in alpha, beta and stable state.

1 of the new feature which is can really good in the world on AI and other use case is "Support for image volumes". 

I am not going to focus on the different use cases for now, but was trying to test this locally using Minikube.

Env Setup 

• Mac (14.1.2)
• Minikube (v1.33.1)
• Driver - docker
• Container Runtime - cri-o(v1.31) hard requirement for now

Minikube Setup

$minikube start --feature-gates=ImageVolume=true --driver=docker     --nodes 1     --cni calico     --cpus=2     --memory=4g     --kubernetes-version=v1.31.0     --container-runtime=cri-o     --profile crio

😄  [crio] minikube v1.33.1 on Darwin 14.1.2 (arm64)

    ▪ KUBECONFIG=/Users/kulsharm2/OSB/aks/staaks

❗  Specified Kubernetes version 1.31.0 is newer than the newest supported version: v1.30.0. Use `minikube config defaults kubernetes-version` for details.

❗  Specified Kubernetes version 1.31.0 not found in Kubernetes version list

🤔  Searching the internet for Kubernetes version...

✅  Kubernetes version 1.31.0 found in GitHub version list

✨  Using the docker driver based on user configuration

📌  Using Docker Desktop driver with root privileges

👍  Starting "crio" primary control-plane node in "crio" cluster

🚜  Pulling base image v0.0.44 ...

🔥  Creating docker container (CPUs=2, Memory=2048MB) ...

🎁  Preparing Kubernetes v1.31.0 on CRI-O 1.24.6 ...

❌  Unable to load cached images: loading cached images: stat /Users/kulsharm2/.minikube/cache/images/arm64/registry.k8s.io/coredns/coredns_v1.10.1: no such file or directory

    ▪ Generating certificates and keys ...

    ▪ Booting up control plane ...

    ▪ Configuring RBAC rules ...

🔗  Configuring Calico (Container Networking Interface) ...

🔎  Verifying Kubernetes components...

    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5

🌟  Enabled addons: storage-provisioner, default-storageclass

🏄  Done! kubectl is now configured to use "crio" cluster and "default" namespace by default

Check for the Kubernetes version and Container runtime version. You will notice that K8S is v1.31 which we need, but cri-o version is v1.24.6. 

$ kubectl get node -o wide

NAME      STATUS   ROLES           AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION   CONTAINER-RUNTIME

crio   Ready    control-plane   15m   v1.31.0   192.168.49.2   <none>        Ubuntu 22.04.4 LTS   6.6.41-0-virt    cri-o://1.24.6

Now to test if you try to test the image volume mounting feature it is going to fail as below:

Test the Image Volume 

Pod Manifest

$cat pod.yaml

apiVersion: v1

kind: Pod

metadata:

  name: image-volume

spec:

  containers:

  - name: shell

    command: ["sleep", "100m"]

    image: quay.io/crio/alpine:3.9

    volumeMounts:

    - name: volume

      mountPath: /volume

  volumes:

  - name: volume

    image:

      reference: quay.io/crio/artifact:v1

      pullPolicy: IfNotPresent

$kubectl get po

NAME           READY   STATUS                 RESTARTS   AGE

image-volume   0/1     CreateContainerError   0          72s

If describe the pod then you will see below error:

    Warning  Failed     2s (x2 over 15s)   kubelet            Error: mount.HostPath is empty

Now since, Minikube doesn't provide OOB option to install specific container runtime. I am going to bit do manual setup for cri-o setup with version v1.31. For this you can go through https://github.com/cri-o/cri-o/blob/main/install.md for more details.

Build Cri-o runtime from source code

• Ssh to the Minikube node
• $minikube ssh -p crio (Profile I am using)
• Check for the installed crio version
• root@k8-1:~# crio version
• INFO[2024-08-18 18:10:59.678939615Z] Starting CRI-O, version: 1.24.6, git: 4bfe15a9feb74ffc95e66a21c04b15fa7bbc2b90(clean)
• Version:          1.24.6
• GitCommit:        4bfe15a9feb74ffc95e66a21c04b15fa7bbc2b90
• GitTreeState:     clean
• BuildDate:        2023-06-14T14:44:50Z
• GoVersion:        go1.18.2
• Compiler:         gc
• Platform:         linux/arm64
• Linkmode:         dynamic
• BuildTags:        apparmor, exclude_graphdriver_devicemapper, containers_image_ostree_stub, seccomp
• SeccompEnabled:   true
• AppArmorEnabled:  false
• Install golang 1.23 which is required (go1.23.0.linux-amd64.tar.gz)
• root@k8-1:~# go version
• -bash: go: command not found
• root@k8-1:~# export GOPATH=/root/go/bin/
• root@k8-1:~# export GOROOT=/root/go/
• root@k8-1:~# export PATH=$PATH:$GOPATH
• root@k8-1:~# go version
• go version go1.23.0 linux/arm64
• Install all the required dependencies
• $apt-get update -qq && apt-get install -y \
•   libbtrfs-dev \
•   containers-common \
•   git \
•   libassuan-dev \
•   libglib2.0-dev \
•   libc6-dev \
•   libgpgme-dev \
•   libgpg-error-dev \
•   libseccomp-dev \
•   libsystemd-dev \
•   libselinux1-dev \
•   pkg-config \
•   go-md2man \
•   cri-o-runc \
•   libudev-dev \
•   software-properties-common \
•   gcc \
•   make
• Clone the cri-o source code
• $git clone https://github.com/cri-o/cri-o (skip SSL verification incase you get issue http.sslVerify=false)
• $cd cri-o
• $make
• $make install
• Verify the cri-o version
• docker@k8-1:~/cri-o$ crio version
• INFO[2024-08-18 19:33:20.276411929Z] Updating config from single file: /etc/crio/crio.conf
• INFO[2024-08-18 19:33:20.276446721Z] Updating config from drop-in file: /etc/crio/crio.conf
• INFO[2024-08-18 19:33:20.276917513Z] Updating config from path: /etc/crio/crio.conf.d
• INFO[2024-08-18 19:33:20.277628471Z] Updating config from drop-in file: /etc/crio/crio.conf.d/01-crio-runc.conf
• INFO[2024-08-18 19:33:20.278858429Z] Updating config from drop-in file: /etc/crio/crio.conf.d/02-crio.conf
• INFO[2024-08-18 19:33:20.279481513Z] Updating config from drop-in file: /etc/crio/crio.conf.d/10-crio.conf
• Version:        1.31.0
• GitCommit:      67290a12649b37b45c3d4de343dbda7668308afb
• GitCommitDate:  2024-08-16T14:35:55Z
• GitTreeState:   clean
• BuildDate:      2024-08-18T19:32:19Z
• GoVersion:      go1.23.0
• Compiler:       gc
• Platform:       linux/arm64
• Linkmode:       dynamic
• BuildTags:
•   containers_image_ostree_stub
•   containers_image_openpgp
•   seccomp
•   selinux
•   exclude_graphdriver_devicemapper
• LDFlags:          unknown
• SeccompEnabled:   true
• AppArmorEnabled:  false
• Restart the Minikube to make the changes in effect
• $minikube stop --profile crio
• ✋  Stopping node "crio"  ...
• 🛑  Powering off "crio" via SSH ...
• 🛑  1 node stopped.
• $ minikube start -p crio
• 😄  [crio] minikube v1.33.1 on Darwin 14.1.2 (arm64)
• ❗  Specified Kubernetes version 1.31.0 is newer than the newest supported version: v1.30.0. Use `minikube config defaults kubernetes-version` for details.
• ❗  Specified Kubernetes version 1.31.0 not found in Kubernetes version list
• 🤔  Searching the internet for Kubernetes version...
• ✅  Kubernetes version 1.31.0 found in GitHub version list
• ✨  Using the docker driver based on existing profile
• 👍  Starting "crio" primary control-plane node in "crio" cluster
• 🚜  Pulling base image v0.0.44 ...
• 🏃  Updating the running docker "crio" container ...
• 🎁  Preparing Kubernetes v1.31.0 on CRI-O 1.31.0 ...
• ❌  Unable to load cached images: loading cached images: stat /Users/kulsharm2/.minikube/cache/images/arm64/registry.k8s.io/pause_3.9: no such file or directory
• 🔎  Verifying Kubernetes components...
•     ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
• 🌟  Enabled addons: storage-provisioner, default-storageclass
• 🏄  Done! kubectl is now configured to use "crio" cluster and "default" namespace by default
• Verify the Kubernetes version and Container runtime version
• $ kubectl get node -o wide
• NAME      STATUS   ROLES           AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION   CONTAINER-RUNTIME
• crio   Ready    control-plane   15m   v1.31.0   192.168.49.2   <none>        Ubuntu 22.04.4 LTS   6.6.41-0-virt    cri-o://1.31.0

Test the Image Volume

pod manifest 

$cat pod.yaml

apiVersion: v1

kind: Pod

metadata:

  name: image-volume

spec:

  containers:

  - name: shell

    command: ["sleep", "infinity"]

    image: quay.io/crio/alpine:3.9

    volumeMounts:

    - name: volume

      mountPath: /volume

  volumes:

  - name: volume

    image:

      reference: quay.io/crio/artifact:v1

      pullPolicy: IfNotPresent

$kubectl apply -f pod1.yaml

pod/image-volume created

 Check the pod status

$kubectl get po

NAME           READY   STATUS    RESTARTS   AGE

image-volume   1/1     Running   0          15s

 Verify the volume attached on the container

$kubectl exec -it image-volume -- df -h /volume

Filesystem                Size      Used Available Use% Mounted on

overlay                  97.9G     30.8G     62.1G  33% /volume

$kubectl exec -it image-volume -- ls -l /volume

total 8

drwxr-xr-x    2 1000     users         4096 Jun 18 09:02 dir

-rw-r--r--    1 1000     users            2 Jun 18 09:02 file