With Kubernetes new version 1.31 (https://kubernetes.io/blog/2024/08/13/kubernetes-v1-31-release/) there are so many features releases for which are in alpha, beta and stable state.
1 of the new feature which is can really good in the world on AI and other use case is "Support for image volumes".
I am not going to focus on the different use cases for now, but was trying to test this locally using Minikube.
Env Setup
- Mac (14.1.2)
- Minikube (v1.33.1)
- Driver - docker
- Container Runtime - cri-o(v1.31) hard requirement for now
Minikube Setup
😄 [crio] minikube v1.33.1 on Darwin 14.1.2 (arm64)▪ KUBECONFIG=/Users/kulsharm2/OSB/aks/staaks❗ Specified Kubernetes version 1.31.0 is newer than the newest supported version: v1.30.0. Use `minikube config defaults kubernetes-version` for details.❗ Specified Kubernetes version 1.31.0 not found in Kubernetes version list🤔 Searching the internet for Kubernetes version...✅ Kubernetes version 1.31.0 found in GitHub version list✨ Using the docker driver based on user configuration📌 Using Docker Desktop driver with root privileges👍 Starting "crio" primary control-plane node in "crio" cluster🚜 Pulling base image v0.0.44 ...🔥 Creating docker container (CPUs=2, Memory=2048MB) ...🎁 Preparing Kubernetes v1.31.0 on CRI-O 1.24.6 ...❌ Unable to load cached images: loading cached images: stat /Users/kulsharm2/.minikube/cache/images/arm64/registry.k8s.io/coredns/coredns_v1.10.1: no such file or directory▪ Generating certificates and keys ...▪ Booting up control plane ...▪ Configuring RBAC rules ...🔗 Configuring Calico (Container Networking Interface) ...🔎 Verifying Kubernetes components...▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5🌟 Enabled addons: storage-provisioner, default-storageclass🏄 Done! kubectl is now configured to use "crio" cluster and "default" namespace by default
Check for the Kubernetes version and Container runtime version. You will notice that K8S is v1.31 which we need, but cri-o version is v1.24.6.
$ kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
crio Ready control-plane 15m v1.31.0 192.168.49.2 <none> Ubuntu 22.04.4 LTS 6.6.41-0-virt cri-o://1.24.6
Now to test if you try to test the image volume mounting feature it is going to fail as below:
Test the Image Volume
Pod Manifest
$cat pod.yamlapiVersion: v1kind: Podmetadata:name: image-volumespec:containers:- name: shellcommand: ["sleep", "100m"]image: quay.io/crio/alpine:3.9volumeMounts:- name: volumemountPath: /volumevolumes:- name: volumeimage:reference: quay.io/crio/artifact:v1pullPolicy: IfNotPresent
$kubectl get po
NAME READY STATUS RESTARTS AGEimage-volume 0/1 CreateContainerError 0 72s
Now since, Minikube doesn't provide OOB option to install specific container runtime. I am going to bit do manual setup for cri-o setup with version v1.31. For this you can go through https://github.com/cri-o/cri-o/blob/main/install.md for more details.
Build Cri-o runtime from source code
- Ssh to the Minikube node
- $minikube ssh -p crio (Profile I am using)
- Check for the installed crio version
- root@k8-1:~# crio version
- INFO[2024-08-18 18:10:59.678939615Z] Starting CRI-O, version: 1.24.6, git: 4bfe15a9feb74ffc95e66a21c04b15fa7bbc2b90(clean)
- Version: 1.24.6
- GitCommit: 4bfe15a9feb74ffc95e66a21c04b15fa7bbc2b90
- GitTreeState: clean
- BuildDate: 2023-06-14T14:44:50Z
- GoVersion: go1.18.2
- Compiler: gc
- Platform: linux/arm64
- Linkmode: dynamic
- BuildTags: apparmor, exclude_graphdriver_devicemapper, containers_image_ostree_stub, seccomp
- SeccompEnabled: true
- AppArmorEnabled: false
- Install golang 1.23 which is required (go1.23.0.linux-amd64.tar.gz)
- root@k8-1:~# go version
- -bash: go: command not found
- root@k8-1:~# export GOPATH=/root/go/bin/
- root@k8-1:~# export GOROOT=/root/go/
- root@k8-1:~# export PATH=$PATH:$GOPATH
- root@k8-1:~# go version
- go version go1.23.0 linux/arm64
- Install all the required dependencies
- $apt-get update -qq && apt-get install -y \
- libbtrfs-dev \
- containers-common \
- git \
- libassuan-dev \
- libglib2.0-dev \
- libc6-dev \
- libgpgme-dev \
- libgpg-error-dev \
- libseccomp-dev \
- libsystemd-dev \
- libselinux1-dev \
- pkg-config \
- go-md2man \
- cri-o-runc \
- libudev-dev \
- software-properties-common \
- gcc \
- make
- Clone the cri-o source code
- $git clone https://github.com/cri-o/cri-o (skip SSL verification incase you get issue http.sslVerify=false)
- $cd cri-o
- $make
- $make install
- Verify the cri-o version
- docker@k8-1:~/cri-o$ crio version
- INFO[2024-08-18 19:33:20.276411929Z] Updating config from single file: /etc/crio/crio.conf
- INFO[2024-08-18 19:33:20.276446721Z] Updating config from drop-in file: /etc/crio/crio.conf
- INFO[2024-08-18 19:33:20.276917513Z] Updating config from path: /etc/crio/crio.conf.d
- INFO[2024-08-18 19:33:20.277628471Z] Updating config from drop-in file: /etc/crio/crio.conf.d/01-crio-runc.conf
- INFO[2024-08-18 19:33:20.278858429Z] Updating config from drop-in file: /etc/crio/crio.conf.d/02-crio.conf
- INFO[2024-08-18 19:33:20.279481513Z] Updating config from drop-in file: /etc/crio/crio.conf.d/10-crio.conf
- Version: 1.31.0
- GitCommit: 67290a12649b37b45c3d4de343dbda7668308afb
- GitCommitDate: 2024-08-16T14:35:55Z
- GitTreeState: clean
- BuildDate: 2024-08-18T19:32:19Z
- GoVersion: go1.23.0
- Compiler: gc
- Platform: linux/arm64
- Linkmode: dynamic
- BuildTags:
- containers_image_ostree_stub
- containers_image_openpgp
- seccomp
- selinux
- exclude_graphdriver_devicemapper
- LDFlags: unknown
- SeccompEnabled: true
- AppArmorEnabled: false
- Restart the Minikube to make the changes in effect
- $minikube stop --profile crio
- ✋ Stopping node "crio" ...
- 🛑 Powering off "crio" via SSH ...
- 🛑 1 node stopped.
- $ minikube start -p crio
- 😄 [crio] minikube v1.33.1 on Darwin 14.1.2 (arm64)
- ❗ Specified Kubernetes version 1.31.0 is newer than the newest supported version: v1.30.0. Use `minikube config defaults kubernetes-version` for details.
- ❗ Specified Kubernetes version 1.31.0 not found in Kubernetes version list
- 🤔 Searching the internet for Kubernetes version...
- ✅ Kubernetes version 1.31.0 found in GitHub version list
- ✨ Using the docker driver based on existing profile
- 👍 Starting "crio" primary control-plane node in "crio" cluster
- 🚜 Pulling base image v0.0.44 ...
- 🏃 Updating the running docker "crio" container ...
- 🎁 Preparing Kubernetes v1.31.0 on CRI-O 1.31.0 ...
- ❌ Unable to load cached images: loading cached images: stat /Users/kulsharm2/.minikube/cache/images/arm64/registry.k8s.io/pause_3.9: no such file or directory
- 🔎 Verifying Kubernetes components...
- ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
- 🌟 Enabled addons: storage-provisioner, default-storageclass
- 🏄 Done! kubectl is now configured to use "crio" cluster and "default" namespace by default
- Verify the Kubernetes version and Container runtime version
- $ kubectl get node -o wide
- NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
- crio Ready control-plane 15m v1.31.0 192.168.49.2 <none> Ubuntu 22.04.4 LTS 6.6.41-0-virt cri-o://1.31.0
Test the Image Volume
pod manifest
$cat pod.yamlapiVersion: v1kind: Podmetadata:name: image-volumespec:containers:- name: shellcommand: ["sleep", "infinity"]image: quay.io/crio/alpine:3.9volumeMounts:- name: volumemountPath: /volumevolumes:- name: volumeimage:reference: quay.io/crio/artifact:v1pullPolicy: IfNotPresent$kubectl apply -f pod1.yamlpod/image-volume created
Check the pod status
$kubectl get poNAME READY STATUS RESTARTS AGEimage-volume 1/1 Running 0 15s
Verify the volume attached on the container
$kubectl exec -it image-volume -- df -h /volumeFilesystem Size Used Available Use% Mounted onoverlay 97.9G 30.8G 62.1G 33% /volume$kubectl exec -it image-volume -- ls -l /volumetotal 8drwxr-xr-x 2 1000 users 4096 Jun 18 09:02 dir-rw-r--r-- 1 1000 users 2 Jun 18 09:02 file