Throughout my career there has always been a disconnect between the documentation and the practical implementation. The Kubernetes (k8s) ecosystem is no stranger to this problem. This guide is a simple approach to installing Kubernetes and some REALLY useful tools. We will walk through installing all the following.
We will need a few tools for this guide. Hopefully everything at handled by my air_gap_all_the_things.sh script. For this guide all the commands will be in the script.
For those that like videos, Watch the video.
Table of Contents:
The prerequisites are fairly simple. We need 4 Rocky Linux servers. Centos or Rhel work just as well. Start with 4 core, 8Gb Ram, and 80Gb hard drive. All the servers will need access to a yum repo server. One of the servers will need access to the internet. The other three should be on the other side of the airgap. The servers can be bare metal or your favorite vm of choice. For the video I am going to use Harvester running on a 1u server. We will need an ssh client to connect to the servers. DNS is a great to have but not necessary.
Before you build take a look at all the files in the repo at https://github.com/rancherfederal/rke_airgap_install
Because we are moving bits across an air gap we need a server with access to the internet. Let's ssh into the build server to start the download/build process. There are a few tools we will need like Skopeo and Helm. We will walk through getting everything needed. We will need root for all three servers. The following instructions are going to be high level. The script air_gap_all_the_things.sh will take care of almost everything.
All the steps below are covered in geeky detail in the scripts build function.
/opt/ranchercopy All Images, Including RegistryThere are quite a few steps in building the Tar.
At the time of writing this guide the compressed zst is 5.8G. The output zst is in the /opt/ directory. Time to move it across the air gap.
At a high level we are going to install RKE2 on the first air gapped node. Let's start with copying the zst to the first node, let's call it airgap1. There are some needed packages on all three nodes. This packages as well as come kernel tuning can be found in the air_gap_all_the_things.sh script. We will need to apply the same to all the nodes.
Move the rke2_rancher_longhorn.zst to /opt/. From there we can uncompress it.
yum install -y zstd
mkdir /opt/rancher
tar -I zstd -vxf rke2_rancher_longhorn.zst -C /opt/rancherWe should now see all the files in the /opt/rancher directory.
[root@airgap1 opt]# ls -asl rancher/
total 24
0 drwxr-xr-x. 5 root root 84 Nov 30 15:45 .
0 drwxr-xr-x. 3 root root 54 Nov 30 15:50 ..
20 -rwxr-xr-x. 1 root root 18634 Nov 30 15:28 air_gap_all_the_things.sh
0 drwxr-xr-x. 3 root root 143 Nov 30 15:45 helm
0 drwxr-xr-x. 7 root root 78 Nov 30 15:45 images
4 drwxr-xr-x. 2 root root 4096 Nov 30 15:45 rke2_1.24.8
Fantastic, let's build the first control node.
With everything uncompressed we can now setup the RKE2 on the same, first, node. The high level steps are as follows:
etcd user to the hostlocalhost:5000Of course, there is a script deploy_control function that gives all the commands.
Now that we have the first node running with RKE2 we can turn our attention to the worker nodes. The design of this guide is to use NFS to server out all the files to the cluster from the first node. This will same a ton of time on copying files around. We will also use the NFS for the registry storage. Here are the high level steps. On each of the worker nodes.
mkdir /opt/ranchermount $IP:/opt/rancher/Rinse and Repeat. The in depth commands can be found in the deploy worker function.
Now that we have the cluster built we can focus our attention to Rancher and Longhorn.
For Rancher we are going to use the Helm Chart we imported. The good news is that everything we need is already loaded. The first chart we need to deploy is cert-manager. Cert Manager is used for creating certificates for Rancher. Please pay attention to all the options for the Helm command. We need to make sure we have the correct image locations and chart locations. Thanks to the NFS mount we are sharing the images to all the nodes. This is not meant for production. But it works well for a development/POC environment.
Note that the hostname=rancher.awesome.sauce will need to be changed to reflect your domain/DNS/network.
helm upgrade -i cert-manager /opt/rancher/helm/cert-manager-v1.10.0.tgz
--namespace cert-manager --create-namespace --set installCRDs=true
--set image.repository=localhost:5000/cert-manager-controller
--set webhook.image.repository=localhost:5000/cert-manager-webhook
--set cainjector.image.repository=localhost:5000/cert-manager-cainjector
--set startupapicheck.image.repository=localhost:5000/cert-manager-ctl
helm upgrade -i rancher /opt/rancher/helm/rancher-2.7.0.tgz --namespace cattle-system
--create-namespace --set bootstrapPassword=bootStrapAllTheThings --set replicas=1
--set auditLog.level=2 --set auditLog.destination=hostPath
--set useBundledSystemChart=true
--set rancherImage=localhost:5000/rancher/rancher
--set systemDefaultRegistry=localhost:5000
--set hostname=rancher.awesome.sauceAnd not to disappoint there is a function, rancher, in the script to help with the deployment. Check out the tl:dr section. The script will need to be edited.
./air_gap_all_the_things.sh rancherOnce deployed you can log into you URL with https and start the bootstrapping process. The initial bootstrap password is bootStrapAllTheThings.
Longhorn is a little more simple than Rancher. It is a single Helm Chart. Once again please change the export DOMAIN=awesome.sauce.
helm upgrade -i longhorn /opt/rancher/helm/longhorn-1.3.2.tgz
--namespace longhorn-system
--create-namespace --set ingress.enabled=true
--set global.cattle.systemDefaultRegistry=localhost:5000
--set ingress.host=longhorn.awesome.sauceAlong with Longhorn the Neuvector is a simple Helm install. Once again please change the manager.ingress.host=neuvector.awesome.sauce.
helm upgrade -i neuvector /opt/rancher/helm/core-2.5.5.tgz
--namespace neuvector --create-namespace --set imagePullSecrets=regsecret
--set k3s.enabled=true --set k3s.runtimePath=/run/k3s/containerd/containerd.sock
--set manager.ingress.enabled=true --set controller.pvc.enabled=true
--set controller.pvc.capacity=500Mi --set registry=localhost:5000
--set tag=5.0.5 --set controller.image.repository=neuvector/controller
--set enforcer.image.repository=neuvector/enforcer
--set manager.image.repository=neuvector/manager
--set cve.updater.image.repository=neuvector/updater
--set manager.ingress.host=neuvector..awesome.sauceAgain the script will simplify things.
We are going to use curl to get the script from Github. Keep in mind that the script is always being updated. Again this in the first node that has access to the internet.
mkdir /opt/rancher
cd /opt/rancher
curl -#OL
https://raw.githubusercontent.com/clemenko/rke_airgap_install/main/air_gap_all_the_things.sh
chmod 755 air_gap_all_the_things.shEdit air_gap_all_the_things.sh and validate the versions are correct.
Please be patient as it is pulling 15Gb from the interwebs.
./air_gap_all_the_things.sh buildThe result will be all the files under /opt/rancher/ and the tar that needs to be moved /opt/rke2_rancher_longhorn.zst.
To the first node.
./air_gap_all_the_things.sh controlThis will setup RKE2, deploy the registry and start NFS.
./air_gap_all_the_things.sh workervi ./air_gap_all_the_things.sh./air_gap_all_the_things.sh rancher./air_gap_all_the_things.sh longhorn./air_gap_all_the_things.sh neuvectorAs a nice to have here is a command to validate the images are loaded from the local registry. Note that some of the images are pre-loaded and will look like they were loaded from the internet.
kubectl get pods -A -o jsonpath="{.items[*].spec.containers[*].image}"
| tr -s '[[:space:]]' '\n' |sort | uniq -cWhy yes there is a Dockerfile to run the build phase. Take a look at the Dockerfile in the repo.
At this point we really good foundation for installing RKE2, Neuvector, Longhorn, and Rancher air gapped. The script is meant to be readable if the process needs to be broken down. Case in point if there a registry available internally then load the images there. The Helm commands will need to be changed to point to that registry.
If there are any issues, please feel free to reach out.
“This publication was prepared or accomplished by the author in a personal capacity. All opinions expressed by the author of this publication are solely their current opinions and do not reflect the opinions of Rancher Federal, Inc., respective parent companies, or affiliates with which the author is affiliated. The author's opinions are based upon information they consider reliable, but neither Rancher Federal, Inc., nor its affiliates, nor the companies with which the author is affiliated, warrant its completeness or accuracy, and it should not be relied upon as such.”