Mitigate the Docker Dilemma with a Proxy Cache

For the better part of a decade, we developers and operators have relied on the Docker Hub and their image repository for our needs. This has been great for the growth of the technology, but this growth has necessitated a change to how Docker makes money. As such, Docker is rate limiting anonymous and authenticated users.

For the average individual developer who is building software, there is likely little impact to their daily work. But for those of us running Kubernetes and applications at scale, this change will likely break our larger clusters, especially those that use the AlwaysImagePullPolicy.

As organizations that are air gapping their setups, this is of little consequence. They’re already accustomed to regularly updating their internal repositories. But, for those of us who need to, we need a way to quickly and easily serve images to our clusters and developers and limit the amount of traffic to Docker Hub. A proxy cache is a great tool to effectively accomplish this goal.

What is a Proxy Cache?

An image proxy cache is a feature of many container image repository server technologies. It allows servers and developers to pull images from a centralized location, and uses upstream connections to upstream image repositories such as Docker Hub to make sure the local copy of the image is up to date. If it is, no pull request is made to the upstream repository, so the server or developer’s pull does not count against the upstream server's rate limit. If the upstream image has changed, it is add pulled once to the local proxy cache and then delivered to the server or developer requesting it.

Setting up a proxy cache is technology specific and outside the scope of this blog post, but know there are many options available. Here are a few examples.

• Harbor
• Docker Registry
• Nexus
• Artifactory

At this point, it’s just a matter of setting up your Kubernetes clusters to use a private repository. Let’s take a look at how to do that with RKE, K3s and RKE Government.

Setup RKE to Use an Image Proxy Cache

Per the RKE Documentation you’ll use the private_registries setting to set up each private registry by its url. An example of this might look like this.

  private_registries:
      - url: mydockermirror.local
        user: username
        password: password
      - url: myquaymirror.local
        user: myusername
        password: mypassword

If you’re using the Terraform RKE Provider, you would use the private_registries argument of the cluster resource.

The same example from above would look like this.

  private_registries {
    url      = "mydockermirror.local"
    user     = "username"
    password = "password"
  }
  private_registries {
    url      = "myquaymirror.local"
    user     = "myusername"
    password = "mypassword"
  }

Set Up K3s to Use an Image Proxy Cache

Per the K3s Documentation, each node in a K3s cluster will need a registries.yaml configuration file at /etc/rancher/k3s/registries.yaml. This includes both the server and the agent, if the server is not tainted to prevent deployments. This registries file allows for multiple endpoints per url. This allows you to specify the local registry, but you could include the public endpoint as a backup. In addition, if the local registry is secured via TLS, you may need to specify certificates and use https as part of the URL. Here’s the above registry example, configured for K3s. We’ll assume the quay mirror is unsecured for this example.

  mirrors:
    docker.io:
      endpoint:
        - "https://mydockermirror.local"
        - "https://docker.io"
    quay.io:
      endpoint:
        - "http://myquaymirror.local"
        - "https://quay.io"
  configs:
    "mydockermirror.local":
      auth:
        username: username
        password: password
      tls:
        cert_file: /mycerts/client.crt
        key_file:  /mycerts/clientkey.crt
        ca_file:   /mycerts/ca.crt
    "docker.io":
      auth:
        username: dockerhubusername
        password: dockerhubpassword
    "myquaymirror.local":
      auth:
        username: myusername
        password: mypassword
    "quay.io":
      auth:
        username: quayiousername
        password: quayiopassword

Set Up RKE Government to Use an Image Proxy Cache

Per the RKE Government/Documentation, the method is nearly identical to the method for K3s. Each node in an RKE Government cluster will need a registries.yaml configuration file at /etc/rancher/rke2/registries.yaml. Like a K3s cluster, if the server is not appropriately tainted, then this node will also require a registries file. This registries file allows for multiple endpoints per url. This allows you to specify the local registry, but you could include the public endpoint as a backup. In addition, if the local registry is secured via TLS, you may need to specify certificates and use https as part of the URL. Here’s the above registry example, as it would look for RKE Government. Again, we’ll assume the quay mirror is unsecured for this example.

  mirrors:
    docker.io:
      endpoint:
        - "https://mydockermirror.local"
        - "https://docker.io"
    quay.io:
      endpoint:
        - "http://myquaymirror.local"
        - "https://quay.io"
  configs:
    "mydockermirror.local":
      auth:
        username: username
        password: password
      tls:
        cert_file: /mycerts/client.crt
        key_file:  /mycerts/clientkey.crt
        ca_file:   /mycerts/ca.crt
    "docker.io":
      auth:
        username: dockerhubusername
        password: dockerhubpassword
    "myquaymirror.local":
      auth:
        username: myusername
        password: mypassword
    "quay.io":
      auth:
        username: quayiousername
        password: quayiopassword

Conclusion

Using these methods alongside highly available image repositories may prevent you from over running your image repository rate limits and give you more control of your upstream container image dependencies.

“This publication was prepared or accomplished by the author in a personal capacity. All opinions expressed by the author of this publication are solely their current opinions and do not reflect the opinions of Rancher Federal, Inc., respective parent companies, or affiliates with which the author is affiliated. The author's opinions are based upon information they consider reliable, but neither Rancher Federal, Inc., nor its affiliates, nor the companies with which the author is affiliated, warrant its completeness or accuracy, and it should not be relied upon as such.”