Introducing "Bottles": A Proof of Concept for Multi-Version CRD Management in Kubernetes
a project by aruiz
Updated 6 months ago. 2 hacker ♥️. 3 followers.

Description

As we delve deeper into the complexities of managing multiple CRD versions within a single Kubernetes cluster, I want to introduce "Bottles" - a proof of concept that aims to address these challenges.

Bottles propose a novel approach to isolating and deploying different CRD versions in a self-contained environment. This would allow for greater flexibility and efficiency in managing diverse workloads.

Goals

  • Evaluate Feasibility: determine if this approach is technically viable, as well as identifying possible obstacles and limitations.
  • Reuse existing technology: leverage existing products whenever possible, e.g. build on top of Kubewarden as admission controller.
  • Focus on Rancher's use case: the ultimate goal is to be able to use this approach to solve Rancher users' needs.

Resources

Core concepts:

  • ConfigMaps: Bottles could be defined and configured using ConfigMaps.
  • Admission Controller: An admission controller will detect "bootled" CRDs being installed and replace the resource name used to store them.
  • Aggregated API Server: By analyzing the author of a request, the aggregated API server will determine the correct bottle and route the request accordingly, making it transparent for the user.

Join this project

Looking for hackers with the skills:

rancher kubernetes poc

This project is part of:

Hack Week 24

Activity

  • 6 months ago: rapetz joined this project.
  • 6 months ago: rapetz liked this project.
  • 6 months ago: moio liked this project.
  • 6 months ago: aruiz started this project.
  • 6 months ago: aruiz added keyword "rancher" to this project.
  • 6 months ago: aruiz added keyword "kubernetes" to this project.
  • 6 months ago: aruiz added keyword "poc" to this project.
  • 6 months ago: aruiz originated this project.


    • Comments

    • aruiz
      5 months ago by aruiz | Reply

      We started the week by creating a rough plan of the areas we wanted to explore, in order to divide the problem into smaller parts and identify further areas of work.

      Rough Plan

      • Create example CRDs that allowed experimenting in our local cluster without breaking it.
        • Nothing really fancy, manually crafted. E.g. copy an existing one from the cluster and just rename it.
      • A Kubernetes controller skeleton to use as the base.
        • Likely based on Kubebuilder/controller-runtime
      • Scripts for bringing up&down a test/dev environment.
      • Explore admission controllers:
        • Can we use kubewarden? at least for some parts?
        • Requirements and possible functional alternatives.
      • Explore APIServices
        • What's the state-of-the-art for building APIServices? Does controller-runtime support it?
        • Is it possible to extract user information from requests? Hopefully without having to terminate auth here.
        • Does it support redirects?

      I talked to Rafa about the project and going over the different areas of exploration.

    • aruiz
      5 months ago by aruiz | Reply

      I started by exploring Kubewarden to check if we could just use a policy to manage the mapping of CRDs being installed along with a Bottle; the idea was to perform this transformation before the CRD is stored in Kubernetes, so we would need to use Admission/Mutating webhooks.

      I concluded that Kuberwarden was not a good fit for this because:

      • Go policies are compiled with TinyGo, due to limited support for WebAssembly in the official toolchain. TinyGo still has some limitations for building the Go standard library, which prevents us from using the k8s.io libraries, which we would need to perform the desired steps, as they will require a Kubernetes client.
      • CEL policies won't provide enough flexibility for our purpose.
      • Rust-based policies could be an option, but probably require a bigger effort to implement (and differ from the language used from the rest of the project).

      I confirmed that Kubebuilder has support for writing defaulting webhooks, so we could use it for modifying the CRDs before they get persisted.

      This framework also creates scripts and resources for building the controller's image, as well as the manifests to install it in Kubernetes. Although it's very focused on adding your own APIs, with some tweaks we could use it to generate such boiler-plate for a built-in type (CRDs).

      These "difficulties" made me think of alternatives approaches for our goal. Assuming that Helm charts is the selected installation mechanism, we could:

      • Create an offline tool whose input is the complete manifests (including the Bottle spec), and produces the required modifications, so that they can be directly applied to Kubernetes.
      • This could be a kubectl (krew) or helm plugin.
      • An offline transformation is a simpler solution, since it moves the processing client-side (allows dry-run, store definitive manifest for GitOps, etc.), while makes the approach less transparent.

    • aruiz
      5 months ago by aruiz | Reply

      Since we had already used a few days, I decided to prioritize exploring the rest of the areas instead of keep building on the admission part, since this was easier to fake in order for the rest to keep working.

      So I started looking into our options for implementing the Aggregation layer in Kubernetes. Besides creating our own CRDs, which is not what we were looking for, the docs suggested 2 options (note that Kubebuilder/controller-runtime is not intended for this use case and have no support for this):

      • Use kubernetes-incubator/apiserver-builder (now named kubernetes-sigs/apiserver-builder-alpha), which aims to provide a similar pattern to Kubebuilder.
        • However, it seems to not be actively maintained (latest release was >2 years ago, no Go modules support, and lack of activity in general.)
      • Use the sample-apiserver project, which seems to be the base for the apiserver-builder generator, but is more up-to-date.
      • Both options seem to build on top of the apiserver-runtime framework, which as I understand is equivalent to controller-runtime for regular controllers. Nonetheless, its last commit was almost one year ago.

      I put some efforts into trying to make apiserver-builder work, including to make the generated project Go-modules aware, but then faced many problem upgrading Kubernetes dependencies to recent versions, so I gave up on that option.

      In this situation, forking sample-apiserver and start modifying it to our needs looked like the best option to go forward.

      This sample project does work, but has very low-level requirements. In particular, it's meant to have access to Etcd itself, in order to serve the APIService. This option was not in our initial plans, as it would make it harder to run our controller. Nonetheless, I also found that the interface implemented does not necessarily have to be backed by Etcd, which brings the option of using a different storage (e.g. any database), as long as the interface methods are implemented. I decided to not pursue this route just yet, though, since it was out of the initial scope and was running out of time. For the sake of the experiment, I tweaked the endpoint to use a Kubernetes client to try to obtain the original data from the main API server and then transforming it. However, this obviously produced an infinite loop, since the control plane would just redirect such requests back to our APIService.

      The last thing that time allowed me to experiment with was the authorization part, as we need to identify which user produced the requests. Even though there is functionality for this, I couldn't manage to make it available to my handlers implementation, and wasn't able to identify why. I need to read more docs about how the workflows for Aggregated APIs, maybe authorization is meant to be resolved by APIServices directly? Sadly, the apiserver-runtime library is not very well documented.

    • Jasbon
      2 months ago by Jasbon | Reply

      This looks like a detailed breakdown of the "Bottles" proof of concept for managing multi-version CRDs in Kubernetes. Are you looking for a summary, feedback, or help with a specific part of the project?

    Similar Projects

    This project is one of its kind!