Comments

• 

  11 months ago by aruiz | Reply

  We started the week by creating a rough plan of the areas we wanted to explore, in order to divide the problem into smaller parts and identify further areas of work.

  Rough Plan

  • Create example CRDs that allowed experimenting in our local cluster without breaking it.
    • Nothing really fancy, manually crafted. E.g. copy an existing one from the cluster and just rename it.
  • A Kubernetes controller skeleton to use as the base.
    • Likely based on Kubebuilder/controller-runtime
  • Scripts for bringing up&down a test/dev environment.
  • Explore admission controllers:
    • Can we use kubewarden? at least for some parts?
    • Requirements and possible functional alternatives.
  • Explore APIServices
    • What's the state-of-the-art for building APIServices? Does controller-runtime support it?
    • Is it possible to extract user information from requests? Hopefully without having to terminate auth here.
    • Does it support redirects?

  I talked to Rafa about the project and going over the different areas of exploration.

×
Edit Comment 9379
# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

- Use hyphens
- for unordereed
- lists

This is an [link to example.com](http://example.com/)

This is an image ![an openSUSE geeko icon](https://en.opensuse.org/images/d/d0/Icon-distribution.png)

This is a user link @hans

This is a project link hw#some-cool-title
More Complex Markdown Help
Formatting Help
• Edit
• Preview
We started the week by creating a rough plan of the areas we wanted to explore, in order to divide the problem into smaller parts and identify further areas of work. #### Rough Plan - Create example CRDs that allowed experimenting in our local cluster without breaking it. - Nothing really fancy, manually crafted. E.g. copy an existing one from the cluster and just rename it. - A Kubernetes controller skeleton to use as the base. - Likely based on Kubebuilder/controller-runtime - Scripts for bringing up&down a test/dev environment. - Explore admission controllers: - Can we use kubewarden? at least for some parts? - Requirements and possible functional alternatives. - Explore APIServices - What's the state-of-the-art for building APIServices? Does controller-runtime support it? - Is it possible to extract user information from requests? Hopefully without having to terminate auth here. - Does it support redirects? I talked to Rafa about the project and going over the different areas of exploration.
 
×
Reply to aruiz
We started the week by creating a rough plan of the areas we wanted to explore, in order to divide the problem into smaller parts and identify further areas of work.
Rough Plan
• Create example CRDs that allowed experimenting in our local cluster without breaking it.
  • Nothing really fancy, manually crafted. E.g. copy an existing one from the cluster and just rename it.
• A Kubernetes controller skeleton to use as the base.
  • Likely based on Kubebuilder/controller-runtime
• Scripts for bringing up&down a test/dev environment.
• Explore admission controllers:
  • Can we use kubewarden? at least for some parts?
  • Requirements and possible functional alternatives.
• Explore APIServices
  • What's the state-of-the-art for building APIServices? Does controller-runtime support it?
  • Is it possible to extract user information from requests? Hopefully without having to terminate auth here.
  • Does it support redirects?
I talked to Rafa about the project and going over the different areas of exploration.
---
# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

- Use hyphens
- for unordereed
- lists

This is an [link to example.com](http://example.com/)

This is an image ![an openSUSE geeko icon](https://en.opensuse.org/images/d/d0/Icon-distribution.png)

This is a user link @hans

This is a project link hw#some-cool-title
More Complex Markdown Help
Formatting Help
• Edit
• Preview
 
• 

  11 months ago by aruiz | Reply

  I started by exploring Kubewarden to check if we could just use a policy to manage the mapping of CRDs being installed along with a Bottle; the idea was to perform this transformation before the CRD is stored in Kubernetes, so we would need to use Admission/Mutating webhooks.

  I concluded that Kuberwarden was not a good fit for this because:

  • Go policies are compiled with TinyGo, due to limited support for WebAssembly in the official toolchain. TinyGo still has some limitations for building the Go standard library, which prevents us from using the k8s.io libraries, which we would need to perform the desired steps, as they will require a Kubernetes client.
  • CEL policies won't provide enough flexibility for our purpose.
  • Rust-based policies could be an option, but probably require a bigger effort to implement (and differ from the language used from the rest of the project).

  I confirmed that Kubebuilder has support for writing defaulting webhooks, so we could use it for modifying the CRDs before they get persisted.

  This framework also creates scripts and resources for building the controller's image, as well as the manifests to install it in Kubernetes. Although it's very focused on adding your own APIs, with some tweaks we could use it to generate such boiler-plate for a built-in type (CRDs).

  These "difficulties" made me think of alternatives approaches for our goal. Assuming that Helm charts is the selected installation mechanism, we could:

  • Create an offline tool whose input is the complete manifests (including the Bottle spec), and produces the required modifications, so that they can be directly applied to Kubernetes.
  • This could be a kubectl (krew) or helm plugin.
  • An offline transformation is a simpler solution, since it moves the processing client-side (allows dry-run, store definitive manifest for GitOps, etc.), while makes the approach less transparent.

×
Edit Comment 9382
# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

- Use hyphens
- for unordereed
- lists

This is an [link to example.com](http://example.com/)

This is an image ![an openSUSE geeko icon](https://en.opensuse.org/images/d/d0/Icon-distribution.png)

This is a user link @hans

This is a project link hw#some-cool-title
More Complex Markdown Help
Formatting Help
• Edit
• Preview
I started by exploring Kubewarden to check if we could just use a policy to manage the mapping of CRDs being installed along with a *Bottle*; the idea was to perform this transformation before the CRD is stored in Kubernetes, so we would need to use Admission/Mutating webhooks. I concluded that Kuberwarden was not a good fit for this because: - Go policies are compiled with TinyGo, due to limited support for WebAssembly in the official toolchain. TinyGo still has some limitations for building the Go standard library, which prevents us from using the k8s.io libraries, which we would need to perform the desired steps, as they will require a Kubernetes client. - CEL policies won't provide enough flexibility for our purpose. - Rust-based policies could be an option, but probably require a bigger effort to implement (and differ from the language used from the rest of the project). I confirmed that [Kubebuilder has support for writing defaulting webhooks](https://book.kubebuilder.io/cronjob-tutorial/webhook-implementation), so we could use it for modifying the CRDs before they get persisted. This framework also creates scripts and resources for building the controller's image, as well as the manifests to install it in Kubernetes. Although it's very focused on adding your own APIs, with some tweaks we could use it to generate such boiler-plate for a built-in type (CRDs). These "difficulties" made me think of alternatives approaches for our goal. Assuming that Helm charts is the selected installation mechanism, we could: - Create an offline tool whose input is the complete manifests (including the *Bottle* spec), and produces the required modifications, so that they can be directly applied to Kubernetes. - This could be a `kubectl` (krew) or helm plugin. - An offline transformation is a simpler solution, since it moves the processing client-side (allows dry-run, store definitive manifest for GitOps, etc.), while makes the approach less transparent.
 
×
Reply to aruiz
I started by exploring Kubewarden to check if we could just use a policy to manage the mapping of CRDs being installed along with a Bottle; the idea was to perform this transformation before the CRD is stored in Kubernetes, so we would need to use Admission/Mutating webhooks.
I concluded that Kuberwarden was not a good fit for this because:
• Go policies are compiled with TinyGo, due to limited support for WebAssembly in the official toolchain. TinyGo still has some limitations for building the Go standard library, which prevents us from using the k8s.io libraries, which we would need to perform the desired steps, as they will require a Kubernetes client.
• CEL policies won't provide enough flexibility for our purpose.
• Rust-based policies could be an option, but probably require a bigger effort to implement (and differ from the language used from the rest of the project).
I confirmed that Kubebuilder has support for writing defaulting webhooks, so we could use it for modifying the CRDs before they get persisted.
This framework also creates scripts and resources for building the controller's image, as well as the manifests to install it in Kubernetes. Although it's very focused on adding your own APIs, with some tweaks we could use it to generate such boiler-plate for a built-in type (CRDs).
These "difficulties" made me think of alternatives approaches for our goal. Assuming that Helm charts is the selected installation mechanism, we could:
• Create an offline tool whose input is the complete manifests (including the Bottle spec), and produces the required modifications, so that they can be directly applied to Kubernetes.
• This could be a kubectl (krew) or helm plugin.
• An offline transformation is a simpler solution, since it moves the processing client-side (allows dry-run, store definitive manifest for GitOps, etc.), while makes the approach less transparent.
---
# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

- Use hyphens
- for unordereed
- lists

This is an [link to example.com](http://example.com/)

This is an image ![an openSUSE geeko icon](https://en.opensuse.org/images/d/d0/Icon-distribution.png)

This is a user link @hans

This is a project link hw#some-cool-title
More Complex Markdown Help
Formatting Help
• Edit
• Preview
 
• 

  11 months ago by aruiz | Reply

  Since we had already used a few days, I decided to prioritize exploring the rest of the areas instead of keep building on the admission part, since this was easier to fake in order for the rest to keep working.

  So I started looking into our options for implementing the Aggregation layer in Kubernetes. Besides creating our own CRDs, which is not what we were looking for, the docs suggested 2 options (note that Kubebuilder/controller-runtime is not intended for this use case and have no support for this):

  • Use kubernetes-incubator/apiserver-builder (now named kubernetes-sigs/apiserver-builder-alpha), which aims to provide a similar pattern to Kubebuilder.
    • However, it seems to not be actively maintained (latest release was >2 years ago, no Go modules support, and lack of activity in general.)
  • Use the sample-apiserver project, which seems to be the base for the apiserver-builder generator, but is more up-to-date.
  • Both options seem to build on top of the apiserver-runtime framework, which as I understand is equivalent to controller-runtime for regular controllers. Nonetheless, its last commit was almost one year ago.

  I put some efforts into trying to make apiserver-builder work, including to make the generated project Go-modules aware, but then faced many problem upgrading Kubernetes dependencies to recent versions, so I gave up on that option.

  In this situation, forking sample-apiserver and start modifying it to our needs looked like the best option to go forward.

  This sample project does work, but has very low-level requirements. In particular, it's meant to have access to Etcd itself, in order to serve the APIService. This option was not in our initial plans, as it would make it harder to run our controller. Nonetheless, I also found that the interface implemented does not necessarily have to be backed by Etcd, which brings the option of using a different storage (e.g. any database), as long as the interface methods are implemented. I decided to not pursue this route just yet, though, since it was out of the initial scope and was running out of time. For the sake of the experiment, I tweaked the endpoint to use a Kubernetes client to try to obtain the original data from the main API server and then transforming it. However, this obviously produced an infinite loop, since the control plane would just redirect such requests back to our APIService.

  The last thing that time allowed me to experiment with was the authorization part, as we need to identify which user produced the requests. Even though there is functionality for this, I couldn't manage to make it available to my handlers implementation, and wasn't able to identify why. I need to read more docs about how the workflows for Aggregated APIs, maybe authorization is meant to be resolved by APIServices directly? Sadly, the apiserver-runtime library is not very well documented.

×
Edit Comment 9415
# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

- Use hyphens
- for unordereed
- lists

This is an [link to example.com](http://example.com/)

This is an image ![an openSUSE geeko icon](https://en.opensuse.org/images/d/d0/Icon-distribution.png)

This is a user link @hans

This is a project link hw#some-cool-title
More Complex Markdown Help
Formatting Help
• Edit
• Preview
Since we had already used a few days, I decided to prioritize exploring the rest of the areas instead of keep building on the admission part, since this was easier to fake in order for the rest to keep working. So I started looking into our options for implementing the [Aggregation layer in Kubernetes](https://kubernetes.io/docs/tasks/extend-kubernetes/setup-extension-api-server/). Besides creating our own CRDs, which is not what we were looking for, the docs suggested 2 options (note that Kubebuilder/controller-runtime is not intended for this use case and have no support for this): - Use `kubernetes-incubator/apiserver-builder` (now named [`kubernetes-sigs/apiserver-builder-alpha`](https://github.com/kubernetes-sigs/apiserver-builder-alpha)), which aims to provide a similar pattern to Kubebuilder. - However, it seems to not be actively maintained (latest release was >2 years ago, no Go modules support, and lack of activity in general.) - Use the [`sample-apiserver`](https://github.com/kubernetes/sample-apiserver) project, which seems to be the base for the `apiserver-builder` generator, but is more up-to-date. - Both options seem to build on top of the [`apiserver-runtime`](https://github.com/kubernetes-sigs/apiserver-runtime) framework, which as I understand is equivalent to `controller-runtime` for regular controllers. Nonetheless, its last commit was almost one year ago. I put some efforts into trying to make `apiserver-builder` work, including to make the generated project Go-modules aware, but then faced many problem upgrading Kubernetes dependencies to recent versions, so I gave up on that option. In this situation, forking `sample-apiserver` and start modifying it to our needs looked like the best option to go forward. This sample project does work, but has very low-level requirements. In particular, it's meant to have access to Etcd itself, in order to serve the APIService. This option was not in our initial plans, as it would make it harder to run our controller. Nonetheless, I also found that the interface implemented does not necessarily have to be backed by Etcd, which brings the option of using a different storage (e.g. any database), as long as the interface methods are implemented. I decided to not pursue this route just yet, though, since it was out of the initial scope and was running out of time. For the sake of the experiment, I tweaked the endpoint to use a Kubernetes client to try to obtain the original data from the main API server and then transforming it. However, this obviously produced an infinite loop, since the control plane would just redirect such requests back to our APIService. The last thing that time allowed me to experiment with was the authorization part, as we need to identify which user produced the requests. Even though [there is functionality](https://pkg.go.dev/k8s.io/apiserver/pkg/endpoints/request#UserFrom) for this, I couldn't manage to make it available to my handlers implementation, and wasn't able to identify why. I need to read more docs about how the workflows for Aggregated APIs, maybe authorization is meant to be resolved by APIServices directly? Sadly, the `apiserver-runtime` library is not very well documented.
 
×
Reply to aruiz
Since we had already used a few days, I decided to prioritize exploring the rest of the areas instead of keep building on the admission part, since this was easier to fake in order for the rest to keep working.
So I started looking into our options for implementing the Aggregation layer in Kubernetes. Besides creating our own CRDs, which is not what we were looking for, the docs suggested 2 options (note that Kubebuilder/controller-runtime is not intended for this use case and have no support for this):
• Use kubernetes-incubator/apiserver-builder (now named kubernetes-sigs/apiserver-builder-alpha), which aims to provide a similar pattern to Kubebuilder.
  • However, it seems to not be actively maintained (latest release was >2 years ago, no Go modules support, and lack of activity in general.)
• Use the sample-apiserver project, which seems to be the base for the apiserver-builder generator, but is more up-to-date.
• Both options seem to build on top of the apiserver-runtime framework, which as I understand is equivalent to controller-runtime for regular controllers. Nonetheless, its last commit was almost one year ago.
I put some efforts into trying to make apiserver-builder work, including to make the generated project Go-modules aware, but then faced many problem upgrading Kubernetes dependencies to recent versions, so I gave up on that option.
In this situation, forking sample-apiserver and start modifying it to our needs looked like the best option to go forward.
This sample project does work, but has very low-level requirements. In particular, it's meant to have access to Etcd itself, in order to serve the APIService. This option was not in our initial plans, as it would make it harder to run our controller. Nonetheless, I also found that the interface implemented does not necessarily have to be backed by Etcd, which brings the option of using a different storage (e.g. any database), as long as the interface methods are implemented. I decided to not pursue this route just yet, though, since it was out of the initial scope and was running out of time. For the sake of the experiment, I tweaked the endpoint to use a Kubernetes client to try to obtain the original data from the main API server and then transforming it. However, this obviously produced an infinite loop, since the control plane would just redirect such requests back to our APIService.
The last thing that time allowed me to experiment with was the authorization part, as we need to identify which user produced the requests. Even though there is functionality for this, I couldn't manage to make it available to my handlers implementation, and wasn't able to identify why. I need to read more docs about how the workflows for Aggregated APIs, maybe authorization is meant to be resolved by APIServices directly? Sadly, the apiserver-runtime library is not very well documented.
---
# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

- Use hyphens
- for unordereed
- lists

This is an [link to example.com](http://example.com/)

This is an image ![an openSUSE geeko icon](https://en.opensuse.org/images/d/d0/Icon-distribution.png)

This is a user link @hans

This is a project link hw#some-cool-title
More Complex Markdown Help
Formatting Help
• Edit
• Preview
 
• 

  8 months ago by Jasbon | Reply

  This looks like a detailed breakdown of the "Bottles" proof of concept for managing multi-version CRDs in Kubernetes. Are you looking for a summary, feedback, or help with a specific part of the project?

×
Edit Comment 9687
# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

- Use hyphens
- for unordereed
- lists

This is an [link to example.com](http://example.com/)

This is an image ![an openSUSE geeko icon](https://en.opensuse.org/images/d/d0/Icon-distribution.png)

This is a user link @hans

This is a project link hw#some-cool-title
More Complex Markdown Help
Formatting Help
• Edit
• Preview
This looks like a detailed breakdown of the "Bottles" proof of concept for managing multi-version CRDs in Kubernetes. Are you looking for a summary, feedback, or help with a specific part of the project?
 
×
Reply to Jasbon
This looks like a detailed breakdown of the "Bottles" proof of concept for managing multi-version CRDs in Kubernetes. Are you looking for a summary, feedback, or help with a specific part of the project?
---
# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

- Use hyphens
- for unordereed
- lists

This is an [link to example.com](http://example.com/)

This is an image ![an openSUSE geeko icon](https://en.opensuse.org/images/d/d0/Icon-distribution.png)

This is a user link @hans

This is a project link hw#some-cool-title
More Complex Markdown Help
Formatting Help
• Edit
• Preview