ClusterOps - Easily install and manage your personal kubernetes cluster
an invention by andreabenini
Description
ClusterOps is a Kubernetes installer and operator designed to streamline the initial configuration
and ongoing maintenance of kubernetes clusters. The focus of this project is primarily on personal
or local installations. However, the goal is to expand its use to encompass all installations of
Kubernetes for local development purposes.
It simplifies cluster management by automating tasks and providing just one user-friendly YAML-based
configuration config.yml.
Overview
- Simplified Configuration: Define your desired cluster state in a simple YAML file, and ClusterOps will handle the rest.
- Automated Setup: Automates initial cluster configuration, including network settings, storage provisioning, special requirements (for example GPUs) and essential components installation.
- Ongoing Maintenance: Performs routine maintenance tasks such as upgrades, security updates, and resource monitoring.
- Extensibility: Easily extend functionality with custom plugins and configurations.
- Self-Healing: Detects and recovers from common cluster issues, ensuring stability, idempotence and reliability. Same operation can be performed multiple times without changing the result.
- Discreet: It works only on what it knows, if you are manually configuring parts of your kubernetes and this configuration does not interfere with it you can happily continue to work on several parts and use this tool only for what is needed.
Features
- distribution and engine independence. Install your favorite kubernetes engine with your package
manager, execute one script and you'll have a complete working environment at your disposal.
- Basic config approach. One single
config.ymlfile with configuration requirements (add/remove features): human readable, plain and simple. All fancy configs managed automatically (ingress, balancers, services, proxy, ...). - Local Builtin ContainerHub. The default installation provides a fully configured ContainerHub available locally along with the kubernetes installation. This configuration allows the user to build, upload and deploy custom container images as they were provided from external sources. Internet public sources are still available but local development can be kept in this localhost server. Builtin ClusterOps operator will be fetched from this ContainerHub registry too.
- Kubernetes official dashboard installed as a plugin, others planned too (k9s for example).
- Kubevirt plugin installed and properly configured. Unleash the power of classic virtualization (KVM+QEMU) on top of Kubernetes and manage your entire system from there, libvirtd and virsh libs are required.
- One operator to rule them all. The installation script configures your machine automatically during installation and adds one kubernetes operator to manage your local cluster. From there the operator takes care of the cluster on your behalf.
- Clean installation and removal. Just test it, when you are done just use the same program to uninstall everything without leaving configs (or pods) behind.
Planned features (Wishlist / TODOs)
- Containerized Data Importer (CDI). Persistent storage management add-on for Kubernetes to provide a declarative way of building and importing Virtual Machine Disks on PVCs for Kubevirt VMs.
- Source2Image utility. Transform your favorite program (python, go, bash, ...) in a container in a matter of minutes, kubectl apply and create it as a Pod or a Deployment quickly.
- Kubevirt VMs startup management. Since your personal cluster might not be up and running all the time this feature will provide basic startup, shutdown, order list commands; it resembles other VM bare metal configuration suites from the past.
- Lightweight k9s console automatically installed as a plugin from the configuration file
- Add other distributions: suse, debian, rocky/rhel, gentoo, MacOS
- Add other kubernetes engines: minicube, KIND, vanilla k8s, CRC
- Monitoring and observation features, alerting with IM notifications (telegram, signal)
- Remote storage, LAN network volumes, S3 buckets, object storage (CEPH, Longhorn)
- Automatic configuration and support for: Nvidia CUDA, Vulkan drivers. Containers downloaded from Nvidia ContainerHub and relative websites should be used directly without additional configuration.
- Cloud Controller Manager (CCM). A Kubernetes control plane component that embeds cloud specific control logic. This component with a specific automation tool easily allows to migrate local working environment to external (private | hybrid | public) clouds.
Project Resources
- github project repository: clusterops
- @andreabenini @SUSE
- complete README.md (document from where this description has been extracted)
- feel free to reach me on slack, email, submit issues, MR, ...
Looking for hackers with the skills:
kubernetes k3s kubevirt kvm operator personal development webui easy containers pods go golang python
This project is part of:
Hack Week 24
Activity
Comments
-
6 months ago by andreabenini | Reply
Day one
Project established. github presence in place, hackweek README project created. Basic libraries in place for the installer/removal utility. I'm now considering k3s because it's easy to manage locally, other engines will be added once main results will be achieved.
Adding SUSE OSes will surely be trivial and I can barely add them all in one shot. I'm now focusing on the k8s operator in order to have minimal functionalities available from it: kubevirt, Web UI, network setup, traefik setup (on local lan, not just localhost).
I'm now using kubebuilder for managing kubernetes operator, its first task will be around adding the default kubernetes dashboard to the system -
6 months ago by andreabenini | Reply
Day two
Created user's ContainerHub, now you can easily create your images locally and upload them, the hub is also used from kubernetes for fetching images.
First dummy (but working) operator has been created and uploaded to localhost ContainerHub and it can be installed directly in the k3s installation at startup. ContainerHub has been created as a systemd service and automatically configured from the same clusterops installation script.
Forced k3s dependency makes also easy to have them loaded at startup when required.> systemctl enable clusterops # Start clusterops (with ContainerHub) and k3s on startup
> systemctl start clusterops # Start ClusterOps+ContainerHub+k3s manually -
6 months ago by andreabenini | Reply
Day three
Finally Kubevirt has joined the group and now represents one of the important pillars of this software collection, it relies on community made vanilla Operator and it just works as it's supposed to be. System's requirements are basically QEMU+KVM and libvirt on which libvirtd is built. After a simple test withvirt-host-validateyou can easily have it at your disposal. Full integration with basic components and builtin clusterops Operator is not stable yet but results are promising.
YAML example files are ready and they can be customized by users to easily create or import virtual machines on top of kubernetes in literally a matter of minutes. -
6 months ago by andreabenini | Reply
Day Four, integration mashup
Here's an update on the progress:
- I've modified the installer to seamlessly integrate the ContainerHub service, which is now a legitimate systemd service. This service will be automatically created and updated during installation to ensure consistency.
- Dashboard configuration and Kubevirt settings will also be automatically set during installation, streamlining the process and centralizing these components.
- The Kubernetes Operator will utilize the same configuration file and maintain a stable state across changes, even in cases where parts of a working system are intentionally deleted (excluding the operator itself, of course!).
- Final step will be to unify all external yaml files and enable their automatic use based on user requests. -
6 months ago by andreabenini | Reply
Day Five, final thoughts,
All day has been spent refining these addons: ContainerHub, KubeVirt. Removing pending tasks and tidying up the code in the python installer was important too. I finally have a working environment and installation/setup/removal procedures can now be considered stable with K3S.
OS configuration: the installer is now reduced to the minimum and porting between different distributions should be rather easy. I'll start now with all SUSE related linux distro porting: SLES, Tumbleweed, OpenSUSE. It's already working on a low spec laptop (company laptop) but I'm trying to collect more data before declaring it stable.
I'll add all RHEL related distros (Rocky, Alma, Fedora, RHEL) after it and Debian at the end to mark my interest on all these platforms. Minor changes should be applied but from what I've seen there's no real deal on adding platforms. Questions might be tricky with Security Enhanced libraries (selinux and apparmor mostly) but until I keep installation and configurations on user's profiles it won't hurt Security Roles or Domains that much.
I'll surely stick on k3s for a while because I'm mostly interested in refining my builtin operator, it's barely working but I'll now add new features to autorecover intentional (or unintentional) misconfigurations or removing pods, namespaces, features. Final goal is keeping the kubernetes installation healthy from the inside and it should survive to everything but intentionally removing the operator from the inside (but in that case the external setup should recover it too !). -
5 months ago by andreabenini | Reply
Installation process is now stable and it's fully working.
I have added all SUSE related OSes: Tumbleweed, SLES, OpenSUSE and I'm heavily testing them all in order to avoid typos or gross errors; considering where this project came from it's a relevant topic as you might understand. Apparmor might be noisy so I'm also taking some extra care with it.
I'll surely add the platform named 'suse' to the installer in the next few days to ensure everything works as expected, I don't have a real test bed and I'm applying tests on snapshotted images. I'll consider it as Beta RC for a couple of days before release.
Quickly after that I'll surely add a few interesting platforms to me: Rocky/Alma/Fedora based distros and Debian based before adding new engines (minicube will probably be the next one). -
14 days ago by ericmenk33 | Reply
ClusterOps simplifies the setup and management of Kubernetes clusters, especially for local or personal installations. It offers a single YAML configuration for automated setup and maintenance tasks like upgrades and security updates. ClusterOps can manage your Kubernetes environment, including network, storage, and even GPU configurations, all through easy-to-use automation. If you're into cutting-edge tools like Kubernetes and looking for efficient cluster management, it's worth checking out vagabond manga, where technology and innovation also thrive.
Similar Projects
This project is one of its kind!