Self-Scaling LLM Infrastructure Powered by Rancher by ademicev0

Self-Scaling LLM Infrastructure Powered by Rancher

---

Description

The Problem

Running LLMs can get expensive and complex pretty quickly.

Today there are typically two choices:

1. Use cloud APIs like OpenAI or Anthropic. Easy to start with, but costs add up at scale.
2. Self-host everything - set up Kubernetes, figure out GPU scheduling, handle scaling, manage model serving... it's a lot of work.

What if there was a middle ground?

What if infrastructure scaled itself instead of making you scale it?

Can we use existing Rancher capabilities like CAPI, autoscaling, and GitOps to make this simpler instead of building everything from scratch?

Project Repository: github.com/alexander-demicev/llmserverless

---

What This Project Does

A key feature is hybrid deployment: requests can be routed based on complexity or privacy needs. Simple or low-sensitivity queries can use public APIs (like OpenAI), while complex or private requests are handled in-house on local infrastructure. This flexibility allows balancing cost, privacy, and performance - using cloud for routine tasks and on-premises resources for sensitive or demanding workloads.

A complete, self-scaling LLM infrastructure that:

• Scales to zero when idle (no idle costs)
• Scales up automatically when requests come in
• Adds more nodes when needed, removes them when demand drops
• Runs on any infrastructure - laptop, bare metal, or cloud

Think of it as "serverless for LLMs" - focus on building, the infrastructure handles itself.

How It Works

A combination of open source tools working together:

Flow:

• Users interact with OpenWebUI (chat interface)
• Requests go to LiteLLM Gateway
• LiteLLM routes requests to:
  • Ollama (Knative) for local model inference (auto-scales pods)
  • Or cloud APIs for fallback

---

Rancher Cluster Lifecycle Visualizer by jferraz

Description

Rancher’s v2 provisioning system represents each downstream cluster with several Kubernetes custom resources across multiple API groups, such as clusters.provisioning.cattle.io and clusters.management.cattle.io. Understanding why a cluster is stuck in states like "Provisioning", "Updating", or "Unavailable" often requires jumping between these resources, reading conditions, and correlating them with agent connectivity and known failure modes. This project will build a Cluster Lifecycle Visualizer: a small, read-only controller that runs in the Rancher management cluster and generates a single, human-friendly view per cluster. It will watch Rancher cluster CRDs, derive a simplified lifecycle phase, keep a history of phase transitions from installation time onward, and attach a short, actionable recommendation string that hints at what the operator should check or do next.

Goals

• Provide a compact lifecycle summary for each Rancher-managed cluster (e.g. Provisioning, WaitingForClusterAgent, Active, Updating, Error) derived from provisioning.cattle.io/v1 Cluster and management.cattle.io/v3 Cluster status and conditions.
• Maintain a phase history for each cluster, allowing operators to see how its state evolved over time since the visualizer was installed.
• Attach a recommended action to the current phase using a small ruleset based on common Rancher failure modes (for example, cluster agent not connected, cluster still stabilizing after an upgrade, or generic error states), to improve the day-to-day debugging experience.
• Deliver an easy-to-install, read-only component (single YAML or small Helm chart) that Rancher users can deploy to their management cluster and inspect via kubectl get/describe, without UI changes or direct access to downstream clusters.
• Use idiomatic Go, wrangler, and Rancher APIs.

Resources

• Rancher Manager documentation on RKE2 and K3s cluster configuration and provisioning flows.
• Rancher API Go types for provisioning.cattle.io/v1 and management.cattle.io/v3 (from the rancher/rancher repository or published Go packages).
• Existing Rancher architecture docs and internal notes about cluster provisioning, cluster agents, and node agents.
• A local Rancher management cluster (k3s or RKE2) with a few test downstream clusters to validate phase detection, history tracking, and recommendations.

---

The Agentic Rancher Experiment: Do Androids Dream of Electric Cattle? by moio

Rancher is a beast of a codebase. Let's investigate if the new 2025 generation of GitHub Autonomous Coding Agents and Copilot Workspaces can actually tame it.

---

The Plan

Create a sandbox GitHub Organization, clone in key Rancher repositories, and let the AI loose to see if it can handle real-world enterprise OSS maintenance - or if it just hallucinates new breeds of Kubernetes resources!

Specifically, throw "Agentic Coders" some typical tasks in a complex, long-lived open-source project, such as:

❥ The Grunt Work: generate missing GoDocs, unit tests, and refactorings. Rebase PRs.

❥ The Complex Stuff: fix actual (historical) bugs and feature requests to see if they can traverse the complexity without (too much) human hand-holding.

❥ Hunting Down Gaps: find areas lacking in docs, areas of improvement in code, dependency bumps, and so on.

If time allows, also experiment with Model Context Protocol (MCP) to give agents context on our specific build pipelines and CI/CD logs.

Why?

We know AI can write "Hello World." and also moderately complex programs from a green field. But can it rebase a 3-month-old PR with conflicts in rancher/rancher? I want to find the breaking point of current AI agents to determine if and how they can help us to reduce our technical debt, work faster and better. At the same time, find out about pitfalls and shortcomings.

The CONCLUSION!!!

A State of the Union document was compiled to summarize lessons learned this week. For more gory details, just read on the diary below!

---

Cluster API Provider for Harvester by rcase

Project Description

The Cluster API "infrastructure provider" for Harvester, also named CAPHV, makes it possible to use Harvester with Cluster API. This enables people and organisations to create Kubernetes clusters running on VMs created by Harvester using a declarative spec.

The project has been bootstrapped in HackWeek 23, and its code is available here.

Work done in HackWeek 2023

• Have a early working version of the provider available on Rancher Sandbox : *DONE *
• Demonstrated the created cluster can be imported using Rancher Turtles: DONE
• Stretch goal - demonstrate using the new provider with CAPRKE2: DONE and the templates are available on the repo

DONE in HackWeek 24:

• Add more Unit Tests
• Improve Status Conditions for some phases
• Add cloud provider config generation
• Testing with Harvester v1.3.2
• Template improvements
• Issues creation

DONE in 2025 (out of Hackweek)

• Support of ClusterClass
• Add to clusterctl community providers, you can add it directly with clusterctl
• Testing on newer versions of Harvester v1.4.X and v1.5.X
• Support for clusterctl generate cluster ...
• Improve Status Conditions to reflect current state of Infrastructure
• Improve CI (some bugs for release creation)

Goals for HackWeek 2025

• FIRST and FOREMOST, any topic is important to you
• Add e2e testing
• Certify the provider for Rancher Turtles
• Add Machine pool labeling
• Add PCI-e passthrough capabilities.
• Other improvement suggestions are welcome!

Thanks to @isim and Dominic Giebert for their contributions!

Resources

Looking for help from anyone interested in Cluster API (CAPI) or who wants to learn more about Harvester.

This will be an infrastructure provider for Cluster API. Some background reading for the CAPI aspect:

• Cluster infrastructure provider contract
• Machine infrastructure provider contract

---

Liz - Prompt autocomplete by ftorchia

Description

Liz is the Rancher AI assistant for cluster operations.

Goals

We want to help users when sending new messages to Liz, by adding an autocomplete feature to complete their requests based on the context.

Example:

• User prompt: "Can you show me the list of p"
• Autocomplete suggestion: "Can you show me the list of p...od in local cluster?"

Example:

• User prompt: "Show me the logs of #rancher-"
• Chat console: It shows a drop-down widget, next to the # character, with the list of available pod names starting with "rancher-".

Technical Overview

1. The AI agent should expose a new ws/autocomplete endpoint to proxy autocomplete messages to the LLM.
2. The UI extension should be able to display prompt suggestions and allow users to apply the autocomplete to the Prompt via keyboard shortcuts.

Resources

GitHub repository

---

Technical talks at universities by agamez

Description

This project aims to empower the next generation of tech professionals by offering hands-on workshops on containerization and Kubernetes, with a strong focus on open-source technologies. By providing practical experience with these cutting-edge tools and fostering a deep understanding of open-source principles, we aim to bridge the gap between academia and industry.

For now, the scope is limited to Spanish universities, since we already have the contacts and have started some conversations.

Goals

• Technical Skill Development: equip students with the fundamental knowledge and skills to build, deploy, and manage containerized applications using open-source tools like Kubernetes.
• Open-Source Mindset: foster a passion for open-source software, encouraging students to contribute to open-source projects and collaborate with the global developer community.
• Career Readiness: prepare students for industry-relevant roles by exposing them to real-world use cases, best practices, and open-source in companies.

Resources

• Instructors: experienced open-source professionals with deep knowledge of containerization and Kubernetes.
• SUSE Expertise: leverage SUSE's expertise in open-source technologies to provide insights into industry trends and best practices.

---

Rewrite Distrobox in go (POC) by fabriziosestito

Description

Rewriting Distrobox in Go.

Main benefits:

• Easier to maintain and to test
• Adapter pattern for different container backends (LXC, systemd-nspawn, etc.)

Goals

• Build a minimal starting point with core commands
• Keep the CLI interface compatible: existing users shouldn't notice any difference
• Use a clean Go architecture with adapters for different container backends
• Keep dependencies minimal and binary size small
• Benchmark against the original shell script

Resources

• Upstream project: https://github.com/89luca89/distrobox/
• Distrobox site: https://distrobox.it/
• ArchWiki: https://wiki.archlinux.org/title/Distrobox

---

Port the classic browser game HackTheNet to PHP 8 by dgedon

Description

The classic browser game HackTheNet from 2004 still runs on PHP 4/5 and MySQL 5 and needs a port to PHP 8 and e.g. MariaDB.

Goals

• Port the game to PHP 8 and MariaDB 11
• Create a container where the game server can simply be started/stopped

Resources

• https://github.com/nodeg/hackthenet

---

Help Create A Chat Control Resistant Turnkey Chatmail/Deltachat Relay Stack - Rootless Podman Compose, OpenSUSE BCI, Hardened, & SELinux by 3nd5h1771fy

Description

The Mission: Decentralized & Sovereign Messaging

FYI: If you have never heard of "Chatmail", you can visit their site here, but simply put it can be thought of as the underlying protocol/platform decentralized messengers like DeltaChat use for their communications. Do not confuse it with the honeypot looking non-opensource paid for prodect with better seo that directs you to chatmailsecure(dot)com

In an era of increasing centralized surveillance by unaccountable bad actors (aka BigTech), "Chat Control," and the erosion of digital privacy, the need for sovereign communication infrastructure is critical. Chatmail is a pioneering initiative that bridges the gap between classic email and modern instant messaging, offering metadata-minimized, end-to-end encrypted (E2EE) communication that is interoperable and open.

However, unless you are a seasoned sysadmin, the current recommended deployment method of a Chatmail relay is rigid, fragile, difficult to properly secure, and effectively takes over the entire host the "relay" is deployed on.

Why This Matters

A simple, host agnostic, reproducible deployment lowers the entry cost for anyone wanting to run a privacy‑preserving, decentralized messaging relay. In an era of perpetually resurrected chat‑control legislation threats, EU digital‑sovereignty drives, and many dangers of using big‑tech messaging platforms (Apple iMessage, WhatsApp, FB Messenger, Instagram, SMS, Google Messages, etc...) for any type of communication, providing an easy‑to‑use alternative empowers:

• Censorship resistance - No single entity controls the relay; operators can spin up new nodes quickly.
  
• Surveillance mitigation - End‑to‑end OpenPGP encryption ensures relay operators never see plaintext.
  
• Digital sovereignty - Communities can host their own infrastructure under local jurisdiction, aligning with national data‑policy goals.
  

By turning the Chatmail relay into a plug‑and‑play container stack, we enable broader adoption, foster a resilient messaging fabric, and give developers, activists, and hobbyists a concrete tool to defend privacy online.

Goals

As I indicated earlier, this project aims to drastically simplify the deployment of Chatmail relay. By converting this architecture into a portable, containerized stack using Podman and OpenSUSE base container images, we can allow anyone to deploy their own censorship-resistant, privacy-preserving communications node in minutes.

Our goal for Hack Week: package every component into containers built on openSUSE/MicroOS base images, initially orchestrated with a single container-compose.yml (podman-compose compatible). The stack will:

• Run on any host that supports Podman (including optimizations and enhancements for SELinux‑enabled systems).
  
• Allow network decoupling by refactoring configurations to move from file-system constrained Unix sockets to internal TCP networking, allowing containers achieve stricter isolation.
• Utilize Enhanced Security with SELinux by using purpose built utilities such as udica we can quickly generate custom SELinux policies for the container stack, ensuring strict confinement superior to standard/typical Docker deployments.
• Allow the use of bind or remote mounted volumes for shared data (/var/vmail, DKIM keys, TLS certs, etc.).
  
• Replace the local DNS server requirement with a remote DNS‑provider API for DKIM/TXT record publishing.
  

By delivering a turnkey, host agnostic, reproducible deployment, we lower the barrier for individuals and small communities to launch their own chatmail relays, fostering a decentralized, censorship‑resistant messaging ecosystem that can serve DeltaChat users and/or future services adopting this protocol

Resources

• The links included above
• https://chatmail.at/doc/relay/
• https://delta.chat/en/help
• Project repo -> https://codeberg.org/EndShittification/containerized-chatmail-relay

---

terraform-provider-feilong by e_bischoff

Project Description

People need to test operating systems and applications on s390 platform. While this is straightforward with KVM, this is very difficult with z/VM.

IBM Cloud Infrastructure Center (ICIC) harnesses the Feilong API, but you can use Feilong without installing ICIC(see this schema).

What about writing a terraform Feilong provider, just like we have the terraform libvirt provider? That would allow to transparently call Feilong from your main.tf files to deploy and destroy resources on your z/VM system.

Goal for Hackweek 23

I would like to be able to easily deploy and provision VMs automatically on a z/VM system, in a way that people might enjoy even outside of SUSE.

My technical preference is to write a terraform provider plugin, as it is the approach that involves the least software components for our deployments, while remaining clean, and compatible with our existing development infrastructure.

Goals for Hackweek 24

Feilong provider works and is used internally by SUSE Manager team. Let's push it forward!

Let's add support for fiberchannel disks and multipath.

Goals for Hackweek 25

Modernization, maturity, and maintenance: support for SLES 16 and openTofu, new API calls, fixes...

Resources

• Feilong TSC and demo
• Call APIs with custom Framework Providers

Outcome

• provider, for terraform 1.5.5
• provider, for terraform 0.13.4
• go client library, for Feilong API 1.0

---

Rewrite Distrobox in go (POC) by fabriziosestito

Description

Rewriting Distrobox in Go.

Main benefits:

• Easier to maintain and to test
• Adapter pattern for different container backends (LXC, systemd-nspawn, etc.)

Goals

• Build a minimal starting point with core commands
• Keep the CLI interface compatible: existing users shouldn't notice any difference
• Use a clean Go architecture with adapters for different container backends
• Keep dependencies minimal and binary size small
• Benchmark against the original shell script

Resources

• Upstream project: https://github.com/89luca89/distrobox/
• Distrobox site: https://distrobox.it/
• ArchWiki: https://wiki.archlinux.org/title/Distrobox

---

Contribute to terraform-provider-libvirt by pinvernizzi

Description

The SUSE Manager (SUMA) teams' main tool for infrastructure automation, Sumaform, largely relies on terraform-provider-libvirt. That provider is also widely used by other teams, both inside and outside SUSE.

It would be good to help the maintainers of this project and give back to the community around it, after all the amazing work that has been already done.

If you're interested in any of infrastructure automation, Terraform, virtualization, tooling development, Go (...) it is also a good chance to learn a bit about them all by putting your hands on an interesting, real-use-case and complex project.

Goals

• Get more familiar with Terraform provider development and libvirt bindings in Go
• Solve some issues and/or implement some features
• Get in touch with the community around the project

Resources

• CONTRIBUTING readme
• Go libvirt library in use by the project
• Terraform plugin development
• "Good first issue" list

---

go-git: unlocking SHA256-based repository cloning ahead of git v3 by pgomes

Description

The go-git library implements the git internals in pure Go, so that any Go application can handle not only Git repositories, but also lower-level primitives (e.g. packfiles, idxfiles, etc) without needing to shell out to the git binary.

The focus for this Hackweek is to fast track key improvements for the project ahead of the upstream release of Git V3, which may take place at some point next year.

Goals

• Add support for cloning SHA256 repositories.
• Decrease memory churn for very large repositories (e.g. Linux Kernel repository).
• Cut the first alpha version for go-git/v6.

Stretch goals

• Review and update the official documentation.
• Optimise use of go-git in Fleet.
• Create RFC/example for go-git plugins to improve extensibility.
• Investigate performance bottlenecks for Blame and Status.

Resources

• https://github.com/go-git/go-git/
• https://go-git.github.io/docs/

---

Play with the userfaultfd(2) system call and download on demand using HTTP Range Requests with Golang by rbranco

Description

The userfaultfd(2) is a cool system call to handle page faults in user-space. This should allow me to list the contents of an ISO or similar archive without downloading the whole thing. The userfaultfd(2) part can also be done in theory with the PROT_NONE mprotect + SIGSEGV trick, for complete Unix portability, though reportedly being slower.

Goals

1. Create my own library for userfaultfd(2) in Golang.
2. Create my own library for HTTP Range Requests.
3. Complete portability with Unix.
4. Benchmarks.
5. Contribute some tests to LTP.

Resources

1. https://docs.kernel.org/admin-guide/mm/userfaultfd.html
2. https://www.cons.org/cracauer/cracauer-userfaultfd.html

---