The Agentic Rancher Experiment: Do Androids Dream of Electric Cattle? by moio

Rancher is a beast of a codebase. Let's investigate if the new 2025 generation of GitHub Autonomous Coding Agents and Copilot Workspaces can actually tame it.

---

The Plan

Create a sandbox GitHub Organization, clone in key Rancher repositories, and let the AI loose to see if it can handle real-world enterprise OSS maintenance - or if it just hallucinates new breeds of Kubernetes resources!

Specifically, throw "Agentic Coders" some typical tasks in a complex, long-lived open-source project, such as:

❥ The Grunt Work: generate missing GoDocs, unit tests, and refactorings. Rebase PRs.

❥ The Complex Stuff: fix actual (historical) bugs and feature requests to see if they can traverse the complexity without (too much) human hand-holding.

❥ Hunting Down Gaps: find areas lacking in docs, areas of improvement in code, dependency bumps, and so on.

If time allows, also experiment with Model Context Protocol (MCP) to give agents context on our specific build pipelines and CI/CD logs.

Why?

We know AI can write "Hello World." and also moderately complex programs from a green field. But can it rebase a 3-month-old PR with conflicts in rancher/rancher? I want to find the breaking point of current AI agents to determine if and how they can help us to reduce our technical debt, work faster and better. At the same time, find out about pitfalls and shortcomings.

The CONCLUSION!!!

A State of the Union document was compiled to summarize lessons learned this week. For more gory details, just read on the diary below!

---

Liz - Prompt autocomplete by ftorchia

Description

Liz is the Rancher AI assistant for cluster operations.

Goals

We want to help users when sending new messages to Liz, by adding an autocomplete feature to complete their requests based on the context.

Example:

• User prompt: "Can you show me the list of p"
• Autocomplete suggestion: "Can you show me the list of p...od in local cluster?"

Example:

• User prompt: "Show me the logs of #rancher-"
• Chat console: It shows a drop-down widget, next to the # character, with the list of available pod names starting with "rancher-".

Technical Overview

1. The AI agent should expose a new ws/autocomplete endpoint to proxy autocomplete messages to the LLM.
2. The UI extension should be able to display prompt suggestions and allow users to apply the autocomplete to the Prompt via keyboard shortcuts.

Resources

GitHub repository

---

Update M2Crypto by mcepl

There are couple of projects I work on, which need my attention and putting them to shape:

• M2Crypto

Goal for this Hackweek

• Put M2Crypto into better shape (most issues closed, all pull requests processed)
• More fun to learn jujutsu
• Play more with Gemini, how much it help (or not).
• Perhaps, also (just slightly related), help to fix vis to work with LuaJIT, particularly to make vis-lspc working.

---

Flaky Tests AI Finder for Uyuni and MLM Test Suites by oscar-barrios

Description

Our current Grafana dashboards provide a great overview of test suite health, including a panel for "Top failed tests." However, identifying which of these failures are due to legitimate bugs versus intermittent "flaky tests" is a manual, time-consuming process. These flaky tests erode trust in our test suites and slow down development.

This project aims to build a simple but powerful Python script that automates flaky test detection. The script will directly query our Prometheus instance for the historical data of each failed test, using the jenkins_build_test_case_failure_age metric. It will then format this data and send it to the Gemini API with a carefully crafted prompt, asking it to identify which tests show a flaky pattern.

The final output will be a clean JSON list of the most probable flaky tests, which can then be used to populate a new "Top Flaky Tests" panel in our existing Grafana test suite dashboard.

Goals

By the end of Hack Week, we aim to have a single, working Python script that:

1. Connects to Prometheus and executes a query to fetch detailed test failure history.
2. Processes the raw data into a format suitable for the Gemini API.
3. Successfully calls the Gemini API with the data and a clear prompt.
4. Parses the AI's response to extract a simple list of flaky tests.
5. Saves the list to a JSON file that can be displayed in Grafana.
6. New panel in our Dashboard listing the Flaky tests

Resources

• Jenkins Prometheus Exporter: https://github.com/uyuni-project/jenkins-exporter/
• Data Source: Our internal Prometheus server.
• Key Metric: jenkins_build_test_case_failure_age{jobname, buildid, suite, case, status, failedsince}.
• Existing Query for Reference: count by (suite) (max_over_time(jenkins_build_test_case_failure_age{status=~"FAILED|REGRESSION", jobname="$jobname"}[$__range])).
• AI Model: The Google Gemini API.
• Example about how to interact with Gemini API: https://github.com/srbarrios/FailTale/
• Visualization: Our internal Grafana Dashboard.
• Internal IaC: https://gitlab.suse.de/galaxy/infrastructure/-/tree/master/srv/salt/monitoring

Outcome

• Jenkins Flaky Test Detector: https://github.com/srbarrios/jenkins-flaky-tests-detector and its container
• IaC on MLM Team: https://gitlab.suse.de/galaxy/infrastructure/-/tree/master/srv/salt/monitoring/jenkinsflakytestsdetector?reftype=heads, https://gitlab.suse.de/galaxy/infrastructure/-/blob/master/srv/salt/monitoring/grafana/dashboards/flaky-tests.json?ref_type=heads, and others.
• Grafana Dashboard: https://grafana.mgr.suse.de/d/flaky-tests/flaky-tests-detection @ @ text

---

MCP Trace Suite by r1chard-lyu

Description

This project plans to create an MCP Trace Suite, a system that consolidates commonly used Linux debugging tools such as bpftrace, perf, and ftrace.

The suite is implemented as an MCP Server. This architecture allows an AI agent to leverage the server to diagnose Linux issues and perform targeted system debugging by remotely executing and retrieving tracing data from these powerful tools.

• Repo: https://github.com/r1chard-lyu/systracesuite
• Demo: Slides

Goals

1. Build an MCP Server that can integrate various Linux debugging and tracing tools, including bpftrace, perf, ftrace, strace, and others, with support for future expansion of additional tools.

2. Perform testing by intentionally creating bugs or issues that impact system performance, allowing an AI agent to analyze the root cause and identify the underlying problem.

Resources

• Gemini CLI: https://geminicli.com/
• eBPF: https://ebpf.io/
• bpftrace: https://github.com/bpftrace/bpftrace/
• perf: https://perfwiki.github.io/main/
• ftrace: https://github.com/r1chard-lyu/tracium/

---

Kubernetes-Based ML Lifecycle Automation by lmiranda

Description

This project aims to build a complete end-to-end Machine Learning pipeline running entirely on Kubernetes, using Go, and containerized ML components.

The pipeline will automate the lifecycle of a machine learning model, including:

• Data ingestion/collection
• Model training as a Kubernetes Job
• Model artifact storage in an S3-compatible registry (e.g. Minio)
• A Go-based deployment controller that automatically deploys new model versions to Kubernetes using Rancher
• A lightweight inference service that loads and serves the latest model
• Monitoring of model performance and service health through Prometheus/Grafana

The outcome is a working prototype of an MLOps workflow that demonstrates how AI workloads can be trained, versioned, deployed, and monitored using the Kubernetes ecosystem.

Goals

By the end of Hack Week, the project should:

1. Produce a fully functional ML pipeline running on Kubernetes with:

   • Data collection job
   • Training job container
   • Storage and versioning of trained models
   • Automated deployment of new model versions
   • Model inference API service
   • Basic monitoring dashboards

2. Showcase a Go-based deployment automation component, which scans the model registry and automatically generates & applies Kubernetes manifests for new model versions.

3. Enable continuous improvement by making the system modular and extensible (e.g., additional models, metrics, autoscaling, or drift detection can be added later).

4. Prepare a short demo explaining the end-to-end process and how new models flow through the system.

Resources

Project Repository

Updates

1. Training pipeline and datasets
2. Inference Service py

---

Exploring Modern AI Trends and Kubernetes-Based AI Infrastructure by jluo

Description

Build a solid understanding of the current landscape of Artificial Intelligence and how modern cloud-native technologies—especially Kubernetes—support AI workloads.

Goals

Use Gemini Learning Mode to guide the exploration, surface relevant concepts, and structure the learning journey:

• Gain insight into the latest AI trends, tools, and architectural concepts.
• Understand how Kubernetes and related cloud-native technologies are used in the AI ecosystem (model training, deployment, orchestration, MLOps).

Resources

• Red Hat AI Topic Articles

  • https://www.redhat.com/en/topics/ai

• Kubeflow Documentation

  • https://www.kubeflow.org/docs/

• Q4 2025 CNCF Technology Landscape Radar report:

  • https://www.cncf.io/announcements/2025/11/11/cncf-and-slashdata-report-finds-leading-ai-tools-gaining-adoption-in-cloud-native-ecosystems/
  • https://www.cncf.io/wp-content/uploads/2025/11/cncfreporttechradar_111025a.pdf

• Agent-to-Agent (A2A) Protocol

  • https://developers.googleblog.com/en/a2a-a-new-era-of-agent-interoperability/

---

Create a Cloud-Native policy engine with notifying capabilities to optimize resource usage by gbazzotti

Description

The goal of this project is to begin the initial phase of development of an all-in-one Cloud-Native Policy Engine that notifies resource owners when their resources infringe predetermined policies. This was inspired by a current issue in the CES-SRE Team where other solutions seemed to not exactly correspond to the needs of the specific workloads running on the Public Cloud Team space.

The initial architecture can be checked out on the Repository listed under Resources.

Among the features that will differ this project from other monitoring/notification systems:

• Pre-defined sensible policies written at the software-level, avoiding a learning curve by requiring users to write their own policies
• All-in-one functionality: logging, mailing and all other actions are not required to install any additional plugins/packages
• Easy account management, being able to parse all required configuration by a single JSON file
• Eliminate integrations by not requiring metrics to go through a data-agreggator

Goals

• Create a minimal working prototype following the workflow specified on the documentation
• Provide instructions on installation/usage
• Work on email notifying capabilities

Resources

• GitHub Repository

---