GenAI-Powered Systemic Bug Evaluation and Management Assistant by rtsvetkov

Motivation

What is the decision critical question which one can ask on a bug? How this question affects the decision on a bug and why?

Let's make GenAI look on the bug from the systemic point and evaluate what we don't know. Which piece of information is missing to take a decision?

Description

To build a tool that takes a raw bug report (including error messages and context) and uses a large language model (LLM) to generate a series of structured, Socratic-style or Systemic questions designed to guide a the integration and development toward the root cause, rather than just providing a direct, potentially incorrect fix.

Goals

Set up a Python environment

Set the environment and get a Gemini API key. 2. Collect 5-10 realistic bug reports (from open-source projects, personal projects, or public forums like Stack Overflow—include the error message and the initial context).

Build the Dialogue Loop

1. Write a basic Python script using the Gemini API.
2. Implement a simple conversational loop: User Input (Bug) -> AI Output (Question) -> User Input (Answer to AI's question) -> AI Output (Next Question). Code Implementation

Socratic/Systemic Strategy Implementation

1. Refine the logic to ensure the questions follow a Socratic and Systemic path (e.g., from symptom-> context -> assumptions -> -> critical parts -> ).
2. Implement Function Calling (an advanced feature of the Gemini API) to suggest specific actions to the user, like "Run a ping test" or "Check the database logs."
3. Implement Bugzillla call to collect the
4. Implement Questioning Framework as LLVM pre-conditioning
5. Define set of instructions
6. Assemble the Tool

Resources

What are Systemic Questions?

Systemic questions explore the relationships, patterns, and interactions within a system rather than focusing on isolated elements.
In IT, they help uncover hidden dependencies, feedback loops, assumptions, and side-effects during debugging or architecture analysis.

Gitlab Project

gitlab.suse.de/sle-prjmgr/BugDecisionCritical_Question

---

Self-Scaling LLM Infrastructure Powered by Rancher by ademicev0

Self-Scaling LLM Infrastructure Powered by Rancher

---

Description

The Problem

Running LLMs can get expensive and complex pretty quickly.

Today there are typically two choices:

1. Use cloud APIs like OpenAI or Anthropic. Easy to start with, but costs add up at scale.
2. Self-host everything - set up Kubernetes, figure out GPU scheduling, handle scaling, manage model serving... it's a lot of work.

What if there was a middle ground?

What if infrastructure scaled itself instead of making you scale it?

Can we use existing Rancher capabilities like CAPI, autoscaling, and GitOps to make this simpler instead of building everything from scratch?

Project Repository: github.com/alexander-demicev/llmserverless

---

What This Project Does

A key feature is hybrid deployment: requests can be routed based on complexity or privacy needs. Simple or low-sensitivity queries can use public APIs (like OpenAI), while complex or private requests are handled in-house on local infrastructure. This flexibility allows balancing cost, privacy, and performance - using cloud for routine tasks and on-premises resources for sensitive or demanding workloads.

A complete, self-scaling LLM infrastructure that:

• Scales to zero when idle (no idle costs)
• Scales up automatically when requests come in
• Adds more nodes when needed, removes them when demand drops
• Runs on any infrastructure - laptop, bare metal, or cloud

Think of it as "serverless for LLMs" - focus on building, the infrastructure handles itself.

How It Works

A combination of open source tools working together:

Flow:

• Users interact with OpenWebUI (chat interface)
• Requests go to LiteLLM Gateway
• LiteLLM routes requests to:
  • Ollama (Knative) for local model inference (auto-scales pods)
  • Or cloud APIs for fallback

---

Flaky Tests AI Finder for Uyuni and MLM Test Suites by oscar-barrios

Description

Our current Grafana dashboards provide a great overview of test suite health, including a panel for "Top failed tests." However, identifying which of these failures are due to legitimate bugs versus intermittent "flaky tests" is a manual, time-consuming process. These flaky tests erode trust in our test suites and slow down development.

This project aims to build a simple but powerful Python script that automates flaky test detection. The script will directly query our Prometheus instance for the historical data of each failed test, using the jenkins_build_test_case_failure_age metric. It will then format this data and send it to the Gemini API with a carefully crafted prompt, asking it to identify which tests show a flaky pattern.

The final output will be a clean JSON list of the most probable flaky tests, which can then be used to populate a new "Top Flaky Tests" panel in our existing Grafana test suite dashboard.

Goals

By the end of Hack Week, we aim to have a single, working Python script that:

1. Connects to Prometheus and executes a query to fetch detailed test failure history.
2. Processes the raw data into a format suitable for the Gemini API.
3. Successfully calls the Gemini API with the data and a clear prompt.
4. Parses the AI's response to extract a simple list of flaky tests.
5. Saves the list to a JSON file that can be displayed in Grafana.
6. New panel in our Dashboard listing the Flaky tests

Resources

• Jenkins Prometheus Exporter: https://github.com/uyuni-project/jenkins-exporter/
• Data Source: Our internal Prometheus server.
• Key Metric: jenkins_build_test_case_failure_age{jobname, buildid, suite, case, status, failedsince}.
• Existing Query for Reference: count by (suite) (max_over_time(jenkins_build_test_case_failure_age{status=~"FAILED|REGRESSION", jobname="$jobname"}[$__range])).
• AI Model: The Google Gemini API.
• Example about how to interact with Gemini API: https://github.com/srbarrios/FailTale/
• Visualization: Our internal Grafana Dashboard.
• Internal IaC: https://gitlab.suse.de/galaxy/infrastructure/-/tree/master/srv/salt/monitoring

Outcome

• Jenkins Flaky Test Detector: https://github.com/srbarrios/jenkins-flaky-tests-detector and its container
• IaC on MLM Team: https://gitlab.suse.de/galaxy/infrastructure/-/tree/master/srv/salt/monitoring/jenkinsflakytestsdetector?reftype=heads, https://gitlab.suse.de/galaxy/infrastructure/-/blob/master/srv/salt/monitoring/grafana/dashboards/flaky-tests.json?ref_type=heads, and others.
• Grafana Dashboard: https://grafana.mgr.suse.de/d/flaky-tests/flaky-tests-detection @ @ text

---

Song Search with CLAP by gcolangiuli

Description

Contrastive Language-Audio Pretraining (CLAP) is an open-source library that enables the training of a neural network on both Audio and Text descriptions, making it possible to search for Audio using a Text input. Several pre-trained models for song search are already available on huggingface

Goals

Evaluate how CLAP can be used for song searching and determine which types of queries yield the best results by developing a Minimum Viable Product (MVP) in Python. Based on the results of this MVP, future steps could include:

• Music Tagging;
• Free text search;
• Integration with an LLM (for example, with MCP or the OpenAI API) for music suggestions based on your own library.

The code for this project will be entirely written using AI to better explore and demonstrate AI capabilities.

Result

In this MVP we implemented:

• Async Song Analysis with Clap model
• Free Text Search of the songs
• Similar song search based on vector representation
• Containerised version with web interface

We also documented what went well and what can be improved in the use of AI.

You can have a look at the result here:

• Result Presentation
• hw25-song-search Github repo

Future implementation can be related to performance improvement and stability of the analysis.

References

• CLAP: The main model being researched;
• huggingface: Pre-trained models for CLAP;
• Free Music Archive: Creative Commons songs that can be used for testing;

---

Try out Neovim Plugins supporting AI Providers by enavarro_suse

Description

Experiment with several Neovim plugins that integrate AI model providers such as Gemini and Ollama.

Goals

Evaluate how these plugins enhance the development workflow, how they differ in capabilities, and how smoothly they integrate into Neovim for day-to-day coding tasks.

Resources

• Neovim 0.11.5
• AI-enabled Neovim plugins:
  • avante.nvim: https://github.com/yetone/avante.nvim
  • Gp.nvim: https://github.com/Robitx/gp.nvim
  • parrot.nvim: https://github.com/frankroeder/parrot.nvim
  • gemini.nvim: https://dotfyle.com/plugins/kiddos/gemini.nvim
  • ...
• Accounts or API keys for AI model providers.
• Local model serving setup (e.g., Ollama)
• Test projects or codebases for practical evaluation:
  • OBS: https://build.opensuse.org/
  • OBS blog and landing page: https://openbuildservice.org/
  • ...

---

Self-Scaling LLM Infrastructure Powered by Rancher by ademicev0

Self-Scaling LLM Infrastructure Powered by Rancher

---

Description

The Problem

Running LLMs can get expensive and complex pretty quickly.

Today there are typically two choices:

1. Use cloud APIs like OpenAI or Anthropic. Easy to start with, but costs add up at scale.
2. Self-host everything - set up Kubernetes, figure out GPU scheduling, handle scaling, manage model serving... it's a lot of work.

What if there was a middle ground?

What if infrastructure scaled itself instead of making you scale it?

Can we use existing Rancher capabilities like CAPI, autoscaling, and GitOps to make this simpler instead of building everything from scratch?

Project Repository: github.com/alexander-demicev/llmserverless

---

What This Project Does

A key feature is hybrid deployment: requests can be routed based on complexity or privacy needs. Simple or low-sensitivity queries can use public APIs (like OpenAI), while complex or private requests are handled in-house on local infrastructure. This flexibility allows balancing cost, privacy, and performance - using cloud for routine tasks and on-premises resources for sensitive or demanding workloads.

A complete, self-scaling LLM infrastructure that:

• Scales to zero when idle (no idle costs)
• Scales up automatically when requests come in
• Adds more nodes when needed, removes them when demand drops
• Runs on any infrastructure - laptop, bare metal, or cloud

Think of it as "serverless for LLMs" - focus on building, the infrastructure handles itself.

How It Works

A combination of open source tools working together:

Flow:

• Users interact with OpenWebUI (chat interface)
• Requests go to LiteLLM Gateway
• LiteLLM routes requests to:
  • Ollama (Knative) for local model inference (auto-scales pods)
  • Or cloud APIs for fallback

---

Extended private brain - RAG my own scripts and data into offline LLM AI by tjyrinki_suse

Description

For purely studying purposes, I'd like to find out if I could teach an LLM some of my own accumulated knowledge, to use it as a sort of extended brain.

I might use qwen3-coder or something similar as a starting point.

Everything would be done 100% offline without network available to the container, since I prefer to see when network is needed, and make it so it's never needed (other than initial downloads).

Goals

1. Learn something about RAG, LLM, AI.
2. Find out if everything works offline as intended.
3. As an end result have a new way to access my own existing know-how, but so that I can query the wisdom in them.
4. Be flexible to pivot in any direction, as long as there are new things learned.

Resources

To be found on the fly.

Timeline

Day 1 (of 4)

• Tried out a RAG demo, expanded on feeding it my own data
• Experimented with qwen3-coder to add a persistent chat functionality, and keeping vectors in a pickle file
• Optimizations to keep everything within context window
• Learn and add a bit of PyTest

Day 2

• More experimenting and more data
• Study ChromaDB
• Add a Web UI that works from another computer even though the container sees network is down

Day 3

• The above RAG is working well enough for demonstration purposes.
• Pivot to trying out OpenCode, configuring local Ollama qwen3-coder there, to analyze the RAG demo.
• Figured out how to configure Ollama template to be usable under OpenCode. OpenCode locally is super slow to just running qwen3-coder alone.

Day 4 (final day)

• Battle with OpenCode that was both slow and kept on piling up broken things.
• Call it success as after all the agentic AI was working locally.
• Clean up the mess left behind a bit.

Blog Post

Summarized the findings at blog post.

---

Creating test suite using LLM on existing codebase of a solar router by fcrozat

Description

Two years ago, I evaluated solar routers as part of hackweek24, I've assembled one and it is running almost smoothly.

However, its code quality is not perfect and the codebase doesn't have any testcase (which is tricky, since it is embedded code and rely on getting external data to react).

Before improving the code itself, a testsuite should be created to ensure code additional don't cause regression.

Goals

Create a testsuite, allowing to test solar router code in a virtual environment. Using LLM to help to create this test suite.

If succesful, try to improve the codebase itself by having it reviewed by LLM.

Resources

Solar router github project

---

SUSE Observability MCP server by drutigliano

Description

The idea is to implement the SUSE Observability Model Context Protocol (MCP) Server as a specialized, middle-tier API designed to translate the complex, high-cardinality observability data from StackState (topology, metrics, and events) into highly structured, contextually rich, and LLM-ready snippets.

This MCP Server abstract the StackState APIs. Its primary function is to serve as a Tool/Function Calling target for AI agents. When an AI receives an alert or a user query (e.g., "What caused the outage?"), the AI calls an MCP Server endpoint. The server then fetches the relevant operational facts, summarizes them, normalizes technical identifiers (like URNs and raw metric names) into natural language concepts, and returns a concise JSON or YAML payload. This payload is then injected directly into the LLM's prompt, ensuring the final diagnosis or action is grounded in real-time, accurate SUSE Observability data, effectively minimizing hallucinations.

Goals

• Grounding AI Responses: Ensure that all AI diagnoses, root cause analyses, and action recommendations are strictly based on verifiable, real-time data retrieved from the SUSE Observability StackState platform.
• Simplifying Data Access: Abstract the complexity of StackState's native APIs (e.g., Time Travel, 4T Data Model) into simple, semantic functions that can be easily invoked by LLM tool-calling mechanisms.
• Data Normalization: Convert complex, technical identifiers (like component URNs, raw metric names, and proprietary health states) into standardized, natural language terms that an LLM can easily reason over.
• Enabling Automated Remediation: Define clear, action-oriented MCP endpoints (e.g., execute_runbook) that allow the AI agent to initiate automated operational workflows (e.g., restarts, scaling) after a diagnosis, closing the loop on observability.

 Hackweek STEP

• Create a functional MCP endpoint exposing one (or more) tool(s) to answer queries like "What is the health of service X?") by fetching, normalizing, and returning live StackState data in an LLM-ready format.

 Scope

• Implement read-only MCP server that can:
  • Connect to a live SUSE Observability instance and authenticate (with API token)
  • Use tools to fetch data for a specific component URN (e.g., current health state, metrics, possibly topology neighbors, ...).
  • Normalize response fields (e.g., URN to "Service Name," health state DEVIATING to "Unhealthy", raw metrics).
  • Return the data as a structured JSON payload compliant with the MCP specification.

Deliverables

• MCP Server v0.1 A running Golang MCP server with at least one tool.
• A README.md and a test script (e.g., curl commands or a simple notebook) showing how an AI agent would call the endpoint and the resulting JSON payload.

Outcome A functional and testable API endpoint that proves the core concept: translating complex StackState data into a simple, LLM-ready format. This provides the foundation for developing AI-driven diagnostics and automated remediation.

Resources

• https://www.honeycomb.io/blog/its-the-end-of-observability-as-we-know-it-and-i-feel-fine
• https://www.datadoghq.com/blog/datadog-remote-mcp-server
• https://modelcontextprotocol.io/specification/2025-06-18/index
• https://modelcontextprotocol.io/docs/develop/build-server

 Basic implementation

• https://github.com/drutigliano19/suse-observability-mcp-server

---

Results

Successfully developed and delivered a fully functional SUSE Observability MCP Server that bridges language models with SUSE Observability's operational data. This project demonstrates how AI agents can perform intelligent troubleshooting and root cause analysis using structured access to real-time infrastructure data.

Example execution

---

Song Search with CLAP by gcolangiuli

Description

Contrastive Language-Audio Pretraining (CLAP) is an open-source library that enables the training of a neural network on both Audio and Text descriptions, making it possible to search for Audio using a Text input. Several pre-trained models for song search are already available on huggingface

Goals

Evaluate how CLAP can be used for song searching and determine which types of queries yield the best results by developing a Minimum Viable Product (MVP) in Python. Based on the results of this MVP, future steps could include:

• Music Tagging;
• Free text search;
• Integration with an LLM (for example, with MCP or the OpenAI API) for music suggestions based on your own library.

The code for this project will be entirely written using AI to better explore and demonstrate AI capabilities.

Result

In this MVP we implemented:

• Async Song Analysis with Clap model
• Free Text Search of the songs
• Similar song search based on vector representation
• Containerised version with web interface

We also documented what went well and what can be improved in the use of AI.

You can have a look at the result here:

• Result Presentation
• hw25-song-search Github repo

Future implementation can be related to performance improvement and stability of the analysis.

References

• CLAP: The main model being researched;
• huggingface: Pre-trained models for CLAP;
• Free Music Archive: Creative Commons songs that can be used for testing;

---

dynticks-testing: analyse perf / trace-cmd output and aggregate data by m.crivellari

Description

dynticks-testing is a project started years ago by Frederic Weisbecker. One of the feature is to check the actual configuration (isolcpus, irqaffinity etc etc) and give feedback on it.

An important goal of this tool is to parse the output of trace-cmd / perf and provide more readable data, showing the duration of every events grouped by PID (showing also the CPU number, if the tasks has been migrated etc).

An example of data captured on my laptop (incomplete!!):

          -0     [005] dN.2. 20310.270699: sched_wakeup:         WaylandProxy:46380 [120] CPU:005
          -0     [005] d..2. 20310.270702: sched_switch:         swapper/5:0 [120] R ==> WaylandProxy:46380 [120]
...
    WaylandProxy-46380 [004] d..2. 20310.295397: sched_switch:         WaylandProxy:46380 [120] S ==> swapper/4:0 [120]
          -0     [006] d..2. 20310.295397: sched_switch:         swapper/6:0 [120] R ==> firefox:46373 [120]
         firefox-46373 [006] d..2. 20310.295408: sched_switch:         firefox:46373 [120] S ==> swapper/6:0 [120]
          -0     [004] dN.2. 20310.295466: sched_wakeup:         WaylandProxy:46380 [120] CPU:004

Output of noise_parse.py:

Task: WaylandProxy Pid: 46380 cpus: {4, 5} (Migrated!!!)
        Wakeup Latency                                Nr:        24     Duration:          89
        Sched switch: kworker/12:2                    Nr:         1     Duration:           6

My first contribution is around Nov. 2024!

Goals

• add more features (eg cpuset)
• test / bugfix

Resources

• Frederic's public repository: https://git.kernel.org/pub/scm/linux/kernel/git/frederic/dynticks-testing.git/
• https://docs.kernel.org/timers/no_hz.html#testing

Progresses

isolcpus and cpusets implemented and merged in master: dynticks-testing.git commit

---

early stage kdump support by mbrugger

Project Description

When we experience a early boot crash, we are not able to analyze the kernel dump, as user-space wasn't able to load the crash system. The idea is to make the crash system compiled into the host kernel (think of initramfs) so that we can create a kernel dump really early in the boot process.

Goal for the Hackweeks

1. Investigate if this is possible and the implications it would have (done in HW21)
2. Hack up a PoC (done in HW22 and HW23)
3. Prepare RFC series (giving it's only one week, we are entering wishful thinking territory here).

update HW23

• I was able to include the crash kernel into the kernel Image.
• I'll need to find a way to load that from init/main.c:start_kernel() probably after kcsan_init()
• I workaround for a smoke test was to hack kexec_file_load() systemcall which has two problems:
  1. My initramfs in the porduction kernel does not have a new enough kexec version, that's not a blocker but where the week ended
  2. As the crash kernel is part of init.data it will be already stale once I can call kexec_file_load() from user-space.

The solution is probably to rewrite the POC so that the invocation can be done from init.text (that's my theory) but I'm not sure if I can reuse the kexec infrastructure in the kernel from there, which I rely on heavily.

update HW24

• Day1
  • rebased on v6.12 with no problems others then me breaking the config
  • setting up a new compilation and qemu/virtme env
  • getting desperate as nothing works that used to work
• Day 2
  • getting to call the invocation of loading the early kernel from __init after kcsan_init()

• Day 3

  • fix problem of memdup not being able to alloc so much memory... use 64K page sizes for now
  • code refactoring
  • I'm now able to load the crash kernel
  • When using virtme I can boot into the crash kernel, also it doesn't boot completely (major milestone!), crash in elfcorehdr_read_notes()

• Day 4

  • crash systems crashes (no pun intended) in copy_old_mempage() link; will need to understand elfcorehdr...
  • call path vmcore_init() -> parse_crash_elf_headers() -> elfcorehdr_read() -> read_from_oldmem() -> copy_oldmem_page() -> copy_to_iter()

• Day 5

  • hacking arch/arm64/kernel/crash_dump.c:copy_old_mempage() to see if crash system really starts. It does.
  • fun fact: retested with more reserved memory and with UEFI FW, host kernel crashes in init but directly starts the crash kernel, so it works (somehow) \o/

update HW25

• Day 1
  • rebased crash-kernel on v6.12.59 (for now), still crashing

---

bpftrace contribution by mkoutny

Description

bpftrace is a great tool, no need to sing odes to it here. It can access any kernel data and process them in real time. It provides helpers for some common Linux kernel structures but not all.

Goals

• set up bpftrace toolchain
• learn about bpftrace implementation and internals
• implement support for percpu_counters
• look into some of the first issues
• send a refined PR (on Thu)

Resources

• https://bpftrace.org/
• https://docs.kernel.org/

---

Add Qualcomm Snapdragon 765G (SM7250) basic device tree to mainline linux kernel by pvorel

Qualcomm Snapdragon 765G (SM7250) (smartphone SoC) has no support in the linux kernel, nor in u-boot. Try to add basic device tree support. The hardest part will be to create boot.img which will be accepted by phone.

UART is available for smartphone :).

---

Improve UML page fault handler by ptesarik

Description

Improve UML handling of segmentation faults in kernel mode. Although such page faults are generally caused by a kernel bug, it is annoying if they cause an infinite loop, or panic the kernel. More importantly, a robust implementation allows to write KUnit tests for various guard pages, preventing potential kernel self-protection regressions.

Goals

Convert the UML page fault handler to use oops_* helpers, go through a few review rounds and finally get my patch series merged in 6.14.

Resources

Wrong initial attempt: https://lore.kernel.org/lkml/20231215121431.680-1-petrtesarik@huaweicloud.com/T/

---