Explore LLM evaluation metrics by thbertoldi

Description

Learn the best practices for evaluating LLM performance with an open-source framework such as DeepEval.

Goals

Curate the knowledge learned during practice and present it to colleagues.

-> Maybe publish a blog post on SUSE's blog?

Resources

https://deepeval.com

https://docs.pactflow.io/docs/bi-directional-contract-testing

---

Exploring Modern AI Trends and Kubernetes-Based AI Infrastructure by jluo

Description

Build a solid understanding of the current landscape of Artificial Intelligence and how modern cloud-native technologies—especially Kubernetes—support AI workloads.

Goals

Use Gemini Learning Mode to guide the exploration, surface relevant concepts, and structure the learning journey:

• Gain insight into the latest AI trends, tools, and architectural concepts.
• Understand how Kubernetes and related cloud-native technologies are used in the AI ecosystem (model training, deployment, orchestration, MLOps).

Resources

• Red Hat AI Topic Articles

  • https://www.redhat.com/en/topics/ai

• Kubeflow Documentation

  • https://www.kubeflow.org/docs/

• Q4 2025 CNCF Technology Landscape Radar report:

  • https://www.cncf.io/announcements/2025/11/11/cncf-and-slashdata-report-finds-leading-ai-tools-gaining-adoption-in-cloud-native-ecosystems/
  • https://www.cncf.io/wp-content/uploads/2025/11/cncfreporttechradar_111025a.pdf

• Agent-to-Agent (A2A) Protocol

  • https://developers.googleblog.com/en/a2a-a-new-era-of-agent-interoperability/

---

"what is it" file and directory analysis via MCP and local LLM, for console and KDE by rsimai

Description

Users sometimes wonder what files or directories they find on their local PC are good for. If they can't determine from the filename or metadata, there should an easy way to quickly analyze the content and at least guess the meaning. An LLM could help with that, through the use of a filesystem MCP and to-text-converters for typical file types. Ideally this is integrated into the desktop environment but works as well from a console. All data is processed locally or "on premise", no artifacts remain or leave the system.

Goals

• The user can run a command from the console, to check on a file or directory
• The filemanager contains the "analyze" feature within the context menu
• The local LLM could serve for other use cases where privacy matters

TBD

• Find or write capable one-shot and interactive MCP client
• Find or write simple+secure file access MCP server
• Create local LLM service with appropriate footprint, containerized
• Shell command with options
• KDE integration (Dolphin)
• Package
• Document

Resources

---

Liz - Prompt autocomplete by ftorchia

Description

Liz is the Rancher AI assistant for cluster operations.

Goals

We want to help users when sending new messages to Liz, by adding an autocomplete feature to complete their requests based on the context.

Example:

• User prompt: "Can you show me the list of p"
• Autocomplete suggestion: "Can you show me the list of p...od in local cluster?"

Example:

• User prompt: "Show me the logs of #rancher-"
• Chat console: It shows a drop-down widget, next to the # character, with the list of available pod names starting with "rancher-".

Technical Overview

1. The AI agent should expose a new ws/autocomplete endpoint to proxy autocomplete messages to the LLM.
2. The UI extension should be able to display prompt suggestions and allow users to apply the autocomplete to the Prompt via keyboard shortcuts.

Resources

GitHub repository

---

Try AI training with ROCm and LoRA by bmwiedemann

Description

I want to setup a Radeon RX 9600 XT 16 GB at home with ROCm on Slowroll.

Goals

I want to test how fast AI inference can get with the GPU and if I can use LoRA to re-train an existing free model for some task.

Resources

• https://rocm.docs.amd.com/en/latest/compatibility/compatibility-matrix.html
• https://build.opensuse.org/project/show/science:GPU:ROCm
• https://src.opensuse.org/ROCm/
• https://www.suse.com/c/lora-fine-tuning-llms-for-text-classification/

Results

got inference working with llama.cpp:

export LLAMACPP_ROCM_ARCH=gfx1200
HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
cmake -S . -B build -DGGML_HIP=ON -DAMDGPU_TARGETS=$LLAMACPP_ROCM_ARCH \
-DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=ON \
-Dhipblas_DIR=/usr/lib64/cmake/hipblaslt/ \
&& cmake --build build --config Release -j8
m=models/gpt-oss-20b-mxfp4.gguf
cd $P/llama.cpp && build/bin/llama-server --model $m --threads 8 --port 8005 --host 0.0.0.0 --device ROCm0 --n-gpu-layers 999

Without the --device option it faulted. Maybe because my APU also appears there?

I updated/fixed various related packages: https://src.opensuse.org/ROCm/rocm-examples/pulls/1 https://src.opensuse.org/ROCm/hipblaslt/pulls/1 SR 1320959

benchmark

I benchmarked inference with llama.cpp + gpt-oss-20b-mxfp4.gguf and ROCm offloading to a Radeon RX 9060 XT 16GB. I varied the number of layers that went to the GPU:

• 0 layers 14.49 tokens/s (8 CPU cores)
• 9 layers 17.79 tokens/s 34% VRAM
• 15 layers 22.39 tokens/s 51% VRAM
• 20 layers 27.49 tokens/s 64% VRAM
• 24 layers 41.18 tokens/s 74% VRAM
• 25+ layers 86.63 tokens/s 75% VRAM (only 200% CPU load)

So there is a significant performance-boost if the whole model fits into the GPU's VRAM.

---

GenAI-Powered Systemic Bug Evaluation and Management Assistant by rtsvetkov

Motivation

What is the decision critical question which one can ask on a bug? How this question affects the decision on a bug and why?

Let's make GenAI look on the bug from the systemic point and evaluate what we don't know. Which piece of information is missing to take a decision?

Description

To build a tool that takes a raw bug report (including error messages and context) and uses a large language model (LLM) to generate a series of structured, Socratic-style or Systemic questions designed to guide a the integration and development toward the root cause, rather than just providing a direct, potentially incorrect fix.

Goals

Set up a Python environment

Set the environment and get a Gemini API key. 2. Collect 5-10 realistic bug reports (from open-source projects, personal projects, or public forums like Stack Overflow—include the error message and the initial context).

Build the Dialogue Loop

1. Write a basic Python script using the Gemini API.
2. Implement a simple conversational loop: User Input (Bug) -> AI Output (Question) -> User Input (Answer to AI's question) -> AI Output (Next Question). Code Implementation

Socratic/Systemic Strategy Implementation

1. Refine the logic to ensure the questions follow a Socratic and Systemic path (e.g., from symptom-> context -> assumptions -> -> critical parts -> ).
2. Implement Function Calling (an advanced feature of the Gemini API) to suggest specific actions to the user, like "Run a ping test" or "Check the database logs."
3. Implement Bugzillla call to collect the
4. Implement Questioning Framework as LLVM pre-conditioning
5. Define set of instructions
6. Assemble the Tool

Resources

What are Systemic Questions?

Systemic questions explore the relationships, patterns, and interactions within a system rather than focusing on isolated elements.
In IT, they help uncover hidden dependencies, feedback loops, assumptions, and side-effects during debugging or architecture analysis.

Gitlab Project

gitlab.suse.de/sle-prjmgr/BugDecisionCritical_Question

---

SUSE Observability MCP server by drutigliano

Description

The idea is to implement the SUSE Observability Model Context Protocol (MCP) Server as a specialized, middle-tier API designed to translate the complex, high-cardinality observability data from StackState (topology, metrics, and events) into highly structured, contextually rich, and LLM-ready snippets.

This MCP Server abstract the StackState APIs. Its primary function is to serve as a Tool/Function Calling target for AI agents. When an AI receives an alert or a user query (e.g., "What caused the outage?"), the AI calls an MCP Server endpoint. The server then fetches the relevant operational facts, summarizes them, normalizes technical identifiers (like URNs and raw metric names) into natural language concepts, and returns a concise JSON or YAML payload. This payload is then injected directly into the LLM's prompt, ensuring the final diagnosis or action is grounded in real-time, accurate SUSE Observability data, effectively minimizing hallucinations.

Goals

• Grounding AI Responses: Ensure that all AI diagnoses, root cause analyses, and action recommendations are strictly based on verifiable, real-time data retrieved from the SUSE Observability StackState platform.
• Simplifying Data Access: Abstract the complexity of StackState's native APIs (e.g., Time Travel, 4T Data Model) into simple, semantic functions that can be easily invoked by LLM tool-calling mechanisms.
• Data Normalization: Convert complex, technical identifiers (like component URNs, raw metric names, and proprietary health states) into standardized, natural language terms that an LLM can easily reason over.
• Enabling Automated Remediation: Define clear, action-oriented MCP endpoints (e.g., execute_runbook) that allow the AI agent to initiate automated operational workflows (e.g., restarts, scaling) after a diagnosis, closing the loop on observability.

 Hackweek STEP

• Create a functional MCP endpoint exposing one (or more) tool(s) to answer queries like "What is the health of service X?") by fetching, normalizing, and returning live StackState data in an LLM-ready format.

 Scope

• Implement read-only MCP server that can:
  • Connect to a live SUSE Observability instance and authenticate (with API token)
  • Use tools to fetch data for a specific component URN (e.g., current health state, metrics, possibly topology neighbors, ...).
  • Normalize response fields (e.g., URN to "Service Name," health state DEVIATING to "Unhealthy", raw metrics).
  • Return the data as a structured JSON payload compliant with the MCP specification.

Deliverables

• MCP Server v0.1 A running Golang MCP server with at least one tool.
• A README.md and a test script (e.g., curl commands or a simple notebook) showing how an AI agent would call the endpoint and the resulting JSON payload.

Outcome A functional and testable API endpoint that proves the core concept: translating complex StackState data into a simple, LLM-ready format. This provides the foundation for developing AI-driven diagnostics and automated remediation.

Resources

• https://www.honeycomb.io/blog/its-the-end-of-observability-as-we-know-it-and-i-feel-fine
• https://www.datadoghq.com/blog/datadog-remote-mcp-server
• https://modelcontextprotocol.io/specification/2025-06-18/index
• https://modelcontextprotocol.io/docs/develop/build-server

 Basic implementation

• https://github.com/drutigliano19/suse-observability-mcp-server

---

Results

Successfully developed and delivered a fully functional SUSE Observability MCP Server that bridges language models with SUSE Observability's operational data. This project demonstrates how AI agents can perform intelligent troubleshooting and root cause analysis using structured access to real-time infrastructure data.

Example execution

---

Create a Cloud-Native policy engine with notifying capabilities to optimize resource usage by gbazzotti

Description

The goal of this project is to begin the initial phase of development of an all-in-one Cloud-Native Policy Engine that notifies resource owners when their resources infringe predetermined policies. This was inspired by a current issue in the CES-SRE Team where other solutions seemed to not exactly correspond to the needs of the specific workloads running on the Public Cloud Team space.

The initial architecture can be checked out on the Repository listed under Resources.

Among the features that will differ this project from other monitoring/notification systems:

• Pre-defined sensible policies written at the software-level, avoiding a learning curve by requiring users to write their own policies
• All-in-one functionality: logging, mailing and all other actions are not required to install any additional plugins/packages
• Easy account management, being able to parse all required configuration by a single JSON file
• Eliminate integrations by not requiring metrics to go through a data-agreggator

Goals

• Create a minimal working prototype following the workflow specified on the documentation
• Provide instructions on installation/usage
• Work on email notifying capabilities

Resources

• GitHub Repository

---