Gen-AI chatbots and test-automation of generated responses by mdati

Description

Start experimenting the generative SUSE-AI chat bot, asking questions on different areas of knowledge or science and possibly analyze the quality of the LLM model response, specific and comparative, checking the answers provided by different LLM models to a same query, using proper quality metrics or tools or methodologies.

Try to define basic guidelines and requirements for quality test automation of AI-generated responses.

First approach of investigation can be based on manual testing: methodologies, findings and data can be useful then to organize valid automated testing.

Goals

• Identify criteria and measuring scales for assessment of a text content.
• Define quality of an answer/text based on defined criteria .
• Identify some knowledge sectors and a proper list of problems/questions per sector.
• Manually run query session and apply evaluation criteria to answers.
• Draft requirements for test automation of AI answers.

Resources

• Announcement of SUSE-AI for Hack Week in Slack
• Openplatform and related 3 LLM models gemma:2b, llama3.1:8b, qwen2.5-coder:3b.

Notes

• Foundation models (FMs):
  are large deep learning neural networks, trained on massive datasets, that have changed the way data scientists approach machine learning (ML). Rather than develop artificial intelligence (AI) from scratch, data scientists use a foundation model as a starting point to develop ML models that power new applications more quickly and cost-effectively.

• Large language models (LLMs):
  are a category of foundation models pre-trained on immense amounts of data acquiring abilities by learning statistical relationships from vast amounts of text during a self- and semi-supervised training process, making them capable of understanding and generating natural language and other types of content , to perform a wide range of tasks.
  LLMs can be used for generative AI (artificial intelligence) to produce content based on input prompts in human language.

Validation of a AI-generated answer is not an easy task to perform, as manually as automated.
An LLM answer text shall contain a given level of informations: correcness, completeness, reasoning description etc.
We shall rely in properly applicable and measurable criteria of validation to get an assessment in a limited amount of time and resources.

---

Automated Test Report reviewer by oscar-barrios

Description

In SUMA/Uyuni team we spend a lot of time reviewing test reports, analyzing each of the test cases failing, checking if the test is a flaky test, checking logs, etc.

Goals

Speed up the review by automating some parts through AI, in a way that we can consume some summary of that report that could be meaningful for the reviewer.

Resources

No idea about the resources yet, but we will make use of:

• HTML/JSON Report (text + screenshots)
• The Test Suite Status GithHub board (via API)
• The environment tested (via SSH)
• The test framework code (via files)

---

SUSE AI Meets the Game Board by moio

Use tabletopgames.ai’s open source TAG and PyTAG frameworks to apply Statistical Forward Planning and Deep Reinforcement Learning to two board games of our own design. On an all-green, all-open source, all-AWS stack!

---

Results: Infrastructure Achievements

We successfully built and automated a containerized stack to support our AI experiments. This included:

• a Fully-Automated, One-Command, GPU-accelerated Kubernetes setup: we created an OpenTofu based script, tofu-tag, to deploy SUSE's RKE2 Kubernetes running on CUDA-enabled nodes in AWS, powered by openSUSE with GPU drivers and gpu-operator
• Containerization of the TAG and PyTAG frameworks: TAG (Tabletop AI Games) and PyTAG were patched for seamless deployment in containerized environments. We automated the container image creation process with GitHub Actions. Our forks (PRs upstream upcoming):
  • https://github.com/moio/TabletopGames/tree/hackweek24
  • https://github.com/moio/PyTAG/tree/hackweek24

./deploy.sh and voilà - Kubernetes running PyTAG (k9s, above) with GPU acceleration (nvtop, below)

Results: Game Design Insights

Our project focused on modeling and analyzing two card games of our own design within the TAG framework:

• Game Modeling: We implemented models for Dario's "Bamboo" and Silvio's "Totoro" and "R3" games, enabling AI agents to play thousands of games ...in minutes!
• AI-driven optimization: By analyzing statistical data on moves, strategies, and outcomes, we iteratively tweaked the game mechanics and rules to achieve better balance and player engagement.
• Advanced analytics: Leveraging AI agents with Monte Carlo Tree Search (MCTS) and random action selection, we compared performance metrics to identify optimal strategies and uncover opportunities for game refinement .
  • more about Bamboo on Dario's site
  • more about R3 on Silvio's site (italian, translation coming)
  • more about Totoro on Silvio's site

A family picture of our card games in progress. From the top: Bamboo, Totoro, R3

Results: Learning, Collaboration, and Innovation

Beyond technical accomplishments, the project showcased innovative approaches to coding, learning, and teamwork:

• "Trio programming" with AI assistance: Our "trio programming" approach—two developers and GitHub Copilot—was a standout success, especially in handling slightly-repetitive but not-quite-exactly-copypaste tasks. Java as a language tends to be verbose and we found it to be fitting particularly well.
• AI tools for reporting and documentation: We extensively used AI chatbots to streamline writing and reporting. (Including writing this report! ...but this note was added manually during edit!)
• GPU compute expertise: Overcoming challenges with CUDA drivers and cloud infrastructure deepened our understanding of GPU-accelerated workloads in the open-source ecosystem.
• Game design as a learning platform: By blending AI techniques with creative game design, we learned not only about AI strategies but also about making games fun, engaging, and balanced.

Last but not least we had a lot of fun! ...and this was definitely not a chatbot generated line!

The Context: AI + Board Games

---

Use local/private LLM for semantic knowledge search by digitaltomm

Description

Use a local LLM, based on SUSE AI (ollama, openwebui) to power geeko search (public instance: https://geeko.port0.org/).

Goals

Build a SUSE internal instance of https://geeko.port0.org/ that can operate on internal resources, crawling confluence.suse.com, gitlab.suse.de, etc.

Resources

Repo: https://github.com/digitaltom/semantic-knowledge-search

Public instance: https://geeko.port0.org/

Results

Internal instance:

I have an internal test instance running which has indexed a couple of internal wiki pages from the SCC team. It's using the ollama (llama3.1:8b) backend of suse-ai.openplatform.suse.com to create embedding vectors for indexed resources and to create a chat response. The semantic search for documents is done with a vector search inside of sqlite, using sqlite-vec.

---

Research how LLMs could help to Linux developers and/or users by anicka

Description

Large language models like ChatGPT have demonstrated remarkable capabilities across a variety of applications. However, their potential for enhancing the Linux development and user ecosystem remains largely unexplored. This project seeks to bridge that gap by researching practical applications of LLMs to improve workflows in areas such as backporting, packaging, log analysis, system migration, and more. By identifying patterns that LLMs can leverage, we aim to uncover new efficiencies and automation strategies that can benefit developers, maintainers, and end users alike.

Goals

• Evaluate Existing LLM Capabilities: Research and document the current state of LLM usage in open-source and Linux development projects, noting successes and limitations.
• Prototype Tools and Scripts: Develop proof-of-concept scripts or tools that leverage LLMs to perform specific tasks like automated log analysis, assisting with backporting patches, or generating packaging metadata.
• Assess Performance and Reliability: Test the tools' effectiveness on real-world Linux data and analyze their accuracy, speed, and reliability.
• Identify Best Use Cases: Pinpoint which tasks are most suitable for LLM support, distinguishing between high-impact and impractical applications.
• Document Findings and Recommendations: Summarize results with clear documentation and suggest next steps for potential integration or further development.

Resources

• Local LLM Implementations: Access to locally hosted LLMs such as LLaMA, GPT-J, or similar open-source models that can be run and fine-tuned on local hardware.
• Computing Resources: Workstations or servers capable of running LLMs locally, equipped with sufficient GPU power for training and inference.
• Sample Data: Logs, source code, patches, and packaging data from openSUSE or SUSE repositories for model training and testing.
• Public LLMs for Benchmarking: Access to APIs from platforms like OpenAI or Hugging Face for comparative testing and performance assessment.
• Existing NLP Tools: Libraries such as spaCy, Hugging Face Transformers, and PyTorch for building and interacting with local LLMs.
• Technical Documentation: Tutorials and resources focused on setting up and optimizing local LLMs for tasks relevant to Linux development.
• Collaboration: Engagement with community experts and teams experienced in AI and Linux for feedback and joint exploration.

---

FamilyTrip Planner: A Personalized Travel Planning Platform for Families by pherranz

Description

FamilyTrip Planner is an innovative travel planning application designed to optimize travel experiences for families with children. By integrating APIs for flights, accommodations, and local activities, the app generates complete itineraries tailored to each family’s unique interests and needs. Recommendations are based on customizable parameters such as destination, trip duration, children’s ages, and personal preferences. FamilyTrip Planner not only simplifies the travel planning process but also offers a comprehensive, personalized experience for families.

Goals

This project aims to: - Create a user-friendly platform that assists families in planning complete trips, from flight and accommodation options to recommended family-friendly activities. - Provide intelligent, personalized travel itineraries using artificial intelligence to enhance travel enjoyment and minimize time and cost. - Serve as an educational project for exploring Go programming and artificial intelligence, with the goal of building proficiency in both.

Resources

To develop FamilyTrip Planner, the project will leverage: - APIs such as Skyscanner, Google Places, and TripAdvisor to source real-time information on flights, accommodations, and activities. - Go programming language to manage data integration, API connections, and backend development. - Basic machine learning libraries to implement AI-driven itinerary suggestions tailored to family needs and preferences.

---

Agama installer on-line demo by lslezak

Description

The Agama installer provides a quite complex user interface. We have some screenshots on the web page but as it is basically a web application it would be nice to have some on-line demo where users could click and check it live.

The problem is that the Agama server directly accesses the hardware (storage probing) and loads installation repositories. We cannot easily mock this in the on-line demo so the easiest way is to have just a read-only demo. You could explore the configuration options but you could not change anything, all changes would be ignored.

The read-only demo would be a bit limited but I still think it would be useful for potential users get the feeling of the new Agama installer and get familiar with it before using in a real installation.

As a proof of concept I already created this on-line demo.

The implementation basically builds Agama in two modes - recording mode where it saves all REST API responses and replay mode where it for the REST API requests returns the previously recorded responses. Recording in the browser is inconvenient and error prone, there should be some scripting instead (see below).

Goals

• Create an Agama on-line demo which can be easily tested by users
• The Agama installer is still in alpha phase and in active development, the online demo needs to be easily rebuilt with the latest Agama version
• Ideally there should be some automation so the demo page is rebuilt automatically without any developer interactions (once a day or week?)

TODO

• Use OpenAPI to get all Agama REST API endpoints, write a script which queries all the endpoints automatically and saves the collected data to a file (see this related PR).
• Write a script for starting an Agama VM (use libvirt/qemu?), the script should ensure we always use the same virtual HW so if we need to dump the latest REST API state we get the same (or very similar data). This should ensure the demo page does not change much regarding the storage proposal etc...
• Fix changing the product, currently it gets stuck after clicking the "Select" button.
• Move the mocking data (the recorded REST API responses) outside the Agama sources, it's too big and will be probably often updated. To avoid messing the history keep it in a separate GitHub repository
• Allow changing the UI language
• Display some note (watermark) in the page so it is clear it is a read-only demo (probably with some version or build date to know how old it is)
• Automation for building new demo page from the latest sources. There should be some check which ensures the recorded data still matches the OpenAPI specification.

Changing the UI language

This will be quite tricky because selecting the proper translation file is done on the server side. We would probably need to completely re-implement the logic in the browser side and adapt the server for that.

Also some REST API responses contain translated texts (storage proposal, pattern names in software). We would need to query the respective endpoints in all supported languages and return the correct response in runtime according to the currently selected language.

Resources

• Agama sources
• Experimental proof of concept demo
• The respective source code change

---

Agama installer on-line demo by lslezak

Description

The Agama installer provides a quite complex user interface. We have some screenshots on the web page but as it is basically a web application it would be nice to have some on-line demo where users could click and check it live.

The problem is that the Agama server directly accesses the hardware (storage probing) and loads installation repositories. We cannot easily mock this in the on-line demo so the easiest way is to have just a read-only demo. You could explore the configuration options but you could not change anything, all changes would be ignored.

The read-only demo would be a bit limited but I still think it would be useful for potential users get the feeling of the new Agama installer and get familiar with it before using in a real installation.

As a proof of concept I already created this on-line demo.

The implementation basically builds Agama in two modes - recording mode where it saves all REST API responses and replay mode where it for the REST API requests returns the previously recorded responses. Recording in the browser is inconvenient and error prone, there should be some scripting instead (see below).

Goals

• Create an Agama on-line demo which can be easily tested by users
• The Agama installer is still in alpha phase and in active development, the online demo needs to be easily rebuilt with the latest Agama version
• Ideally there should be some automation so the demo page is rebuilt automatically without any developer interactions (once a day or week?)

TODO

• Use OpenAPI to get all Agama REST API endpoints, write a script which queries all the endpoints automatically and saves the collected data to a file (see this related PR).
• Write a script for starting an Agama VM (use libvirt/qemu?), the script should ensure we always use the same virtual HW so if we need to dump the latest REST API state we get the same (or very similar data). This should ensure the demo page does not change much regarding the storage proposal etc...
• Fix changing the product, currently it gets stuck after clicking the "Select" button.
• Move the mocking data (the recorded REST API responses) outside the Agama sources, it's too big and will be probably often updated. To avoid messing the history keep it in a separate GitHub repository
• Allow changing the UI language
• Display some note (watermark) in the page so it is clear it is a read-only demo (probably with some version or build date to know how old it is)
• Automation for building new demo page from the latest sources. There should be some check which ensures the recorded data still matches the OpenAPI specification.

Changing the UI language

This will be quite tricky because selecting the proper translation file is done on the server side. We would probably need to completely re-implement the logic in the browser side and adapt the server for that.

Also some REST API responses contain translated texts (storage proposal, pattern names in software). We would need to query the respective endpoints in all supported languages and return the correct response in runtime according to the currently selected language.

Resources

• Agama sources
• Experimental proof of concept demo
• The respective source code change

---

Longhorn UI Extension (POC) by yiya.chen

Description

The goal is to create a Longhorn UI extension within Rancher using existing resources.
Longhorn’s UI is built using React, while Rancher’s UI extensions are built using Vue. Developers will explore different approaches to integrate and extend Longhorn’s UI within Rancher’s Vue-based ecosystem, aiming to create a seamless, functional UI extension.

Goals

• Build a Longhorn UI extension (look and feel)
• Support theme switching to align with Rancher’s UI

Results

• https://github.com/a110605/longhorn-hackday
• https://github.com/a110605/longhorn-ui/tree/darkmode
• https://github.com/houhoucoop/hackweek/tree/main/hackweek24

Resources

• Longhorn UI: https://github.com/longhorn/longhorn-ui
• Rancher UI Extension: https://extensions.rancher.io/extensions/next/home
• darkreader: https://www.npmjs.com/package/darkreader
• veaury: https://github.com/gloriasoft/veaury
• module federation: https://webpack.js.org/concepts/module-federation/

---