Small healthcheck tool for Longhorn by mbrookhuis

Project Description

We have often problems (e.g. pods not starting) that are related to PVCs not running, cluster (nodes) not all up or deployments not running or completely running. This all prevents administration activities. Having something that can regular be run to validate the status of the cluster would be helpful, and not as of today do a lot of manual tasks.

As addition (read enough time), we could add changing reservation, adding new disks, etc. --> This didn't made it. But the scripts can easily be adopted.

This tool would decrease troubleshooting time, giving admins rights to the rancher GUI and could be used in automation.

Goal for this Hackweek

At the end we should have a small python tool that is doing a (very) basic health check on nodes, deployments and PVCs. First attempt was to make it in golang, but that was taking to much time.

Overview

This tool will run a simple healthcheck on a kubernetes cluster. It will perform the following actions:

• node check: This will check all nodes, and display the status and the k3s version. If the status of the nodes is not "Ready" (this should be only reported), the cluster will be reported as having problems

• deployment check: This check will list all deployments, and display the number of expected replicas and the used replica. If there are unused replicas this will be displayed. The cluster will be reported as having problems.

• pvc check: This check will list of all pvc's, and display the status and the robustness. If the robustness is not "Healthy", the cluster will be reported as having problems.

If there is a problem registered in the checks, there will be a warning that the cluster is not healthy and the program will exit with 1.

The script has 1 mandatory parameter and that is the kubeconf of the cluster or of a node off the cluster.

The code is writen for Python 3.11, but will also work on 3.6 (the default with SLES15.x). There is a venv present that will contain all needed packages. Also, the script can be run on the cluster itself or any other linux server.

Installation

To install this project, perform the following steps:

• Create the directory /opt/k8s-check

mkdir /opt/k8s-check

• Copy all the file to this directory and make the following changes:

chmod +x k8s-check.py

---

Symbol Relations by hli

Description

There are tools to build function call graphs based on parsing source code, for example, cscope.

This project aims to achieve a similar goal by directly parsing the disasembly (i.e. objdump) of a compiled binary. The assembly code is what the CPU sees, therefore more "direct". This may be useful in certain scenarios, such as gdb/crash debugging.

Detailed description and Demos can be found in the README file:

Supports x86 for now (because my customers only use x86 machines), but support for other architectures can be added easily.

Tested with python3.6

Goals

Any comments are welcome.

Resources

https://github.com/lhb-cafe/SymbolRelations

symrellib.py: mplements the symbol relation graph and the disassembly parser

symrel_tracer*.py: implements tracing (-t option)

symrel.py: "cli parser"

---

Enhance UV openQA helper script by mdonis

Description

A couple months ago an UV openQA helper script was created to help/automate the searching phase inside openQA for a given MU to test. The script searches inside all our openQA job groups (qam-sle) related with a given MU and generates an output suitable to add (copy & paste) inside the update log.

This is still a WIP and could use some enhancements.

Goals

• Move script from bash to python: this would be useful in case we want to include this into MTUI in the future. The script will be separate from MTUI for now. The idea is to have this as a CLI tool using the click library or something similar.
• Add option to look for jobs in other sections inside aggregated updates: right now, when looking for regression tests under aggregated updates for a given MU, the script only looks inside the Core MU job group. This is where most of the regression tests we need are located, but some MUs have their regression tests under the YaST/Containers/Security MU job groups. We should keep the Core MU group as a default, but add an option to be able to look into other job groups under aggregated updates.
• Remove the -a option: this option is used to indicate the update ID and is mandatory right now. This is a bit weird and goes against posix stardards. It was developed this way in order to avoid using positional parameters. This problem should be fixed if we move the script to python.

Some other ideas to consider:

• Look into the QAM dashboard API. This has more info on each MU, could use this to link general openQA build results, whether the related RR is approved or not, etc
• Make it easier to see if there's regression tests for a package in an openQA test build. Check if there's a possibility to search for tests that have the package name in them inside each testsuite.
• Unit testing?

More ideas TBD

Resources

https://github.com/os-autoinst/scripts/blob/master/openqa-search-maintenance-core-jobs

https://confluence.suse.com/display/maintenanceqa/Guide+on+how+to+test+Updates

Post-Hackweek update

All major features were implemented. Unit tests are still in progress, and project will be moved to the SUSE github org once everything's done. https://github.com/mjdonis/oqa-search

---

Selenium with Python by xguo

Description

Try to create test case about Selenium base on Python

Goals

• Knowledge about Selenium with Python
• Create new test case about Selenium

Resources

https://selenium-python.readthedocs.io/ https://www.selenium.dev/

---

Make more sense of openQA test results using AI by livdywan

Description

AI has the potential to help with something many of us spend a lot of time doing which is making sense of openQA logs when a job fails.

User Story

Allison Average has a puzzled look on their face while staring at log files that seem to make little sense. Is this a known issue, something completely new or maybe related to infrastructure changes?

Goals

• Leverage a chat interface to help Allison
• Create a model from scratch based on data from openQA
• Proof of concept for automated analysis of openQA test results

Bonus

• Use AI to suggest solutions to merge conflicts
  • This would need a merge conflict editor that can suggest solving the conflict
• Use image recognition for needles

Resources

• TensorFlow
• open-webui
• openQA instance of openSUSE
• openQA documentation

Timeline

Day 1

• Conversing with open-webui to teach me how to create a model based on openQA test results
  • Asking for example code using TensorFlow in Python
  • Discussing log files to explore what to analyze
  • Drafting a new project called Testimony (based on Implementing a containerized Python action) - the project name was also suggested by the assistant

Day 2

• Using NotebookLLM (Gemini) to produce conversational versions of blog posts
  • Writing openQA tests in Python
  • Developing in distrobox containers
• Researching the possibility of creating a project logo with AI
  • Asking open-webui, persons with prior experience and conducting a web search for advice

Highlights

• I briefly tested compared models to see if they would make me more productive. Between llama, gemma and mistral there was no amazing difference in the results for my case.
• Convincing the chat interface to produce code specific to my use case required very explicit instructions.
• Asking for advice on how to use open-webui itself better was frustratingly unfruitful both in trivial and more advanced regards.
• Documentation on source materials used by LLM's and tools for this purpose seems virtually non-existent - specifically if a logo can be generated based on particular licenses

Outcomes

• Chat interface-supported development is providing good starting points and open-webui being open source is more flexible than Gemini. Although currently some fancy features such as grounding and generated podcasts are missing.
• Allison still has to be very experienced with openQA to use a chat interface for test review. Publicly available system prompts would make that easier, though.

---

Saline (state deployment control and monitoring tool for SUSE Manager/Uyuni) by vizhestkov

Project Description

Saline is an addition for salt used in SUSE Manager/Uyuni aimed to provide better control and visibility for states deploymend in the large scale environments.

In current state the published version can be used only as a Prometheus exporter and missing some of the key features implemented in PoC (not published). Now it can provide metrics related to salt events and state apply process on the minions. But there is no control on this process implemented yet.

Continue with implementation of the missing features and improve the existing implementation:

• authentication (need to decide how it should be/or not related to salt auth)

• web service providing the control of states deployment

Goal for this Hackweek

• Implement missing key features

• Implement the tool for state deployment control with CLI

Resources

https://github.com/openSUSE/saline

---

SUSE AI Meets the Game Board by moio

Use tabletopgames.ai’s open source TAG and PyTAG frameworks to apply Statistical Forward Planning and Deep Reinforcement Learning to two board games of our own design. On an all-green, all-open source, all-AWS stack!

---

Results: Infrastructure Achievements

We successfully built and automated a containerized stack to support our AI experiments. This included:

• a Fully-Automated, One-Command, GPU-accelerated Kubernetes setup: we created an OpenTofu based script, tofu-tag, to deploy SUSE's RKE2 Kubernetes running on CUDA-enabled nodes in AWS, powered by openSUSE with GPU drivers and gpu-operator
• Containerization of the TAG and PyTAG frameworks: TAG (Tabletop AI Games) and PyTAG were patched for seamless deployment in containerized environments. We automated the container image creation process with GitHub Actions. Our forks (PRs upstream upcoming):
  • https://github.com/moio/TabletopGames/tree/hackweek24
  • https://github.com/moio/PyTAG/tree/hackweek24

./deploy.sh and voilà - Kubernetes running PyTAG (k9s, above) with GPU acceleration (nvtop, below)

Results: Game Design Insights

Our project focused on modeling and analyzing two card games of our own design within the TAG framework:

• Game Modeling: We implemented models for Dario's "Bamboo" and Silvio's "Totoro" and "R3" games, enabling AI agents to play thousands of games ...in minutes!
• AI-driven optimization: By analyzing statistical data on moves, strategies, and outcomes, we iteratively tweaked the game mechanics and rules to achieve better balance and player engagement.
• Advanced analytics: Leveraging AI agents with Monte Carlo Tree Search (MCTS) and random action selection, we compared performance metrics to identify optimal strategies and uncover opportunities for game refinement .
  • more about Bamboo on Dario's site
  • more about R3 on Silvio's site (italian, translation coming)
  • more about Totoro on Silvio's site

A family picture of our card games in progress. From the top: Bamboo, Totoro, R3

Results: Learning, Collaboration, and Innovation

Beyond technical accomplishments, the project showcased innovative approaches to coding, learning, and teamwork:

• "Trio programming" with AI assistance: Our "trio programming" approach—two developers and GitHub Copilot—was a standout success, especially in handling slightly-repetitive but not-quite-exactly-copypaste tasks. Java as a language tends to be verbose and we found it to be fitting particularly well.
• AI tools for reporting and documentation: We extensively used AI chatbots to streamline writing and reporting. (Including writing this report! ...but this note was added manually during edit!)
• GPU compute expertise: Overcoming challenges with CUDA drivers and cloud infrastructure deepened our understanding of GPU-accelerated workloads in the open-source ecosystem.
• Game design as a learning platform: By blending AI techniques with creative game design, we learned not only about AI strategies but also about making games fun, engaging, and balanced.

Last but not least we had a lot of fun! ...and this was definitely not a chatbot generated line!

The Context: AI + Board Games

---

Symbol Relations by hli

Description

There are tools to build function call graphs based on parsing source code, for example, cscope.

This project aims to achieve a similar goal by directly parsing the disasembly (i.e. objdump) of a compiled binary. The assembly code is what the CPU sees, therefore more "direct". This may be useful in certain scenarios, such as gdb/crash debugging.

Detailed description and Demos can be found in the README file:

Supports x86 for now (because my customers only use x86 machines), but support for other architectures can be added easily.

Tested with python3.6

Goals

Any comments are welcome.

Resources

https://github.com/lhb-cafe/SymbolRelations

symrellib.py: mplements the symbol relation graph and the disassembly parser

symrel_tracer*.py: implements tracing (-t option)

symrel.py: "cli parser"

---

ClusterOps - Easily install and manage your personal kubernetes cluster by andreabenini

Description

ClusterOps is a Kubernetes installer and operator designed to streamline the initial configuration and ongoing maintenance of kubernetes clusters. The focus of this project is primarily on personal or local installations. However, the goal is to expand its use to encompass all installations of Kubernetes for local development purposes.
It simplifies cluster management by automating tasks and providing just one user-friendly YAML-based configuration config.yml.

Overview

• Simplified Configuration: Define your desired cluster state in a simple YAML file, and ClusterOps will handle the rest.
• Automated Setup: Automates initial cluster configuration, including network settings, storage provisioning, special requirements (for example GPUs) and essential components installation.
• Ongoing Maintenance: Performs routine maintenance tasks such as upgrades, security updates, and resource monitoring.
• Extensibility: Easily extend functionality with custom plugins and configurations.
• Self-Healing: Detects and recovers from common cluster issues, ensuring stability, idempotence and reliability. Same operation can be performed multiple times without changing the result.
• Discreet: It works only on what it knows, if you are manually configuring parts of your kubernetes and this configuration does not interfere with it you can happily continue to work on several parts and use this tool only for what is needed.

Features

• distribution and engine independence. Install your favorite kubernetes engine with your package manager, execute one script and you'll have a complete working environment at your disposal.
  
• Basic config approach. One single config.yml file with configuration requirements (add/remove features): human readable, plain and simple. All fancy configs managed automatically (ingress, balancers, services, proxy, ...).
• Local Builtin ContainerHub. The default installation provides a fully configured ContainerHub available locally along with the kubernetes installation. This configuration allows the user to build, upload and deploy custom container images as they were provided from external sources. Internet public sources are still available but local development can be kept in this localhost server. Builtin ClusterOps operator will be fetched from this ContainerHub registry too.
• Kubernetes official dashboard installed as a plugin, others planned too (k9s for example).
• Kubevirt plugin installed and properly configured. Unleash the power of classic virtualization (KVM+QEMU) on top of Kubernetes and manage your entire system from there, libvirtd and virsh libs are required.
• One operator to rule them all. The installation script configures your machine automatically during installation and adds one kubernetes operator to manage your local cluster. From there the operator takes care of the cluster on your behalf.
• Clean installation and removal. Just test it, when you are done just use the same program to uninstall everything without leaving configs (or pods) behind.

Planned features (Wishlist / TODOs)

• Containerized Data Importer (CDI). Persistent storage management add-on for Kubernetes to provide a declarative way of building and importing Virtual Machine Disks on PVCs for

---