mgr-ansible-ssh - Intelligent, Lightweight CLI for Distributed Remote Execution by deve5h

Description

By the end of Hack Week, the target will be to deliver a minimal functional version 1 (MVP) of a custom command-line tool named mgr-ansible-ssh (a unified wrapper for BOTH ad-hoc shell & playbooks) that allows operators to:

1. Execute arbitrary shell commands on thousand of remote machines simultaneously using Ansible Runner with artifacts saved locally.
2. Pass runtime options such as inventory file, remote command string/ playbook execution, parallel forks, limits, dry-run mode, or no-std-ansible-output.
3. Leverage existing SSH trust relationships without additional setup.
4. Provide a clean, intuitive CLI interface with --help for ease of use. It should provide consistent UX & CI-friendly interface.
5. Establish a foundation that can later be extended with advanced features such as logging, grouping, interactive shell mode, safe-command checks, and parallel execution tuning.

The MVP should enable day-to-day operations to efficiently target thousands of machines with a single, consistent interface.

Goals

Primary Goals (MVP):

Build a functional CLI tool (mgr-ansible-ssh) capable of executing shell commands on multiple remote hosts using Ansible Runner. Test the tool across a large distributed environment (1000+ machines) to validate its performance and reliability.

Looking forward to significantly reducing the zypper deployment time across all 351 RMT VM servers in our MLM cluster by eliminating the dependency on the taskomatic service, bringing execution down to a fraction of the current duration. The tool should also support multiple runtime flags, such as:

mgr-ansible-ssh: Remote command execution wrapper using Ansible Runner

Usage: mgr-ansible-ssh [--help] [--version] [--inventory INVENTORY]
                   [--run RUN] [--playbook PLAYBOOK] [--limit LIMIT]
                   [--forks FORKS] [--dry-run] [--no-ansible-output]

Required Arguments
--inventory, -i      Path to Ansible inventory file to use

Any One of the Arguments Is Required
--run, -r            Execute the specified shell command on target hosts
--playbook, -p       Execute the specified Ansible playbook on target hosts

Optional Arguments
--help, -h           Show the help message and exit
--version, -v        Show the version and exit
--limit, -l          Limit execution to specific hosts or groups
--forks, -f          Number of parallel Ansible forks
--dry-run            Run in Ansible check mode (requires -p or --playbook)
--no-ansible-output  Suppress Ansible stdout output

Secondary/Stretched Goals (if time permits):

1. Add pretty output formatting (success/failure summary per host).
2. Implement basic logging of executed commands and results.
3. Introduce safety checks for risky commands (shutdown, rm -rf, etc.).
4. Package the tool so it can be installed with pip or stored internally.

Resources

Collaboration is welcome from anyone interested in CLI tooling, automation, or distributed systems. Skills that would be particularly valuable include:

1. Python especially around CLI dev (argparse, click, rich)

---

Uyuni Saltboot rework by oholecek

Description

When Uyuni switched over to the containerized proxies we had to abandon salt based saltboot infrastructure we had before. Uyuni already had integration with a Cobbler provisioning server and saltboot infra was re-implemented on top of this Cobbler integration.

What was not obvious from the start was that Cobbler, having all it's features, woefully slow when dealing with saltboot size environments. We did some improvements in performance, introduced transactions, and generally tried to make this setup usable. However the underlying slowness remained.

Goals

This project is not something trying to invent new things, it is just finally implementing saltboot infrastructure directly with the Uyuni server core.

Instead of generating grub and pxelinux configurations by Cobbler for all thousands of systems and branches, we will provide a GET access point to retrieve grub or pxelinux file during the boot:

/saltboot/group/grub/$fqdn and similar for systems /saltboot/system/grub/$mac

Next we adapt our tftpd translator to query these points when asked for default or mac based config.

Lastly similar thing needs to be done on our apache server when HTTP UEFI boot is used.

Resources

---

Move Uyuni Test Framework from Selenium to Playwright + AI by oscar-barrios

Description

This project aims to migrate the existing Uyuni Test Framework from Selenium to Playwright. The move will improve the stability, speed, and maintainability of our end-to-end tests by leveraging Playwright's modern features. We'll be rewriting the current Selenium code in Ruby to Playwright code in TypeScript, which includes updating the test framework runner, step definitions, and configurations. This is also necessary because we're moving from Cucumber Ruby to CucumberJS.

If you're still curious about the AI in the title, it was just a way to grab your attention. Thanks for your understanding.

Nah, let's be honest AI helped a lot to vibe code a good part of the Ruby methods of the Test framework, moving them to Typescript, along with the migration from Capybara to Playwright. I've been using "Cline" as plugin for WebStorm IDE, using Gemini API behind it.

---

Goals

• Migrate Core tests including Onboarding of clients
• Improve test reliabillity: Measure and confirm a significant reduction of flakiness.
• Implement a robust framework: Establish a well-structured and reusable Playwright test framework using the CucumberJS

---

Resources

• Existing Uyuni Test Framework (Cucumber Ruby + Capybara + Selenium)
• My Template for CucumberJS + Playwright in TypeScript
• Started Hackweek Project

---

Uyuni Health-check Grafana AI Troubleshooter by ygutierrez

Description

This project explores the feasibility of using the open-source Grafana LLM plugin to enhance the Uyuni Health-check tool with LLM capabilities. The idea is to integrate a chat-based "AI Troubleshooter" directly into existing dashboards, allowing users to ask natural-language questions about errors, anomalies, or performance issues.

Goals

• Investigate if and how the grafana-llm-app plug-in can be used within the Uyuni Health-check tool.
• Investigate if this plug-in can be used to query LLMs for troubleshooting scenarios.
• Evaluate support for local LLMs and external APIs through the plugin.
• Evaluate if and how the Uyuni MCP server could be integrated as another source of information.

Resources

Grafana LMM plug-in

Uyuni Health-check

---

Flaky Tests AI Finder for Uyuni and MLM Test Suites by oscar-barrios

Description

Our current Grafana dashboards provide a great overview of test suite health, including a panel for "Top failed tests." However, identifying which of these failures are due to legitimate bugs versus intermittent "flaky tests" is a manual, time-consuming process. These flaky tests erode trust in our test suites and slow down development.

This project aims to build a simple but powerful Python script that automates flaky test detection. The script will directly query our Prometheus instance for the historical data of each failed test, using the jenkins_build_test_case_failure_age metric. It will then format this data and send it to the Gemini API with a carefully crafted prompt, asking it to identify which tests show a flaky pattern.

The final output will be a clean JSON list of the most probable flaky tests, which can then be used to populate a new "Top Flaky Tests" panel in our existing Grafana test suite dashboard.

Goals

By the end of Hack Week, we aim to have a single, working Python script that:

1. Connects to Prometheus and executes a query to fetch detailed test failure history.
2. Processes the raw data into a format suitable for the Gemini API.
3. Successfully calls the Gemini API with the data and a clear prompt.
4. Parses the AI's response to extract a simple list of flaky tests.
5. Saves the list to a JSON file that can be displayed in Grafana.
6. New panel in our Dashboard listing the Flaky tests

Resources

• Jenkins Prometheus Exporter: https://github.com/uyuni-project/jenkins-exporter/
• Data Source: Our internal Prometheus server.
• Key Metric: jenkins_build_test_case_failure_age{jobname, buildid, suite, case, status, failedsince}.
• Existing Query for Reference: count by (suite) (max_over_time(jenkins_build_test_case_failure_age{status=~"FAILED|REGRESSION", jobname="$jobname"}[$__range])).
• AI Model: The Google Gemini API.
• Example about how to interact with Gemini API: https://github.com/srbarrios/FailTale/
• Visualization: Our internal Grafana Dashboard.
• Internal IaC: https://gitlab.suse.de/galaxy/infrastructure/-/tree/master/srv/salt/monitoring

Outcome

• Jenkins Flaky Test Detector: https://github.com/srbarrios/jenkins-flaky-tests-detector and its container
• IaC on MLM Team: https://gitlab.suse.de/galaxy/infrastructure/-/tree/master/srv/salt/monitoring/jenkinsflakytestsdetector?reftype=heads, https://gitlab.suse.de/galaxy/infrastructure/-/blob/master/srv/salt/monitoring/grafana/dashboards/flaky-tests.json?ref_type=heads, and others.
• Grafana Dashboard: https://grafana.mgr.suse.de/d/flaky-tests/flaky-tests-detection @ @ text

---

Set Uyuni to manage edge clusters at scale by RDiasMateus

Description

Prepare a Poc on how to use MLM to manage edge clusters. Those cluster are normally equal across each location, and we have a large number of them.

The goal is to produce a set of sets/best practices/scripts to help users manage this kind of setup.

Goals

step 1: Manual set-up

Goal: Have a running application in k3s and be able to update it using System Update Controler (SUC)

• Deploy Micro 6.2 machine

• Deploy k3s - single node

  • https://docs.k3s.io/quick-start

• Build/find a simple web application (static page)

  • Build/find a helmchart to deploy the application

• Deploy the application on the k3s cluster

• Install App updates through helm update

• Install OS updates using MLM

step 2: Automate day 1

Goal: Trigger the application deployment and update from MLM

• Salt states For application (with static data)
  • Deploy the application helmchart, if not present
  • install app updates through helmchart parameters
• Link it to GIT
  • Define how to link the state to the machines (based in some pillar data? Using configuration channels by importing the state? Naming convention?)
  • Use git update to trigger helmchart app update
• Recurrent state applying configuration channel?

step 3: Multi-node cluster

Goal: Use SUC to update a multi-node cluster.

• Create a multi-node cluster
• Deploy application
  • call the helm update/install only on control plane?
• Install App updates through helm update
• Prepare a SUC for OS update (k3s also? How?)
  • https://github.com/rancher/system-upgrade-controller
  • https://documentation.suse.com/cloudnative/k3s/latest/en/upgrades/automated.html
  • Update/deploy the SUC?
  • Update/deploy the SUC CRD with the update procedure

---

SUSE Edge Image Builder MCP by eminguez

Description

Based on my other hackweek project, SUSE Edge Image Builder's Json Schema I would like to build also a MCP to be able to generate EIB config files the AI way.

Realistically I don't think I'll be able to have something consumable at the end of this hackweek but at least I would like to start exploring MCPs, the difference between an API and MCP, etc.

Goals

• Familiarize myself with MCPs
• Unrealistic: Have an MCP that can generate an EIB config file

Resources

• https://hackweek.opensuse.org/25/projects/suse-edge-image-builder-json-schema
• Anything else :)

Result

https://github.com/e-minguez/eib-mcp

I've extensively used antigravity and its agent mode to code this. This heavily uses https://hackweek.opensuse.org/25/projects/suse-edge-image-builder-json-schema for the MCP to be built.

I've ended up learning a lot of things about "prompting", json schemas in general, some golang, MCPs and AI in general :)

Example:

Generate an Edge Image Builder configuration for an ISO image based on slmicro-6.2.iso, targeting x86_64 architecture. The output name should be 'my-edge-image' and it should install to /dev/sda. It should deploy a 3 nodes kubernetes cluster with nodes names "node1", "node2" and "node3" as: * hostname: node1, IP: 1.1.1.1, role: initializer * hostname: node2, IP: 1.1.1.2, role: agent * hostname: node3, IP: 1.1.1.3, role: agent The kubernetes version should be k3s 1.33.4-k3s1 and it should deploy a cert-manager helm chart (the latest one available according to https://cert-manager.io/docs/installation/helm/). It should create a user called "suse" with password "suse" and set ntp to "foo.ntp.org". The VIP address for the API should be 1.2.3.4

Generates:

``` apiVersion: "1.0" image: arch: x86_64 baseImage: slmicro-6.2.iso imageType: iso outputImageName: my-edge-image kubernetes: helm: charts: - name: cert-manager repositoryName: jetstack

---

SUSE Edge Image Builder json schema by eminguez

Description

Current SUSE Edge Image Builder tool doesn't provide a json schema (yes, I know EIB uses yaml but it seems JSON Schema can be used to validate YAML documents yay!) that defines the configuration file syntax, values, etc.

Having a json schema will make integrations straightforward, as once the json schema is in place, it can be used as the interface for other tools to consume and generate EIB definition files (like TUI wizards, web UIs, etc.)

I'll make use of AI tools for this so I'd learn more about vibe coding, agents, etc.

Goals

• Learn about json schemas
• Try to implement something that can take the EIB source code and output an initial json schema definition
• Create a PR for EIB to be adopted
• Learn more about AI tools and how those can help on similar projects.

Resources

• json-schema.org
• suse-edge/edge-image-builder
• Any AI tool that can help me!

Result

Pull Request created! https://github.com/suse-edge/edge-image-builder/pull/821

I've extensively used gemini via the VScode "gemini code assist" plugin but I found it not too good... my workstation froze for minutes using it... I have a pretty beefy macbook pro M2 and AFAIK the model is being executed on the cloud... so I basically spent a few days fighting with it... Then I switched to antigravity and its agent mode... and it worked much better.

I've ended up learning a few things about "prompting", json schemas in general, some golang and AI in general :)

---