Uyuni: re-architecting code with Akka
Simplify the codebase by using a more modern toolkit to accelerate maintenance and future development.
Enjoy Hakkaweek!
Problem
Uyuni's codebase has a 10+ year history, with contributions from dozens of individuals at varying points in their technical experience, understanding of the project, goals and timelines.
Inevitably, solutions to some of the common problems have grown organically, with different people solving slightly different instances of the same problem in different and possibly incomplete ways. This makes maintenance and future evolution of Uyuni more difficult, as developers and designers need to keep specifics of all such solutions into consideration.
One such area is the handling of asynchronous tasks within Java code, where currently we have at least 4 different implementations, none of which is really complete.
Spirit of this HackWeek effort
Write a proof-of-concept to replace as many existing, incomplete implementations as possible using one piece of existing, proven open source technology which already fulfills all known requirements (akka.io).
Should such a proof-of-concept succeed, integrating it into the project would:
- Simplify the code base, and possibly user experience, given there would be 1 solution instead of 4
- Modernize our project, by using a new toolkit instead of several older, homegrown solutions
- Accelerate new development of Uyuni, by leveraging existing know-how of Akka, and not requiring developers to know all current solutions
Failure... is definitely an option!
Learning why this can't work will be as valuable as having it working!
Not learning and not having fun... are NOT options!
That's HackWeek folks!
Hacking details
For asynchronous task handling we currently have:
- Taskomatic's basic facilities, including one-shot tasks
- Some specific task facilities (eg.
ErrataCacheDriver,SSHPushDriver) - Tomcat's
MessageQueuefacility - Tomcat's Salt event handler (
PGEventListener)
Open problems that in various degrees affect all of the above are:
- enforcing dependencies between asynchronous tasks
- enforcing dependencies between asynchronous tasks and other parts of the codebase
- establishing priority of tasks
- tuning, performance
- observability
akka.io seems to have all needed mechanisms:
- dependencies can be expressed by actors firing messages to other actors when done
- any other part of the code base can ask an actor to do something by sending it a message
- Akka has a concept of priority messages to actors
- Akka seems to be regarded as high performance, has lots of tuning knobs
- Akka has a lot of utilities for monitoring
Moreover Akka can run actors on different processes or even hosts transparently to the code. We could exploit this to improve the Tomcat/Taskomatic interface or even offload some tasks to separate machines.
Battle plan
- spend one day researching Akka and understanding how it works
- attack the simplest subsystem first, try to have it working somehow with Akka
- attack as many other subsystems as possible
- last half day: prepare demo for Monday : : :
Looking for hackers with the skills:
This project is part of:
Hack Week 19
Activity
Comments
-
almost 5 years ago by moio | Reply
Diary - day 1: learning
We learned many concepts:
- some immediately relevant for this HackWeek: Actor, ActorSystem, ActorContext, typed Actor, Actor hierarchy, Behavior, Mailbox, Dispatcher, Routing, at-most-once delivery
- others not immediately relevant, still interesting: Supervision, Artery, Akka Clustering, Akka Streams, backpressure, the Phi Accrual Failure Detector, Akka Persistence/Event Sourcing, consistent hashing
- on top of that: some new related tech bits: vavr.io, aeron.io, logback, Lightbend Telemetry, HOCON
We also played with some "hello world" examples.
-
almost 5 years ago by moio | Reply
Diary - day 2: MessageQueue classes
- "hello world" examples moved inside the Uyuni code base and ran in Uyuni
- replaced "hello world" examples with code to run three MessageQueue tasks, which work after the port to akka!
- added reflection-based glue code to automatically register Actors, minimizing boilerplate for next iterations
- in the process, developed several refactoring patches to simplify our codebase that can be accepted even outside of the scope of this effort (and a bugfix and a contribution to an unrelated PR)
- learnt about the Reflections library
Always-up-to-date-Uyuni PR with latest results available
To try:
- deploy an Uyuni Server (Master branch)
- use manager-build.xml to resolve Ivy dependencies and deploy
-
almost 5 years ago by moio | Reply
Diary - day 3: traditional MessageQueue is obsoleted
- created a better mechanism to start asynchronous tasks after a Transaction has ended. Current code requires polling every 10ms!
- more use cases of MessageQueue were moved to Akka. By end of day only the ones for Salt management remain
- configuration mechanism added to decide concurrency limits (per message type!)
- some dead code removed
-
almost 5 years ago by moio | Reply
Diary - day 4: MessageQueue and Salt event thread pool are removed
- many more use cases are switched to Akka (total is ~35)
- MessageQueue and related classes are removed completely, together with all surrounding likely unsafe code
- deferring mechanism moved to race-condition prone plain Java to Actor
- hash routing implemented in a straightforward patch compared to a much more complicated one we had, touching 3 programming languages
-
-
almost 5 years ago by moio | Reply
Diary - HackWeekEnd: (a very first) distributed Uyuni works!
- we feel like we just can't leave this code alone
- created a worker node alongside the Uyuni Server
- flagged the Minion Start Event worker (Akka-ified on day 4) to run in the worker
- fixed bugs and added hacks until registration works, moving load from the Server to the worker!
- adding more workers distributes load round-robin!
Soooo...
[
- we feel like we just can't leave this code alone
- created a worker node alongside the Uyuni Server
- flagged the Minion Start Event worker (Akka-ified on day 4) to run in the worker
- fixed bugs and added hacks until registration works, moving load from the Server to the worker!
- adding more workers distributes load round-robin!
Reply to moio
Diary - HackWeekEnd: (a very first) distributed Uyuni works!
Soooo...
[
# A First Level Header ## A Second Level Header Use one asterisk to *emphasize* Use two asterisks for **strong emphasis** - Use hyphens - for unordereed - lists This is an [link to example.com](http://example.com/) This is an image  This is a user link @hans This is a project link hw#some-cool-title
Similar Projects
SUSE AI Meets the Game Board by moio
Use tabletopgames.ai’s open source TAG and PyTAG frameworks to apply Statistical Forward Planning and Deep Reinforcement Learning to two board games of our own design. On an all-green, all-open source, all-AWS stack!
Results: Infrastructure Achievements
We successfully built and automated a containerized stack to support our AI experiments. This included:
- a Fully-Automated, One-Command, GPU-accelerated Kubernetes setup: we created an OpenTofu based script, tofu-tag, to deploy SUSE's RKE2 Kubernetes running on CUDA-enabled nodes in AWS, powered by openSUSE with GPU drivers and gpu-operator
- Containerization of the TAG and PyTAG frameworks: TAG (Tabletop AI Games) and PyTAG were patched for seamless deployment in containerized environments. We automated the container image creation process with GitHub Actions. Our forks (PRs upstream upcoming):
./deploy.sh and voilà - Kubernetes running PyTAG (k9s, above) with GPU acceleration (nvtop, below)
Results: Game Design Insights
Our project focused on modeling and analyzing two card games of our own design within the TAG framework:
- Game Modeling: We implemented models for Dario's "Bamboo" and Silvio's "Totoro" and "R3" games, enabling AI agents to play thousands of games ...in minutes!
- AI-driven optimization: By analyzing statistical data on moves, strategies, and outcomes, we iteratively tweaked the game mechanics and rules to achieve better balance and player engagement.
- Advanced analytics: Leveraging AI agents with Monte Carlo Tree Search (MCTS) and random action selection, we compared performance metrics to identify optimal strategies and uncover opportunities for game refinement .
- more about Bamboo on Dario's site
- more about R3 on Silvio's site (italian, translation coming)
- more about Totoro on Silvio's site
A family picture of our card games in progress. From the top: Bamboo, Totoro, R3
Results: Learning, Collaboration, and Innovation
Beyond technical accomplishments, the project showcased innovative approaches to coding, learning, and teamwork:
- "Trio programming" with AI assistance: Our "trio programming" approach—two developers and GitHub Copilot—was a standout success, especially in handling slightly-repetitive but not-quite-exactly-copypaste tasks. Java as a language tends to be verbose and we found it to be fitting particularly well.
- AI tools for reporting and documentation: We extensively used AI chatbots to streamline writing and reporting. (Including writing this report! ...but this note was added manually during edit!)
- GPU compute expertise: Overcoming challenges with CUDA drivers and cloud infrastructure deepened our understanding of GPU-accelerated workloads in the open-source ecosystem.
- Game design as a learning platform: By blending AI techniques with creative game design, we learned not only about AI strategies but also about making games fun, engaging, and balanced.
Last but not least we had a lot of fun! ...and this was definitely not a chatbot generated line!
The Context: AI + Board Games
Testing and adding GNU/Linux distributions on Uyuni by juliogonzalezgil
Join the Gitter channel! https://gitter.im/uyuni-project/hackweek
Uyuni is a configuration and infrastructure management tool that saves you time and headaches when you have to manage and update tens, hundreds or even thousands of machines. It also manages configuration, can run audits, build image containers, monitor and much more!
Currently there are a few distributions that are completely untested on Uyuni or SUSE Manager (AFAIK) or just not tested since a long time, and could be interesting knowing how hard would be working with them and, if possible, fix whatever is broken.
For newcomers, the easiest distributions are those based on DEB or RPM packages. Distributions with other package formats are doable, but will require adapting the Python and Java code to be able to sync and analyze such packages (and if salt does not support those packages, it will need changes as well). So if you want a distribution with other packages, make sure you are comfortable handling such changes.
No developer experience? No worries! We had non-developers contributors in the past, and we are ready to help as long as you are willing to learn. If you don't want to code at all, you can also help us preparing the documentation after someone else has the initial code ready, or you could also help with testing :-)
The idea is testing Salt and Salt-ssh clients, but NOT traditional clients, which are deprecated.
To consider that a distribution has basic support, we should cover at least (points 3-6 are to be tested for both salt minions and salt ssh minions):
- Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)
- Onboarding (salt minion from UI, salt minion from bootstrap scritp, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator)
- Package management (install, remove, update...)
- Patching
- Applying any basic salt state (including a formula)
- Salt remote commands
- Bonus point: Java part for product identification, and monitoring enablement
- Bonus point: sumaform enablement (https://github.com/uyuni-project/sumaform)
- Bonus point: Documentation (https://github.com/uyuni-project/uyuni-docs)
- Bonus point: testsuite enablement (https://github.com/uyuni-project/uyuni/tree/master/testsuite)
If something is breaking: we can try to fix it, but the main idea is research how supported it is right now. Beyond that it's up to each project member how much to hack :-)
- If you don't have knowledge about some of the steps: ask the team
- If you still don't know what to do: switch to another distribution and keep testing.
This card is for EVERYONE, not just developers. Seriously! We had people from other teams helping that were not developers, and added support for Debian and new SUSE Linux Enterprise and openSUSE Leap versions :-)
Pending
FUSS
FUSS is a complete GNU/Linux solution (server, client and desktop/standalone) based on Debian for managing an educational network.
https://fuss.bz.it/
Seems to be a Debian 12 derivative, so adding it could be quite easy.
[W]Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)[W]Onboarding (salt minion from UI, salt minion from bootstrap script, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator) --> Working for all 3 options (salt minion UI, salt minion bootstrap script and salt-ssh minion from the UI).[W]Package management (install, remove, update...) --> Installing a new package works, needs to test the rest.[I]Patching (if patch information is available, could require writing some code to parse it, but IIRC we have support for Ubuntu already). No patches detected. Do we support patches for Debian at all?[W]Applying any basic salt state (including a formula)[W]Salt remote commands[ ]Bonus point: Java part for product identification, and monitoring enablement
Automated Test Report reviewer by oscar-barrios
Description
In SUMA/Uyuni team we spend a lot of time reviewing test reports, analyzing each of the test cases failing, checking if the test is a flaky test, checking logs, etc.
Goals
Speed up the review by automating some parts through AI, in a way that we can consume some summary of that report that could be meaningful for the reviewer.
Resources
No idea about the resources yet, but we will make use of:
- HTML/JSON Report (text + screenshots)
- The Test Suite Status GithHub board (via API)
- The environment tested (via SSH)
- The test framework code (via files)
Enable the containerized Uyuni server to run on different host OS by j_renner
Description
The Uyuni server is provided as a container, but we still require it to run on Leap Micro? This is not how people expect to use containerized applications, so it would be great if we tested other host OSs and enabled them by providing builds of necessary tools for (e.g. mgradm). Interesting candidates should be:
- openSUSE Leap
- Cent OS 7
- Ubuntu
- ???
Goals
Make it really easy for anyone to run the Uyuni containerized server on whatever OS they want (with support for containers of course).
Uyuni developer-centric documentation by deneb_alpha
Description
While we currently have extensive documentation on user-oriented tasks such as adding minions, patching, fine-tuning, etc, there is a notable gap when it comes to centralizing and documenting core functionalities for developers.
The number of functionalities and side tools we have in Uyuni can be overwhelming. It would be nice to have a centralized place with descriptive list of main/core functionalities.
Goals
Create, aggregate and review on the Uyuni wiki a set of resources, focused on developers, that include also some known common problems/troubleshooting.
The documentation will be helpful not only for everyone who is trying to learn the functionalities with all their inner processes like newcomer developers or community enthusiasts, but also for anyone who need a refresh.
Resources
The resources are currently aggregated here: https://github.com/uyuni-project/uyuni/wiki
Saline (state deployment control and monitoring tool for SUSE Manager/Uyuni) by vizhestkov
Project Description
Saline is an addition for salt used in SUSE Manager/Uyuni aimed to provide better control and visibility for states deploymend in the large scale environments.
In current state the published version can be used only as a Prometheus exporter and missing some of the key features implemented in PoC (not published). Now it can provide metrics related to salt events and state apply process on the minions. But there is no control on this process implemented yet.
Continue with implementation of the missing features and improve the existing implementation:
authentication (need to decide how it should be/or not related to salt auth)
web service providing the control of states deployment
Goal for this Hackweek
Implement missing key features
Implement the tool for state deployment control with CLI
Resources
https://github.com/openSUSE/saline
Saltboot ability to deploy OEM images by oholecek
Description
Saltboot is a system deployment part of Uyuni. It is the mechanism behind deploying Kiwi built system images from central Uyuni server location.
System image is when the image is only of one partition and does not contain whole disk image and deployment system has to take care of partitioning, fstab on top of integrity validation.
However systems like Aeon, SUSE Linux Enterprise Micro and similar are distributed as disk images (also so called OEM images). Saltboot currently cannot deploy these systems.
The main problem to saltboot is however that currently saltboot support is built into the image itself. This step is not desired when using OEM images.
Goals
Saltboot needs to be standalone and be able to deploy OEM images. Responsibility of saltboot would then shrink to selecting correct image, image integrity validation, deployment and boot to deployed system.
Resources
- Saltboot - https://github.com/uyuni-project/retail/tree/master
- Uyuni - https://github.com/uyuni-project/uyuni
Improve Development Environment on Uyuni by mbussolotto
Description
Currently create a dev environment on Uyuni might be complicated. The steps are:
- add the correct repo
- download packages
- configure your IDE (checkstyle, format rules, sonarlint....)
- setup debug environment
- ...
The current doc can be improved: some information are hard to be find out, some others are completely missing.
Dev Container might solve this situation.
Goals
Uyuni development in no time:
- using VSCode:
- setting.json should contains all settings (for all languages in Uyuni, with all checkstyle rules etc...)
- dev container should contains all dependencies
- setup debug environment
- implement a GitHub Workspace solution
- re-write documentation
Lots of pieces are already implemented: we need to connect them in a consistent solution.
Resources
- https://github.com/uyuni-project/uyuni/wiki
Create SUSE Manager users from ldap/ad groups by mbrookhuis
Description
This tool is used to create users in SUSE Manager Server based on LDAP/AD groups. For each LDAP/AD group a role within SUSE Manager Server is defined. Also, the tool will check if existing users still have the role they should have, and, if not, it will be corrected. The same for if a user is disabled, it will be enabled again. If a users is not present in the LDAP/AD groups anymore, it will be disabled or deleted, depending on the configuration.
The code is written for Python 3.6 (the default with SLES15.x), but will also work with newer versions. And works against SUSE Manger 4.3 and 5.x
Goals
Create a python and/or golang utility that will manage users in SUSE Manager based on LDAP/AD group-membership. In a configuration file is defined which roles the members of a group will get.
Table of contents
Installation
To install this project, perform the following steps:
- Be sure that python 3.6 is installed and also the module python3-PyYAML. Also the ldap3 module is needed:
bash
zypper in python3 python3-PyYAML
pip install yaml
On the server or PC, where it should run, create a directory. On linux, e.g. /opt/sm-ldap-users
Copy all the file to this directory.
Edit the configsm.yaml. All parameters should be entered. Tip: for the ldap information, the best would be to use the same as for SSSD.
Be sure that the file sm-ldap-users.py is executable. It would be good to change the owner to root:root and only root can read and execute:
bash
chmod 600 *
chmod 700 sm-ldap-users.py
chown root:root *
Usage
This is very simple. Once the configsm.yaml contains the correct information, executing the following will do the magic:
bash
/sm-ldap-users.py
repository link
https://github.com/mbrookhuis/sm-ldap-users
Testing and adding GNU/Linux distributions on Uyuni by juliogonzalezgil
Join the Gitter channel! https://gitter.im/uyuni-project/hackweek
Uyuni is a configuration and infrastructure management tool that saves you time and headaches when you have to manage and update tens, hundreds or even thousands of machines. It also manages configuration, can run audits, build image containers, monitor and much more!
Currently there are a few distributions that are completely untested on Uyuni or SUSE Manager (AFAIK) or just not tested since a long time, and could be interesting knowing how hard would be working with them and, if possible, fix whatever is broken.
For newcomers, the easiest distributions are those based on DEB or RPM packages. Distributions with other package formats are doable, but will require adapting the Python and Java code to be able to sync and analyze such packages (and if salt does not support those packages, it will need changes as well). So if you want a distribution with other packages, make sure you are comfortable handling such changes.
No developer experience? No worries! We had non-developers contributors in the past, and we are ready to help as long as you are willing to learn. If you don't want to code at all, you can also help us preparing the documentation after someone else has the initial code ready, or you could also help with testing :-)
The idea is testing Salt and Salt-ssh clients, but NOT traditional clients, which are deprecated.
To consider that a distribution has basic support, we should cover at least (points 3-6 are to be tested for both salt minions and salt ssh minions):
- Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)
- Onboarding (salt minion from UI, salt minion from bootstrap scritp, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator)
- Package management (install, remove, update...)
- Patching
- Applying any basic salt state (including a formula)
- Salt remote commands
- Bonus point: Java part for product identification, and monitoring enablement
- Bonus point: sumaform enablement (https://github.com/uyuni-project/sumaform)
- Bonus point: Documentation (https://github.com/uyuni-project/uyuni-docs)
- Bonus point: testsuite enablement (https://github.com/uyuni-project/uyuni/tree/master/testsuite)
If something is breaking: we can try to fix it, but the main idea is research how supported it is right now. Beyond that it's up to each project member how much to hack :-)
- If you don't have knowledge about some of the steps: ask the team
- If you still don't know what to do: switch to another distribution and keep testing.
This card is for EVERYONE, not just developers. Seriously! We had people from other teams helping that were not developers, and added support for Debian and new SUSE Linux Enterprise and openSUSE Leap versions :-)
Pending
FUSS
FUSS is a complete GNU/Linux solution (server, client and desktop/standalone) based on Debian for managing an educational network.
https://fuss.bz.it/
Seems to be a Debian 12 derivative, so adding it could be quite easy.
[W]Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)[W]Onboarding (salt minion from UI, salt minion from bootstrap script, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator) --> Working for all 3 options (salt minion UI, salt minion bootstrap script and salt-ssh minion from the UI).[W]Package management (install, remove, update...) --> Installing a new package works, needs to test the rest.[I]Patching (if patch information is available, could require writing some code to parse it, but IIRC we have support for Ubuntu already). No patches detected. Do we support patches for Debian at all?[W]Applying any basic salt state (including a formula)[W]Salt remote commands[ ]Bonus point: Java part for product identification, and monitoring enablement
Saline (state deployment control and monitoring tool for SUSE Manager/Uyuni) by vizhestkov
Project Description
Saline is an addition for salt used in SUSE Manager/Uyuni aimed to provide better control and visibility for states deploymend in the large scale environments.
In current state the published version can be used only as a Prometheus exporter and missing some of the key features implemented in PoC (not published). Now it can provide metrics related to salt events and state apply process on the minions. But there is no control on this process implemented yet.
Continue with implementation of the missing features and improve the existing implementation:
authentication (need to decide how it should be/or not related to salt auth)
web service providing the control of states deployment
Goal for this Hackweek
Implement missing key features
Implement the tool for state deployment control with CLI
Resources
https://github.com/openSUSE/saline
SUSE Prague claw machine by anstalker
Project Description
The idea is to build a claw machine similar to e.g. this one:
Why? Well, it could be a lot of fun!
But also it's a great way to dispense SUSE and openSUSE merch like little Geekos at events like conferences, career fairs and open house events.
Goal for this Hackweek
Build an arcade claw machine.
Resources
In French, an article about why you always lose in claw machine games:
We're looking for handy/crafty people in the Prague office:
- woodworking XP or equipment
- arduino/raspi embedded programming knowledge
- Anthony can find a budget for going to GM and buying servos and such ;)