Simplify the codebase by using a more modern toolkit to accelerate maintenance and future development.

Enjoy Hakkaweek!

Problem

Uyuni's codebase has a 10+ year history, with contributions from dozens of individuals at varying points in their technical experience, understanding of the project, goals and timelines.

Inevitably, solutions to some of the common problems have grown organically, with different people solving slightly different instances of the same problem in different and possibly incomplete ways. This makes maintenance and future evolution of Uyuni more difficult, as developers and designers need to keep specifics of all such solutions into consideration.

One such area is the handling of asynchronous tasks within Java code, where currently we have at least 4 different implementations, none of which is really complete.

Spirit of this HackWeek effort

Write a proof-of-concept to replace as many existing, incomplete implementations as possible using one piece of existing, proven open source technology which already fulfills all known requirements (akka.io).

Should such a proof-of-concept succeed, integrating it into the project would:

  • Simplify the code base, and possibly user experience, given there would be 1 solution instead of 4
  • Modernize our project, by using a new toolkit instead of several older, homegrown solutions
  • Accelerate new development of Uyuni, by leveraging existing know-how of Akka, and not requiring developers to know all current solutions

Failure... is definitely an option!

Learning why this can't work will be as valuable as having it working!

Not learning and not having fun... are NOT options!

That's HackWeek folks!

Hacking details

For asynchronous task handling we currently have:

  • Taskomatic's basic facilities, including one-shot tasks
  • Some specific task facilities (eg. ErrataCacheDriver, SSHPushDriver)
  • Tomcat's MessageQueue facility
  • Tomcat's Salt event handler (PGEventListener)

Open problems that in various degrees affect all of the above are:

  1. enforcing dependencies between asynchronous tasks
  2. enforcing dependencies between asynchronous tasks and other parts of the codebase
  3. establishing priority of tasks
  4. tuning, performance
  5. observability

akka.io seems to have all needed mechanisms:

  1. dependencies can be expressed by actors firing messages to other actors when done
  2. any other part of the code base can ask an actor to do something by sending it a message
  3. Akka has a concept of priority messages to actors
  4. Akka seems to be regarded as high performance, has lots of tuning knobs
  5. Akka has a lot of utilities for monitoring

Moreover Akka can run actors on different processes or even hosts transparently to the code. We could exploit this to improve the Tomcat/Taskomatic interface or even offload some tasks to separate machines.

Battle plan

  • spend one day researching Akka and understanding how it works
  • attack the simplest subsystem first, try to have it working somehow with Akka
  • attack as many other subsystems as possible
  • last half day: prepare demo for Monday : : :

Looking for hackers with the skills:

java scala akka uyuni susemanager fun functionalprogramming

This project is part of:

Hack Week 19

Activity

  • almost 5 years ago: fkobzik liked this project.
  • almost 5 years ago: RDiasMateus joined this project.
  • almost 5 years ago: j_renner liked this project.
  • almost 5 years ago: dleidi liked this project.
  • almost 5 years ago: RDiasMateus liked this project.
  • almost 5 years ago: pagarcia liked this project.
  • almost 5 years ago: moio liked this project.
  • almost 5 years ago: moio added keyword "java" to this project.
  • almost 5 years ago: moio added keyword "scala" to this project.
  • almost 5 years ago: moio added keyword "akka" to this project.
  • almost 5 years ago: moio added keyword "uyuni" to this project.
  • almost 5 years ago: moio added keyword "susemanager" to this project.
  • almost 5 years ago: moio added keyword "fun" to this project.
  • almost 5 years ago: moio added keyword "functionalprogramming" to this project.
  • almost 5 years ago: moio started this project.
  • almost 5 years ago: moio originated this project.

  • Comments

    • moio
      almost 5 years ago by moio | Reply

      Diary - day 1: learning

      We learned many concepts:

      • some immediately relevant for this HackWeek: Actor, ActorSystem, ActorContext, typed Actor, Actor hierarchy, Behavior, Mailbox, Dispatcher, Routing, at-most-once delivery
      • others not immediately relevant, still interesting: Supervision, Artery, Akka Clustering, Akka Streams, backpressure, the Phi Accrual Failure Detector, Akka Persistence/Event Sourcing, consistent hashing
      • on top of that: some new related tech bits: vavr.io, aeron.io, logback, Lightbend Telemetry, HOCON

      We also played with some "hello world" examples.

    • moio
      almost 5 years ago by moio | Reply

      Diary - day 2: MessageQueue classes

      • "hello world" examples moved inside the Uyuni code base and ran in Uyuni
      • replaced "hello world" examples with code to run three MessageQueue tasks, which work after the port to akka!
      • added reflection-based glue code to automatically register Actors, minimizing boilerplate for next iterations
      • in the process, developed several refactoring patches to simplify our codebase that can be accepted even outside of the scope of this effort (and a bugfix and a contribution to an unrelated PR)
      • learnt about the Reflections library

      Always-up-to-date-Uyuni PR with latest results available

      To try:

      • deploy an Uyuni Server (Master branch)
      • use manager-build.xml to resolve Ivy dependencies and deploy

    • moio
      almost 5 years ago by moio | Reply

      Diary - day 3: traditional MessageQueue is obsoleted

      • created a better mechanism to start asynchronous tasks after a Transaction has ended. Current code requires polling every 10ms!
      • more use cases of MessageQueue were moved to Akka. By end of day only the ones for Salt management remain
      • configuration mechanism added to decide concurrency limits (per message type!)
      • some dead code removed

    • moio
      almost 5 years ago by moio | Reply

      Diary - day 4: MessageQueue and Salt event thread pool are removed

    • moio
      almost 5 years ago by moio | Reply

      Diary - day 5: introduce Akka Clustering

      • cleaned up code and revision history until this point
      • started adding Akka Clustering to the codebase. Objective: move jobs ported to Akka to different nodes!
      • Hello World of Akka Clustering works in Uyuni

    • moio
      almost 5 years ago by moio | Reply

      Diary - HackWeekEnd: (a very first) distributed Uyuni works!

      • we feel like we just can't leave this code alone
      • created a worker node alongside the Uyuni Server
      • flagged the Minion Start Event worker (Akka-ified on day 4) to run in the worker
      • fixed bugs and added hacks until registration works, moving load from the Server to the worker!
      • adding more workers distributes load round-robin!

      Screenshot-from-2020-02-15-22-24-43.png

      Soooo...

      [EVERYTHING IS AWESOME

    Similar Projects

    SUSE AI Meets the Game Board by moio

    Use tabletopgames.ai’s open source TAG and PyTAG frameworks to apply Statistical Forward Planning and Deep Reinforcement Learning to two board games of our own design. On an all-green, all-open source, all-AWS stack!
    A chameleon playing chess in a train car, as a metaphor of SUSE AI applied to games


    Results: Infrastructure Achievements

    We successfully built and automated a containerized stack to support our AI experiments. This included:

    A screenshot of k9s and nvtop showing PyTAG running in Kubernetes with GPU acceleration

    ./deploy.sh and voilà - Kubernetes running PyTAG (k9s, above) with GPU acceleration (nvtop, below)

    Results: Game Design Insights

    Our project focused on modeling and analyzing two card games of our own design within the TAG framework:

    • Game Modeling: We implemented models for Dario's "Bamboo" and Silvio's "Totoro" and "R3" games, enabling AI agents to play thousands of games ...in minutes!
    • AI-driven optimization: By analyzing statistical data on moves, strategies, and outcomes, we iteratively tweaked the game mechanics and rules to achieve better balance and player engagement.
    • Advanced analytics: Leveraging AI agents with Monte Carlo Tree Search (MCTS) and random action selection, we compared performance metrics to identify optimal strategies and uncover opportunities for game refinement .

    Cards from the three games

    A family picture of our card games in progress. From the top: Bamboo, Totoro, R3

    Results: Learning, Collaboration, and Innovation

    Beyond technical accomplishments, the project showcased innovative approaches to coding, learning, and teamwork:

    • "Trio programming" with AI assistance: Our "trio programming" approach—two developers and GitHub Copilot—was a standout success, especially in handling slightly-repetitive but not-quite-exactly-copypaste tasks. Java as a language tends to be verbose and we found it to be fitting particularly well.
    • AI tools for reporting and documentation: We extensively used AI chatbots to streamline writing and reporting. (Including writing this report! ...but this note was added manually during edit!)
    • GPU compute expertise: Overcoming challenges with CUDA drivers and cloud infrastructure deepened our understanding of GPU-accelerated workloads in the open-source ecosystem.
    • Game design as a learning platform: By blending AI techniques with creative game design, we learned not only about AI strategies but also about making games fun, engaging, and balanced.

    Last but not least we had a lot of fun! ...and this was definitely not a chatbot generated line!

    The Context: AI + Board Games


    Testing and adding GNU/Linux distributions on Uyuni by juliogonzalezgil

    Join the Gitter channel! https://gitter.im/uyuni-project/hackweek

    Uyuni is a configuration and infrastructure management tool that saves you time and headaches when you have to manage and update tens, hundreds or even thousands of machines. It also manages configuration, can run audits, build image containers, monitor and much more!

    Currently there are a few distributions that are completely untested on Uyuni or SUSE Manager (AFAIK) or just not tested since a long time, and could be interesting knowing how hard would be working with them and, if possible, fix whatever is broken.

    For newcomers, the easiest distributions are those based on DEB or RPM packages. Distributions with other package formats are doable, but will require adapting the Python and Java code to be able to sync and analyze such packages (and if salt does not support those packages, it will need changes as well). So if you want a distribution with other packages, make sure you are comfortable handling such changes.

    No developer experience? No worries! We had non-developers contributors in the past, and we are ready to help as long as you are willing to learn. If you don't want to code at all, you can also help us preparing the documentation after someone else has the initial code ready, or you could also help with testing :-)

    The idea is testing Salt and Salt-ssh clients, but NOT traditional clients, which are deprecated.

    To consider that a distribution has basic support, we should cover at least (points 3-6 are to be tested for both salt minions and salt ssh minions):

    1. Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)
    2. Onboarding (salt minion from UI, salt minion from bootstrap scritp, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator)
    3. Package management (install, remove, update...)
    4. Patching
    5. Applying any basic salt state (including a formula)
    6. Salt remote commands
    7. Bonus point: Java part for product identification, and monitoring enablement
    8. Bonus point: sumaform enablement (https://github.com/uyuni-project/sumaform)
    9. Bonus point: Documentation (https://github.com/uyuni-project/uyuni-docs)
    10. Bonus point: testsuite enablement (https://github.com/uyuni-project/uyuni/tree/master/testsuite)

    If something is breaking: we can try to fix it, but the main idea is research how supported it is right now. Beyond that it's up to each project member how much to hack :-)

    • If you don't have knowledge about some of the steps: ask the team
    • If you still don't know what to do: switch to another distribution and keep testing.

    This card is for EVERYONE, not just developers. Seriously! We had people from other teams helping that were not developers, and added support for Debian and new SUSE Linux Enterprise and openSUSE Leap versions :-)

    Pending

    FUSS

    FUSS is a complete GNU/Linux solution (server, client and desktop/standalone) based on Debian for managing an educational network.

    https://fuss.bz.it/

    Seems to be a Debian 12 derivative, so adding it could be quite easy.

    • [x] Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)
    • [P] Onboarding (salt minion from UI, salt minion from bootstrap scritp, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator) --> Works from UI, need to test the rest.
    • [P] Package management (install, remove, update...) --> Installing a new package works, needs to test the rest.
    • [ ] Patching (if patch information is available, could require writing some code to parse it, but IIRC we have support for Ubuntu already)
    • [ ] Applying any basic salt state (including a formula)
    • [ ] Salt remote commands
    • [ ] Bonus point: Java part for product identification, and monitoring enablement


    Automated Test Report reviewer by oscar-barrios

    Description

    In SUMA/Uyuni team we spend a lot of time reviewing test reports, analyzing each of the test cases failing, checking if the test is a flaky test, checking logs, etc.

    Goals

    Speed up the review by automating some parts through AI, in a way that we can consume some summary of that report that could be meaningful for the reviewer.

    Resources

    No idea about the resources yet, but we will make use of:

    • HTML/JSON Report (text + screenshots)
    • The Test Suite Status GithHub board (via API)
    • The environment tested (via SSH)
    • The test framework code (via files)


    Improve Development Environment on Uyuni by mbussolotto

    Description

    Currently create a dev environment on Uyuni might be complicated. The steps are:

    • add the correct repo
    • download packages
    • configure your IDE (checkstyle, format rules, sonarlint....)
    • setup debug environment
    • ...

    The current doc can be improved: some information are hard to be find out, some others are completely missing.

    Dev Container might solve this situation.

    Goals

    Uyuni development in no time:

    • using VSCode:
      • setting.json should contains all settings (for all languages in Uyuni, with all checkstyle rules etc...)
      • dev container should contains all dependencies
      • setup debug environment
    • implement a GitHub Workspace solution
    • re-write documentation

    Lots of pieces are already implemented: we need to connect them in a consistent solution.

    Resources

    • https://github.com/uyuni-project/uyuni/wiki


    Uyuni developer-centric documentation by deneb_alpha

    Description

    While we currently have extensive documentation on user-oriented tasks such as adding minions, patching, fine-tuning, etc, there is a notable gap when it comes to centralizing and documenting core functionalities for developers.

    The number of functionalities and side tools we have in Uyuni can be overwhelming. It would be nice to have a centralized place with descriptive list of main/core functionalities.

    Goals

    Create, aggregate and review on the Uyuni wiki a set of resources, focused on developers, that include also some known common problems/troubleshooting.

    The documentation will be helpful not only for everyone who is trying to learn the functionalities with all their inner processes like newcomer developers or community enthusiasts, but also for anyone who need a refresh.

    Resources

    The resources are currently aggregated here: https://github.com/uyuni-project/uyuni/wiki


    Testing and adding GNU/Linux distributions on Uyuni by juliogonzalezgil

    Join the Gitter channel! https://gitter.im/uyuni-project/hackweek

    Uyuni is a configuration and infrastructure management tool that saves you time and headaches when you have to manage and update tens, hundreds or even thousands of machines. It also manages configuration, can run audits, build image containers, monitor and much more!

    Currently there are a few distributions that are completely untested on Uyuni or SUSE Manager (AFAIK) or just not tested since a long time, and could be interesting knowing how hard would be working with them and, if possible, fix whatever is broken.

    For newcomers, the easiest distributions are those based on DEB or RPM packages. Distributions with other package formats are doable, but will require adapting the Python and Java code to be able to sync and analyze such packages (and if salt does not support those packages, it will need changes as well). So if you want a distribution with other packages, make sure you are comfortable handling such changes.

    No developer experience? No worries! We had non-developers contributors in the past, and we are ready to help as long as you are willing to learn. If you don't want to code at all, you can also help us preparing the documentation after someone else has the initial code ready, or you could also help with testing :-)

    The idea is testing Salt and Salt-ssh clients, but NOT traditional clients, which are deprecated.

    To consider that a distribution has basic support, we should cover at least (points 3-6 are to be tested for both salt minions and salt ssh minions):

    1. Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)
    2. Onboarding (salt minion from UI, salt minion from bootstrap scritp, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator)
    3. Package management (install, remove, update...)
    4. Patching
    5. Applying any basic salt state (including a formula)
    6. Salt remote commands
    7. Bonus point: Java part for product identification, and monitoring enablement
    8. Bonus point: sumaform enablement (https://github.com/uyuni-project/sumaform)
    9. Bonus point: Documentation (https://github.com/uyuni-project/uyuni-docs)
    10. Bonus point: testsuite enablement (https://github.com/uyuni-project/uyuni/tree/master/testsuite)

    If something is breaking: we can try to fix it, but the main idea is research how supported it is right now. Beyond that it's up to each project member how much to hack :-)

    • If you don't have knowledge about some of the steps: ask the team
    • If you still don't know what to do: switch to another distribution and keep testing.

    This card is for EVERYONE, not just developers. Seriously! We had people from other teams helping that were not developers, and added support for Debian and new SUSE Linux Enterprise and openSUSE Leap versions :-)

    Pending

    FUSS

    FUSS is a complete GNU/Linux solution (server, client and desktop/standalone) based on Debian for managing an educational network.

    https://fuss.bz.it/

    Seems to be a Debian 12 derivative, so adding it could be quite easy.

    • [x] Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)
    • [P] Onboarding (salt minion from UI, salt minion from bootstrap scritp, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator) --> Works from UI, need to test the rest.
    • [P] Package management (install, remove, update...) --> Installing a new package works, needs to test the rest.
    • [ ] Patching (if patch information is available, could require writing some code to parse it, but IIRC we have support for Ubuntu already)
    • [ ] Applying any basic salt state (including a formula)
    • [ ] Salt remote commands
    • [ ] Bonus point: Java part for product identification, and monitoring enablement


    Enable the containerized Uyuni server to run on different host OS by j_renner

    Description

    The Uyuni server is provided as a container, but we still require it to run on Leap Micro? This is not how people expect to use containerized applications, so it would be great if we tested other host OSs and enabled them by providing builds of necessary tools for (e.g. mgradm). Interesting candidates should be:

    • openSUSE Leap
    • Cent OS 7
    • Ubuntu
    • ???

    Goals

    Make it really easy for anyone to run the Uyuni containerized server on whatever OS they want (with support for containers of course).


    Create SUSE Manager users from ldap/ad groups by mbrookhuis

    Description

    This tool is used to create users in SUSE Manager Server based on LDAP/AD groups. For each LDAP/AD group a role within SUSE Manager Server is defined. Also, the tool will check if existing users still have the role they should have, and, if not, it will be corrected. The same for if a user is disabled, it will be enabled again. If a users is not present in the LDAP/AD groups anymore, it will be disabled or deleted, depending on the configuration.

    The code is written for Python 3.6 (the default with SLES15.x), but will also work with newer versions. And works against SUSE Manger 4.3 and 5.x

    Goals

    Create a python and/or golang utility that will manage users in SUSE Manager based on LDAP/AD group-membership. In a configuration file is defined which roles the members of a group will get.

    Table of contents

    Installation

    To install this project, perform the following steps:

    • Be sure that python 3.6 is installed and also the module python3-PyYAML. Also the ldap3 module is needed:

    bash zypper in python3 python3-PyYAML pip install yaml

    • On the server or PC, where it should run, create a directory. On linux, e.g. /opt/sm-ldap-users

    • Copy all the file to this directory.

    • Edit the configsm.yaml. All parameters should be entered. Tip: for the ldap information, the best would be to use the same as for SSSD.

    • Be sure that the file sm-ldap-users.py is executable. It would be good to change the owner to root:root and only root can read and execute:

    bash chmod 600 * chmod 700 sm-ldap-users.py chown root:root *

    Usage

    This is very simple. Once the configsm.yaml contains the correct information, executing the following will do the magic:

    bash /sm-ldap-users.py

    repository link

    https://github.com/mbrookhuis/sm-ldap-users


    Improve Development Environment on Uyuni by mbussolotto

    Description

    Currently create a dev environment on Uyuni might be complicated. The steps are:

    • add the correct repo
    • download packages
    • configure your IDE (checkstyle, format rules, sonarlint....)
    • setup debug environment
    • ...

    The current doc can be improved: some information are hard to be find out, some others are completely missing.

    Dev Container might solve this situation.

    Goals

    Uyuni development in no time:

    • using VSCode:
      • setting.json should contains all settings (for all languages in Uyuni, with all checkstyle rules etc...)
      • dev container should contains all dependencies
      • setup debug environment
    • implement a GitHub Workspace solution
    • re-write documentation

    Lots of pieces are already implemented: we need to connect them in a consistent solution.

    Resources

    • https://github.com/uyuni-project/uyuni/wiki


    Testing and adding GNU/Linux distributions on Uyuni by juliogonzalezgil

    Join the Gitter channel! https://gitter.im/uyuni-project/hackweek

    Uyuni is a configuration and infrastructure management tool that saves you time and headaches when you have to manage and update tens, hundreds or even thousands of machines. It also manages configuration, can run audits, build image containers, monitor and much more!

    Currently there are a few distributions that are completely untested on Uyuni or SUSE Manager (AFAIK) or just not tested since a long time, and could be interesting knowing how hard would be working with them and, if possible, fix whatever is broken.

    For newcomers, the easiest distributions are those based on DEB or RPM packages. Distributions with other package formats are doable, but will require adapting the Python and Java code to be able to sync and analyze such packages (and if salt does not support those packages, it will need changes as well). So if you want a distribution with other packages, make sure you are comfortable handling such changes.

    No developer experience? No worries! We had non-developers contributors in the past, and we are ready to help as long as you are willing to learn. If you don't want to code at all, you can also help us preparing the documentation after someone else has the initial code ready, or you could also help with testing :-)

    The idea is testing Salt and Salt-ssh clients, but NOT traditional clients, which are deprecated.

    To consider that a distribution has basic support, we should cover at least (points 3-6 are to be tested for both salt minions and salt ssh minions):

    1. Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)
    2. Onboarding (salt minion from UI, salt minion from bootstrap scritp, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator)
    3. Package management (install, remove, update...)
    4. Patching
    5. Applying any basic salt state (including a formula)
    6. Salt remote commands
    7. Bonus point: Java part for product identification, and monitoring enablement
    8. Bonus point: sumaform enablement (https://github.com/uyuni-project/sumaform)
    9. Bonus point: Documentation (https://github.com/uyuni-project/uyuni-docs)
    10. Bonus point: testsuite enablement (https://github.com/uyuni-project/uyuni/tree/master/testsuite)

    If something is breaking: we can try to fix it, but the main idea is research how supported it is right now. Beyond that it's up to each project member how much to hack :-)

    • If you don't have knowledge about some of the steps: ask the team
    • If you still don't know what to do: switch to another distribution and keep testing.

    This card is for EVERYONE, not just developers. Seriously! We had people from other teams helping that were not developers, and added support for Debian and new SUSE Linux Enterprise and openSUSE Leap versions :-)

    Pending

    FUSS

    FUSS is a complete GNU/Linux solution (server, client and desktop/standalone) based on Debian for managing an educational network.

    https://fuss.bz.it/

    Seems to be a Debian 12 derivative, so adding it could be quite easy.

    • [x] Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)
    • [P] Onboarding (salt minion from UI, salt minion from bootstrap scritp, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator) --> Works from UI, need to test the rest.
    • [P] Package management (install, remove, update...) --> Installing a new package works, needs to test the rest.
    • [ ] Patching (if patch information is available, could require writing some code to parse it, but IIRC we have support for Ubuntu already)
    • [ ] Applying any basic salt state (including a formula)
    • [ ] Salt remote commands
    • [ ] Bonus point: Java part for product identification, and monitoring enablement


    Saline (state deployment control and monitoring tool for SUSE Manager/Uyuni) by vizhestkov

    Project Description

    Saline is an addition for salt used in SUSE Manager/Uyuni aimed to provide better control and visibility for states deploymend in the large scale environments.

    In current state the published version can be used only as a Prometheus exporter and missing some of the key features implemented in PoC (not published). Now it can provide metrics related to salt events and state apply process on the minions. But there is no control on this process implemented yet.

    Continue with implementation of the missing features and improve the existing implementation:

    • authentication (need to decide how it should be/or not related to salt auth)

    • web service providing the control of states deployment

    Goal for this Hackweek

    • Implement missing key features

    • Implement the tool for state deployment control with CLI

    Resources

    https://github.com/openSUSE/saline


    SUSE Prague claw machine by anstalker

    Project Description

    The idea is to build a claw machine similar to e.g. this one:

    example image

    Why? Well, it could be a lot of fun!

    But also it's a great way to dispense SUSE and openSUSE merch like little Geekos at events like conferences, career fairs and open house events.

    Goal for this Hackweek

    Build an arcade claw machine.

    Resources

    In French, an article about why you always lose in claw machine games:

    We're looking for handy/crafty people in the Prague office:

    • woodworking XP or equipment
    • arduino/raspi embedded programming knowledge
    • Anthony can find a budget for going to GM and buying servos and such ;)