Uyuni: re-architecting code with Akka
Simplify the codebase by using a more modern toolkit to accelerate maintenance and future development.
Enjoy Hakkaweek!
Problem
Uyuni's codebase has a 10+ year history, with contributions from dozens of individuals at varying points in their technical experience, understanding of the project, goals and timelines.
Inevitably, solutions to some of the common problems have grown organically, with different people solving slightly different instances of the same problem in different and possibly incomplete ways. This makes maintenance and future evolution of Uyuni more difficult, as developers and designers need to keep specifics of all such solutions into consideration.
One such area is the handling of asynchronous tasks within Java code, where currently we have at least 4 different implementations, none of which is really complete.
Spirit of this HackWeek effort
Write a proof-of-concept to replace as many existing, incomplete implementations as possible using one piece of existing, proven open source technology which already fulfills all known requirements (akka.io).
Should such a proof-of-concept succeed, integrating it into the project would:
- Simplify the code base, and possibly user experience, given there would be 1 solution instead of 4
- Modernize our project, by using a new toolkit instead of several older, homegrown solutions
- Accelerate new development of Uyuni, by leveraging existing know-how of Akka, and not requiring developers to know all current solutions
Failure... is definitely an option!
Learning why this can't work will be as valuable as having it working!
Not learning and not having fun... are NOT options!
That's HackWeek folks!
Hacking details
For asynchronous task handling we currently have:
- Taskomatic's basic facilities, including one-shot tasks
- Some specific task facilities (eg.
ErrataCacheDriver,SSHPushDriver) - Tomcat's
MessageQueuefacility - Tomcat's Salt event handler (
PGEventListener)
Open problems that in various degrees affect all of the above are:
- enforcing dependencies between asynchronous tasks
- enforcing dependencies between asynchronous tasks and other parts of the codebase
- establishing priority of tasks
- tuning, performance
- observability
akka.io seems to have all needed mechanisms:
- dependencies can be expressed by actors firing messages to other actors when done
- any other part of the code base can ask an actor to do something by sending it a message
- Akka has a concept of priority messages to actors
- Akka seems to be regarded as high performance, has lots of tuning knobs
- Akka has a lot of utilities for monitoring
Moreover Akka can run actors on different processes or even hosts transparently to the code. We could exploit this to improve the Tomcat/Taskomatic interface or even offload some tasks to separate machines.
Battle plan
- spend one day researching Akka and understanding how it works
- attack the simplest subsystem first, try to have it working somehow with Akka
- attack as many other subsystems as possible
- last half day: prepare demo for Monday : : :
Looking for hackers with the skills:
This project is part of:
Hack Week 19
Activity
Comments
-
over 4 years ago by moio | Reply
Diary - day 1: learning
We learned many concepts:
- some immediately relevant for this HackWeek: Actor, ActorSystem, ActorContext, typed Actor, Actor hierarchy, Behavior, Mailbox, Dispatcher, Routing, at-most-once delivery
- others not immediately relevant, still interesting: Supervision, Artery, Akka Clustering, Akka Streams, backpressure, the Phi Accrual Failure Detector, Akka Persistence/Event Sourcing, consistent hashing
- on top of that: some new related tech bits: vavr.io, aeron.io, logback, Lightbend Telemetry, HOCON
We also played with some "hello world" examples.
-
over 4 years ago by moio | Reply
Diary - day 2: MessageQueue classes
- "hello world" examples moved inside the Uyuni code base and ran in Uyuni
- replaced "hello world" examples with code to run three MessageQueue tasks, which work after the port to akka!
- added reflection-based glue code to automatically register Actors, minimizing boilerplate for next iterations
- in the process, developed several refactoring patches to simplify our codebase that can be accepted even outside of the scope of this effort (and a bugfix and a contribution to an unrelated PR)
- learnt about the Reflections library
Always-up-to-date-Uyuni PR with latest results available
To try:
- deploy an Uyuni Server (Master branch)
- use manager-build.xml to resolve Ivy dependencies and deploy
-
over 4 years ago by moio | Reply
Diary - day 3: traditional MessageQueue is obsoleted
- created a better mechanism to start asynchronous tasks after a Transaction has ended. Current code requires polling every 10ms!
- more use cases of MessageQueue were moved to Akka. By end of day only the ones for Salt management remain
- configuration mechanism added to decide concurrency limits (per message type!)
- some dead code removed
-
over 4 years ago by moio | Reply
Diary - day 4: MessageQueue and Salt event thread pool are removed
- many more use cases are switched to Akka (total is ~35)
- MessageQueue and related classes are removed completely, together with all surrounding likely unsafe code
- deferring mechanism moved to race-condition prone plain Java to Actor
- hash routing implemented in a straightforward patch compared to a much more complicated one we had, touching 3 programming languages
-
-
over 4 years ago by moio | Reply
Diary - HackWeekEnd: (a very first) distributed Uyuni works!
- we feel like we just can't leave this code alone
- created a worker node alongside the Uyuni Server
- flagged the Minion Start Event worker (Akka-ified on day 4) to run in the worker
- fixed bugs and added hacks until registration works, moving load from the Server to the worker!
- adding more workers distributes load round-robin!
Soooo...
[
- we feel like we just can't leave this code alone
- created a worker node alongside the Uyuni Server
- flagged the Minion Start Event worker (Akka-ified on day 4) to run in the worker
- fixed bugs and added hacks until registration works, moving load from the Server to the worker!
- adding more workers distributes load round-robin!
Reply to moio
Diary - HackWeekEnd: (a very first) distributed Uyuni works!
Soooo...
[
# A First Level Header ## A Second Level Header Use one asterisk to *emphasize* Use two asterisks for **strong emphasis** * Use asterisks * for lists This is an [example link](http://example.com/) This is an  This is a user link @hans This is a project link hw#some-cool-title
Similar Projects