Uyuni: improve spacewalk-repo-sync performance
Let's make reposync faster
Every day,
Multiple times a day,
Every SUSE Manager customer,
Every Red Hat Satellite customer,
Every Spacewalk user,
And every Uyuni user...
...spends a lot of CPU and wall clock time in reposyncing.
Intro
A lot of that time is wasted by an old, overcomplicated and most of all inefficient algorithm that contributes heavily on heat dissipation and user patience depletion!
HackWeek hackers, we can change that!
Past attempts only partially succeeded: https://trello.com/c/inl9Wu0p/40-reduce-global-warming, https://trello.com/c/dYAR0J8K/13-reduce-global-warming-take-2
But we have better tools now!
Tooling
py-spy to the rescue: introduction
Install with:
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python get-pip.py
pip install py-spy
Trace a running spacewalk-repo-sync with:
py-spy --nonblocking --pid `ps aux | grep spacewalk-repo-sync | grep -v grep | awk '{print $2}'` --flame output.svg --duration 10
Look at the results with:
python -m SimpleHTTPServer 8666
And point your browser to http://:8666/output.html. Here is one such example:
Current remarks:
- we currently spend a lot of time in lookup functions
- lookup functions SELECT rows at every INSERT
- this is especially bad for checksums, capabilities and some other cases
- design comes from Oracle and can probably be changed!
Looking for hackers with the skills:
This project is part of:
Hack Week 18
Activity
Comments
-
over 6 years ago by ebischoff | Reply
See also this fate request "Have a synchronization that does not take hours (or days)"
-
over 6 years ago by joachimwerner | Reply
Related, but probably out of scope for your hack week project: Once we've optimized the syncing code, I think we could also reduce the scope of what needs to be synced for many customers: Especially for pilots, but also in real life, many of the older updates (e.g. several complete kernels, several Java updates) are never going to be needed, but still synced. We should investigate how we can offer something like a "JeR" ("Just enough Repo") to speed things up even more. This could be done server-side (provide alternative repo metadata for a "current stuff only" repo or client-side (but then some dependency resolution magic is probably needed).
-
Similar Projects
Bring to Cockpit + System Roles capabilities from YAST by miguelpc
Bring to Cockpit + System Roles features from YAST
Cockpit and System Roles have been added to SLES 16 There are several capabilities in YAST that are not yet present in Cockpit and System Roles We will follow the principle of "automate first, UI later" being System Roles the automation component and Cockpit the UI one.
Goals
The idea is to implement service configuration in System Roles and then add an UI to manage these in Cockpit. For some capabilities it will be required to have an specific Cockpit Module as they will interact with a reasource already configured.
Resources
A plan on capabilities missing and suggested implementation is available here: https://docs.google.com/spreadsheets/d/1ZhX-Ip9MKJNeKSYV3bSZG4Qc5giuY7XSV0U61Ecu9lo/edit
Linux System Roles: https://linux-system-roles.github.io/