Project Description

For a few months, openSUSE community has the ability to host the openSUSE rpm repositories on a commercial CDN and it is in a slowly rolling out phase. There are however remaining potential bottlenecks and optimisation opportunities. My goal for this hack week is to investigate them and make reasonable progress on resolving them.

Topics that are in scope and being investigated:

  • Switch of repository metadata from gzip to the more modern (faster to decompress, smaller sizes) zstd
  • investigate zypper ref performance overheads
  • Understand Zsync and ZChunk and benchmark for tradeoffs
  • Leverage the CDN for delivering repository metadata and a mirrorlist so that the roundtrips to download.o.org can be reduced for users outside europe.
  • Investigate performance of HTTP/3 to see if it would benefit us

Goal for this Hackweek

Focus on openSUSE Tumbleweed

Looking for hackers with the skills:

opensuse cdn zypper rpm performance benchmark

This project is part of:

Hack Week 23

Activity

  • 12 months ago: mlschroe joined this project.
  • 12 months ago: favogt joined this project.
  • 12 months ago: favogt liked this project.
  • about 1 year ago: dirkmueller added keyword "cdn" to this project.
  • about 1 year ago: dirkmueller added keyword "zypper" to this project.
  • about 1 year ago: dirkmueller added keyword "rpm" to this project.
  • about 1 year ago: dirkmueller added keyword "performance" to this project.
  • about 1 year ago: dirkmueller added keyword "benchmark" to this project.
  • about 1 year ago: dirkmueller added keyword "opensuse" to this project.
  • about 1 year ago: dirkmueller started this project.
  • about 1 year ago: dirkmueller originated this project.

  • Comments

    • dirkmueller
      about 1 year ago by dirkmueller | Reply

      It turns out that createrepo_c was already preparing the switch to Zstd and Zchunk, so the bulk of the work has been to fix various places in the Open Build Service and product building logic to handle that. The submissions are planned to go live on Nov 13th.

      This provides overall a 10-30% reduction in download size for repository data and removes a few hundred ms in decompression time.

    • dirkmueller
      about 1 year ago by dirkmueller | Reply

      @mlschroe made a patch to libsolv that removes ~ 500ms from the parsing time for the tumbleweed repositories: https://github.com/openSUSE/libsolv/commit/23cbed3219bd07b5c3fa1ed8a6f2fa6c478c0fdb

    • dirkmueller
      about 1 year ago by dirkmueller | Reply

      I've spend several days on profiling and tuning zchunk support for the tumbleweed usecase (updating from tumbleweed snapshots) and made some upstream contributions to createrepo_c and zchunk to allow for these tunings.

    • dirkmueller
      about 1 year ago by dirkmueller | Reply

      @mlschroe worked on reviving zsync support in libzypp's default multicurl implementation

    • dirkmueller
      about 1 year ago by dirkmueller | Reply

      I've submitted necessary changes to enable curl with http/3 support. However this requires either switching to gnutls (which the security team doesn't like) or include the quictls patches for openssl. submitted the latter and waiting for maintainer review.

    Similar Projects

    Update Haskell ecosystem in Tumbleweed to GHC-9.10.x by psimons

    Description

    We are currently at GHC-9.8.x, which a bit old. So I'd like to take a shot at the latest version of the compiler, GHC-9.10.x. This is gonna be interesting because the new version requires major updates to all kinds of libraries and base packages, which typically means patching lots of packages to make them build again.

    Goals

    Have working builds of GHC-9.10.x and the required Haskell packages in 'devel:languages:haskell` so that we can compile:

    • git-annex
    • pandoc
    • xmonad
    • cabal-install

    Resources

    • https://build.opensuse.org/project/show/devel:languages:haskell/
    • https://github.com/opensuse-haskell/configuration/
    • #discuss-haskell
    • https://www.twitch.tv/peti343


    Digital art wallpapers for openSUSE Leap and Tumbleweed by lkocman

    Description

    We've enrolled set of new wallpapers to both Leap 16 and Tumbleweed as part of https://news.opensuse.org/2024/10/26/leap-tw-get-makeovers/

    We've previewed digital art wallpapers which were not part of the initial drop. I'd like to spend time on hackweek to finialize my current Taipei (mountains) and Mauritius digital art wallpapers.

    Goals

    Finalize existing two digital art wallpapers for Leap and Tumbleweed https://github.com/openSUSE/branding/issues/155 Make them available as part of leap16 dir in https://github.com/openSUSE/wallpapers and update (This makes is available to Tumbleweed users as well). Update https://build.opensuse.org/package/show/X11:common:Factory/wallpapers-openSUSE-extra && Leap:16.0 && Factory.

    Resources

    https://github.com/openSUSE/branding/issues/155 The mauritius draft can be found in https://github.com/lkocman/geo-wallpapers


    New openSUSE-welcome by lkocman

    Project Description

    Let's revisit our existing openSUSE welcome app.

    My goal was to show Leap 16 in a new coat. Welcome app adds to the first time use experience. We've recently added donation button to our existing welcome.

    Some things that I recently wanted to address were EOL and possibly upgrade notification.

    I've already done some experiments with mint welcome app, but not sure if it's better than the existing one.

    There is also a PR to rework existing app https://github.com/openSUSE/openSUSE-welcome/pull/36 (this should be considered as an option too)

    Goal for this Hackweek

    New welcome app, possibly with EOL notification for Leap.

    1) Welcome application(s) with (rebrand changes) maintained under github.com/openSUSE

    2) Application is submitted to openSUSE:Factory && openSUSE:Leap:16.0

    3) Updated needles in openQA (probably post hackweek)

    Resources

    Reddit discussion about the best welcome app out there.

    Github repo for the current welcome app.


    YQPkg - Bringing the Single Package Selection Back to Life by shundhammer

    tl;dr

    Rip out the high-level YQPackageSelector widget from YaST and make it a standalone Qt program without any YaST dependencies.

    See section "Result" at the bottom for the current status after the hack week.

    The Past and the Present

    We used to have and still have a powerful software selection with the YaST sw_single module (and the YaST patterns counterpart): You can select software down to the package level, you can easily select one of many available package versions, you can select entire patterns - or just view them and pick individual packages from patterns.

    You can search packages based on name, description, "requires" or "provides" level, and many more things.

    The Future

    YaST is on its way out, to be replaced by the new Agama installer and Cockpit for system administration. Those tools can do many things, but fine-grained package selection is not among them. And there are also no other Open Source tools available for that purpose that even come close to the YaST package selection.

    Many aspects of YaST have become obsolete over the years; many subsystems now come with a good default configuration, or they can configure themselves automatically. Just think about sound or X11 configuration; when did you last need to touch them?

    For others, the desktops bring their own tools (e.g. printers), or there are FOSS configuration tools (NetworkManager, BlueMan). Most YaST modules are no longer needed, and for many others there is a replacement in tools like Cockpit.

    But no longer having a powerful fine-grained package selection like in YaST sw_single will hurt. Big time. At least until there is an adequate replacement, many users will want to keep it.

    The Idea

    YaST sw_single always revolved around a powerful high-level widget on the abstract UI level. Libyui has low-level widgets like YPushButton, YCheckBox, YInputField, more advanced ones like YTable, YTree; and some few very high-level ones like YPackageSelector and YPatternSelector that do the whole package selection thing alone, working just on the libzypp level and changing the status of packages or patterns there.

    For the YaST Qt UI, the YQPackageSelector / YQPatternSelector widgets work purely on the Qt and libzypp level; no other YaST infrastructure involved, in particular no Ruby (or formerly YCP) interpreter, no libyui-level widgets, no bindings between Qt / C++ and Ruby / YaST-core, nothing. So it's not too hard to rip all that part out of YaST and create a standalone program from it.

    For the NCurses UI, the NCPackageSelector / NCPatternSelector create a lot of libyui widgets (inheriting YWidget / NCWidget) and use a lot of libyui calls to glue them together; and all that of course still needs a lot of YaST / libyui / libyui-ncurses infrastructure. So NCurses is out of scope here.

    Preparatory Work: Initializing the Package Subsystem

    To see if this is feasible at all, the existing UI examples needed some fixing to check what is needed on that level. That was the make-or-break decision: Would it be realistically possible to set the needed environment in libzypp up (without being stranded in the middle of that task alone at the end of the hack week)?

    Yes, it is: That part is already working:

    https://github.com/yast/yast-ycp-ui-bindings/pull/71

    Go there for a screenshot

    That's already halfway there.

    The complete Ruby code of this example is here. The real thing will be pure C++ without any YaST dependencies.

    The Plan


    New migration tool for Leap by lkocman

    Update

    I will call a meeting with other interested people at 11:00 CET https://meet.opensuse.org/migrationtool

    Description

    SLES 16 plans to have no yast tool in it. Leap 16 might keep some bits, however, we need a new tool for Leap to SLES migration, as this was previously handled by a yast2-migration-sle

    Goals

    A tool able to migrate Leap 16 to SLES 16, I would like to cover also other scenarios within openSUSE, as in many cases users would have to edit repository files manually.

    • Leap -> Leap n+1 (minor and major version updates)
    • Leap -> SLES docs
    • Leap -> Tumbleweed
    • Leap -> Slowroll
    • Leap Micro -> Leap Micro n+1 (minor and major version updates)
    • Leap Micro -> MicroOS

    Hackweek 24 update

    Marcela and I were working on the project from Brno coworking as well as finalizing pieces after the hackweek. We've tested several migration scenarios and it works. But it needs further polishing and testing.

    Projected was renamed to opensuse-migration-tool and was submitted to devel project https://build.opensuse.org/requests/1227281

    Repository

    https://github.com/openSUSE/opensuse-migration-tool

    Out of scope is any migration to an immutable system. I know Richard already has some tool for that.

    Resources

    Tracker for yast stack reduction code-o-o/leap/features#173 YaST stack reduction


    "autoremove" functionality for zypper by e_bischoff

    The purpose would be to have the equivalent of Ubuntu's "apt-get autoremove" functionality.

    When you install package P, it might draw in dependancies D1, D2, ... Dn automatically.

    When you later uninstall P, the dependancies D1, D2, ..., Dn might remain on your system.

    If you keep installing and uninstalling packages, after a while your system remains cluttered with things you don't need.

    The idea would be to mark all dependancies that were installed but not explicitely requested as "installed automatically". Then a command like "zypper autoremove" could remove them at once if they are not needed anymore.

    Approaches seen online

    After scouring forms for solutions these are some alias's used to replicate the functionality

    1. zypper packages --unneeded | awk -F'|' 'NR==0 || NR==1 || NR==2 || NR==3 || NR==4 {next} {print $3}' | grep -v Name | sudo xargs zypper remove --clean-deps

    2. This one is a script and has bashisms bash mapfile -t unneeded < <(zypper --quiet pa --unneeded | awk '$1 == "i" { print $5, "-", $7 }') (( ${#unneeded@]} )) && sudo zypper --quiet rm --clean-deps --details "${unneeded@]}"

    3. sudo zypper rm $(zypper pa --unneeded | awk '/i / {print $3}' FS='|' | uniq | tr -d ' ')

    Based on testing zypper packages --orphaned provides packages that are not in any repo, even if a user has explicitly installed them, so --orphaned may not be the way to go, instead focusing on --unneeded


    Ansible for add-on management by lmanfredi

    Description

    Machines can contains various combinations of add-ons and are often modified during the time.

    The list of repos can change so I would like to create an automation able to reset the status to a given state, based on metadata available for these machines

    Goals

    Create an Ansible automation able to take care of add-on (repo list) configuration using metadata as reference

    Resources

    Results

    Created WIP project Ansible-add-on-openSUSE


    Switch software-o-o to parse repomd data by hennevogel

    Currently software.opensuse.org search is using the OBS binary search for everything, even for packages inside the openSUSE distributions. Let's switch this to use repomd data from download.opensuse.org


    A CLI for Harvester by mohamed.belgaied

    [comment]: # Harvester does not officially come with a CLI tool, the user is supposed to interact with Harvester mostly through the UI [comment]: # Though it is theoretically possible to use kubectl to interact with Harvester, the manipulation of Kubevirt YAML objects is absolutely not user friendly. [comment]: # Inspired by tools like multipass from Canonical to easily and rapidly create one of multiple VMs, I began the development of Harvester CLI. Currently, it works but Harvester CLI needs some love to be up-to-date with Harvester v1.0.2 and needs some bug fixes and improvements as well.

    Project Description

    Harvester CLI is a command line interface tool written in Go, designed to simplify interfacing with a Harvester cluster as a user. It is especially useful for testing purposes as you can easily and rapidly create VMs in Harvester by providing a simple command such as: harvester vm create my-vm --count 5 to create 5 VMs named my-vm-01 to my-vm-05.

    asciicast

    Harvester CLI is functional but needs a number of improvements: up-to-date functionality with Harvester v1.0.2 (some minor issues right now), modifying the default behaviour to create an opensuse VM instead of an ubuntu VM, solve some bugs, etc.

    Github Repo for Harvester CLI: https://github.com/belgaied2/harvester-cli

    Done in previous Hackweeks

    • Create a Github actions pipeline to automatically integrate Harvester CLI to Homebrew repositories: DONE
    • Automatically package Harvester CLI for OpenSUSE / Redhat RPMs or DEBs: DONE

    Goal for this Hackweek

    The goal for this Hackweek is to bring Harvester CLI up-to-speed with latest Harvester versions (v1.3.X and v1.4.X), and improve the code quality as well as implement some simple features and bug fixes.

    Some nice additions might be: * Improve handling of namespaced objects * Add features, such as network management or Load Balancer creation ? * Add more unit tests and, why not, e2e tests * Improve CI * Improve the overall code quality * Test the program and create issues for it

    Issue list is here: https://github.com/belgaied2/harvester-cli/issues

    Resources

    The project is written in Go, and using client-go the Kubernetes Go Client libraries to communicate with the Harvester API (which is Kubernetes in fact). Welcome contributions are:

    • Testing it and creating issues
    • Documentation
    • Go code improvement

    What you might learn

    Harvester CLI might be interesting to you if you want to learn more about:

    • GitHub Actions
    • Harvester as a SUSE Product
    • Go programming language
    • Kubernetes API