Project Description

For a few months, openSUSE community has the ability to host the openSUSE rpm repositories on a commercial CDN and it is in a slowly rolling out phase. There are however remaining potential bottlenecks and optimisation opportunities. My goal for this hack week is to investigate them and make reasonable progress on resolving them.

Topics that are in scope and being investigated:

  • Switch of repository metadata from gzip to the more modern (faster to decompress, smaller sizes) zstd
  • investigate zypper ref performance overheads
  • Understand Zsync and ZChunk and benchmark for tradeoffs
  • Leverage the CDN for delivering repository metadata and a mirrorlist so that the roundtrips to download.o.org can be reduced for users outside europe.
  • Investigate performance of HTTP/3 to see if it would benefit us

Goal for this Hackweek

Focus on openSUSE Tumbleweed

Looking for hackers with the skills:

opensuse cdn zypper rpm performance benchmark

This project is part of:

Hack Week 23

Activity

  • 2 months ago: mlschroe joined this project.
  • 3 months ago: favogt joined this project.
  • 3 months ago: favogt liked this project.
  • 3 months ago: dirkmueller added keyword "cdn" to this project.
  • 3 months ago: dirkmueller added keyword "zypper" to this project.
  • 3 months ago: dirkmueller added keyword "rpm" to this project.
  • 3 months ago: dirkmueller added keyword "performance" to this project.
  • 3 months ago: dirkmueller added keyword "benchmark" to this project.
  • 3 months ago: dirkmueller added keyword "opensuse" to this project.
  • 3 months ago: dirkmueller started this project.
  • 3 months ago: dirkmueller originated this project.

  • Comments

    • dirkmueller
      3 months ago by dirkmueller | Reply

      It turns out that createrepo_c was already preparing the switch to Zstd and Zchunk, so the bulk of the work has been to fix various places in the Open Build Service and product building logic to handle that. The submissions are planned to go live on Nov 13th.

      This provides overall a 10-30% reduction in download size for repository data and removes a few hundred ms in decompression time.

    • dirkmueller
      3 months ago by dirkmueller | Reply

      @mlschroe made a patch to libsolv that removes ~ 500ms from the parsing time for the tumbleweed repositories: https://github.com/openSUSE/libsolv/commit/23cbed3219bd07b5c3fa1ed8a6f2fa6c478c0fdb

    • dirkmueller
      3 months ago by dirkmueller | Reply

      I've spend several days on profiling and tuning zchunk support for the tumbleweed usecase (updating from tumbleweed snapshots) and made some upstream contributions to createrepo_c and zchunk to allow for these tunings.

    • dirkmueller
      3 months ago by dirkmueller | Reply

      @mlschroe worked on reviving zsync support in libzypp's default multicurl implementation

    • dirkmueller
      3 months ago by dirkmueller | Reply

      I've submitted necessary changes to enable curl with http/3 support. However this requires either switching to gnutls (which the security team doesn't like) or include the quictls patches for openssl. submitted the latter and waiting for maintainer review.

    Similar Projects

    Update Haskell ecosystem in Tumbleweed to GHC-9.6.x by psimons

    [comment]: # (Please use the project descriptio...


    Containerized home mirror by lkocman

    I'm running a simple home mirror, but I managed...


    Basic calendar interface for upcoming openSUSE meetings by lkocman

    [comment]: # (Please use the project descriptio...


    Apple Silicon openSUSE spin by vgrinco

    Project Description

    The folks at [Asahi lin...


    Extend GObject based introspectable API to libzypp by zbenjamin

    [comment]: # (Please use the project descriptio...


    Switch software-o-o to parse repomd data by hennevogel

    Currently software.opensuse.org search is using...


    Tumbleweed support for Raspberry Pi 4 with Quad SATA HAT by jbaier_cz

    [comment]: # (Please use the project descriptio...


    LSP server for SPEC files by mcepl

    Has anybody ever heard about RPM Spec LSP serve...


    A set of utilities to produce a "from scratch" OCI/Docker container using Opensuse/SLE rpms by ldragon

    [comment]: # (Please use the project descriptio...


    A CLI for Harvester by mohamed.belgaied

    [comment]: # Harvester does not officially come...


    Visualization of historical sar(1) archives by ggherdovich

    Project Description

    The sar(1) tool, fr...