Optimize KIWI product build in Open Build Service (OBS)
an invention by lslezak
Updated almost 6 years ago. 2 hacker ♥️.

Introduction

OBS can build not only build traditional RPM packages, but it can even build appliances or installation ISO images.

We use this feature in YaST:Head:installer project to build an installation image with the latest YaST (which is not included in Factory yet). This allows as to try new things without worrying to break the installer in Factory or in the upcoming openSUSE distribution.

The Problem

Because the public build service is sometimes quite slow I have deployed a private OBS instance (see the OBS download page) at my workstation.

I found out that product building is relatively slow, even when running in the private instance locally. A quick look revealed that for building ISO image the build service downloads too many packages including source packages, debugsource and debuginfo packages which are not need for our installation ISO image. The amount of the unnecessary packages is several gigabytes even for our small mini installation DVD (which contains packages only for Minimal X pattern, no KDE nor GNOME...).

Focus of the Project

This project is focused on decreasing package download for product ISO image build.

The project should make building ISO images in OBS faster (especially in a private instance linking to the public OBS server), saving disk space and download bandwidth.

Result

I have successfully added kiwi file parsing (to know whether the debuginfo and source packages are needed or not) and skipping debuginfo and source packages into both OBS scheduler (which checks and evaluates dependencies and triggers builds) and worker (which downloads the packages during package/ISO build).

Outcome of the Project

  • Significantly decreased amount of downloaded packages, over 7GB (!!) even for our small miniDVD
  • This results in faster builds, disk space and network bandwidth savings
  • Created a GitHub pull request to include this improvement upstream
  • I learned a lot about OBS internals
  • And refreshed my perl knowledge ;-)

Looking for hackers with the skills:

obs kiwi build iso_image

This project is part of:

Hack Week 10

Activity

  • over 11 years ago: k0da liked this project.
  • over 11 years ago: fcrozat liked this project.
  • over 11 years ago: lslezak added keyword "iso_image" to this project.
  • over 11 years ago: lslezak added keyword "obs" to this project.
  • over 11 years ago: lslezak added keyword "kiwi" to this project.
  • over 11 years ago: lslezak added keyword "build" to this project.
  • over 11 years ago: lslezak started this project.
  • over 11 years ago: lslezak originated this project.


    • Comments

    • lslezak
      over 11 years ago by lslezak | Reply

      I have implemented this optimization in my private OBS instance at https://muffin.suse.cz, for building our miniDVD.

      In this case it saves over 7GB package download for the ISO image build!!

    • lslezak
      over 11 years ago by lslezak | Reply

      I have created a GitHub pull request to include this improvement in the next OBS release (and likely in the public instance as well), see https://github.com/openSUSE/open-build-service/pull/426.

    • lslezak
      over 11 years ago by lslezak | Reply

      Details for the miniDVD build:

      Skipped 3831 (3299 MB) debuginfo packages
Skipped 1511 (3793 MB) source packages

      That's pretty nice saving!

    • lslezak
      over 11 years ago by lslezak | Reply

      There are still possible some more improvements, OBS still downloads some more not needed packages:

      • If you build x86_64 only medium also i586 packages will be downloaded although not needed
      • It downloads packages from all repositories, if the kiwi file contains more repositories the packages will be downloaded from all of them, but during build kiwi will properly choose only one package (depending on the repository priority attribute)
      • It downloads all binary packages which are build from a source package. For example if your kiwi file only requires e.g. augeas package in the target product ISO then OBS will still download augeas-devel, augeas-devel-32bit, augeas-lenses and augeas-lenses-tests packages.

    Similar Projects

    Fix RSpec tests in order to replace the ruby-ldap rubygem in OBS by enavarro_suse

    Description

    "LDAP mode is not official supported by OBS!". See: config/options.yml.example#L100-L102

    However, there is an RSpec file which tests LDAP mode in OBS. These tests use the ruby-ldap rubygem, mocking the results returned by a LDAP server.

    The ruby-ldap rubygem seems no longer maintaned, and also prevents from updating to a more recent Ruby version. A good alternative is to replace it with the net-ldap rubygem.

    Before replacing the ruby-ldap rubygem, we should modify the tests so the don't mock the responses of a LDAP server. Instead, we should modify the tests and run them against a real LDAP server.

    Goals

    Goals of this project:

    • Modify the RSpec tests and run them against a real LDAP server
    • Replace the net-ldap rubygem with the ruby-ldap rubygem

    Achieving the above mentioned goals will:

    • Permit upgrading OBS from Ruby 3.1 to Ruby 3.2
    • Make a step towards officially supporting LDAP in OBS.

    Resources

    Learn obs/ibs sync tool by xlai

    Description

    Once images/repo are built from IBS/OBS, there is a tool to sync the image from IBS/OBS to openqa asset directory and trigger openqa jobs accordingly.

    Goals

    Check how the tool is implemented, and be capable to add/modify our needed images/repo in future by ourselves.

    Resources

    • https://github.com/os-autoinst/openqa-trigger-from-obs
    • https://gitlab.suse.de/openqa/openqa-trigger-from-ibs-plugin/-/tree/master?ref_type=heads

    Git CI to automate the creation of product definition by gyribeiro

    Description

    Automate the creation of product definition

    Goals

    Create a Git CI that will:

    • automatically be triggered once a change (commit) in package list is done.
    • run tool responsible to update product definition based on the changes in package list
    • test the updated product definition in OBS
    • submit a pull request updating the product definition in the repository

    NOTE: this Git CI may also be triggered manually

    Resources

    • https://docs.gitlab.com/ee/ci/
    • https://openbuildservice.org/2021/05/31/scm-integration/
    • https://github.com/openSUSE/openSUSE-release-tools

    Research openqa-trigger-from-obs and openqa-trigger-from-ibs-plugin by qwang

    Description

    openqa-trigger-from-obs project is a framework that OSD is using it to automatically sync the defined images and repositories from OBS/IBS to its assets for testing. This framework very likely will be used for the synchronize to each location's openqa include openqa.qa2.suse.asia Beijing local procy scc scc-proxy.suse.asia(although it's not a MUST to our testing) it's now rewriting requests to openqa.qa2.suse.asia instead of openqa.suse.de, the assets/repo should be consistent the format Beijing local openQA is maintaining an own script but still need many manually activities when new build comes, and not consistent to OSD, that will request many test code change due to CC network change

    Goals

    Research this framework in case it will be re-used for Beijing local openQA, and will need to be setup and maintained by ourselves

    Resources

    https://github.com/os-autoinst/openqa-trigger-from-obs/tree/master https://gitlab.suse.de/openqa/openqa-trigger-from-ibs-plugin

    beijing :rainbow machine

    Automation of ABI compatibility checks by ateixeira

    Description

    ABI compatibility checks could be further automated by using the OBS API to download built RPMs and using existing tools to analyze ABI compatibility between the libraries contained in those packages. This project aims to explore these possibilities and figure out a way to make ABI checks as painless and fast as possible for package maintainers.

    Resources

    https://github.com/openSUSE/abi-compliance-checker

    https://github.com/lvc/abi-compliance-checker

    https://sourceware.org/libabigail/