Introduction
OBS can build not only build traditional RPM packages, but it can even build appliances or installation ISO images.
We use this feature in YaST:Head:installer project to build an installation image with the latest YaST (which is not included in Factory yet). This allows as to try new things without worrying to break the installer in Factory or in the upcoming openSUSE distribution.
The Problem
Because the public build service is sometimes quite slow I have deployed a private OBS instance (see the OBS download page) at my workstation.
I found out that product building is relatively slow, even when running in the private instance locally. A quick look revealed that for building ISO image the build service downloads too many packages including source packages, debugsource and debuginfo packages which are not need for our installation ISO image. The amount of the unnecessary packages is several gigabytes even for our small mini installation DVD (which contains packages only for Minimal X pattern, no KDE nor GNOME...).
Focus of the Project
This project is focused on decreasing package download for product ISO image build.
The project should make building ISO images in OBS faster (especially in a private instance linking to the public OBS server), saving disk space and download bandwidth.
Result
I have successfully added kiwi file parsing (to know whether the debuginfo and source packages are needed or not) and skipping debuginfo and source packages into both OBS scheduler (which checks and evaluates dependencies and triggers builds) and worker (which downloads the packages during package/ISO build).
Outcome of the Project
- Significantly decreased amount of downloaded packages, over 7GB (!!) even for our small miniDVD
- This results in faster builds, disk space and network bandwidth savings
- Created a GitHub pull request to include this improvement upstream
- I learned a lot about OBS internals
- And refreshed my perl knowledge ;-)
This project is part of:
Hack Week 10
Activity
Comments
-
about 11 years ago by lslezak | Reply
I have implemented this optimization in my private OBS instance at https://muffin.suse.cz, for building our miniDVD.
In this case it saves over 7GB package download for the ISO image build!!
-
about 11 years ago by lslezak | Reply
I have created a GitHub pull request to include this improvement in the next OBS release (and likely in the public instance as well), see https://github.com/openSUSE/open-build-service/pull/426.
-
-
about 11 years ago by lslezak | Reply
There are still possible some more improvements, OBS still downloads some more not needed packages:
- If you build x86_64 only medium also i586 packages will be downloaded although not needed
- It downloads packages from all repositories, if the kiwi file contains more repositories the packages will be downloaded from all of them, but during build kiwi will properly choose only one package (depending on the repository
priorityattribute) - It downloads all binary packages which are build from a source package. For example if your kiwi file only requires e.g. augeas package in the target product ISO then OBS will still download augeas-devel, augeas-devel-32bit, augeas-lenses and augeas-lenses-tests packages.
Similar Projects
New features in openqa-trigger-from-obs for openQA by jlausuch
Description
Implement new features in openqa-trigger-from-obs to make xml more flexible.
Goals
One of the features to be implemented: - Possibility to define "VERSION" and "ARCH" variables per flavor instead of global.
Resources
https://github.com/os-autoinst/openqa-trigger-from-obs
Research openqa-trigger-from-obs and openqa-trigger-from-ibs-plugin by qwang
Description
openqa-trigger-from-obs project is a framework that OSD is using it to automatically sync the defined images and repositories from OBS/IBS to its assets for testing. This framework very likely will be used for the synchronize to each location's openqa include openqa.qa2.suse.asia Beijing local procy scc scc-proxy.suse.asia(although it's not a MUST to our testing) it's now rewriting requests to openqa.qa2.suse.asia instead of openqa.suse.de, the assets/repo should be consistent the format Beijing local openQA is maintaining an own script but still need many manually activities when new build comes, and not consistent to OSD, that will request many test code change due to CC network change
Goals
Research this framework in case it will be re-used for Beijing local openQA, and will need to be setup and maintained by ourselves
Resources
https://github.com/os-autoinst/openqa-trigger-from-obs/tree/master https://gitlab.suse.de/openqa/openqa-trigger-from-ibs-plugin
beijing :rainbow machine
Git CI to automate the creation of product definition by gyribeiro
Description
Automate the creation of product definition
Goals
Create a Git CI that will:
- automatically be triggered once a change (commit) in package list is done.
- run tool responsible to update product definition based on the changes in package list
- test the updated product definition in OBS
- submit a pull request updating the product definition in the repository
NOTE: this Git CI may also be triggered manually
Resources
- https://docs.gitlab.com/ee/ci/
- https://openbuildservice.org/2021/05/31/scm-integration/
- https://github.com/openSUSE/openSUSE-release-tools
Bootstrap openSUSE on LoongArch by glaubitz
Description
LoongArch is a new architecture from China which has its roots in the MIPS architecture. It has been created by Loongson and is already supported by Debian Ports, Gentoo and Loongnix.
Upstream support for LoongArch is already quite complete which includes LLVM, Rust, Golang, GRUB, QEMU, LibreOffice and many more. In Debian Ports, where the port is called "loong64", more than 95% of the whole Debian archive have been successfully built for LoongArch.
QEMU support is rather complete and stable such that packages can be built in emulated environments. Hardware can also be requested by Loongson on request for free. Access to real hardware is also provided through the GCC Compile Farm.
Goals
The initial goal should be to add LoongArch to OBS and build a minimal set of packages.
Resources
- Introduction to LoongArch: https://docs.kernel.org/arch/loongarch/introduction.html
- LoongArch community on Github: https://github.com/loongarchlinux
- Debian Ports repository for loong64: http://ftp.ports.debian.org/debian-ports/pool-loong64/main/
- Gentoo stage3 for loong: https://www.gentoo.org/downloads/#loong
Results
- An initial set of packages for openSUSE loongarch64 has been successfully bootstrapped
- An OBS project has been set up to build packages for openSUSE loongarch64 with more than 3000 packages being built already
- A work-in-progress guide on how to bootstrap a new openSUSE port from Debian has been created
- A work-in-progress guide on how to add a new target to the openSUSE toolchain has been created
Acknowledgements
- Thanks to Adrian Schröter and Rüdiger Oertl for the help with setting up the FTP space and OBS project
- Thanks to Dirk Müller for the input on how to get started with a new port
- Thanks to Richard Biener for quickly accepting my submit requests to add loongarch64 support to the toolchain
Explore the integration between OBS and GitHub by pdostal
Project Description
The goals:
1) When GitHub pull request is created or modified the OBS project will be forked and the build results reported back to GitHub. 2) When new version of the GitHub project will be published the OBS will redownload the source and rebuild the project.
Goal for this Hackweek
Do as much as possible, blog about it and maybe use it another existing project.
Resources
- The Blog post
- Issue: poo#123858 - build.opensuse.org: /usr/lib/obs/service//go_modules.service No such file or directory