Project Description

Most developers are comfortable with the workflows of git hosting services like gitlab and github, including their CI/CD capabilities. This project aims to experiment with new downstream package development and maintenance workflows based on upstream git repositories cloned at gitlab.suse.de. I'll be using the libvirt package for these experiments since it typically contains a healthy mixture of downstream-only patches along with upstream cherry picks.

I'd like to hide as much obs/ibs as I can behind some helper scripts for use by the package maintainer. Package developers and contributors would use the standard git workflows and could conveniently ignore the build service. The kernel package is developed in a similar fashion. I'd like to create a simple template that is usable by less complex packages.

Goals for this Hackweek

Create some helpers that a package maintainer can use to generate the artifacts needed for an obs/ibs package submission from an upstream git repository.

Create gitlab CI configuration and scripts that can be used to execute tests on remote runners in the Provo devlab.

Resources

https://gitlab.suse.de/virtualization/libvirt

https://gitlab.suse.de/virtualization/libvirt-rpm

Looking for hackers with the skills:

obs gitlab-ci git gitlab buildservice

This project is part of:

Hack Week 21

Activity

  • over 2 years ago: archanaserver liked this project.
  • over 2 years ago: archanaserver joined this project.
  • over 2 years ago: jzerebecki liked this project.
  • over 2 years ago: jfehlig added keyword "buildservice" to this project.
  • over 2 years ago: jfehlig added keyword "git" to this project.
  • over 2 years ago: jfehlig added keyword "gitlab" to this project.
  • over 2 years ago: jfehlig removed keyword ci/cd from this project.
  • over 2 years ago: jfehlig added keyword "gitlab-ci" to this project.
  • over 2 years ago: jfehlig added keyword "ci/cd" to this project.
  • over 2 years ago: jfehlig added keyword "obs" to this project.
  • over 2 years ago: jfehlig started this project.
  • over 2 years ago: jfehlig originated this project.

  • Comments

    • jfehlig
      over 2 years ago by jfehlig | Reply

      Illness and life-belated project update:

      Upstream libvirt is mirrored from gitlab.com to gitlab.suse.de via a systemd timer service running in a VM in the Provo devlab. Other virtualization-related projects also mirror their upstream repositories in a similar fashion. For this reason the VM is called "code-mirror". Developers and contributors are free to work with these mirrored repositories using common git workflows.

      Products, including Factory/Tumbleweed, are maintained in branches based on upstream release tags. E.g. for SLE15 SP4, which contains libvirt 8.0.0, the gitlab.suse.de libvirt mirror contains a 'v8.0.0-sle15sp4' branch where maintenance can occur. The branch is based on the upstream v8.0.0 tag and contains SUSE-specific downstream patches and upstream cherry picks. Fixing a SLE15 SP4 bug generally means checking out the 'v8.0.0-sle15sp4' branch, cherry picking the necessary commit(s) from 'upstream-master', and pushing back to the 'v8.0.0-sle15sp4' on gitlab.suse.de.

      Downstream-only content such as build service and package management artifacts are maintained in a separate git repository on gitlab.suse.de named libvirt-rpm. This includes a downstream-specific CI/CD config file (aka .gitlab-ci.yml), a spec file template, the changelog, lintrc files, miscellaneous downstream source files, and any scripts needed to implement the build, deploy, test, and submit stages of a CI pipeline. The libvirt-rpm repository contains maintenance branches that share the same name as the maintenance branches in the main libvirt repository, allowing maintenance of the downstream-only content on a per-product basis as well.

      When creating a new maintenance branch in the main libvirt repository, the first commit in the branch is to add the libvirt-rpm repository as a submodule under the rpm directory. The second commit sets the submodule branch to the same name as the new maintenance branch. Subsequent commits include downstream patches and upstream cherry picks needed for the maintained product.

      When pushing a change to a maintenance branch, a new CI pipeline is triggered based on the instructions in the custom .gitlab-ci.yml. The first stage of the pipeline builds a package. The second stage deploys the package to a candidate host, the third stage tests the package, and finally the last stage submits the package to the associated obs/ibs devel project. Submitting the the package from the devel project to the target product is deferred to the package maintainer.

      The first stage of the pipeline is the most interesting, particularly extracting the artifacts needed for a package build from a branch in the mirrored upstream repository. On a push or merge request event, gitlab will clone the associated branch (and optionally recurse submodules) at $CIPROJECTDIR on a gitlab runner. For the libvirt project, this means the libvirt-rpm submodule is also expanded on the runner, making the spec file template, changelog, CI/CD scripts, and other downstream-only content available. One of the CI/CD scripts prepares the build service artifacts in a workspace by first creating an archive based on the libvirt release tag. This is essentially a pristine tarball of an upstream release. All downstream and cherry picked patches are then extracted from the branch. A spec file template is then populated with the list of extracted patches. Other build service artifacts are copied from the libvirt-rpm submodule to the workspace, where finally 'osc build' is invoked.

      Nothing precludes using the gitlab.suse.de libvirt repository to develop upstream features and bug fixes. Development branches can be pushed to the repository for safe keeping until they are suitable for upstream submission. Once committed upstream and mirrored back to the gitlab.suse.de repository, the commit(s) can be cherry picked to the product maintenance branches.

      https://gitlab.suse.de/virtualization/libvirt https://gitlab.suse.de/virtualization/libvirt-rpm

    Similar Projects

    Git CI to automate the creation of product definition by gyribeiro

    Description

    Automate the creation of product definition

    Goals

    Create a Git CI that will:

    • automatically be triggered once a change (commit) in package list is done.
    • run tool responsible to update product definition based on the changes in package list
    • test the updated product definition in OBS
    • submit a pull request updating the product definition in the repository

    NOTE: this Git CI may also be triggered manually

    Resources

    • https://docs.gitlab.com/ee/ci/
    • https://openbuildservice.org/2021/05/31/scm-integration/
    • https://github.com/openSUSE/openSUSE-release-tools


    Explore the integration between OBS and GitHub by pdostal

    Project Description

    The goals:

    1) When GitHub pull request is created or modified the OBS project will be forked and the build results reported back to GitHub. 2) When new version of the GitHub project will be published the OBS will redownload the source and rebuild the project.

    Goal for this Hackweek

    Do as much as possible, blog about it and maybe use it another existing project.

    Resources


    New features in openqa-trigger-from-obs for openQA by jlausuch

    Description

    Implement new features in openqa-trigger-from-obs to make xml more flexible.

    Goals

    One of the features to be implemented: - Possibility to define "VERSION" and "ARCH" variables per flavor instead of global.

    Resources

    https://github.com/os-autoinst/openqa-trigger-from-obs


    Bootstrap openSUSE on LoongArch by glaubitz

    Description

    LoongArch is a new architecture from China which has its roots in the MIPS architecture. It has been created by Loongson and is already supported by Debian Ports, Gentoo and Loongnix.

    Upstream support for LoongArch is already quite complete which includes LLVM, Rust, Golang, GRUB, QEMU, LibreOffice and many more. In Debian Ports, where the port is called "loong64", more than 95% of the whole Debian archive have been successfully built for LoongArch.

    QEMU support is rather complete and stable such that packages can be built in emulated environments. Hardware can also be requested by Loongson on request for free. Access to real hardware is also provided through the GCC Compile Farm.

    Goals

    The initial goal should be to add LoongArch to OBS and build a minimal set of packages.

    Resources

    Results

    Acknowledgements

    • Thanks to Adrian Schröter and Rüdiger Oertl for the help with setting up the FTP space and OBS project
    • Thanks to Dirk Müller for the input on how to get started with a new port
    • Thanks to Richard Biener for quickly accepting my submit requests to add loongarch64 support to the toolchain


    Fix RSpec tests in order to replace the ruby-ldap rubygem in OBS by enavarro_suse

    Description

    "LDAP mode is not official supported by OBS!". See: config/options.yml.example#L100-L102

    However, there is an RSpec file which tests LDAP mode in OBS. These tests use the ruby-ldap rubygem, mocking the results returned by a LDAP server.

    The ruby-ldap rubygem seems no longer maintaned, and also prevents from updating to a more recent Ruby version. A good alternative is to replace it with the net-ldap rubygem.

    Before replacing the ruby-ldap rubygem, we should modify the tests so the don't mock the responses of a LDAP server. Instead, we should modify the tests and run them against a real LDAP server.

    Goals

    Goals of this project:

    • Modify the RSpec tests and run them against a real LDAP server
    • Replace the net-ldap rubygem with the ruby-ldap rubygem

    Achieving the above mentioned goals will:

    • Permit upgrading OBS from Ruby 3.1 to Ruby 3.2
    • Make a step towards officially supporting LDAP in OBS.

    Resources


    Port git-fixup to POSIX shell script and submit to git/git by mcepl

    Description

    https://github.com/keis/git-fixup is an exceedingly useful program, which I use daily, and I would love to every git user could bask in its awesomeness. Alas, it is a bash script, so it is not appropriate for the inclusion in git proper.

    Goals

    Port the script to plain POSIX shell and submit for consideration to git@vger.kernel.org

    Resources


    Explore the integration between OBS and GitHub by pdostal

    Project Description

    The goals:

    1) When GitHub pull request is created or modified the OBS project will be forked and the build results reported back to GitHub. 2) When new version of the GitHub project will be published the OBS will redownload the source and rebuild the project.

    Goal for this Hackweek

    Do as much as possible, blog about it and maybe use it another existing project.

    Resources