Package workflow experiments
a project by jfehlig
Updated over 2 years ago. 2 hacker ♥️. 2 followers.

Project Description

Most developers are comfortable with the workflows of git hosting services like gitlab and github, including their CI/CD capabilities. This project aims to experiment with new downstream package development and maintenance workflows based on upstream git repositories cloned at gitlab.suse.de. I'll be using the libvirt package for these experiments since it typically contains a healthy mixture of downstream-only patches along with upstream cherry picks.

I'd like to hide as much obs/ibs as I can behind some helper scripts for use by the package maintainer. Package developers and contributors would use the standard git workflows and could conveniently ignore the build service. The kernel package is developed in a similar fashion. I'd like to create a simple template that is usable by less complex packages.

Goals for this Hackweek

Create some helpers that a package maintainer can use to generate the artifacts needed for an obs/ibs package submission from an upstream git repository.

Create gitlab CI configuration and scripts that can be used to execute tests on remote runners in the Provo devlab.

Resources

https://gitlab.suse.de/virtualization/libvirt

https://gitlab.suse.de/virtualization/libvirt-rpm

Join this project

Looking for hackers with the skills:

obs gitlab-ci git gitlab buildservice

This project is part of:

Hack Week 21

Activity

  • over 2 years ago: archanaserver liked this project.
  • over 2 years ago: archanaserver joined this project.
  • over 2 years ago: jzerebecki liked this project.
  • over 2 years ago: jfehlig added keyword "buildservice" to this project.
  • over 2 years ago: jfehlig added keyword "git" to this project.
  • over 2 years ago: jfehlig added keyword "gitlab" to this project.
  • over 2 years ago: jfehlig removed keyword ci/cd from this project.
  • over 2 years ago: jfehlig added keyword "gitlab-ci" to this project.
  • over 2 years ago: jfehlig added keyword "ci/cd" to this project.
  • over 2 years ago: jfehlig added keyword "obs" to this project.
  • over 2 years ago: jfehlig started this project.
  • over 2 years ago: jfehlig originated this project.


    • Comments

    • jfehlig
      over 2 years ago by jfehlig | Reply

      Illness and life-belated project update:

      Upstream libvirt is mirrored from gitlab.com to gitlab.suse.de via a systemd timer service running in a VM in the Provo devlab. Other virtualization-related projects also mirror their upstream repositories in a similar fashion. For this reason the VM is called "code-mirror". Developers and contributors are free to work with these mirrored repositories using common git workflows.

      Products, including Factory/Tumbleweed, are maintained in branches based on upstream release tags. E.g. for SLE15 SP4, which contains libvirt 8.0.0, the gitlab.suse.de libvirt mirror contains a 'v8.0.0-sle15sp4' branch where maintenance can occur. The branch is based on the upstream v8.0.0 tag and contains SUSE-specific downstream patches and upstream cherry picks. Fixing a SLE15 SP4 bug generally means checking out the 'v8.0.0-sle15sp4' branch, cherry picking the necessary commit(s) from 'upstream-master', and pushing back to the 'v8.0.0-sle15sp4' on gitlab.suse.de.

      Downstream-only content such as build service and package management artifacts are maintained in a separate git repository on gitlab.suse.de named libvirt-rpm. This includes a downstream-specific CI/CD config file (aka .gitlab-ci.yml), a spec file template, the changelog, lintrc files, miscellaneous downstream source files, and any scripts needed to implement the build, deploy, test, and submit stages of a CI pipeline. The libvirt-rpm repository contains maintenance branches that share the same name as the maintenance branches in the main libvirt repository, allowing maintenance of the downstream-only content on a per-product basis as well.

      When creating a new maintenance branch in the main libvirt repository, the first commit in the branch is to add the libvirt-rpm repository as a submodule under the rpm directory. The second commit sets the submodule branch to the same name as the new maintenance branch. Subsequent commits include downstream patches and upstream cherry picks needed for the maintained product.

      When pushing a change to a maintenance branch, a new CI pipeline is triggered based on the instructions in the custom .gitlab-ci.yml. The first stage of the pipeline builds a package. The second stage deploys the package to a candidate host, the third stage tests the package, and finally the last stage submits the package to the associated obs/ibs devel project. Submitting the the package from the devel project to the target product is deferred to the package maintainer.

      The first stage of the pipeline is the most interesting, particularly extracting the artifacts needed for a package build from a branch in the mirrored upstream repository. On a push or merge request event, gitlab will clone the associated branch (and optionally recurse submodules) at $CIPROJECTDIR on a gitlab runner. For the libvirt project, this means the libvirt-rpm submodule is also expanded on the runner, making the spec file template, changelog, CI/CD scripts, and other downstream-only content available. One of the CI/CD scripts prepares the build service artifacts in a workspace by first creating an archive based on the libvirt release tag. This is essentially a pristine tarball of an upstream release. All downstream and cherry picked patches are then extracted from the branch. A spec file template is then populated with the list of extracted patches. Other build service artifacts are copied from the libvirt-rpm submodule to the workspace, where finally 'osc build' is invoked.

      Nothing precludes using the gitlab.suse.de libvirt repository to develop upstream features and bug fixes. Development branches can be pushed to the repository for safe keeping until they are suitable for upstream submission. Once committed upstream and mirrored back to the gitlab.suse.de repository, the commit(s) can be cherry picked to the product maintenance branches.

      https://gitlab.suse.de/virtualization/libvirt https://gitlab.suse.de/virtualization/libvirt-rpm

    Similar Projects

    Explore the integration between OBS and GitHub by pdostal

    Project Description

    The goals:

    1) When GitHub pull request is created or modified the OBS project will be forked and the build results reported back to GitHub. 2) When new version of the GitHub project will be published the OBS will redownload the source and rebuild the project.

    Goal for this Hackweek

    Do as much as possible, blog about it and maybe use it another existing project.

    Resources

    Implement a full OBS api client in Rust by nbelouin

    Description

    I recently started to work on tooling for OBS using rust, to do so I started a Rust create to interact with OBS API, I only implemented a few routes/resources for what I needed. What about making it a full fledged OBS client library.

    Goals

    • Implement more routes/resources
    • Implement a test suite against the actual OBS implementation
    • Bonus: Create an osc like cli in Rust using the library

    Resources

    • https://github.com/suse-edge/obs-tools/tree/main/obs-client
    • https://api.opensuse.org/apidocs/

    Switch software-o-o to parse repomd data by hennevogel

    Currently software.opensuse.org search is using the OBS binary search for everything, even for packages inside the openSUSE distributions. Let's switch this to use repomd data from download.opensuse.org

    Learn about OSB and contribute to `kustomize` and `k9s` packages to add ARM arch by dpock

    Description

    There are already k9s and kustomize packages that exist for openSUSE today. These could be used as the source for these binaries in our rancher projects. By using them we would benefit from CVE fixes included in our distribution of the packages not in cluded upstream. However they are not providing arm package builds which are required.

    Goals

    • [ ] Update the kustomize package in OBS to use the newest version and send change request

    Resources

    • k9s: https://build.opensuse.org/package/show/openSUSE:Factory/k9s
    • kustomize: https://build.opensuse.org/package/show/openSUSE:Factory/kustomize
    • Learning Docs: https://confluence.suse.com/display/packaging/Training%2C+Talks+and+Videos

    Testing and adding GNU/Linux distributions on Uyuni by juliogonzalezgil

    Join the Gitter channel! https://gitter.im/uyuni-project/hackweek

    Uyuni is a configuration and infrastructure management tool that saves you time and headaches when you have to manage and update tens, hundreds or even thousands of machines. It also manages configuration, can run audits, build image containers, monitor and much more!

    Currently there are a few distributions that are completely untested on Uyuni or SUSE Manager (AFAIK) or just not tested since a long time, and could be interesting knowing how hard would be working with them and, if possible, fix whatever is broken.

    For newcomers, the easiest distributions are those based on DEB or RPM packages. Distributions with other package formats are doable, but will require adapting the Python and Java code to be able to sync and analyze such packages (and if salt does not support those packages, it will need changes as well). So if you want a distribution with other packages, make sure you are comfortable handling such changes.

    No developer experience? No worries! We had non-developers contributors in the past, and we are ready to help as long as you are willing to learn. If you don't want to code at all, you can also help us preparing the documentation after someone else has the initial code ready, or you could also help with testing :-)

    The idea is testing Salt and Salt-ssh clients, but NOT traditional clients, which are deprecated.

    To consider that a distribution has basic support, we should cover at least (points 3-6 are to be tested for both salt minions and salt ssh minions):

    1. Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)
    2. Onboarding (salt minion from UI, salt minion from bootstrap scritp, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator)
    3. Package management (install, remove, update...)
    4. Patching
    5. Applying any basic salt state (including a formula)
    6. Salt remote commands
    7. Bonus point: Java part for product identification, and monitoring enablement
    8. Bonus point: sumaform enablement (https://github.com/uyuni-project/sumaform)
    9. Bonus point: Documentation (https://github.com/uyuni-project/uyuni-docs)
    10. Bonus point: testsuite enablement (https://github.com/uyuni-project/uyuni/tree/master/testsuite)

    If something is breaking: we can try to fix it, but the main idea is research how supported it is right now. Beyond that it's up to each project member how much to hack :-)

    • If you don't have knowledge about some of the steps: ask the team
    • If you still don't know what to do: switch to another distribution and keep testing.

    This card is for EVERYONE, not just developers. Seriously! We had people from other teams helping that were not developers, and added support for Debian and new SUSE Linux Enterprise and openSUSE Leap versions :-)

    Pending

    FUSS

    FUSS is a complete GNU/Linux solution (server, client and desktop/standalone) based on Debian for managing an educational network.

    https://fuss.bz.it/

    Seems to be a Debian 12 derivative, so adding it could be quite easy.

    • [W] Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)
    • [W] Onboarding (salt minion from UI, salt minion from bootstrap script, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator) --> Working for all 3 options (salt minion UI, salt minion bootstrap script and salt-ssh minion from the UI).
    • [W] Package management (install, remove, update...) --> Installing a new package works, needs to test the rest.
    • [I] Patching (if patch information is available, could require writing some code to parse it, but IIRC we have support for Ubuntu already). No patches detected. Do we support patches for Debian at all?
    • [W] Applying any basic salt state (including a formula)
    • [W] Salt remote commands
    • [ ] Bonus point: Java part for product identification, and monitoring enablement

    Port git-fixup to POSIX shell script and submit to git/git by mcepl

    Description

    https://github.com/keis/git-fixup is an exceedingly useful program, which I use daily, and I would love to every git user could bask in its awesomeness. Alas, it is a bash script, so it is not appropriate for the inclusion in git proper.

    Goals

    Port the script to plain POSIX shell and submit for consideration to git@vger.kernel.org

    Resources

    Explore the integration between OBS and GitHub by pdostal

    Project Description

    The goals:

    1) When GitHub pull request is created or modified the OBS project will be forked and the build results reported back to GitHub. 2) When new version of the GitHub project will be published the OBS will redownload the source and rebuild the project.

    Goal for this Hackweek

    Do as much as possible, blog about it and maybe use it another existing project.

    Resources