Investigate possibilities for the distributed builds for Ceph to speed up builds.

This task could have 2 scopes.

  • replace build and vstart for developers to run in containers for any target base (openSUSE, Ubuntu and etc)

  • distribute build jobs across nodes, probably on k8s cluster

There are some projects already to distribute the build:

distcc

icecream

As the first step maybe get to know already existent system and how much they speed up the build to have a reference point.

Idea is to build container (and reuse it later) with all dependencies (./install-deps.sh on some base for devs, or chroot before build phase for osc/obs) and start number of build jobs on k8s cluster (or podman if local dev env).

That could help: * developers to build, test their changes on any dist base, locally or on k8s * speed up builds for IBS/OBS to distribute build on k8s cluster

For dev environment buildah could be used to get base OS and actually run ./install-deps.sh and get current base for the build. For osc/isc, that tool already prepares chroot, so that might be consumed as container base.

Some tool needs to be used/developed to generate k8s manifests to run build based on that container, run those manifests locally with podman play kube or remotely scheduling those manifest on k8s cluster and some how gather the results from the jobs and store resulted binaries/rpms/containers somewhere to run/store them.

Looking for hackers with the skills:

icecream distcc ceph osc obs

This project is part of:

Hack Week 19

Activity

  • almost 5 years ago: rgrigorev started this project.
  • almost 5 years ago: tbechtold liked this project.
  • almost 5 years ago: denisok added keyword "icecream" to this project.
  • almost 5 years ago: denisok added keyword "distcc" to this project.
  • almost 5 years ago: denisok added keyword "ceph" to this project.
  • almost 5 years ago: denisok added keyword "osc" to this project.
  • almost 5 years ago: denisok added keyword "obs" to this project.
  • almost 5 years ago: denisok originated this project.

  • Comments

    • rgrigorev
      almost 5 years ago by rgrigorev | Reply

      I did some measurements on our node: Main build node + icecream manage ses-client-6 main storage device /dev/nvme0n1p3 Drive speed: 'Timing buffered disk reads: 5388 MB in 3.00 seconds = 1795.90 MB/sec' CPU Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz/ 16Threads/ https://ark.intel.com/content/www/us/en/ark/products/123547/intel-xeon-silver-4110-processor-11m-cache-2-10-ghz.html 64G RAM minion nodes ses-client-7 and ses-client-8

      description real user sys 1 node 210m38.879s 202m4.993s 12m45.631s 1 node -j 6 47m44.742s 254m36.362s 15m27.561s 1 node -j 11 34m38.795s 318m48.671s 17m22.380s 2 nodes -j 12 26m44.379s 156m59.053s 11m44.222s 3 nodes -j 18 25m4.973s 63m48.105s 7m53.588s 3 nodes -j 33 17m27.299s 55m18.695s 7m38.526s 3 nodes -j 33, 25G network 17m0.235s 46m24.425s 7m13.615s .... + ram disk 17m2.661s 47m0.784s 7m11.908s .... + fix in boost 13m22.895s 45m16.003s 4m55.285s

    • rgrigorev
      almost 5 years ago by rgrigorev | Reply

      I did some measurements on our node:

      main build node + icecream manage ses-client-6
      main storage device /dev/nvme0n1p3
      Drive speed: 'Timing buffered disk reads: 5388 MB in 3.00 seconds = 1795.90 MB/sec'
      CPU Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz/ 16Threads/ full cpu description
      64G RAM
      minion nodes ses-client-7 and ses-client-8

      > description ________ real __________ user __________ sys
      > 1 node ______________ 210m38.879s __ 202m4.993s ___ 12m45.631s
      > 1 node -j 6 ___________ 47m44.742s ___ 254m36.362s __ 15m27.561s
      >1 node -j 11 __________ 34m38.795s ___ 318m48.671s __ 17m22.380s
      >2 nodes -j 12 _________ 26m44.379s ___ 156m59.053s __ 11m44.222s
      >3 nodes -j 18 _________ 25m4.973s ____ 63m48.105s ___ 7m53.588s
      >3 nodes -j 33 _________ 17m27.299s ___ 55m18.695s ___ 7m38.526s
      >3 nodes -j 33, 25G net _ 17m0.235s _____ 46m24.425s ___ 7m13.615s
      >.... + ram disk ________ 17m2.661s _____ 47m0.784s ____ 7m11.908s
      .... + fix in boost ______ 13m22.895s ____ 45m16.003s ___ 4m55.285s

    • rpenyaev
      almost 5 years ago by rpenyaev | Reply

      Roman, can the following kernel patch also speedup our ceph builds? Seems worth to try

      https://www.phoronix.com/scan.php?page=news_item&px=Linux-Pipe-Parallel-Job-Opt https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0ddad21d3e99c743a3aa473121dc5561679e26bb

    • rgrigorev
      almost 5 years ago by rgrigorev | Reply

      3 nodes -j 33 (~11 per node), hsm network, ram disk, manual fixes in boost files + kernel 5.6.7 pre

      real 13m22.895s user 45m16.003s sys 4m55.285s yes, there is also some improvement

      • rgrigorev
        almost 5 years ago by rgrigorev | Reply

        -j 11, kernel 5.6.0-rc1-197.29 (include new patch from Linux) real 31m13.175s user 321m5.955s sys 16m13.355s

        baseline -j 11 real 34m38.795s user 318m48.671s sys 17m22.380s

    Similar Projects

    Learn obs/ibs sync tool by xlai

    Description

    Once images/repo are built from IBS/OBS, there is a tool to sync the image from IBS/OBS to openqa asset directory and trigger openqa jobs accordingly.

    Goals

    Check how the tool is implemented, and be capable to add/modify our needed images/repo in future by ourselves.

    Resources

    • https://github.com/os-autoinst/openqa-trigger-from-obs
    • https://gitlab.suse.de/openqa/openqa-trigger-from-ibs-plugin/-/tree/master?ref_type=heads


    Git CI to automate the creation of product definition by gyribeiro

    Description

    Automate the creation of product definition

    Goals

    Create a Git CI that will:

    • automatically be triggered once a change (commit) in package list is done.
    • run tool responsible to update product definition based on the changes in package list
    • test the updated product definition in OBS
    • submit a pull request updating the product definition in the repository

    NOTE: this Git CI may also be triggered manually

    Resources

    • https://docs.gitlab.com/ee/ci/
    • https://openbuildservice.org/2021/05/31/scm-integration/
    • https://github.com/openSUSE/openSUSE-release-tools


    Fix RSpec tests in order to replace the ruby-ldap rubygem in OBS by enavarro_suse

    Description

    "LDAP mode is not official supported by OBS!". See: config/options.yml.example#L100-L102

    However, there is an RSpec file which tests LDAP mode in OBS. These tests use the ruby-ldap rubygem, mocking the results returned by a LDAP server.

    The ruby-ldap rubygem seems no longer maintaned, and also prevents from updating to a more recent Ruby version. A good alternative is to replace it with the net-ldap rubygem.

    Before replacing the ruby-ldap rubygem, we should modify the tests so the don't mock the responses of a LDAP server. Instead, we should modify the tests and run them against a real LDAP server.

    Goals

    Goals of this project:

    • Modify the RSpec tests and run them against a real LDAP server
    • Replace the net-ldap rubygem with the ruby-ldap rubygem

    Achieving the above mentioned goals will:

    • Permit upgrading OBS from Ruby 3.1 to Ruby 3.2
    • Make a step towards officially supporting LDAP in OBS.

    Resources


    Switch software-o-o to parse repomd data by hennevogel

    Currently software.opensuse.org search is using the OBS binary search for everything, even for packages inside the openSUSE distributions. Let's switch this to use repomd data from download.opensuse.org


    Bootstrap openSUSE on LoongArch by glaubitz

    Description

    LoongArch is a new architecture from China which has its roots in the MIPS architecture. It has been created by Loongson and is already supported by Debian Ports, Gentoo and Loongnix.

    Upstream support for LoongArch is already quite complete which includes LLVM, Rust, Golang, GRUB, QEMU, LibreOffice and many more. In Debian Ports, where the port is called "loong64", more than 95% of the whole Debian archive have been successfully built for LoongArch.

    QEMU support is rather complete and stable such that packages can be built in emulated environments. Hardware can also be requested by Loongson on request for free. Access to real hardware is also provided through the GCC Compile Farm.

    Goals

    The initial goal should be to add LoongArch to OBS and build a minimal set of packages.

    Resources

    Results

    Acknowledgements

    • Thanks to Adrian Schröter and Rüdiger Oertl for the help with setting up the FTP space and OBS project
    • Thanks to Dirk Müller for the input on how to get started with a new port
    • Thanks to Richard Biener for quickly accepting my submit requests to add loongarch64 support to the toolchain