Faster kernel builds in the OBS
an invention by michal-m
Updated about 2 years ago. 13 hacker ♥️. 1 follower.

The kernel compile and link itself is only a fraction of the OBS kernel-default package build. Several other passes take significant time: - Post build checks, especially the clamav scan - Debuginfo extractioon, which is single-threaded - Writing the RPMs

The post build checks and clamav are disabled in the KOTD projects, but they are still annoying when doing official builds. Clamav is suspiciously slow and should be investigated. The RPM write speed has been improved by switching to bzip2 compression, at the expense of their increased size. There is still room for improvement by writing the RPMs in parallel. Debuginfo extraction is a major pain.

Looking for hackers with the skills:

Nothing? Add some keywords!

This project is part of:

Hack Week 14

Activity

  • almost 5 years ago: cdywan liked this project.
  • almost 8 years ago: alnovak liked this project.
  • almost 8 years ago: mkubecek liked this project.
  • almost 8 years ago: wanghaisu liked this project.
  • almost 8 years ago: ZRen liked this project.
  • almost 8 years ago: a_faerber liked this project.
  • almost 8 years ago: ataschner liked this project.
  • almost 8 years ago: mkoutny liked this project.
  • almost 8 years ago: jdelvare liked this project.
  • almost 8 years ago: dsterba liked this project.
  • almost 8 years ago: jcejka liked this project.
  • almost 8 years ago: mhaefner liked this project.
  • almost 8 years ago: sparschauer liked this project.
  • almost 8 years ago: alnovak joined this project.
  • almost 8 years ago: michal-m started this project.
  • almost 8 years ago: michal-m originated this project.


    • Comments

    • michal-m
      almost 8 years ago by michal-m | Reply

      I started looking into the clamav scan, because it seemed like low-hanging fruit. I have a patch to the clamscan utility to scan the files in parallel, I just need to polish it and submit upstream. I also fixed some issues in the post-build-checks-malwarescan package. I'll test this and submit. Let's see if I can do something about the debuginfo extraction. I tried a few years ago had to give up. But I was young back then :-P.

    • michal-m
      almost 8 years ago by michal-m | Reply

      clamav-devel pull request to add parallel mode to clamscan.

    • michal-m
      almost 8 years ago by michal-m | Reply

      I also submitted the post-build-checks-malwarescan improvements to Base:System (SR#405470).

    • michal-m
      almost 8 years ago by michal-m | Reply

      In home:michal-m:branches:SUSE:SLE-12:Update in the IBS, during kernel-default build, the clamav scan takes 99 + 154 seconds. That's a considerable improvement, since it normally takes something like 6 + 7 minutes. There is still the issue that we run all the post-build checks twice if module signing is enabled. These numbers are with the patched clamav, which is not yet submitted.

    • michal-m
      almost 8 years ago by michal-m | Reply

      I managed to parallelize find-debuginfo.sh as well, hooray. I have yet to submit the patch upstream, but I think I finished the goals I set for this hackweek. So I'll mark it as done.

    • michal-m
      almost 8 years ago by michal-m | Reply

      Upstream submission of the find-debuginfo.sh patches. I also uploaded the patched rpm to home:michal-m:branches:SUSE:SLE-12:Update in the IBS, let's see how fast is the kernel going to build.

    • michal-m
      almost 8 years ago by michal-m | Reply

      Bummer. I found a race in the find-debuginfo.sh patch. Testing a new version.

    Similar Projects

    This project is one of its kind!