Modular kernel packaging
a project by mwilck
Updated almost 2 years ago. 7 hacker ♥️. 4 followers.

Project Description

Create a PoC for a modular packaged kernel. This means:

  • the kernel package itself is minimal
  • all device drivers are packaged as KMPs (kernel module packages)
  • some other modules (e.g. subsystems like netfilter, SCSI, …) are packaged as KMPs, too
  • module dependencies (depmod) translate into package dependencies and are handled by rpm
  • hardware requirements are handled by zypper/YaST via modalias
  • other system requirements (e.g. "this system needs a firewall") would be handled by patterns

Advantages

  • minimal installation size and download size for kernel and drivers — only actually required/used modules are installed
  • reduced build time: drivers don't need to be rebuilt with every kernel update, only if the driver itself (or the APIs it uses) are updated. KABI tracking ensures that drivers remain compatible with updated kernels.
  • vice-versa, a driver update doesn't necessitate a full kernel update.
  • this might also simplify QA procedures: driver testing could be separated from core kernel testing.
  • replacement of in-box drivers by out-of-box drivers simplified; it would become a routine procedure
  • conservative customers can update components selectively, reducing regression risks
  • package management techniques (e.g. download statistics) can be used to track usage of drivers and identify modules to be disabled
  • we could get rid of package bundles like kernel-default-base or kernel-default-extra.

Disadvantages / Challenges

  • "Missing driver problem": e.g. network controller breaks, new controller needs new driver → unavoidable, patterns e.g. "kernel-network-driver-all" or "kernel-usb-driver-all" as precaution (but this would forfeit a lot of the size advantage)
  • weak dependencies, or dependencies that aren't expressed in depmod terms. We handle this partially already for initrd (SUSE INITRD mechanism). We must translate it into package dependencies
  • user space dependencies: many tools might not be prepared for a large number of missing drivers in an (open)SUSE environment. Need to look into dracut etc.
  • is our KABI tracking good enough? So far it works well for the small number of KMPs, but what if we get 1000s of KMPs with complex dependencies?
  • Does the KABI dependency tracking scale to this level, or will dependency checks be extremely slow?
  • It gets more difficult to assess system supportablity. Tooling is required to verify that installed driver packages are compatible.
  • we can't run depmod after every KMP installation, it'd be far too slow (I have a patched depmod available that would make this possible with "incremental" mode).
  • avoid excessive meta data duplication (kernel changelog in every KMP)
  • the KMP concept needs to be revised. The multiversion(kernel) property of KMPs allows to install the same KMP for different KABI versions, but doesn't work well for version updates of the KMP itself. That would be a big problem, because we update some drivers frequently.
  • do we need a means for kernel module management tools (modprobe) to hook into package management (zypper) for pulling in missing modules automatically?
  • …

Goal for this Hackweek

  • Make it possible to build "real" KMPs from the kernel package build. Note that the in-kernel -kmp packages such as dlm-kmp aren't "real" KMPs; they use a hard kernel version dependency rather than KABI dependencies.
  • Hack on the kernel build scripts to enable modular packaging. Perhaps one-KMP-per-module as a first step, to experiment with module interdependencies. 2nd step would be combining modules into reasonable sets. Alternatively, start with a set of modules to be packaged as KMPs (e.g. device drivers) and increase the set step-by-step.
  • Experiment with practical use of the modular kernel. Figure out problems with the interaction of module and package management. Play through the "missing driver" scenario, and try to find a concept for aworkaround.

Later goals

  • Develop a modified KMP concept. Make version of the KABI the KMP was compiled against part of the package name rather than the version (instead of foo-kmp-default-1.0_k5.3.18_59.5, use foo-kmp-default-5.3.18_59.5-1.0, indicating module foo for the KABI of 5.3.18_59.5, version 1.0)
  • Tackle a few of the challenges above
  • Explore possibilities to improve packaging of kernel firmware, too. Currently FW is packaged in big chunks by vendor. Packaging individual drivers might facilitate fine-grained dependencies (driver pulls in required FW packages).

Join this project

Looking for hackers with the skills:

kmp kernel module rpm zypper packagemanagement kabi

This project is part of:

Hack Week 21 Hack Week 22

Activity

  • almost 2 years ago: mkoutny liked this project.
  • almost 2 years ago: mwilck started this project.
  • almost 2 years ago: ptesarik liked this project.
  • almost 2 years ago: mbrugger liked this project.
  • almost 2 years ago: sbahling liked this project.
  • almost 2 years ago: llansky3 liked this project.
  • almost 2 years ago: coolgw liked this project.
  • almost 2 years ago: avicenzi liked this project.
  • almost 2 years ago: mwilck added keyword "kabi" to this project.
  • almost 2 years ago: mwilck added keyword "packagemanagement" to this project.
  • almost 2 years ago: mwilck added keyword "zypper" to this project.
  • almost 2 years ago: mwilck added keyword "rpm" to this project.
  • almost 2 years ago: mwilck added keyword "module" to this project.
  • almost 2 years ago: mwilck added keyword "kernel" to this project.
  • almost 2 years ago: mwilck added keyword "kmp" to this project.
  • almost 2 years ago: mwilck originated this project.


    • Comments

    • mwilck
      almost 2 years ago by mwilck | Reply

      Jun 27

      I spent the day studying the kernel build scripts. Using them and understanding them are two different things. I'll try to use a similar approach as kernel-default-base: Just repackage existing modules. This will speed up development, as I'll avoid long build times for the kernel package. OBS repos created under home:mwilck:ModularKernel (not much to see yet).

    • mwilck
      almost 2 years ago by mwilck | Reply

      Jun 28

      I have a package with just the kernel image now, and at least one auto-built KMP. I am using a variant of the kernel-default-base package to simply repackage stuff that's currently built in the default package.

    • mwilck
      almost 2 years ago by mwilck | Reply

      Jun 29 — Jul 1

      Packaging of both the kernel image and modules works. I need to transform the scripts from kernel-default-base into rpm macros, but that's not a big deal, actually.

      Unfortunately I couldn't make more progress this week due to sickness.

    • mwilck
      almost 2 years ago by mwilck | Reply

      Later

      The kernel-default-image package now packages the whole set of modules from kernel-default-base, each module packaged separately. No actual compilation is done, the compiled modules are just taken from kernel-default, the same way as it's done in kernel-default-base. If we ever want to change this, the real challenge will be how to determine the list of subpackages, as it's currently calculated dynamically from modules.dep. We would basically need to generate the list of subpackages from the list of .ko files built during kernel compilation. I am not sure if that's possible. If not, we will have to do at least one "normal" build for every config to derive the module list first before modularly packaged kernels can be built. But we have a long way to go until then, anyway.

      Dependency resolution works the same way as for kernel-default-base; thus it's sufficient to list a set of desired modules or module-name-matching regular expressions, and the scripts will figure out the full set to satisfy all requirements.

      The module packages have ksym() requires and (if they export anything) also ksym() provides, as well as modalias() supplements. 

      > rpm -qp --supplements ehci-pci-kmp-default-5.14.21-lp154.23.1.g49db222.0.x86_64.rpm 
modalias(kernel-default:pci:v*d*sv*sd*bc0Csc03i20*)
modalias(kernel-default:pci:v0000104Ad0000CC00sv*sd*bc*sc*i*)
> rpm -qp --requires ehci-pci-kmp-default-5.14.21-lp154.23.1.g49db222.0.x86_64.rpm 
coreutils
grep
kernel-default-image
ksym(default:__dynamic_dev_dbg) = 22868a41
ksym(default:__fentry__) = bdfb6dbb
ksym(default:__pci_register_driver) = c2126345
ksym(default:__stack_chk_fail) = d0da656b
ksym(default:_dev_info) = fe960afa
ksym(default:_dev_warn) = 5cba8a10
ksym(default:device_set_wakeup_capable) = 226a1ef2
ksym(default:dma_set_coherent_mask) = 93f12bc3
ksym(default:ehci_init_driver) = d72c5993
ksym(default:ehci_resume) = 645790fb
...
> rpm -qp --provides ehci-hcd-kmp-default-5.14.21-lp154.23.1.g49db222.0.x86_64.rpm 
ehci-hcd-kmp = 5.14.21
ehci-hcd-kmp-default = 5.14.21-lp154.23.1.g49db222.0
ehci-hcd-kmp-default(x86-64) = 5.14.21-lp154.23.1.g49db222.0
kmod(ehci_hcd.ko)
ksym(default:ehci_adjust_port_wakeup_flags) = e6bf909e
ksym(default:ehci_handshake) = 1456664e
ksym(default:ehci_hub_control) = 913627e8
ksym(default:ehci_init_driver) = d72c5993
ksym(default:ehci_reset) = 145925dc
ksym(default:ehci_resume) = 645790fb
ksym(default:ehci_setup) = 4048a7e9
ksym(default:ehci_suspend) = 2c94b03f

      The build log is interesting, too.

      I had to modify the scripts looking for provides and supplements for this to work. Thus I'm shipping these as part of the package.

      Not surprisingly, the package repo contains a long list of packages. Including modules from -extra, I built a set of almost 3000 packages. Not quite texlive, but getting close add-emoji

      I had to stop at this point; so I haven't done any installation tests with the KMPs generated this way yet. The dependencies look quite sane though. What's obviously missing at this point is some patterns to allow installing e.g. all netfilter modules. For drivers, modalias should work (zypper inr --no-recommends), but I havent't tested it. The repo also urgently needs my performance fixes for depmod from last year, which are unfortunately still unfinished.

    • michals
      almost 2 years ago by michals | Reply

      The problem with this approach is that when you include the dependencies the subpackages may overlap, and while rpm itself would be fine with that because the modules coming from one kernel build are identical the packaging policy forbids it.

      • tiwai
        almost 2 years ago by tiwai | Reply

        What do you mean by "the dependencies the subpackages may overlap"? The same ksym() checksum provided by multiple kmps (of various versions)?

        I guess a partial solution would be to embed the hard dependency from the result of depmod. e.g. if a module has dependency on another module abc (as found in depmod), we can put "Requires: abc-kmp-default = %{version}" explicitly at the build time. Also, a softdep could be interpreted with Supplements or such, too.

    Similar Projects

    early stage kdump support by mbrugger

    [comment]: # (Please use the project descriptio...

    Authenticated hashes for BTRFS by dsterba

    Project Description

    Implement a checksum ...

    Model checking the BPF verifier by shunghsiyu

    Project Description

    BPF verifier plays a ...

    Tumbleweed support for Raspberry Pi 4 with Quad SATA HAT by jbaier_cz

    [comment]: # (Please use the project descriptio...

    LSP server for SPEC files by mcepl

    Has anybody ever heard about RPM Spec LSP serve...

    Switch software-o-o to parse repomd data by hennevogel

    Currently software.opensuse.org search is using...

    A CLI for Harvester by mohamed.belgaied

    [comment]: # Harvester does not officially come...

    A set of utilities to produce a "from scratch" OCI/Docker container using Opensuse/SLE rpms by ldragon

    [comment]: # (Please use the project descriptio...

    Investigate zypper/openSUSE repository refresh optimisations by dirkmueller

    [comment]: # (Please use the project descriptio...

    Extend GObject based introspectable API to libzypp by zbenjamin

    [comment]: # (Please use the project descriptio...