SUSE Hack Week: Modular kernel packaging

Project Description

Create a PoC for a modular packaged kernel. This means:

the kernel package itself is minimal
all device drivers are packaged as KMPs (kernel module packages)
some other modules (e.g. subsystems like netfilter, SCSI, …) are packaged as KMPs, too
module dependencies (depmod) translate into package dependencies and are handled by rpm
hardware requirements are handled by zypper/YaST via modalias
other system requirements (e.g. "this system needs a firewall") would be handled by patterns

Advantages

minimal installation size and download size for kernel and drivers — only actually required/used modules are installed
reduced build time: drivers don't need to be rebuilt with every kernel update, only if the driver itself (or the APIs it uses) are updated. KABI tracking ensures that drivers remain compatible with updated kernels.
vice-versa, a driver update doesn't necessitate a full kernel update.
this might also simplify QA procedures: driver testing could be separated from core kernel testing.
replacement of in-box drivers by out-of-box drivers simplified; it would become a routine procedure
conservative customers can update components selectively, reducing regression risks
package management techniques (e.g. download statistics) can be used to track usage of drivers and identify modules to be disabled
we could get rid of package bundles like kernel-default-base or kernel-default-extra.

Disadvantages / Challenges

"Missing driver problem": e.g. network controller breaks, new controller needs new driver → unavoidable, patterns e.g. "kernel-network-driver-all" or "kernel-usb-driver-all" as precaution (but this would forfeit a lot of the size advantage)
weak dependencies, or dependencies that aren't expressed in depmod terms. We handle this partially already for initrd (SUSE INITRD mechanism). We must translate it into package dependencies
user space dependencies: many tools might not be prepared for a large number of missing drivers in an (open)SUSE environment. Need to look into dracut etc.
is our KABI tracking good enough? So far it works well for the small number of KMPs, but what if we get 1000s of KMPs with complex dependencies?
Does the KABI dependency tracking scale to this level, or will dependency checks be extremely slow?
It gets more difficult to assess system supportablity. Tooling is required to verify that installed driver packages are compatible.
we can't run depmod after every KMP installation, it'd be far too slow (I have a patched depmod available that would make this possible with "incremental" mode).
avoid excessive meta data duplication (kernel changelog in every KMP)
the KMP concept needs to be revised. The multiversion(kernel) property of KMPs allows to install the same KMP for different KABI versions, but doesn't work well for version updates of the KMP itself. That would be a big problem, because we update some drivers frequently.
do we need a means for kernel module management tools (modprobe) to hook into package management (zypper) for pulling in missing modules automatically?
…

Goal for this Hackweek

Make it possible to build "real" KMPs from the kernel package build. Note that the in-kernel -kmp packages such as dlm-kmp aren't "real" KMPs; they use a hard kernel version dependency rather than KABI dependencies.
Hack on the kernel build scripts to enable modular packaging. Perhaps one-KMP-per-module as a first step, to experiment with module interdependencies. 2nd step would be combining modules into reasonable sets. Alternatively, start with a set of modules to be packaged as KMPs (e.g. device drivers) and increase the set step-by-step.
Experiment with practical use of the modular kernel. Figure out problems with the interaction of module and package management. Play through the "missing driver" scenario, and try to find a concept for aworkaround.

Later goals

Develop a modified KMP concept. Make version of the KABI the KMP was compiled against part of the package name rather than the version (instead of foo-kmp-default-1.0_k5.3.18_59.5, use foo-kmp-default-5.3.18_59.5-1.0, indicating module foo for the KABI of 5.3.18_59.5, version 1.0)
Tackle a few of the challenges above
Explore possibilities to improve packaging of kernel firmware, too. Currently FW is packaged in big chunks by vendor. Packaging individual drivers might facilitate fine-grained dependencies (driver pulls in required FW packages).

Join this project Leave this project

Looking for hackers with the skills:

kmp kernel module rpm zypper packagemanagement kabi

This project is part of:

Hack Week 21 Hack Week 22

Activity

over 3 years ago: mkoutny liked this project.

over 3 years ago: mwilck started this project.

over 3 years ago: ptesarik liked this project.

over 3 years ago: mbrugger liked this project.

over 3 years ago: sbahling liked this project.

over 3 years ago: llansky3 liked this project.

over 3 years ago: coolgw liked this project.

over 3 years ago: avicenzi liked this project.

over 3 years ago: mwilck added keyword "kabi" to this project.

over 3 years ago: mwilck added keyword "packagemanagement" to this project.

over 3 years ago: mwilck added keyword "zypper" to this project.

over 3 years ago: mwilck added keyword "rpm" to this project.

over 3 years ago: mwilck added keyword "module" to this project.

over 3 years ago: mwilck added keyword "kernel" to this project.

over 3 years ago: mwilck added keyword "kmp" to this project.

over 3 years ago: mwilck originated this project.

Comments

over 3 years ago by mwilck | Reply

Jun 27

I spent the day studying the kernel build scripts. Using them and understanding them are two different things. I'll try to use a similar approach as kernel-default-base: Just repackage existing modules. This will speed up development, as I'll avoid long build times for the kernel package. OBS repos created under home:mwilck:ModularKernel (not much to see yet).

over 3 years ago by mwilck | Reply

Jun 28

I have a package with just the kernel image now, and at least one auto-built KMP. I am using a variant of the kernel-default-base package to simply repackage stuff that's currently built in the default package.

over 3 years ago by mwilck | Reply

Jun 29 — Jul 1

Packaging of both the kernel image and modules works. I need to transform the scripts from kernel-default-base into rpm macros, but that's not a big deal, actually.

Unfortunately I couldn't make more progress this week due to sickness.

over 3 years ago by mwilck | Reply

Later

The kernel-default-image package now packages the whole set of modules from kernel-default-base, each module packaged separately. No actual compilation is done, the compiled modules are just taken from kernel-default, the same way as it's done in kernel-default-base. If we ever want to change this, the real challenge will be how to determine the list of subpackages, as it's currently calculated dynamically from modules.dep. We would basically need to generate the list of subpackages from the list of .ko files built during kernel compilation. I am not sure if that's possible. If not, we will have to do at least one "normal" build for every config to derive the module list first before modularly packaged kernels can be built. But we have a long way to go until then, anyway.

Dependency resolution works the same way as for kernel-default-base; thus it's sufficient to list a set of desired modules or module-name-matching regular expressions, and the scripts will figure out the full set to satisfy all requirements.

The module packages have ksym() requires and (if they export anything) also ksym() provides, as well as modalias() supplements.

> rpm -qp --supplements ehci-pci-kmp-default-5.14.21-lp154.23.1.g49db222.0.x86_64.rpm modalias(kernel-default:pci:v*d*sv*sd*bc0Csc03i20*) modalias(kernel-default:pci:v0000104Ad0000CC00sv*sd*bc*sc*i*) > rpm -qp --requires ehci-pci-kmp-default-5.14.21-lp154.23.1.g49db222.0.x86_64.rpm coreutils grep kernel-default-image ksym(default:__dynamic_dev_dbg) = 22868a41 ksym(default:__fentry__) = bdfb6dbb ksym(default:__pci_register_driver) = c2126345 ksym(default:__stack_chk_fail) = d0da656b ksym(default:_dev_info) = fe960afa ksym(default:_dev_warn) = 5cba8a10 ksym(default:device_set_wakeup_capable) = 226a1ef2 ksym(default:dma_set_coherent_mask) = 93f12bc3 ksym(default:ehci_init_driver) = d72c5993 ksym(default:ehci_resume) = 645790fb ... > rpm -qp --provides ehci-hcd-kmp-default-5.14.21-lp154.23.1.g49db222.0.x86_64.rpm ehci-hcd-kmp = 5.14.21 ehci-hcd-kmp-default = 5.14.21-lp154.23.1.g49db222.0 ehci-hcd-kmp-default(x86-64) = 5.14.21-lp154.23.1.g49db222.0 kmod(ehci_hcd.ko) ksym(default:ehci_adjust_port_wakeup_flags) = e6bf909e ksym(default:ehci_handshake) = 1456664e ksym(default:ehci_hub_control) = 913627e8 ksym(default:ehci_init_driver) = d72c5993 ksym(default:ehci_reset) = 145925dc ksym(default:ehci_resume) = 645790fb ksym(default:ehci_setup) = 4048a7e9 ksym(default:ehci_suspend) = 2c94b03f The build log is interesting, too.

I had to modify the scripts looking for provides and supplements for this to work. Thus I'm shipping these as part of the package.

Not surprisingly, the package repo contains a long list of packages. Including modules from -extra, I built a set of almost 3000 packages. Not quite texlive, but getting close

I had to stop at this point; so I haven't done any installation tests with the KMPs generated this way yet. The dependencies look quite sane though. What's obviously missing at this point is some patterns to allow installing e.g. all netfilter modules. For drivers, modalias should work (zypper inr --no-recommends), but I havent't tested it. The repo also urgently needs my performance fixes for depmod from last year, which are unfortunately still unfinished.

over 3 years ago by michals | Reply

The problem with this approach is that when you include the dependencies the subpackages may overlap, and while rpm itself would be fine with that because the modules coming from one kernel build are identical the packaging policy forbids it.
- over 3 years ago by tiwai | Reply
  
  What do you mean by "the dependencies the subpackages may overlap"? The same ksym() checksum provided by multiple kmps (of various versions)?
  
  I guess a partial solution would be to embed the hard dependency from the result of depmod. e.g. if a module has dependency on another module abc (as found in depmod), we can put "Requires: abc-kmp-default = %{version}" explicitly at the build time. Also, a softdep could be interpreted with Supplements or such, too.

Similar Projects

kernel

Backporting patches using LLM by jankara

Description

Backporting Linux kernel fixes (either for CVE issues or as part of general git-fixes workflow) is boring and mostly mechanical work (dealing with changes in context, renamed variables, new helper functions etc.). The idea of this project is to explore usage of LLM for backporting Linux kernel commits to SUSE kernels using LLM.

Goals

Create safe environment allowing LLM to run and backport patches without exposing the whole filesystem to it (for privacy and security reasons).
Write prompt that will guide LLM through the backporting process. Fine tune it based on experimental results.
Explore success rate of LLMs when backporting various patches.

Resources

Docker
Gemini CLI

Repository

Current version of the container with some instructions for use are at: https://gitlab.suse.de/jankara/gemini-cli-backporter

dynticks-testing: analyse perf / trace-cmd output and aggregate data by m.crivellari

Description

dynticks-testing is a project started years ago by Frederic Weisbecker. One of the feature is to check the actual configuration (isolcpus, irqaffinity etc etc) and give feedback on it.

An important goal of this tool is to parse the output of trace-cmd / perf and provide more readable data, showing the duration of every events grouped by PID (showing also the CPU number, if the tasks has been migrated etc).

An example of data captured on my laptop (incomplete!!):

          -0     [005] dN.2. 20310.270699: sched_wakeup:         WaylandProxy:46380 [120] CPU:005
          -0     [005] d..2. 20310.270702: sched_switch:         swapper/5:0 [120] R ==> WaylandProxy:46380 [120]
...
    WaylandProxy-46380 [004] d..2. 20310.295397: sched_switch:         WaylandProxy:46380 [120] S ==> swapper/4:0 [120]
          -0     [006] d..2. 20310.295397: sched_switch:         swapper/6:0 [120] R ==> firefox:46373 [120]
         firefox-46373 [006] d..2. 20310.295408: sched_switch:         firefox:46373 [120] S ==> swapper/6:0 [120]
          -0     [004] dN.2. 20310.295466: sched_wakeup:         WaylandProxy:46380 [120] CPU:004

Output of noise_parse.py:

Task: WaylandProxy Pid: 46380 cpus: {4, 5} (Migrated!!!)
        Wakeup Latency                                Nr:        24     Duration:          89
        Sched switch: kworker/12:2                    Nr:         1     Duration:           6

My first contribution is around Nov. 2024!

Goals

add more features (eg cpuset)
test / bugfix

Resources

Frederic's public repository: https://git.kernel.org/pub/scm/linux/kernel/git/frederic/dynticks-testing.git/
https://docs.kernel.org/timers/no_hz.html#testing

Progresses

isolcpus and cpusets implemented and merged in master: dynticks-testing.git commit

Improve UML page fault handler by ptesarik

Description

Improve UML handling of segmentation faults in kernel mode. Although such page faults are generally caused by a kernel bug, it is annoying if they cause an infinite loop, or panic the kernel. More importantly, a robust implementation allows to write KUnit tests for various guard pages, preventing potential kernel self-protection regressions.

Goals

Convert the UML page fault handler to use oops_* helpers, go through a few review rounds and finally get my patch series merged in 6.14.

Resources

Wrong initial attempt: https://lore.kernel.org/lkml/20231215121431.680-1-petrtesarik@huaweicloud.com/T/

early stage kdump support by mbrugger

Project Description

When we experience a early boot crash, we are not able to analyze the kernel dump, as user-space wasn't able to load the crash system. The idea is to make the crash system compiled into the host kernel (think of initramfs) so that we can create a kernel dump really early in the boot process.

Goal for the Hackweeks

Investigate if this is possible and the implications it would have (done in HW21)
Hack up a PoC (done in HW22 and HW23)
Prepare RFC series (giving it's only one week, we are entering wishful thinking territory here).

update HW23

I was able to include the crash kernel into the kernel Image.
I'll need to find a way to load that from init/main.c:start_kernel() probably after kcsan_init()
I workaround for a smoke test was to hack kexec_file_load() systemcall which has two problems:
1. My initramfs in the porduction kernel does not have a new enough kexec version, that's not a blocker but where the week ended
2. As the crash kernel is part of init.data it will be already stale once I can call kexec_file_load() from user-space.

The solution is probably to rewrite the POC so that the invocation can be done from init.text (that's my theory) but I'm not sure if I can reuse the kexec infrastructure in the kernel from there, which I rely on heavily.

update HW24

Day1
- rebased on v6.12 with no problems others then me breaking the config
- setting up a new compilation and qemu/virtme env
- getting desperate as nothing works that used to work
Day 2
- getting to call the invocation of loading the early kernel from __init after kcsan_init()
Day 3
- fix problem of memdup not being able to alloc so much memory... use 64K page sizes for now
- code refactoring
- I'm now able to load the crash kernel
- When using virtme I can boot into the crash kernel, also it doesn't boot completely (major milestone!), crash in elfcorehdr_read_notes()
Day 4
- crash systems crashes (no pun intended) in copy_old_mempage() link; will need to understand elfcorehdr...
- call path vmcore_init() -> parse_crash_elf_headers() -> elfcorehdr_read() -> read_from_oldmem() -> copy_oldmem_page() -> copy_to_iter()
Day 5
- hacking arch/arm64/kernel/crash_dump.c:copy_old_mempage() to see if crash system really starts. It does.
- fun fact: retested with more reserved memory and with UEFI FW, host kernel crashes in init but directly starts the crash kernel, so it works (somehow) \o/

update HW25

Day 1
- rebased crash-kernel on v6.12.59 (for now), still crashing

Add Qualcomm Snapdragon 765G (SM7250) basic device tree to mainline linux kernel by pvorel

Qualcomm Snapdragon 765G (SM7250) (smartphone SoC) has no support in the linux kernel, nor in u-boot. Try to add basic device tree support. The hardest part will be to create boot.img which will be accepted by phone.

UART is available for smartphone :).

rpm

Create a page with all devel:languages:perl packages and their versions by tinita

Description

Perl projects now live in git: https://src.opensuse.org/perl

It would be useful to have an easy way to check which version of which perl module is in devel:languages:perl. Also we have meta overrides and patches for various modules, and it would be good to have them at a central place, so it is easier to lookup, and we can share with other vendors.

I did some initial data dump here a while ago: https://github.com/perlpunk/cpan-meta

But I never had the time to automate this.

I can also use the data to check if there are necessary updates (currently it uses data from download.opensuse.org, so there is some delay and it depends on building).

Goals

Have a script that updates a central repository (e.g. https://src.opensuse.org/perl/_metadata) with metadata by looking at https://src.opensuse.org/perl/_ObsPrj (check if there are any changes from the last run)
Create a HTML page with the list of packages (use Javascript and some table library to make it easily searchable)

Resources

Results

Day 1

First part of the code which retrieves data from https://src.opensuse.org/perl/_ObsPrj with submodules and creates a YAML and a JSON file.
Repo: https://github.com/perlpunk/opensuse-perl-meta
Also a first version of the HTML is live: https://perlpunk.github.io/opensuse-perl-meta/

Day 2

HTML Page has now links to src.opensuse.org and the date of the last update, plus a short info at the top
Code is now 100% covered by tests: https://app.codecov.io/gh/perlpunk/opensuse-perl-meta
I used the modern perl class feature, which makes perl classes even nicer and shorter. See example
Tests
- I tried out the mocking feature of the modern Test2::V0 library which provides call tracking. See example
- I tried out comparing data structures with the new Test2::V0 library. It let's you compare parts of the structure with the like function, which only compares the date that is mentioned in the expected data. example

Day 3

Added various things to the table
- Dependencies column
- Show popup with info for cpanspec, patches and dependencies
- Added last date / commit to the data export.

Plan: With the added date / commit we can now daily check _ObsPrj for changes and only fetch the data for changed packages.

Day 4

A CLI for Harvester by mohamed.belgaied

Harvester does not officially come with a CLI tool, the user is supposed to interact with Harvester mostly through the UI. Though it is theoretically possible to use kubectl to interact with Harvester, the manipulation of Kubevirt YAML objects is absolutely not user friendly. Inspired by tools like multipass from Canonical to easily and rapidly create one of multiple VMs, I began the development of Harvester CLI. Currently, it works but Harvester CLI needs some love to be up-to-date with Harvester v1.0.2 and needs some bug fixes and improvements as well.

Project Description

Harvester CLI is a command line interface tool written in Go, designed to simplify interfacing with a Harvester cluster as a user. It is especially useful for testing purposes as you can easily and rapidly create VMs in Harvester by providing a simple command such as: harvester vm create my-vm --count 5 to create 5 VMs named my-vm-01 to my-vm-05.

Harvester CLI is functional but needs a number of improvements: up-to-date functionality with Harvester v1.0.2 (some minor issues right now), modifying the default behaviour to create an opensuse VM instead of an ubuntu VM, solve some bugs, etc.

Github Repo for Harvester CLI: https://github.com/belgaied2/harvester-cli

Done in previous Hackweeks

Create a Github actions pipeline to automatically integrate Harvester CLI to Homebrew repositories: DONE
Automatically package Harvester CLI for OpenSUSE / Redhat RPMs or DEBs: DONE

Goal for this Hackweek

The goal for this Hackweek is to bring Harvester CLI up-to-speed with latest Harvester versions (v1.3.X and v1.4.X), and improve the code quality as well as implement some simple features and bug fixes.

Some nice additions might be: * Improve handling of namespaced objects * Add features, such as network management or Load Balancer creation ? * Add more unit tests and, why not, e2e tests * Improve CI * Improve the overall code quality * Test the program and create issues for it

Issue list is here: https://github.com/belgaied2/harvester-cli/issues

Resources

The project is written in Go, and using client-go the Kubernetes Go Client libraries to communicate with the Harvester API (which is Kubernetes in fact). Welcome contributions are:

Testing it and creating issues
Documentation
Go code improvement

What you might learn

Harvester CLI might be interesting to you if you want to learn more about:

GitHub Actions
Harvester as a SUSE Product
Go programming language
Kubernetes API
Kubevirt API objects (Manipulating VMs and VM Configuration in Kubernetes using Kubevirt)

Switch software-o-o to store repomd in a database by hennevogel

Description

The openSUSE Software portal is a web app to explore binary packages of openSUSE distributions. Kind of like an package manager / app store.

https://software.opensuse.org/

This app has been around forever (August 2007) and it's architecture is a bit brittle. It acts as a frontend to the OBS distributions and published binary search APIs, calculates and caches a lot of stuff in memory and needs code changes nearly every openSUSE release to keep up.

As you can imagine, it's a heavy user of the OBS API, especially when caches are cold.

Goals

I want to change the app to cache repomod data in a (postgres) database structure

Distributions have many Repositories
Repositories have many Packages
Packages have many Patches

The UI workflows will be as following

As an admin I setup Distribution and it's repositories
As an admin I sync all repositories repomd files into to the database
As a user I browse a Distribution by category
As a user I search for Package of a Distribution in it's Repositories
As a user I extend the search to Package build on OBS for this Distribution

This has a couple of pro's:

Less traffic on the OBS API as the usual Packages are inside the database
Easier base to add features to this page. Like comments, ratings, openSUSE specific screenshots etc.
Separating the Distribution package search from searching through OBS will hopefully make more clear for newbies that enabling extra repositories is kind of dangerous.

And one con:

You can't search for packages build for foreign distributions with this app anymore (although we could consume their repomd etc. but I doubt we have the audience on an opensuse.org domain...)

TODO

Introduce a PG database
Add clockworkd as scheduler and delayed_job as ActiveJob backend
Introduce ActiveStorage
Build initial data model
Introduce repomd to database sync
- Adapt repomd sync to Leap 16.0 repomod layout changes (single arch, no update repo)
- Make repomd sync idempotent
Introduce database search
Setup foreman to run rails s and rake jobs:workoff
Adapt UI
- Build Category Browsing
- Build Admin Distribution CRUD interface

zypper

File search subcommand for zypper by mook_work

Description

Zypper currently only supports searching for files in a few pre-defined prefixes (/usr/bin, /usr/sbin, etc.) via using rpm provides. This means that it is difficult to find files that are not in the explicit list.

Doing this in zypper itself seems difficult.

Goals

Create a zypper subcommand (plugin) that can download file lists as needed to search for file contents. As a stretch goal, have a secondary subcommand that can list files in packages regardless of whether they are already installed.

At this point, making the resulting code usable as part of zypper itself is not a goal, as it would probably take more time than available.

Resources

Upstream issue: https://github.com/openSUSE/zypper/issues/469

Stretch goal: https://github.com/openSUSE/zypper/issues/164

Subcommands: https://manpages.opensuse.org/Tumbleweed/zypper/zypper.8.en.html#SUBCOMMANDS

Project Description

Advantages

Disadvantages / Challenges

Goal for this Hackweek

Later goals

Looking for hackers with the skills:

This project is part of:

Activity

Comments

over 3 years ago by mwilck | Reply

Jun 27

over 3 years ago by mwilck | Reply

Jun 28

over 3 years ago by mwilck | Reply

Jun 29 — Jul 1

over 3 years ago by mwilck | Reply

Later

over 3 years ago by michals | Reply

over 3 years ago by tiwai | Reply

Similar Projects

kernel

Backporting patches using LLM by jankara

Description

Goals

Resources

Repository

dynticks-testing: analyse perf / trace-cmd output and aggregate data by m.crivellari

Description

Goals

Resources

Progresses

Improve UML page fault handler by ptesarik

Description

Goals

Resources

early stage kdump support by mbrugger

Project Description

Goal for the Hackweeks

update HW23

update HW24

update HW25

Add Qualcomm Snapdragon 765G (SM7250) basic device tree to mainline linux kernel by pvorel

rpm

Create a page with all devel:languages:perl packages and their versions by tinita

Description

Goals

Resources

Results

Day 1

Day 2

Day 3

Day 4

A CLI for Harvester by mohamed.belgaied

Project Description

Done in previous Hackweeks

Goal for this Hackweek

Resources

What you might learn

Switch software-o-o to store repomd in a database by hennevogel

Description

Goals

TODO

zypper

File search subcommand for zypper by mook_work

Description

Goals

Resources