early stage kdump support by mbrugger

Project Description

When we experience a early boot crash, we are not able to analyze the kernel dump, as user-space wasn't able to load the crash system. The idea is to make the crash system compiled into the host kernel (think of initramfs) so that we can create a kernel dump really early in the boot process.

Goal for the Hackweeks

1. Investigate if this is possible and the implications it would have (done in HW21)
2. Hack up a PoC (done in HW22 and HW23)
3. Prepare RFC series (giving it's only one week, we are entering wishful thinking territory here).

update HW23

• I was able to include the crash kernel into the kernel Image.
• I'll need to find a way to load that from init/main.c:start_kernel() probably after kcsan_init()
• I workaround for a smoke test was to hack kexec_file_load() systemcall which has two problems:
  1. My initramfs in the porduction kernel does not have a new enough kexec version, that's not a blocker but where the week ended
  2. As the crash kernel is part of init.data it will be already stale once I can call kexec_file_load() from user-space.

The solution is probably to rewrite the POC so that the invocation can be done from init.text (that's my theory) but I'm not sure if I can reuse the kexec infrastructure in the kernel from there, which I rely on heavily.

update HW24

• Day1
  • rebased on v6.12 with no problems others then me breaking the config
  • setting up a new compilation and qemu/virtme env
  • getting desperate as nothing works that used to work
• Day 2
  • getting to call the invocation of loading the early kernel from __init after kcsan_init()

• Day 3

  • fix problem of memdup not being able to alloc so much memory... use 64K page sizes for now
  • code refactoring
  • I'm now able to load the crash kernel
  • When using virtme I can boot into the crash kernel, also it doesn't boot completely (major milestone!), crash in elfcorehdr_read_notes()

• Day 4

  • crash systems crashes (no pun intended) in copy_old_mempage() link; will need to understand elfcorehdr...
  • call path vmcore_init() -> parse_crash_elf_headers() -> elfcorehdr_read() -> read_from_oldmem() -> copy_oldmem_page() -> copy_to_iter()

• Day 5

  • hacking arch/arm64/kernel/crash_dump.c:copy_old_mempage() to see if crash system really starts. It does.
  • fun fact: retested with more reserved memory and with UEFI FW, host kernel crashes in init but directly starts the crash kernel, so it works (somehow) \o/

update HW25

• Day 1
  • rebased crash-kernel on v6.12.59 (for now), still crashing

---

pudc - A PID 1 process that barks to the internet by mssola

Description

As a fun exercise in order to dig deeper into the Linux kernel, its interfaces, the RISC-V architecture, and all the dragons in between; I'm building a blog site cooked like this:

• The backend is written in a mixture of C and RISC-V assembly.
• The backend is actually PID1 (for real, not within a container).
• We poll and parse incoming HTTP requests ourselves.
• The frontend is a mere HTML page with htmx.

The project is meant to be Linux-specific, so I'm going to use io_uring, pidfs, namespaces, and Linux-specific features in order to drive all of this.

I'm open for suggestions and so on, but this is meant to be a solo project, as this is more of a learning exercise for me than anything else.

Goals

• Have a better understanding of different Linux features from user space down to the kernel internals.
• Most importantly: have fun.

Resources

• https://github.com/mssola/pudc

---

Add Qualcomm Snapdragon 765G (SM7250) basic device tree to mainline linux kernel by pvorel

Qualcomm Snapdragon 765G (SM7250) (smartphone SoC) has no support in the linux kernel, nor in u-boot. Try to add basic device tree support. The hardest part will be to create boot.img which will be accepted by phone.

UART is available for smartphone :).

---

Backporting patches using LLM by jankara

Description

Backporting Linux kernel fixes (either for CVE issues or as part of general git-fixes workflow) is boring and mostly mechanical work (dealing with changes in context, renamed variables, new helper functions etc.). The idea of this project is to explore usage of LLM for backporting Linux kernel commits to SUSE kernels using LLM.

Goals

• Create safe environment allowing LLM to run and backport patches without exposing the whole filesystem to it (for privacy and security reasons).
• Write prompt that will guide LLM through the backporting process. Fine tune it based on experimental results.
• Explore success rate of LLMs when backporting various patches.

Resources

• Docker
• Gemini CLI

Repository

Current version of the container with some instructions for use are at: https://gitlab.suse.de/jankara/gemini-cli-backporter

---

bpftrace contribution by mkoutny

Description

bpftrace is a great tool, no need to sing odes to it here. It can access any kernel data and process them in real time. It provides helpers for some common Linux kernel structures but not all.

Goals

• set up bpftrace toolchain
• learn about bpftrace implementation and internals
• implement support for percpu_counters
• look into some of the first issues
• send a refined PR (on Thu)

Resources

• https://bpftrace.org/
• https://docs.kernel.org/

---

MCP Trace Suite by r1chard-lyu

Description

This project plans to create an MCP Trace Suite, a system that consolidates commonly used Linux debugging tools such as bpftrace, perf, and ftrace.

The suite is implemented as an MCP Server. This architecture allows an AI agent to leverage the server to diagnose Linux issues and perform targeted system debugging by remotely executing and retrieving tracing data from these powerful tools.

• Repo: https://github.com/r1chard-lyu/systracesuite
• Demo: Slides

Goals

1. Build an MCP Server that can integrate various Linux debugging and tracing tools, including bpftrace, perf, ftrace, strace, and others, with support for future expansion of additional tools.

2. Perform testing by intentionally creating bugs or issues that impact system performance, allowing an AI agent to analyze the root cause and identify the underlying problem.

Resources

• Gemini CLI: https://geminicli.com/
• eBPF: https://ebpf.io/
• bpftrace: https://github.com/bpftrace/bpftrace/
• perf: https://perfwiki.github.io/main/
• ftrace: https://github.com/r1chard-lyu/tracium/

---

RMT.rs: High-Performance Registration Path for RMT using Rust by gbasso

Description

The SUSE Repository Mirroring Tool (RMT) is a critical component for managing software updates and subscriptions, especially for our Public Cloud Team (PCT). In a cloud environment, hundreds or even thousands of new SUSE instances (VPS/EC2) can be provisioned simultaneously. Each new instance attempts to register against an RMT server, creating a "thundering herd" scenario.

We have observed that the current RMT server, written in Ruby, faces performance issues under this high-concurrency registration load. This can lead to request overhead, slow registration times, and outright registration failures, delaying the readiness of new cloud instances.

This Hackweek project aims to explore a solution by re-implementing the performance-critical registration path in Rust. The goal is to leverage Rust's high performance, memory safety, and first-class concurrency handling to create an alternative registration endpoint that is fast, reliable, and can gracefully manage massive, simultaneous request spikes.

The new Rust module will be integrated into the existing RMT Ruby application, allowing us to directly compare the performance of both implementations.

Goals

The primary objective is to build and benchmark a high-performance Rust-based alternative for the RMT server registration endpoint.

Key goals for the week:

1. Analyze & Identify: Dive into the SUSE/rmt Ruby codebase to identify and map out the exact critical path for server registration (e.g., controllers, services, database interactions).
2. Develop in Rust: Implement a functionally equivalent version of this registration logic in Rust.
3. Integrate: Explore and implement a method for Ruby/Rust integration to "hot-wire" the new Rust module into the RMT application. This may involve using FFI, or libraries like rb-sys or magnus.
4. Benchmark: Create a benchmarking script (e.g., using k6, ab, or a custom tool) that simulates the high-concurrency registration load from thousands of clients.
5. Compare & Present: Conduct a comparative performance analysis (requests per second, latency, success/error rates, CPU/memory usage) between the original Ruby path and the new Rust path. The deliverable will be this data and a summary of the findings.

Resources

• RMT Source Code (Ruby):
  • https://github.com/SUSE/rmt
• RMT Documentation:
  • https://documentation.suse.com/sles/15-SP7/html/SLES-all/book-rmt.html
• Tooling & Stacks:
  • RMT/Ruby development environment (for running the base RMT)
  • Rust development environment (rustup, cargo)
• Potential Integration Libraries:
  • rb-sys: https://github.com/oxidize-rb/rb-sys
  • Magnus: https://github.com/matsadler/magnus
• Benchmarking Tools:
  • k6 (https://k6.io/)
  • ab (ApacheBench)

---