Improve the picotm Transaction Manager
Picotm is a system-level transaction manager. It provides transactional semantics to low-level C operations, such as
- memory access,
- modifying data structures,
- (some) file I/O, and
- common interfaces from the C Standard Library and POSIX.
Picotm also handles error detection and recovery for all it's functionality. It's fully modular, so new functionality can be added.
For the Hackweek, I want to dedicate some time to picotm. I want to finish some of the refactoring work that I have been working on. If there's time left, I'd like to investigate two-phase commits and how to support them in picotm.
Picotm is available at http://picotm.org/.
Looking for hackers with the skills:
This project is part of:
Hack Week 25
Activity
Comments
-
7 days ago by tdz | Reply
FYI, source code for this project, including the changes from this hackweek are available from the project repo at https://github.com/picotm/picotm, or from my copy at https://github.com/tdz/picotm.
-
7 days ago by tdz | Reply
Day 1: I have not looked at picotm for far too long. So I spent the first day of the hackweek with getting accommodated to the source code again.
- I made my self familiar with the code base again so that I find my way around. I also ran the included test cases, which all still work. :)
- I cleaned up licensing information of the code.
- I converted the code base to require C23 and to use C23's nullptr. As picotm has always been highly portable, both changes were straight forward.
- I added a .editorconfig file to describe the coding style. Many editors support this file.
-
6 days ago by tdz | Reply
Day 2: Picotm support transactional file I/O at the level of POSIX interfaces. This means transactional semantics for operations on
- file descriptors,
- open file descriptions (i.e., file state), and
- on-disk file buffers (or sockets, pipes, etc).
For example, writing to the file via write() requires tracking the file descriptor against concurrent close() calls, tracking the file write position against concurrent write() calls, and tracking the file content against concurrent read and write operations. This gets more complex as a file buffer can be references by multiple open file descriptions and a single open file description can be referenced by multiple file descriptors.
Before I can make any improvements, I first have to get back into the current implementation. So for most of the day I studied the code I wrote a few years ago.
-
5 days ago by tdz | Reply
Day 3: Today I worked on improving concurrency control for file I/O. For managing resources, picotm needs an identifier that distinguishes the resources against other resources of the same type. For example, each value in memory has a unique address. In the case of Unix file I/O, we have the file descriptors, open file descriptions, and file buffers, as described in the comment of das 2.
File descriptors are unique numbers per process. File buffers usually have a device and inode id that uniquely identifies the buffer.
Open file descriptions have neither. This used to be a problem, as open file descriptions could not distinguished from each other. Solving this required excessive locking, at the expense of parallelism. For example, two write to the same file could often not run concurrently even if they did not share the same open file description.
Since I last worked on picotm's file I/O code, there is now a solution. Since v5.12, the Linux kernel supports the kcmp system call. It allows for testing if two file descriptors refer to the same open file description. Although this is not a unique id, it is good enough to distinguish open file descriptions from each other. Kcmp is even sorted, so binary search trees can be build from it.
To finally make use of kcmp, I updated picotm's transactional file-I/O code. When doing transactional I/O, it finds the correct open file description for each file descriptor and avoid unnecessary interference with other file descriptors.
-
4 days ago by tdz | Reply
Day 4: I clean up the kcmp support. It turns out Linux meanwhile added another way of testing files for equality: fcntl() command FDUPFDQUERY was added in 2024 in v6.10. While kcmp is optional, newer kernels should support FDUPFDQUERY unconditionally. So I added support for the interface as well.
-
3 days ago by rbranco | Reply
Nice! FreeBSD adopted the kcmp system call since 14.1. Will look to add support for FDUPFDQUERY soon.
-
3 days ago by tdz | Reply
Hi! Nice to see that there's some 'standardization' happening around the fcntl() command. IIRC FreeBSD already provides a custom syscall with that functionality. Right now, I only implemented this for Linux, but picotm used to run well on the BSDs. Maybe I'll take a look at a FreeBSD port.
-
-
3 days ago by tdz | Reply
Day 5: Time to wrap up and get things into shape. So today I finished the patches and added them to the project repository. I also found time to do additional refactoring on some of the code's internals.
I'm very happy that I took the time to come back to this old problem and finally fix it. Overall a great hackweek.
Similar Projects
Smart lighting with Pico 2 by jmodak
Description
I am trying to create a smart-lighting project with a Raspberry Pi Pico that reacts to a movie's visuals and audio that involves combining two distinct functions: ambient screen lighting(visual response) and sound-reactive lighting(audio response)
Goals
- Visuals: Capturing the screen's colour requires an external device to analyse screen content and send colour data to the MCU via serial communication.
- Audio: A sound sensor module connected directly to the Pico that can detect sound volume.
- Pico 2W: The MCU receives data fro, both inputs and controls an LED strip.
Resources
- Raspberry Pi Pico 2 W
- RGB LED strip
- Sound detecting sensor
- Power supply
- breadboard and wires
Add a machine-readable output to dmidecode by jdelvare
Description
There have been repeated requests for a machine-friendly dmidecode output over the last decade. During Hack Week 19, 5 years ago, I prepared the code to support alternative output formats, but didn't have the time to go further. Last year, Jiri Hnidek from Red Hat Linux posted a proof-of-concept implementation to add JSON output support. This is a fairly large pull request which needs to be carefully reviewed and tested.
Goals
Review Jiri's work and provide constructive feedback. Merge the code if acceptable. Evaluate the costs and benefits of using a library such as json-c.
x64id: An x86/x64 instruction disassembler by m.crivellari
Description
This is an old side project. An x86/x64 machine code decoder. It is useful to get instructions' length and identify each of its fields.
Example:
C7 85 68 FF FF FF 00 00 00 00
This is the instruction:
MOV DWORD PTR SS:[LOCAL.38],0
What follows are some of the information collected by the disassembler, based on the specific instruction:
RAW bytes (hex): C7 85 68 FF FF FF 00 00 00 00
Instr. length: 10
Print instruction fields:
Located Prefixes 0:
OP: 0xC7
mod_reg_rm: 0x85
disp (4): 0xFFFFFF68
Iimm: 0x0
Lacks the mnemonic representation: from the previous machine code is not able to produce the "MOV..." instruction, for example.
Goals
The goal is almost easy: partially implement the mnemonic representation. I have already started during the weekend, likely tomorrow I will push the branch!
Resources
- The project: https://github.com/DispatchCode/x64-Instruction-Decoder/
- This is useful to avoid gdb and objdump in local: https://defuse.ca/online-x86-assembler.htm
- Another interesting resource is https://godbolt.org/
Progress
- An initial implementation can be found at: https://github.com/DispatchCode/x64-Instruction-Decoder/tree/mnemonic-support It is described under the "Mnemonic translation" in the README file!
Let's consider this example:
[...other bytes...] 43 89 44 B5 00 01 00 [...other bytes...]
pudc - A PID 1 process that barks to the internet by mssola
Description
As a fun exercise in order to dig deeper into the Linux kernel, its interfaces, the RISC-V architecture, and all the dragons in between; I'm building a blog site cooked like this:
- The backend is written in a mixture of C and RISC-V assembly.
- The backend is actually PID1 (for real, not within a container).
- We poll and parse incoming HTTP requests ourselves.
- The frontend is a mere HTML page with htmx.
The project is meant to be Linux-specific, so I'm going to use io_uring, pidfs, namespaces, and Linux-specific features in order to drive all of this.
I'm open for suggestions and so on, but this is meant to be a solo project, as this is more of a learning exercise for me than anything else.
Goals
- Have a better understanding of different Linux features from user space down to the kernel internals.
- Most importantly: have fun.
Resources
Port OTPClient to GTK >= 4.18 by pstivanin
Project Description
OTPClient is currently using GTK3 and cannot easily be ported to GTK4. Since GTK4 came out, there have been quite some big changes. Also, there are now some new deprecation that will take effect with GTK5 (and are active starting from 4.10 as warnings), so I need to think ahead and port OTPClient without using any of those deprecated features.
Goal for this Hackweek
- fix the last 3 opened issues (https://github.com/paolostivanin/OTPClient/issues/402, https://github.com/paolostivanin/OTPClient/issues/404, https://github.com/paolostivanin/OTPClient/issues/406) and release a new version
- continue the rewrite from where we left last year
- if possible, finally close this 6 years old issue: https://github.com/paolostivanin/OTPClient/issues/123
RMT.rs: High-Performance Registration Path for RMT using Rust by gbasso
Description
The SUSE Repository Mirroring Tool (RMT) is a critical component for managing software updates and subscriptions, especially for our Public Cloud Team (PCT). In a cloud environment, hundreds or even thousands of new SUSE instances (VPS/EC2) can be provisioned simultaneously. Each new instance attempts to register against an RMT server, creating a "thundering herd" scenario.
We have observed that the current RMT server, written in Ruby, faces performance issues under this high-concurrency registration load. This can lead to request overhead, slow registration times, and outright registration failures, delaying the readiness of new cloud instances.
This Hackweek project aims to explore a solution by re-implementing the performance-critical registration path in Rust. The goal is to leverage Rust's high performance, memory safety, and first-class concurrency handling to create an alternative registration endpoint that is fast, reliable, and can gracefully manage massive, simultaneous request spikes.
The new Rust module will be integrated into the existing RMT Ruby application, allowing us to directly compare the performance of both implementations.
Goals
The primary objective is to build and benchmark a high-performance Rust-based alternative for the RMT server registration endpoint.
Key goals for the week:
- Analyze & Identify: Dive into the
SUSE/rmtRuby codebase to identify and map out the exact critical path for server registration (e.g., controllers, services, database interactions). - Develop in Rust: Implement a functionally equivalent version of this registration logic in Rust.
- Integrate: Explore and implement a method for Ruby/Rust integration to "hot-wire" the new Rust module into the RMT application. This may involve using FFI, or libraries like
rb-sysormagnus. - Benchmark: Create a benchmarking script (e.g., using
k6,ab, or a custom tool) that simulates the high-concurrency registration load from thousands of clients. - Compare & Present: Conduct a comparative performance analysis (requests per second, latency, success/error rates, CPU/memory usage) between the original Ruby path and the new Rust path. The deliverable will be this data and a summary of the findings.
Resources
- RMT Source Code (Ruby):
https://github.com/SUSE/rmt
- RMT Documentation:
https://documentation.suse.com/sles/15-SP7/html/SLES-all/book-rmt.html
- Tooling & Stacks:
- RMT/Ruby development environment (for running the base RMT)
- Rust development environment (
rustup,cargo)
- Potential Integration Libraries:
- rb-sys:
https://github.com/oxidize-rb/rb-sys - Magnus:
https://github.com/matsadler/magnus
- rb-sys:
- Benchmarking Tools:
k6(https://k6.io/)ab(ApacheBench)