A set of utilities to produce a "from scratch" OCI/Docker container using Opensuse/SLE rpms
a project by ldragon
Updated over 2 years ago. 7 hacker ♥️. 6 followers.

Project Description

I recently used melange and apko to build a from scratch image. The result was a set of auditable and easy to use container and apk repository. The toolkit reduces the work need to make from scratch images with minimal work on the actual docker container(which can be quite painful if you've tried making a from scratch image on your own).

The end goal would be to produce 2 utilities. 1. A utility that can build rpms from yaml thus simplifying the rpm building process. This would not aim to replace the entire rpm spec but rather be a subset of functionality and will not include macros for now. The counterpart to this utility would another utility to produce a container image including a minimal set of dependencies from the opensuse repos.

As for the language used for the project, I believe rust will be a good choice, it provides enough flexibility along with a large collection of utilities we can use for this including but not limited to, rpm, oci-spec-rs along with compile time checks to speed up the development process

Goals for this Hackweek

The goals for this hackweek for the rpm utility are the following: 1. produce a signing keypair for packages 2. run a set of commands to produce a binary/lib that can be copied to a directory that will be converted to a rpm 3. output a rpm 4. make a rpm repository that can be used by the container utility

The goals for the container utility are the following: 1. build a container from a repository 2. output the container as a tarball 3. publish the container to a repo

Resources

Join this project

Looking for hackers with the skills:

containers rust docker podman rpm packaging

This project is part of:

Hack Week 23

Activity

  • about 2 years ago: vliaskovitis liked this project.
  • about 2 years ago: amunoz liked this project.
  • about 2 years ago: amunoz disliked this project.
  • about 2 years ago: amunoz liked this project.
  • about 2 years ago: Amrita42 liked this project.
  • about 2 years ago: tschmitz joined this project.
  • about 2 years ago: ygutierrez liked this project.
  • about 2 years ago: gleidi liked this project.
  • about 2 years ago: rsimai liked this project.
  • about 2 years ago: hennevogel removed keyword rpms from this project.
  • about 2 years ago: hennevogel added keyword "rpm" to this project.
  • over 2 years ago: algir joined this project.
  • over 2 years ago: ldragon liked this project.
  • over 2 years ago: ldragon started this project.
  • over 2 years ago: ldragon added keyword "containers" to this project.
  • over 2 years ago: ldragon added keyword "rpms" to this project.
  • over 2 years ago: ldragon added keyword "rust" to this project.
  • over 2 years ago: ldragon added keyword "docker" to this project.
  • over 2 years ago: ldragon added keyword "podman" to this project.
  • over 2 years ago: ldragon originated this project.


    • Comments

    • ldragon
      about 2 years ago by ldragon | Reply

      @tschmitz and I are working on this together rpmo rpmo is what our tool to produce rpms from yamls is called. As of writing we are able to produce a build env for rpms and download the source(s). What is left is running a pipeline to build everything and finally producing a rpm rpm2container rpm2container should produce a minimal rootfs and turn it into a container using rpms. Currently it is able to produce a rootfs. Todo is generating a config.json to create a bundle to produce a tar ball which can be loaded by podman and executed.

      Under the hood both projects use the tumbleweed container via podman to avoid utilizing host tools and remain cross distro compatible. We did evaluate using bwrap but we needed to use the host for zypper using bind mounts and symlinks however the container already provides the environment we require to bootstrap a build env

    • ldragon
      about 2 years ago by ldragon | Reply

      As of writing rpmo is able to produce rpms. in the future this should be extended to produce a repo file along side the rpm. rpm2container is also nearing the finish line

    • ldragon
      about 2 years ago by ldragon | Reply

      at last rpm2container can also produce containers. Leading to the end of this POC. for future folks picking this up, here's a list of things you will want to add to both projects.

      rpmo:

      • it should output a directory complete with a .repo file and other bits needed to make it a valid repo for zypper

      • support for sources other than archives(eg git or the current directory) and support for local source(eg patches)

      • auto dynamic linking resolution. rpmo can only generate a rpm with a set of predefined dependencies. it should be possible to do elf parsing and determine sonames the binary is linked to and report them.

      • signing. Currently rpmo outputs non-signed rpms and the build env explicitly disabled gpg checks both should be resolved

      • reproducibility. it does work in a hack way. should be made more reliable

      rpm2container:

      • don't skip gpg checks

      • sbom generation

      • pushing image to a remote

      • more on the fly generation of json used in the container.

      along with these both should be made to be arch independent. many vars are hardcoded to the compiled arch. this should be changed and both should support building for non-native arches. Overall as a proof of concept I'm proud of how far in a week I managed to come, and how much i got working

    Similar Projects

    Rewrite Distrobox in go (POC) by fabriziosestito

    Description

    Rewriting Distrobox in Go.

    Main benefits:

    • Easier to maintain and to test
    • Adapter pattern for different container backends (LXC, systemd-nspawn, etc.)

    Goals

    • Build a minimal starting point with core commands
    • Keep the CLI interface compatible: existing users shouldn't notice any difference
    • Use a clean Go architecture with adapters for different container backends
    • Keep dependencies minimal and binary size small
    • Benchmark against the original shell script

    Resources

    • Upstream project: https://github.com/89luca89/distrobox/
    • Distrobox site: https://distrobox.it/
    • ArchWiki: https://wiki.archlinux.org/title/Distrobox

    Technical talks at universities by agamez

    Description

    This project aims to empower the next generation of tech professionals by offering hands-on workshops on containerization and Kubernetes, with a strong focus on open-source technologies. By providing practical experience with these cutting-edge tools and fostering a deep understanding of open-source principles, we aim to bridge the gap between academia and industry.

    For now, the scope is limited to Spanish universities, since we already have the contacts and have started some conversations.

    Goals

    • Technical Skill Development: equip students with the fundamental knowledge and skills to build, deploy, and manage containerized applications using open-source tools like Kubernetes.
    • Open-Source Mindset: foster a passion for open-source software, encouraging students to contribute to open-source projects and collaborate with the global developer community.
    • Career Readiness: prepare students for industry-relevant roles by exposing them to real-world use cases, best practices, and open-source in companies.

    Resources

    • Instructors: experienced open-source professionals with deep knowledge of containerization and Kubernetes.
    • SUSE Expertise: leverage SUSE's expertise in open-source technologies to provide insights into industry trends and best practices.

    AI-Powered Unit Test Automation for Agama by joseivanlopez

    The Agama project is a multi-language Linux installer that leverages the distinct strengths of several key technologies:

    • Rust: Used for the back-end services and the core HTTP API, providing performance and safety.
    • TypeScript (React/PatternFly): Powers the modern web user interface (UI), ensuring a consistent and responsive user experience.
    • Ruby: Integrates existing, robust YaST libraries (e.g., yast-storage-ng) to reuse established functionality.

    The Problem: Testing Overhead

    Developing and maintaining code across these three languages requires a significant, tedious effort in writing, reviewing, and updating unit tests for each component. This high cost of testing is a drain on developer resources and can slow down the project's evolution.

    The Solution: AI-Driven Automation

    This project aims to eliminate the manual overhead of unit testing by exploring and integrating AI-driven code generation tools. We will investigate how AI can:

    1. Automatically generate new unit tests as code is developed.
    2. Intelligently correct and update existing unit tests when the application code changes.

    By automating this crucial but monotonous task, we can free developers to focus on feature implementation and significantly improve the speed and maintainability of the Agama codebase.

    Goals

    • Proof of Concept: Successfully integrate and demonstrate an authorized AI tool (e.g., gemini-cli) to automatically generate unit tests.
    • Workflow Integration: Define and document a new unit test automation workflow that seamlessly integrates the selected AI tool into the existing Agama development pipeline.
    • Knowledge Sharing: Establish a set of best practices for using AI in code generation, sharing the learned expertise with the broader team.

    Contribution & Resources

    We are seeking contributors interested in AI-powered development and improving developer efficiency. Whether you have previous experience with code generation tools or are eager to learn, your participation is highly valuable.

    If you want to dive deep into AI for software quality, please reach out and join the effort!

    • Authorized AI Tools: Tools supported by SUSE (e.g., gemini-cli)
    • Focus Areas: Rust, TypeScript, and Ruby components within the Agama project.

    Interesting Links

    RMT.rs: High-Performance Registration Path for RMT using Rust by gbasso

    Description

    The SUSE Repository Mirroring Tool (RMT) is a critical component for managing software updates and subscriptions, especially for our Public Cloud Team (PCT). In a cloud environment, hundreds or even thousands of new SUSE instances (VPS/EC2) can be provisioned simultaneously. Each new instance attempts to register against an RMT server, creating a "thundering herd" scenario.

    We have observed that the current RMT server, written in Ruby, faces performance issues under this high-concurrency registration load. This can lead to request overhead, slow registration times, and outright registration failures, delaying the readiness of new cloud instances.

    This Hackweek project aims to explore a solution by re-implementing the performance-critical registration path in Rust. The goal is to leverage Rust's high performance, memory safety, and first-class concurrency handling to create an alternative registration endpoint that is fast, reliable, and can gracefully manage massive, simultaneous request spikes.

    The new Rust module will be integrated into the existing RMT Ruby application, allowing us to directly compare the performance of both implementations.

    Goals

    The primary objective is to build and benchmark a high-performance Rust-based alternative for the RMT server registration endpoint.

    Key goals for the week:

    1. Analyze & Identify: Dive into the SUSE/rmt Ruby codebase to identify and map out the exact critical path for server registration (e.g., controllers, services, database interactions).
    2. Develop in Rust: Implement a functionally equivalent version of this registration logic in Rust.
    3. Integrate: Explore and implement a method for Ruby/Rust integration to "hot-wire" the new Rust module into the RMT application. This may involve using FFI, or libraries like rb-sys or magnus.
    4. Benchmark: Create a benchmarking script (e.g., using k6, ab, or a custom tool) that simulates the high-concurrency registration load from thousands of clients.
    5. Compare & Present: Conduct a comparative performance analysis (requests per second, latency, success/error rates, CPU/memory usage) between the original Ruby path and the new Rust path. The deliverable will be this data and a summary of the findings.

    Resources

    • RMT Source Code (Ruby):
      • https://github.com/SUSE/rmt
    • RMT Documentation:
      • https://documentation.suse.com/sles/15-SP7/html/SLES-all/book-rmt.html
    • Tooling & Stacks:
      • RMT/Ruby development environment (for running the base RMT)
      • Rust development environment (rustup, cargo)
    • Potential Integration Libraries:
      • rb-sys: https://github.com/oxidize-rb/rb-sys
      • Magnus: https://github.com/matsadler/magnus
    • Benchmarking Tools:
      • k6 (https://k6.io/)
      • ab (ApacheBench)

    Looking at Rust if it could be an interesting programming language by jsmeix

    Get some basic understanding of Rust security related features from a general point of view.

    This Hack Week project is not to learn Rust to become a Rust programmer. This might happen later but it is not the goal of this Hack Week project.

    The goal of this Hack Week project is to evaluate if Rust could be an interesting programming language.

    An interesting programming language must make it easier to write code that is correct and stays correct when over time others maintain and enhance it than the opposite.

    Modal editor in Rust by acervesato

    Description

    To write a modal editor in Rust inspired by vim and having the following features:

    • vim basic motion commands + insert/visual mode
    • multiple buffers with tabs
    • status bar

    It should be written for terminal only using ratatui library and crossterm.

    Goals

    The goal is to start with a functional prototype that can be extended in the future with the following features (in random order):

    • treesitter support + styles
    • fuzzy finder
    • grep finder
    • integration with git
    • tree viewer
    • internal terminal floating window
    • mailing list workflow integration

    Resources

    A CLI for Harvester by mohamed.belgaied

    Harvester does not officially come with a CLI tool, the user is supposed to interact with Harvester mostly through the UI. Though it is theoretically possible to use kubectl to interact with Harvester, the manipulation of Kubevirt YAML objects is absolutely not user friendly. Inspired by tools like multipass from Canonical to easily and rapidly create one of multiple VMs, I began the development of Harvester CLI. Currently, it works but Harvester CLI needs some love to be up-to-date with Harvester v1.0.2 and needs some bug fixes and improvements as well.

    Project Description

    Harvester CLI is a command line interface tool written in Go, designed to simplify interfacing with a Harvester cluster as a user. It is especially useful for testing purposes as you can easily and rapidly create VMs in Harvester by providing a simple command such as: harvester vm create my-vm --count 5 to create 5 VMs named my-vm-01 to my-vm-05.

    asciicast

    Harvester CLI is functional but needs a number of improvements: up-to-date functionality with Harvester v1.0.2 (some minor issues right now), modifying the default behaviour to create an opensuse VM instead of an ubuntu VM, solve some bugs, etc.

    Github Repo for Harvester CLI: https://github.com/belgaied2/harvester-cli

    Done in previous Hackweeks

    • Create a Github actions pipeline to automatically integrate Harvester CLI to Homebrew repositories: DONE
    • Automatically package Harvester CLI for OpenSUSE / Redhat RPMs or DEBs: DONE

    Goal for this Hackweek

    The goal for this Hackweek is to bring Harvester CLI up-to-speed with latest Harvester versions (v1.3.X and v1.4.X), and improve the code quality as well as implement some simple features and bug fixes.

    Some nice additions might be: * Improve handling of namespaced objects * Add features, such as network management or Load Balancer creation ? * Add more unit tests and, why not, e2e tests * Improve CI * Improve the overall code quality * Test the program and create issues for it

    Issue list is here: https://github.com/belgaied2/harvester-cli/issues

    Resources

    The project is written in Go, and using client-go the Kubernetes Go Client libraries to communicate with the Harvester API (which is Kubernetes in fact). Welcome contributions are:

    • Testing it and creating issues
    • Documentation
    • Go code improvement

    What you might learn

    Harvester CLI might be interesting to you if you want to learn more about:

    • GitHub Actions
    • Harvester as a SUSE Product
    • Go programming language
    • Kubernetes API
    • Kubevirt API objects (Manipulating VMs and VM Configuration in Kubernetes using Kubevirt)

    GHC-9.14 and split Hadrian from GHC build by osukup

    Description

    Prepare openSUSE Tumbleweed project for new GHC Haskell compiler and separate builder (Hadrian) from GHC build

    Goals

    • have GHC-9.14 project with working compiler and if possible filled with packageset
    • have Hadrian in own package built with bootstrap compiler to separate Hadrian bootstrap from ghc bootstrap

    Resources

    devel:languages:haskell

    d:l:h:ghc-9.12.x

    opensuse Haskell rpm macros

    opensuse haskell package gen project