knots
a project by LarsMB
Updated almost 5 years ago. 2 hacker ♥️. 2 followers.

Summary

knots aims to be a multi-node S3 benchmark and load generator written in Rust and on top of WebAssembly, waPC, and waSCC.

Goal for this Hackweek

My goal for HackWeek is to have a prototype actor that can at least run a few GET/PUT requests against a single bucket and tells me how long that took.

Collaboration

Since this is a training wheels exercise for me on multiple fronts (Rust, and the entire WASM/waPC/waSCC world), I'm not actively looking for other contributors yet. I don't learn as much if someone else does the work, and I have precisely zero experience with any of the technologies involved except S3 ;-)

But if you have experience, I don't mind chatting!

Especially if your project maybe includes a krustlet/waSCC ops environment to try and deploy the actor on.

I just want an S3 benchmark, now

I recommend you take a look at these:

  • fio is an excellent C tool for benchmarking all sorts of data IO mechanisms, and is very useful for comparing protocols. I contributed a simple S3 module to it a while ago. If your primary interest is data throughput and latency, fio is your tool of choice.
  • MinIO warp is a versatile tool written in Go for benchmarking various aspects of S3 and probably the most mature choice.
  • cosbench is a very well-known tool for benchmarking S3 written in Java.

Join this project

Looking for hackers with the skills:

s3 wasm wascc wasi ceph krustlet rust containerless

This project is part of:

Hack Week 20

Activity

  • almost 5 years ago: jluis liked this project.
  • almost 5 years ago: LarsMB liked this project.
  • almost 5 years ago: LarsMB added keyword "containerless" to this project.
  • almost 5 years ago: LarsMB started this project.
  • almost 5 years ago: LarsMB added keyword "ceph" to this project.
  • almost 5 years ago: LarsMB added keyword "krustlet" to this project.
  • almost 5 years ago: LarsMB added keyword "rust" to this project.
  • almost 5 years ago: LarsMB added keyword "s3" to this project.
  • almost 5 years ago: LarsMB added keyword "wasm" to this project.
  • almost 5 years ago: LarsMB added keyword "wascc" to this project.
  • almost 5 years ago: LarsMB added keyword "wasi" to this project.
  • almost 5 years ago: LarsMB originated this project.


    • Comments

    Be the first to comment!

    Similar Projects

    Updatecli Autodiscovery supporting WASM plugins by olblak

    Description

    Updatecli is a Golang Update policy engine that allow to write Update policies in YAML manifest. Updatecli already has a plugin ecosystem for common update strategies such as automating Dockerfile or Kubernetes manifest from Git repositories.

    This is what we call autodiscovery where Updatecli generate manifest and apply them dynamically based on some context.

    Obviously, the Updatecli project doesn't accept plugins specific to an organization.

    I saw project using different languages such as python, C#, or JS to generate those manifest.

    It would be great to be able to share and reuse those specific plugins

    During the HackWeek, I'll hang on the Updatecli matrix channel

    https://matrix.to/#/#Updatecli_community:gitter.im

    Goals

    Implement autodiscovery plugins using WASM. I am planning to experiment with https://github.com/extism/extism

    To build a simple WASM autodiscovery plugin and run it from Updatecli

    Resources

    • https://github.com/extism/extism
    • https://github.com/updatecli/updatecli
    • https://www.updatecli.io/docs/core/autodiscovery/
    • https://matrix.to/#/#Updatecli_community:gitter.im

    RMT.rs: High-Performance Registration Path for RMT using Rust by gbasso

    Description

    The SUSE Repository Mirroring Tool (RMT) is a critical component for managing software updates and subscriptions, especially for our Public Cloud Team (PCT). In a cloud environment, hundreds or even thousands of new SUSE instances (VPS/EC2) can be provisioned simultaneously. Each new instance attempts to register against an RMT server, creating a "thundering herd" scenario.

    We have observed that the current RMT server, written in Ruby, faces performance issues under this high-concurrency registration load. This can lead to request overhead, slow registration times, and outright registration failures, delaying the readiness of new cloud instances.

    This Hackweek project aims to explore a solution by re-implementing the performance-critical registration path in Rust. The goal is to leverage Rust's high performance, memory safety, and first-class concurrency handling to create an alternative registration endpoint that is fast, reliable, and can gracefully manage massive, simultaneous request spikes.

    The new Rust module will be integrated into the existing RMT Ruby application, allowing us to directly compare the performance of both implementations.

    Goals

    The primary objective is to build and benchmark a high-performance Rust-based alternative for the RMT server registration endpoint.

    Key goals for the week:

    1. Analyze & Identify: Dive into the SUSE/rmt Ruby codebase to identify and map out the exact critical path for server registration (e.g., controllers, services, database interactions).
    2. Develop in Rust: Implement a functionally equivalent version of this registration logic in Rust.
    3. Integrate: Explore and implement a method for Ruby/Rust integration to "hot-wire" the new Rust module into the RMT application. This may involve using FFI, or libraries like rb-sys or magnus.
    4. Benchmark: Create a benchmarking script (e.g., using k6, ab, or a custom tool) that simulates the high-concurrency registration load from thousands of clients.
    5. Compare & Present: Conduct a comparative performance analysis (requests per second, latency, success/error rates, CPU/memory usage) between the original Ruby path and the new Rust path. The deliverable will be this data and a summary of the findings.

    Resources

    • RMT Source Code (Ruby):
      • https://github.com/SUSE/rmt
    • RMT Documentation:
      • https://documentation.suse.com/sles/15-SP7/html/SLES-all/book-rmt.html
    • Tooling & Stacks:
      • RMT/Ruby development environment (for running the base RMT)
      • Rust development environment (rustup, cargo)
    • Potential Integration Libraries:
      • rb-sys: https://github.com/oxidize-rb/rb-sys
      • Magnus: https://github.com/matsadler/magnus
    • Benchmarking Tools:
      • k6 (https://k6.io/)
      • ab (ApacheBench)

    Mail client with mailing list workflow support in Rust by acervesato

    Description

    To create a mail user interface using Rust programming language, supporting mailing list patches workflow. I know, aerc is already there, but I would like to create something simpler, without integrated protocols. Just a plain user interface that is using some crates to read and create emails which are fetched and sent via external tools.

    I already know Rust, but not the async support, which is needed in this case in order to handle events inside the mail folder and to send notifications.

    Goals

    • simple user interface in the style of aerc, with some vim keybindings for motions and search
    • automatic run of external tools (like mbsync) for checking emails
    • automatic run commands for notifications
    • apply patch set from ML
    • tree-sitter support with styles

    Resources

    • ratatui: user interface (https://ratatui.rs/)
    • notify: folder watcher (https://docs.rs/notify/latest/notify/)
    • mail-parser: parser for emails (https://crates.io/crates/mail-parser)
    • mail-builder: create emails in proper format (https://docs.rs/mail-builder/latest/mail_builder/)
    • gitpatch: ML support (https://crates.io/crates/gitpatch)
    • tree-sitter-rust: support for mail format (https://crates.io/crates/tree-sitter)

    OpenPlatform Self-Service Portal by tmuntan1

    Description

    In SUSE IT, we developed an internal developer platform for our engineers using SUSE technologies such as RKE2, SUSE Virtualization, and Rancher. While it works well for our existing users, the onboarding process could be better.

    To improve our customer experience, I would like to build a self-service portal to make it easy for people to accomplish common actions. To get started, I would have the portal create Jira SD tickets for our customers to have better information in our tickets, but eventually I want to add automation to reduce our workload.

    Goals

    • Build a frontend website (Angular) that helps customers create Jira SD tickets.
    • Build a backend (Rust with Axum) for the backend, which would do all the hard work for the frontend.

    Resources (SUSE VPN only)

    • development site: https://ui-dev.openplatform.suse.com/login?returnUrl=%2Fopenplatform%2Fforms
    • https://gitlab.suse.de/itpe/core/open-platform/op-portal/backend
    • https://gitlab.suse.de/itpe/core/open-platform/op-portal/frontend

    Learn how to use the Relm4 Rust GUI crate by xiaoguang_wang

    Relm4 is based on gtk4-rs and compatible with libadwaita. The gtk4-rs crate provides all the tools necessary to develop applications. Building on this foundation, Relm4 makes developing more idiomatic, simpler, and faster.

    https://github.com/Relm4/Relm4

    Learn a bit of embedded programming with Rust in a micro:bit v2 by aplanas

    Description

    micro:bit is a small single board computer with a ARM Cortex-M4 with the FPU extension, with a very constrain amount of memory and a bunch of sensors and leds.

    The board is very well documented, with schematics and code for all the features available, so is an excellent platform for learning embedded programming.

    Rust is a system programming language that can generate ARM code, and has crates (libraries) to access the micro:bit hardware. There is plenty documentation about how to make small programs that will run in the micro:bit.

    Goals

    Start learning about embedded programming in Rust, and maybe make some code to the small KS4036F Robot car from keyestudio.

    Resources

    Diary

    Day 1

    • Start reading https://mb2.implrust.com/abstraction-layers.html
    • Prepare the dev environment (cross compiler, probe-rs)
    • Flash first code in the board (blinky led)
    • Checking differences between BSP and HAL
    • Compile and install a more complex example, with stack protection
    • Reading about the simplicity of xtask, as alias for workspace execution
    • Reading the CPP code of the official micro:bit libraries. They have a font!

    Day 2

    • There are multiple BSP for the microbit. One is using async code for non-blocking operations
    • Download and study a bit the API for microbit-v2, the nRF official crate
    • Take a look of the KS4036F programming, seems that the communication is multiplexed via I2C
    • The motor speed can be selected via PWM (pulse with modulation): power it longer (high frequency), and it will increase the speed
    • Scrolling some text
    • Debug by printing! defmt is a crate that can be used with probe-rs to emit logs
    • Start reading input from the board: buttons
    • The logo can be touched and detected as a floating point value

    Day 3

    • A bit confused how to read the float value from a pin