knots
a project by LarsMB
Updated over 3 years ago. 2 hacker ♥️. 2 followers.

Summary

knots aims to be a multi-node S3 benchmark and load generator written in Rust and on top of WebAssembly, waPC, and waSCC.

Goal for this Hackweek

My goal for HackWeek is to have a prototype actor that can at least run a few GET/PUT requests against a single bucket and tells me how long that took.

Collaboration

Since this is a training wheels exercise for me on multiple fronts (Rust, and the entire WASM/waPC/waSCC world), I'm not actively looking for other contributors yet. I don't learn as much if someone else does the work, and I have precisely zero experience with any of the technologies involved except S3 ;-)

But if you have experience, I don't mind chatting!

Especially if your project maybe includes a krustlet/waSCC ops environment to try and deploy the actor on.

I just want an S3 benchmark, now

I recommend you take a look at these:

  • fio is an excellent C tool for benchmarking all sorts of data IO mechanisms, and is very useful for comparing protocols. I contributed a simple S3 module to it a while ago. If your primary interest is data throughput and latency, fio is your tool of choice.
  • MinIO warp is a versatile tool written in Go for benchmarking various aspects of S3 and probably the most mature choice.
  • cosbench is a very well-known tool for benchmarking S3 written in Java.

Join this project

Looking for hackers with the skills:

s3 wasm wascc wasi ceph krustlet rust containerless

This project is part of:

Hack Week 20

Activity

  • almost 4 years ago: jluis liked this project.
  • almost 4 years ago: LarsMB liked this project.
  • almost 4 years ago: LarsMB added keyword "containerless" to this project.
  • almost 4 years ago: LarsMB started this project.
  • almost 4 years ago: LarsMB added keyword "ceph" to this project.
  • almost 4 years ago: LarsMB added keyword "krustlet" to this project.
  • almost 4 years ago: LarsMB added keyword "rust" to this project.
  • almost 4 years ago: LarsMB added keyword "s3" to this project.
  • almost 4 years ago: LarsMB added keyword "wasm" to this project.
  • almost 4 years ago: LarsMB added keyword "wascc" to this project.
  • almost 4 years ago: LarsMB added keyword "wasi" to this project.
  • almost 4 years ago: LarsMB originated this project.


    • Comments

    Be the first to comment!

    Similar Projects

    Better diff'ing experience by MSirringhaus

    Description

    For diff-ing directories, I usually like to use meld, but it struggles a lot with large trees. Experiment with writing a TUI meld-clone for diffing directories and files

    Goals

    Get first prototype going of a TUI that can show

    • diffs of text-files
    • diffs of directories.

    Stretch goals

    • Themes
    • Filters (no whitespace, etc.)
    • Live config changes (Show/hide line numbers, etc.)

    Grapesss: a physical Shamir's Secret Sharing application [ESP32-C3 + Mobile] by ecandino

    drawing

    Description

    A couple of years ago I created StegoSecretS, a small cli used to encrypt and split a secret into multiple keys, using the Shamir's Secret Sharing algorithm.

    The idea is to re-implement the project using physical devices. One device alone will be useless, but when close together they can be used to decrypt the secret.

    On a practical side the user encrypts the secret with a mobile application. The same application is used to split the secret, and load the partial keys into different micro-controllers. Another user will be able to decrypt the secret only having at least N devices close together (using the application).

    I'm planning to use a couple of ESP32-C3 I bought, and build a very simple Android mobile application.

    Goals

    • Learn about Rust and micro-controllers (ESP32-C3)
    • Learn about mobile applications (Android and Kotlin)

    Resources

    Hacking on sched_ext by flonnegren

    Description

    Sched_ext upstream has some interesting issues open for grabs:

    Goals

    Send patches to sched_ext upstream

    Also set up perfetto to trace some of the example schedulers.

    Resources

    https://github.com/sched-ext/scx

    Write an url shortener in Rust (And learn in the way) by szarate

    So I have 469.icu :), it's currently doing nothing... (and for sale) but in the meantime, I'd like to write an url shortener from scratch and deploy it on my own server

    https://github.com/foursixnine/url-manager-rs/tree/main

    Agama installer on-line demo by lslezak

    Description

    The Agama installer provides a quite complex user interface. We have some screenshots on the web page but as it is basically a web application it would be nice to have some on-line demo where users could click and check it live.

    The problem is that the Agama server directly accesses the hardware (storage probing) and loads installation repositories. We cannot easily mock this in the on-line demo so the easiest way is to have just a read-only demo. You could explore the configuration options but you could not change anything, all changes would be ignored.

    The read-only demo would be a bit limited but I still think it would be useful for potential users get the feeling of the new Agama installer and get familiar with it before using in a real installation.

    As a proof of concept I already created this on-line demo.

    The implementation basically builds Agama in two modes - recording mode where it saves all REST API responses and replay mode where it for the REST API requests returns the previously recorded responses. Recording in the browser is inconvenient and error prone, there should be some scripting instead (see below).

    Goals

    • Create an Agama on-line demo which can be easily tested by users
    • The Agama installer is still in alpha phase and in active development, the online demo needs to be easily rebuilt with the latest Agama version
    • Ideally there should be some automation so the demo page is rebuilt automatically without any developer interactions (once a day or week?)

    TODO

    • Use OpenAPI to get all Agama REST API endpoints, write a script which queries all the endpoints automatically and saves the collected data to a file (see this related PR).
    • Write a script for starting an Agama VM (use libvirt/qemu?), the script should ensure we always use the same virtual HW so if we need to dump the latest REST API state we get the same (or very similar data). This should ensure the demo page does not change much regarding the storage proposal etc...
    • Fix changing the product, currently it gets stuck after clicking the "Select" button.
    • Move the mocking data (the recorded REST API responses) outside the Agama sources, it's too big and will be probably often updated. To avoid messing the history keep it in a separate GitHub repository
    • Allow changing the UI language
    • Display some note (watermark) in the page so it is clear it is a read-only demo (probably with some version or build date to know how old it is)
    • Automation for building new demo page from the latest sources. There should be some check which ensures the recorded data still matches the OpenAPI specification.

    Changing the UI language

    This will be quite tricky because selecting the proper translation file is done on the server side. We would probably need to completely re-implement the logic in the browser side and adapt the server for that.

    Also some REST API responses contain translated texts (storage proposal, pattern names in software). We would need to query the respective endpoints in all supported languages and return the correct response in runtime according to the currently selected language.

    Resources