Scalability improvement for running tiedot database on SMP systems
tiedot is a general purpose, document database engine designed for both embeddable usage and running stand-alone DB server, implemented in Go.
Despite the effort of partitioning collection data and indexes, a single tiedot process still cannot scale beyond 4 CPU cores due to limitations of both data structure and Go runtime. Last year, an attempt began to partition data across multiple tiedot processes, and use IPC for communication and database logics. Initial benchmarks show promising scalability improvement using the new implementation. So let us complete the new implementation together, to bring the most scalable & embeddable DB engine for Go programs!
Project URL: https://github.com/HouzuoGuo/tiedot
Looking for hackers with the skills:
This project is part of:
Hack Week 12
Activity
Comments
Similar Projects
A CLI for Harvester by mohamed.belgaied
Harvester does not officially come with a CLI tool, the user is supposed to interact with Harvester mostly through the UI. Though it is theoretically possible to use kubectl to interact with Harvester, the manipulation of Kubevirt YAML objects is absolutely not user friendly. Inspired by tools like multipass from Canonical to easily and rapidly create one of multiple VMs, I began the development of Harvester CLI. Currently, it works but Harvester CLI needs some love to be up-to-date with Harvester v1.0.2 and needs some bug fixes and improvements as well.
Project Description
Harvester CLI is a command line interface tool written in Go, designed to simplify interfacing with a Harvester cluster as a user. It is especially useful for testing purposes as you can easily and rapidly create VMs in Harvester by providing a simple command such as:
harvester vm create my-vm --count 5
to create 5 VMs named my-vm-01 to my-vm-05.
Harvester CLI is functional but needs a number of improvements: up-to-date functionality with Harvester v1.0.2 (some minor issues right now), modifying the default behaviour to create an opensuse VM instead of an ubuntu VM, solve some bugs, etc.
Github Repo for Harvester CLI: https://github.com/belgaied2/harvester-cli
Done in previous Hackweeks
- Create a Github actions pipeline to automatically integrate Harvester CLI to Homebrew repositories: DONE
- Automatically package Harvester CLI for OpenSUSE / Redhat RPMs or DEBs: DONE
Goal for this Hackweek
The goal for this Hackweek is to bring Harvester CLI up-to-speed with latest Harvester versions (v1.3.X and v1.4.X), and improve the code quality as well as implement some simple features and bug fixes.
Some nice additions might be: * Improve handling of namespaced objects * Add features, such as network management or Load Balancer creation ? * Add more unit tests and, why not, e2e tests * Improve CI * Improve the overall code quality * Test the program and create issues for it
Issue list is here: https://github.com/belgaied2/harvester-cli/issues
Resources
The project is written in Go, and using client-go the Kubernetes Go Client libraries to communicate with the Harvester API (which is Kubernetes in fact).
Welcome contributions are:
- Testing it and creating issues
- Documentation
- Go code improvement
What you might learn
Harvester CLI might be interesting to you if you want to learn more about:
- GitHub Actions
- Harvester as a SUSE Product
- Go programming language
- Kubernetes API
- Kubevirt API objects (Manipulating VMs and VM Configuration in Kubernetes using Kubevirt)
Add support for todo.sr.ht to git-bug by mcepl
Description
I am a big fan of distributed issue tracking and the best (and possibly) only credible such issue tracker is now git-bug. It has bridges to another centralized issue trackers, so user can download (and modify) issues on GitHub, GitLab, Launchpad, Jira). I am also a fan of SourceHut, which has its own issue tracker, so I would like it bridge the two. Alas, I don’t know much about Go programming language (which the git-bug is written) and absolutely nothing about GraphQL (which todo.sr.ht uses for communication). AI to the rescue. I would like to vibe code (and eventually debug and make functional) bridge to the SourceHut issue tracker.
Goals
Functional fix for https://github.com/git-bug/git-bug/issues/1024
Resources
- anybody how actually understands how GraphQL and authentication on SourceHut (OAuth2) works
Play with the userfaultfd(2) system call and download on demand using HTTP Range Requests with Golang by rbranco
Description
The userfaultfd(2) is a cool system call to handle page faults in user-space. This should allow me to list the contents of an ISO or similar archive without downloading the whole thing. The userfaultfd(2) part can also be done in theory with the PROT_NONE mprotect + SIGSEGV trick, for complete Unix portability, though reportedly being slower.
Goals
- Create my own library for userfaultfd(2) in Golang.
- Create my own library for HTTP Range Requests.
- Complete portability with Unix.
- Benchmarks.
- Contribute some tests to LTP.
Resources
- https://docs.kernel.org/admin-guide/mm/userfaultfd.html
- https://www.cons.org/cracauer/cracauer-userfaultfd.html
HTTP API for nftables by crameleon
Background
The idea originated in https://progress.opensuse.org/issues/164060 and is about building RESTful API which translates authorized HTTP requests to operations in nftables, possibly utilizing libnftables-json(5).
Originally, I started developing such an interface in Go, utilizing https://github.com/google/nftables. The conversion of string networks to nftables set elements was problematic (unfortunately no record of details), and I started a second attempt in Python, which made interaction much simpler thanks to native nftables Python bindings.
Goals
- Find and track the issue with google/nftables
- Revisit and polish the Go or Python code (prefer Go, but possibly depends on implementing missing functionality), primarily the server component
- Finish functionality to interact with nftables sets (retrieving and updating elements), which are of interest for the originating issue
- Align test suite
- Packaging
Resources
- https://git.netfilter.org/nftables/tree/py/src/nftables.py
- https://git.com.de/Georg/nftables-http-api (to be moved to GitHub)
- https://build.opensuse.org/package/show/home:crameleon:containers/pytest-nftables-container
Results
- Started new https://github.com/tacerus/nftables-http-api.
- First Go nftables issue was related to set elements needing to be added with different start and end addresses - coincidentally, this was recently discovered by someone else, who added a useful helper function for this: https://github.com/google/nftables/pull/342.
- Further improvements submitted: https://github.com/google/nftables/pull/347.
Side results
Upon starting to unify the structure and implementing more functionality, missing JSON output support was noticed for some subcommands in libnftables. Submitted patches here as well:
- https://lore.kernel.org/netfilter-devel/20251203131736.4036382-2-georg@syscid.com/T/#u
Create a go module to wrap happy-compta.fr by cbosdonnat
Description
https://happy-compta.fr is a tool for french work councils simple book keeping. While it does the job, it has no API to work with and it is tedious to enter loads of operations.
Goals
Write a go client module to be used as an API to programmatically manipulate the tool.
Writing an example tool to load data from a CSV file would be good too.