Scalability improvement for running tiedot database on SMP systems
an invention by guohouzuo
Updated almost 4 years ago. 2 hacker ♥️.

tiedot is a general purpose, document database engine designed for both embeddable usage and running stand-alone DB server, implemented in Go.

Despite the effort of partitioning collection data and indexes, a single tiedot process still cannot scale beyond 4 CPU cores due to limitations of both data structure and Go runtime. Last year, an attempt began to partition data across multiple tiedot processes, and use IPC for communication and database logics. Initial benchmarks show promising scalability improvement using the new implementation. So let us complete the new implementation together, to bring the most scalable & embeddable DB engine for Go programs!

Project URL: https://github.com/HouzuoGuo/tiedot

Looking for hackers with the skills:

go databases development

This project is part of:

Hack Week 12

Activity

  • almost 11 years ago: psankar liked this project.
  • almost 11 years ago: guohouzuo liked this project.
  • almost 11 years ago: guohouzuo added keyword "go" to this project.
  • almost 11 years ago: guohouzuo added keyword "databases" to this project.
  • almost 11 years ago: guohouzuo added keyword "development" to this project.
  • almost 11 years ago: guohouzuo started this project.
  • almost 11 years ago: guohouzuo originated this project.


    • Comments

    • psankar
      almost 11 years ago by psankar | Reply

      I would have joined this had I had some time. All the best.

    Similar Projects

    Play with the userfaultfd(2) system call and download on demand using HTTP Range Requests with Golang by rbranco

    Description

    The userfaultfd(2) is a cool system call to handle page faults in user-space. This should allow me to list the contents of an ISO or similar archive without downloading the whole thing. The userfaultfd(2) part can also be done in theory with the PROT_NONE mprotect + SIGSEGV trick, for complete Unix portability, though reportedly being slower.

    Goals

    1. Create my own library for userfaultfd(2) in Golang.
    2. Create my own library for HTTP Range Requests.
    3. Complete portability with Unix.
    4. Benchmarks.
    5. Contribute some tests to LTP.

    Resources

    1. https://docs.kernel.org/admin-guide/mm/userfaultfd.html
    2. https://www.cons.org/cracauer/cracauer-userfaultfd.html

    Create a go module to wrap happy-compta.fr by cbosdonnat

    Description

    https://happy-compta.fr is a tool for french work councils simple book keeping. While it does the job, it has no API to work with and it is tedious to enter loads of operations.

    Goals

    Write a go client module to be used as an API to programmatically manipulate the tool.

    Writing an example tool to load data from a CSV file would be good too.

    Rewrite Distrobox in go (POC) by fabriziosestito

    Description

    Rewriting Distrobox in Go.

    Main benefits:

    • Easier to maintain and to test
    • Adapter pattern for different container backends (LXC, systemd-nspawn, etc.)

    Goals

    • Build a minimal starting point with core commands
    • Keep the CLI interface compatible: existing users shouldn't notice any difference
    • Use a clean Go architecture with adapters for different container backends
    • Keep dependencies minimal and binary size small
    • Benchmark against the original shell script

    Resources

    • Upstream project: https://github.com/89luca89/distrobox/
    • Distrobox site: https://distrobox.it/
    • ArchWiki: https://wiki.archlinux.org/title/Distrobox

    Contribute to terraform-provider-libvirt by pinvernizzi

    Description

    The SUSE Manager (SUMA) teams' main tool for infrastructure automation, Sumaform, largely relies on terraform-provider-libvirt. That provider is also widely used by other teams, both inside and outside SUSE.

    It would be good to help the maintainers of this project and give back to the community around it, after all the amazing work that has been already done.

    If you're interested in any of infrastructure automation, Terraform, virtualization, tooling development, Go (...) it is also a good chance to learn a bit about them all by putting your hands on an interesting, real-use-case and complex project.

    Goals

    • Get more familiar with Terraform provider development and libvirt bindings in Go
    • Solve some issues and/or implement some features
    • Get in touch with the community around the project

    Resources

    HTTP API for nftables by crameleon

    Background

    The idea originated in https://progress.opensuse.org/issues/164060 and is about building RESTful API which translates authorized HTTP requests to operations in nftables, possibly utilizing libnftables-json(5).

    Originally, I started developing such an interface in Go, utilizing https://github.com/google/nftables. The conversion of string networks to nftables set elements was problematic (unfortunately no record of details), and I started a second attempt in Python, which made interaction much simpler thanks to native nftables Python bindings.

    Goals

    1. Find and track the issue with google/nftables
    2. Revisit and polish the Go or Python code (prefer Go, but possibly depends on implementing missing functionality), primarily the server component
    3. Finish functionality to interact with nftables sets (retrieving and updating elements), which are of interest for the originating issue
    4. Align test suite
    5. Packaging

    Resources

    • https://git.netfilter.org/nftables/tree/py/src/nftables.py
    • https://git.com.de/Georg/nftables-http-api (to be moved to GitHub)
    • https://build.opensuse.org/package/show/home:crameleon:containers/pytest-nftables-container

    Results

    • Started new https://github.com/tacerus/nftables-http-api.
    • First Go nftables issue was related to set elements needing to be added with different start and end addresses - coincidentally, this was recently discovered by someone else, who added a useful helper function for this: https://github.com/google/nftables/pull/342.
    • Further improvements submitted: https://github.com/google/nftables/pull/347.

    Side results

    Upon starting to unify the structure and implementing more functionality, missing JSON output support was noticed for some subcommands in libnftables. Submitted patches here as well:

    • https://lore.kernel.org/netfilter-devel/20251203131736.4036382-2-georg@syscid.com/T/#u