Scalability improvement for running tiedot database on SMP systems
an invention by guohouzuo
Updated almost 4 years ago. 2 hacker ♥️.

tiedot is a general purpose, document database engine designed for both embeddable usage and running stand-alone DB server, implemented in Go.

Despite the effort of partitioning collection data and indexes, a single tiedot process still cannot scale beyond 4 CPU cores due to limitations of both data structure and Go runtime. Last year, an attempt began to partition data across multiple tiedot processes, and use IPC for communication and database logics. Initial benchmarks show promising scalability improvement using the new implementation. So let us complete the new implementation together, to bring the most scalable & embeddable DB engine for Go programs!

Project URL: https://github.com/HouzuoGuo/tiedot

Looking for hackers with the skills:

go databases development

This project is part of:

Hack Week 12

Activity

  • almost 11 years ago: psankar liked this project.
  • almost 11 years ago: guohouzuo liked this project.
  • almost 11 years ago: guohouzuo added keyword "go" to this project.
  • almost 11 years ago: guohouzuo added keyword "databases" to this project.
  • almost 11 years ago: guohouzuo added keyword "development" to this project.
  • almost 11 years ago: guohouzuo started this project.
  • almost 11 years ago: guohouzuo originated this project.


    • Comments

    • psankar
      almost 11 years ago by psankar | Reply

      I would have joined this had I had some time. All the best.

    Similar Projects

    Play with the userfaultfd(2) system call and download on demand using HTTP Range Requests with Golang by rbranco

    Description

    The userfaultfd(2) is a cool system call to handle page faults in user-space. This should allow me to list the contents of an ISO or similar archive without downloading the whole thing. The userfaultfd(2) part can also be done in theory with the PROT_NONE mprotect + SIGSEGV trick, for complete Unix portability, though reportedly being slower.

    Goals

    1. Create my own library for userfaultfd(2) in Golang.
    2. Create my own library for HTTP Range Requests.
    3. Complete portability with Unix.
    4. Benchmarks.
    5. Contribute some tests to LTP.

    Resources

    1. https://docs.kernel.org/admin-guide/mm/userfaultfd.html
    2. https://www.cons.org/cracauer/cracauer-userfaultfd.html

    SUSE Health Check Tools by roseswe

    SUSE HC Tools Overview

    A collection of tools written in Bash or Go 1.24++ to make life easier with handling of a bunch of tar.xz balls created by supportconfig.

    Background: For SUSE HC we receive a bunch of supportconfig tar balls to check them for misconfiguration, areas for improvement or future changes.

    Main focus on these HC are High Availability (pacemaker), SLES itself and SAP workloads, esp. around the SUSE best practices.

    Goals

    • Overall improvement of the tools
    • Adding new collectors
    • Add support for SLES16

    Resources

    csv2xls* example.sh go.mod listprodids.txt sumtext* trails.go README.md csv2xls.go exceltest.go go.sum m.sh* sumtext.go vercheck.py* config.ini csvfiles/ getrpm* listprodids* rpmdate.sh* sumxls* verdriver* credtest.go example.py getrpm.go listprodids.go sccfixer.sh* sumxls.go verdriver.go

    docollall.sh* extracthtml.go gethostnamectl* go.sum numastat.go cpuvul* extractcluster.go firmwarebug* gethostnamectl.go m.sh* numastattest.go cpuvul.go extracthtml* firmwarebug.go go.mod numastat* xtr_cib.sh*

    $ getrpm -r pacemaker >> Product ID: 2795 (SUSE Linux Enterprise Server for SAP Applications 15 SP7 x86_64), RPM Name: +--------------+----------------------------+--------+--------------+--------------------+ | Package Name | Version | Arch | Release | Repository | +--------------+----------------------------+--------+--------------+--------------------+ | pacemaker | 2.1.10+20250718.fdf796ebc8 | x86_64 | 150700.3.3.1 | sle-ha/15.7/x86_64 | | pacemaker | 2.1.9+20250410.471584e6a2 | x86_64 | 150700.1.9 | sle-ha/15.7/x86_64 | +--------------+----------------------------+--------+--------------+--------------------+ Total packages found: 2

    Create a go module to wrap happy-compta.fr by cbosdonnat

    Description

    https://happy-compta.fr is a tool for french work councils simple book keeping. While it does the job, it has no API to work with and it is tedious to enter loads of operations.

    Goals

    Write a go client module to be used as an API to programmatically manipulate the tool.

    Writing an example tool to load data from a CSV file would be good too.

    Rewrite Distrobox in go (POC) by fabriziosestito

    Description

    Rewriting Distrobox in Go.

    Main benefits:

    • Easier to maintain and to test
    • Adapter pattern for different container backends (LXC, systemd-nspawn, etc.)

    Goals

    • Build a minimal starting point with core commands
    • Keep the CLI interface compatible: existing users shouldn't notice any difference
    • Use a clean Go architecture with adapters for different container backends
    • Keep dependencies minimal and binary size small
    • Benchmark against the original shell script

    Resources

    • Upstream project: https://github.com/89luca89/distrobox/
    • Distrobox site: https://distrobox.it/
    • ArchWiki: https://wiki.archlinux.org/title/Distrobox

    Add support for todo.sr.ht to git-bug by mcepl

    Description

    I am a big fan of distributed issue tracking and the best (and possibly) only credible such issue tracker is now git-bug. It has bridges to another centralized issue trackers, so user can download (and modify) issues on GitHub, GitLab, Launchpad, Jira). I am also a fan of SourceHut, which has its own issue tracker, so I would like it bridge the two. Alas, I don’t know much about Go programming language (which the git-bug is written) and absolutely nothing about GraphQL (which todo.sr.ht uses for communication). AI to the rescue. I would like to vibe code (and eventually debug and make functional) bridge to the SourceHut issue tracker.

    Goals

    Functional fix for https://github.com/git-bug/git-bug/issues/1024

    Resources

    • anybody how actually understands how GraphQL and authentication on SourceHut (OAuth2) works