Scalability improvement for running tiedot database on SMP systems
an invention by guohouzuo
Updated almost 3 years ago. 2 hacker ♥️.

tiedot is a general purpose, document database engine designed for both embeddable usage and running stand-alone DB server, implemented in Go.

Despite the effort of partitioning collection data and indexes, a single tiedot process still cannot scale beyond 4 CPU cores due to limitations of both data structure and Go runtime. Last year, an attempt began to partition data across multiple tiedot processes, and use IPC for communication and database logics. Initial benchmarks show promising scalability improvement using the new implementation. So let us complete the new implementation together, to bring the most scalable & embeddable DB engine for Go programs!

Project URL: https://github.com/HouzuoGuo/tiedot

Looking for hackers with the skills:

go databases development

This project is part of:

Hack Week 12

Activity

  • over 9 years ago: psankar liked this project.
  • over 9 years ago: guohouzuo liked this project.
  • almost 10 years ago: guohouzuo added keyword "go" to this project.
  • almost 10 years ago: guohouzuo added keyword "databases" to this project.
  • almost 10 years ago: guohouzuo added keyword "development" to this project.
  • almost 10 years ago: guohouzuo started this project.
  • almost 10 years ago: guohouzuo originated this project.


    • Comments

    • psankar
      over 9 years ago by psankar | Reply

      I would have joined this had I had some time. All the best.

    Similar Projects

    Jenny Static Site Generator by adam.pickering

    Description

    For my personal site I have been using hugo. It works, but I am not satisfied: every time I want to make a change (which is infrequently) I have to read through the documentation again to understand how hugo works. I don't find the documentation easy to use, and the structure of the repository that hugo requires is unintuitive/more complex than what I need. So, I have decided to write my own simple static site generator in Go. It is named Jenny, after my wife.

    Goals

    • Pages can be written in markdown (which is automatically converted to HTML), but other file types are also allowed
    • Easy to understand and use
      • Intuitive, simple design
      • Clear documentation
      • Hot reloading
      • Binaries provided for download
    • Future maintenance is easy
      • Automated releases

    Resources

    https://github.com/adamkpickering/jenny

    WebUI for your data by avicenzi

    A single place to view every bit of data you have.

    Problem

    You have too much data and you are a data hoarder.

    • Family photos and videos.
    • Lots of eBooks, TV Shows, Movies, and else.
    • Boxes full of papers (taxes, invoices, IDs, certificates, exams, and else).
    • Bank account statements (multiple currencies, countries, and people).

    Maybe you have some data on S3, some on your NAS, and some on your local PC.

    • How do you get it all together?
    • How do you link a bank transaction to a product invoice?
    • How to tag any object type and create a collection out of it (mix videos, photos, PDFs, transactions)?
    • How to store this? file/folder structure does not work, everything is linked together

    Project Description

    The idea is a place where you can throw all your data, photos, videos, documents, binaries, and else.

    Create photo albums, document collections, add tags across multiple file-formats, link content, and else.

    The UI should be easy to use, where the data is not important for now (could be all S3 or local drive).

    Similar proposals

    The closest I found so far is https://perkeep.org/, but this is not what I'm looking for.

    Goal for this Hackweek

    Create a web UI, in Svelte ideally, perhaps React.

    It should be able to show photos and videos at least.

    Resources

    None so far, this is just an idea.

    Hack on rich terminal user interfaces by amanzini

    Description

    TUIs (Textual User Interface) are a big classic of our daily workflow. Many linux users 'live' in the terminal and modern implementations have a lot to offer : unicode fonts, 24 bit colors etc.

    Goals

    • Explore the current available solution on modern languages and implement a PoC , for example a small maze generator, porting of a classic game or just display the HackWeek cute logo.
    • Practice some Go / Rust coding and programming patterns
    • Fiddle around, hack, learn, have fun
    • keep a development diary, practice on project documentation

    Follow this link for source code repository

    Some ideas for inspiration:

    Related projects:

    Resources

    ClusterOps - Easily install and manage your personal kubernetes cluster by andreabenini

    Description

    ClusterOps is a Kubernetes installer and operator designed to streamline the initial configuration and ongoing maintenance of kubernetes clusters. The focus of this project is primarily on personal or local installations. However, the goal is to expand its use to encompass all installations of Kubernetes for local development purposes.
    It simplifies cluster management by automating tasks and providing just one user-friendly YAML-based configuration config.yml.

    Overview

    • Simplified Configuration: Define your desired cluster state in a simple YAML file, and ClusterOps will handle the rest.
    • Automated Setup: Automates initial cluster configuration, including network settings, storage provisioning, special requirements (for example GPUs) and essential components installation.
    • Ongoing Maintenance: Performs routine maintenance tasks such as upgrades, security updates, and resource monitoring.
    • Extensibility: Easily extend functionality with custom plugins and configurations.
    • Self-Healing: Detects and recovers from common cluster issues, ensuring stability, idempotence and reliability. Same operation can be performed multiple times without changing the result.
    • Discreet: It works only on what it knows, if you are manually configuring parts of your kubernetes and this configuration does not interfere with it you can happily continue to work on several parts and use this tool only for what is needed.

    Features

    • distribution and engine independence. Install your favorite kubernetes engine with your package manager, execute one script and you'll have a complete working environment at your disposal.
    • Basic config approach. One single config.yml file with configuration requirements (add/remove features): human readable, plain and simple. All fancy configs managed automatically (ingress, balancers, services, proxy, ...).
    • Local Builtin ContainerHub. The default installation provides a fully configured ContainerHub available locally along with the kubernetes installation. This configuration allows the user to build, upload and deploy custom container images as they were provided from external sources. Internet public sources are still available but local development can be kept in this localhost server. Builtin ClusterOps operator will be fetched from this ContainerHub registry too.
    • Kubernetes official dashboard installed as a plugin, others planned too (k9s for example).
    • Kubevirt plugin installed and properly configured. Unleash the power of classic virtualization (KVM+QEMU) on top of Kubernetes and manage your entire system from there, libvirtd and virsh libs are required.
    • One operator to rule them all. The installation script configures your machine automatically during installation and adds one kubernetes operator to manage your local cluster. From there the operator takes care of the cluster on your behalf.
    • Clean installation and removal. Just test it, when you are done just use the same program to uninstall everything without leaving configs (or pods) behind.

    Planned features (Wishlist / TODOs)

    • Containerized Data Importer (CDI). Persistent storage management add-on for Kubernetes to provide a declarative way of building and importing Virtual Machine Disks on PVCs for

    Contribute to terraform-provider-libvirt by pinvernizzi

    Description

    The SUSE Manager (SUMA) teams' main tool for infrastructure automation, Sumaform, largely relies on terraform-provider-libvirt. That provider is also widely used by other teams, both inside and outside SUSE.

    It would be good to help the maintainers of this project and give back to the community around it, after all the amazing work that has been already done.

    If you're interested in any of infrastructure automation, Terraform, virtualization, tooling development, Go (...) it is also a good chance to learn a bit about them all by putting your hands on an interesting, real-use-case and complex project.

    Goals

    • Get more familiar with Terraform provider development and libvirt bindings in Go
    • Solve some issues and/or implement some features
    • Get in touch with the community around the project

    Resources

    Improve Development Environment on Uyuni by mbussolotto

    Description

    Currently create a dev environment on Uyuni might be complicated. The steps are:

    • add the correct repo
    • download packages
    • configure your IDE (checkstyle, format rules, sonarlint....)
    • setup debug environment
    • ...

    The current doc can be improved: some information are hard to be find out, some others are completely missing.

    Dev Container might solve this situation.

    Goals

    Uyuni development in no time:

    • using VSCode:
      • setting.json should contains all settings (for all languages in Uyuni, with all checkstyle rules etc...)
      • dev container should contains all dependencies
      • setup debug environment
    • implement a GitHub Workspace solution
    • re-write documentation

    Lots of pieces are already implemented: we need to connect them in a consistent solution.

    Resources

    • https://github.com/uyuni-project/uyuni/wiki

    Uyuni developer-centric documentation by deneb_alpha

    Description

    While we currently have extensive documentation on user-oriented tasks such as adding minions, patching, fine-tuning, etc, there is a notable gap when it comes to centralizing and documenting core functionalities for developers.

    The number of functionalities and side tools we have in Uyuni can be overwhelming. It would be nice to have a centralized place with descriptive list of main/core functionalities.

    Goals

    Create, aggregate and review on the Uyuni wiki a set of resources, focused on developers, that include also some known common problems/troubleshooting.

    The documentation will be helpful not only for everyone who is trying to learn the functionalities with all their inner processes like newcomer developers or community enthusiasts, but also for anyone who need a refresh.

    Resources

    The resources are currently aggregated here: https://github.com/uyuni-project/uyuni/wiki

    ClusterOps - Easily install and manage your personal kubernetes cluster by andreabenini

    Description

    ClusterOps is a Kubernetes installer and operator designed to streamline the initial configuration and ongoing maintenance of kubernetes clusters. The focus of this project is primarily on personal or local installations. However, the goal is to expand its use to encompass all installations of Kubernetes for local development purposes.
    It simplifies cluster management by automating tasks and providing just one user-friendly YAML-based configuration config.yml.

    Overview

    • Simplified Configuration: Define your desired cluster state in a simple YAML file, and ClusterOps will handle the rest.
    • Automated Setup: Automates initial cluster configuration, including network settings, storage provisioning, special requirements (for example GPUs) and essential components installation.
    • Ongoing Maintenance: Performs routine maintenance tasks such as upgrades, security updates, and resource monitoring.
    • Extensibility: Easily extend functionality with custom plugins and configurations.
    • Self-Healing: Detects and recovers from common cluster issues, ensuring stability, idempotence and reliability. Same operation can be performed multiple times without changing the result.
    • Discreet: It works only on what it knows, if you are manually configuring parts of your kubernetes and this configuration does not interfere with it you can happily continue to work on several parts and use this tool only for what is needed.

    Features

    • distribution and engine independence. Install your favorite kubernetes engine with your package manager, execute one script and you'll have a complete working environment at your disposal.
    • Basic config approach. One single config.yml file with configuration requirements (add/remove features): human readable, plain and simple. All fancy configs managed automatically (ingress, balancers, services, proxy, ...).
    • Local Builtin ContainerHub. The default installation provides a fully configured ContainerHub available locally along with the kubernetes installation. This configuration allows the user to build, upload and deploy custom container images as they were provided from external sources. Internet public sources are still available but local development can be kept in this localhost server. Builtin ClusterOps operator will be fetched from this ContainerHub registry too.
    • Kubernetes official dashboard installed as a plugin, others planned too (k9s for example).
    • Kubevirt plugin installed and properly configured. Unleash the power of classic virtualization (KVM+QEMU) on top of Kubernetes and manage your entire system from there, libvirtd and virsh libs are required.
    • One operator to rule them all. The installation script configures your machine automatically during installation and adds one kubernetes operator to manage your local cluster. From there the operator takes care of the cluster on your behalf.
    • Clean installation and removal. Just test it, when you are done just use the same program to uninstall everything without leaving configs (or pods) behind.

    Planned features (Wishlist / TODOs)

    • Containerized Data Importer (CDI). Persistent storage management add-on for Kubernetes to provide a declarative way of building and importing Virtual Machine Disks on PVCs for