Description

FastFileCheck is a high-performance, multithreaded file integrity checker for Linux. Designed for speed and efficiency, it utilizes parallel processing and a lightweight database to quickly hash and verify large volumes of files, ensuring their integrity over time.

https://github.com/paolostivanin/FastFileCheck

Goals

  • Release v1.0.0

Design overwiew:

  • Main thread (producer): traverses directories and feeds the queue (one thread is more than enough for most use cases)
  • Dedicated consumer thread: manages queue and distributes work to threadpool
  • Worker threads: compute hashes in parallel

This separation of concerns is efficient because:

  • Directory traversal is I/O bound and works well in a single thread
  • Queue management is centralized, preventing race conditions
  • Hash computation is CPU-intensive and properly parallelized

Looking for hackers with the skills:

c hashing lmdb multithreading integrity-checker

This project is part of:

Hack Week 24

Activity

  • about 1 year ago: pstivanin added keyword "c" to this project.
  • about 1 year ago: pstivanin added keyword "hashing" to this project.
  • about 1 year ago: pstivanin added keyword "lmdb" to this project.
  • about 1 year ago: pstivanin added keyword "multithreading" to this project.
  • about 1 year ago: pstivanin added keyword "integrity-checker" to this project.
  • about 1 year ago: pstivanin started this project.
  • about 1 year ago: pstivanin originated this project.

  • Comments

    • pstivanin
      about 1 year ago by pstivanin | Reply

      most of the stuff have been implemented: https://github.com/paolostivanin/FastFileCheck/releases/tag/v1.0.0-alpha1

    • pstivanin
      about 1 year ago by pstivanin | Reply

      deleted

    Similar Projects

    pudc - A PID 1 process that barks to the internet by mssola

    Description

    As a fun exercise in order to dig deeper into the Linux kernel, its interfaces, the RISC-V architecture, and all the dragons in between; I'm building a blog site cooked like this:

    • The backend is written in a mixture of C and RISC-V assembly.
    • The backend is actually PID1 (for real, not within a container).
    • We poll and parse incoming HTTP requests ourselves.
    • The frontend is a mere HTML page with htmx.

    The project is meant to be Linux-specific, so I'm going to use io_uring, pidfs, namespaces, and Linux-specific features in order to drive all of this.

    I'm open for suggestions and so on, but this is meant to be a solo project, as this is more of a learning exercise for me than anything else.

    Goals

    • Have a better understanding of different Linux features from user space down to the kernel internals.
    • Most importantly: have fun.

    Resources


    Improve the picotm Transaction Manager by tdz

    Picotm is a system-level transaction manager. It provides transactional semantics to low-level C operations, such as

    • memory access,
    • modifying data structures,
    • (some) file I/O, and
    • common interfaces from the C Standard Library and POSIX.

    Picotm also handles error detection and recovery for all it's functionality. It's fully modular, so new functionality can be added.

    For the Hackweek, I want to dedicate some time to picotm. I want to finish some of the refactoring work that I have been working on. If there's time left, I'd like to investigate two-phase commits and how to support them in picotm.

    Picotm is available at http://picotm.org/.


    Port OTPClient to GTK >= 4.18 by pstivanin

    Project Description

    OTPClient is currently using GTK3 and cannot easily be ported to GTK4. Since GTK4 came out, there have been quite some big changes. Also, there are now some new deprecation that will take effect with GTK5 (and are active starting from 4.10 as warnings), so I need to think ahead and port OTPClient without using any of those deprecated features.

    Goal for this Hackweek

    • fix the last 3 opened issues (https://github.com/paolostivanin/OTPClient/issues/402, https://github.com/paolostivanin/OTPClient/issues/404, https://github.com/paolostivanin/OTPClient/issues/406) and release a new version
    • continue the rewrite from where we left last year
    • if possible, finally close this 6 years old issue: https://github.com/paolostivanin/OTPClient/issues/123


    Smart lighting with Pico 2 by jmodak

    Description

    I am trying to create a smart-lighting project with a Raspberry Pi Pico that reacts to a movie's visuals and audio that involves combining two distinct functions: ambient screen lighting(visual response) and sound-reactive lighting(audio response)

    Goals

    • Visuals: Capturing the screen's colour requires an external device to analyse screen content and send colour data to the MCU via serial communication.
    • Audio: A sound sensor module connected directly to the Pico that can detect sound volume.
    • Pico 2W: The MCU receives data fro, both inputs and controls an LED strip.

    Resources

    • Raspberry Pi Pico 2 W
    • RGB LED strip
    • Sound detecting sensor
    • Power supply
    • breadboard and wires