Description

FastFileCheck is a high-performance, multithreaded file integrity checker for Linux. Designed for speed and efficiency, it utilizes parallel processing and a lightweight database to quickly hash and verify large volumes of files, ensuring their integrity over time.

https://github.com/paolostivanin/FastFileCheck

Goals

  • Release v1.0.0

Design overwiew:

  • Main thread (producer): traverses directories and feeds the queue (one thread is more than enough for most use cases)
  • Dedicated consumer thread: manages queue and distributes work to threadpool
  • Worker threads: compute hashes in parallel

This separation of concerns is efficient because:

  • Directory traversal is I/O bound and works well in a single thread
  • Queue management is centralized, preventing race conditions
  • Hash computation is CPU-intensive and properly parallelized

Looking for hackers with the skills:

c hashing lmdb multithreading integrity-checker

This project is part of:

Hack Week 24

Activity

  • 10 months ago: pstivanin added keyword "c" to this project.
  • 10 months ago: pstivanin added keyword "hashing" to this project.
  • 10 months ago: pstivanin added keyword "lmdb" to this project.
  • 10 months ago: pstivanin added keyword "multithreading" to this project.
  • 10 months ago: pstivanin added keyword "integrity-checker" to this project.
  • 10 months ago: pstivanin started this project.
  • 11 months ago: pstivanin originated this project.

  • Comments

    • pstivanin
      11 months ago by pstivanin | Reply

      most of the stuff have been implemented: https://github.com/paolostivanin/FastFileCheck/releases/tag/v1.0.0-alpha1

    • pstivanin
      10 months ago by pstivanin | Reply

      deleted

    Similar Projects

    pudc - A PID 1 process that barks to the internet by mssola

    Description

    As a fun exercise in order to dig deeper into the Linux kernel, its interfaces, the RISC-V architecture, and all the dragons in between; I'm building a blog site cooked like this:

    • The backend is written in a mixture of C and RISC-V assembly.
    • The backend is actually PID1 (for real, not within a container).
    • We poll and parse incoming HTTP requests ourselves.
    • The frontend is a mere HTML page with htmx.

    The project is meant to be Linux-specific, so I'm going to use io_uring, pidfs, namespaces, and Linux-specific features in order to drive all of this.

    I'm open for suggestions and so on, but this is meant to be a solo project, as this is more of a learning exercise for me than anything else.

    Goals

    • Have a better understanding of different Linux features from user space down to the kernel internals.
    • Most importantly: have fun.

    Resources