I'm running a simple home mirror, but I managed to get into a situation where I have to use a bunch of custom excludes. I think we should be able to offer what people want nowadays.\

To my knowledge, hotstuff and other pre-configured rsync modules are no longer maintained and advised not to use. This leads people to maintain their own exclusion list for the rsync.

So I'm trying to improve my current experience at home and reduce the effort for others.

Project Description

I'd like to offer people with limited storage a super-easy way how to deploy a home mirror.

Ideally, Leap Micro + preconfigured combustion script. Boot and pull a pre-configured mirror container via cockpit (unless done in the combustion script) (160GB hotstuff, 320GB, etc ...). Done in like 5-8 minutes and already serving.

That's the easier part^

The more tricky part will be that nobody really touched are pre-configured rsync modules hotstuff etc for quite some time.

So refreshment of our pre-configured hotstuff, exposing it on github.com/openSUSEm, updating docs, and advertising is the next goal. Also, make a pre-recorded demo similar to the one I've done for Micro.

Trigger Ish recently configured a mirror for the Mauritius region and he struggled with quota issues. That only confirmed that I'm not alone who sees the issue. I shared with him my exclusion list and crontab line and he was happy. I think we can do a better job and nobody should be using these exclusion lists that require maintenance.

Goal for this Hackweek

Avoid using exclude list while being on a budget. Offer up-to-date pre-configured options. Offer simple solutions via containers for (not-only) people with rasp pi or similar home NAS.

  • rsync modules for mirrors with quota up2date in git (currently located on pontifex)
  • auto-deployment of mirror config to pontifex? (bonus, otherwise manual pull via request to admin@opensuse.org)
  • Set of container images (per offered rsync module)
  • Update docs (see resources) and publish a demo

Resources

https://en.opensuse.org/openSUSE:Mirror_howto

https://www.youtube.com/watch?v=ft8UVx9elKc

Looking for hackers with the skills:

hotstuff mirror opensuse rsync leapmicro containers raspberrypi

This project is part of:

Hack Week 21

Activity

  • 3 months ago: tjyrinki_suse liked this project.
  • 4 months ago: avicenzi joined this project.
  • 4 months ago: pdostal joined this project.
  • 4 months ago: pdostal liked this project.
  • 5 months ago: Pharaoh_Atem liked this project.
  • 5 months ago: Ishwon joined this project.
  • 5 months ago: lkocman started this project.
  • 5 months ago: lkocman added keyword "raspberrypi" to this project.
  • 5 months ago: lkocman added keyword "hotstuff" to this project.
  • 5 months ago: lkocman added keyword "mirror" to this project.
  • 5 months ago: lkocman added keyword "opensuse" to this project.
  • 5 months ago: lkocman added keyword "rsync" to this project.
  • 5 months ago: lkocman added keyword "leapmicro" to this project.
  • 5 months ago: lkocman added keyword "containers" to this project.
  • 5 months ago: lkocman originated this project.

  • Comments

    • lrupp
      5 months ago by lrupp | Reply

      Really love to see this happening! Just one note: if possible, please provide an RPM package that can easily be installed and updated on the download mirror (pontifex). Single scripts, deployed at some time on a machine and not maintained any longer are a way we really should avoid. I had/have this experience especially with pontifex too often in the past....

      • lkocman
        5 months ago by lkocman | Reply

        Will keep that in mind. Yes I also think we want github and rpm deployment.

    • pjessen
      5 months ago by pjessen | Reply

      I can't remember when, but I hardwired the hotstuff modules, after I realised no one had the insight to fix the knapsack stuff. I try to maintain the other rsync modules, but I don't spend much time on it. Wrt this project, I say go for it, as long as it doesn't mean any extra ongoing maintenance.

      • lkocman
        5 months ago by lkocman | Reply

        Absolutely the goal is to simplify work. There will be some effort initially (refresh itself, initial bugfixes and rework deployment). That's what the hackweek is about.

    • lkocman
      5 months ago by lkocman | Reply

      Initial drop of the current configuration (previously was under svn) https://github.com/openSUSE/opensuse-hotstuff (openSUSE Heroes can commit)

      Very simple rpm utilizing obs_scm and explicit listing of individual files https://build.opensuse.org/request/show/975806 If infra team accepts this I'd like to start deploying via update from the infra repo.

      Next step is refresh of the files :-)

    • lkocman
      5 months ago by lkocman | Reply

      Perhaps a related project https://github.com/Firstyear/opensuse-proxy-cache

    • simotek
      5 months ago by simotek | Reply

      The cache @firstyear wrote (https://github.com/Firstyear/opensuse-proxy-cache) might be more useful for alot of people rather then syncing as a fully populated mirror it fetches packages etc on demand the first time they are requested and caches them for future use. A number of people in Australia are already using it with decent success, I have it running in a container on one of my machines and the rest point to that as there main repo. It means you don't have to download the whole mirror just the stuff you actually use.

    • avicenzi
      4 months ago by avicenzi | Reply

      Hi, just fix hotstuff modules, other problems are solved in https://github.com/opensuse-brasil/mirror-images (I think).

      I still need to clean up and upload my swarm/compose files and create some k8s files.

      I've been running my mirror for probably over a year now on containers, no issues, it works really well.

      My mirror current runs on this repo https://github.com/alexandrevicenzi/opensuse-mirror-docker/, it has some flaws.

      I would really love if modules just included versions for example:

      • Leap 15.4 oss/non-oss and updates
      • TW oss/non-oss and updates

      In my mirror I have current (15.4), current -1 (15.3) and TW, no ports, no debugs, no sources. Just Leap 15.4 with updates uses about 75 GB, that's really good even if you want to host at home. If you add 15.3, and TW that goes up to 700 GB, and if you add debug, source, ports, that goes over 2 TB. Would be nice if it was easier to filter architectures, I have zero interest in hosting IBM/PPC packages, that's often mainframe, and most people would not use openSUSE TW in a mainframe.

      I don't really like the idea that I need to update my include/exclude file every new release, if we can avoid that with hotstuff, that is better. It's easier to maintain modules on our side, than expect many to update their configs every new release.

    • avicenzi
      4 months ago by avicenzi | Reply

      I just had an idea, maybe a silly one, but what if we had a service where the user chooses what he wants to mirror, and they get a custom url that is plain text, and with this url, they can just wget before they run rsync in cron to get their desired include/exclude list.

      I would like sometime like that, and to be honest, not that difficult to be achieved. I would be willing, or I may implement that myself, just for fun.

    • lkocman
      3 months ago by lkocman | Reply

      I'm spending initial 2-3 days on the https://hackweek.opensuse.org/21/projects/opensuse-build-supported-by-suse-it

      I'll be looking into this on Wednesday.

      I did get an access to MySQL database for mirrorcache from Andrii, so scripting this should be possible

    Similar Projects

    Migrate suntorytimed/resourcespace container to an openSUSE base by suntorytimed

    Project Description

    For quite some time I a...


    ignite.opensuse.org by lkocman

    Let's do what we can to get https://github.com/...


    MicroOS Desktop by RBrownSUSE

    [Video Recording of openSUSE Conference sessio...


    Migrate from Raspbian to openSUSE aarch64 for motion detection camera by tjyrinki_suse

    Project Description

    I have had a hobby pr...


    Family with MicroOS by xguo

    Project Description

    Try to family with Mi...


    Uyuni/SUSE Manager Server Helm chart on k3s by moio

    ![Combined icons of k3s and Uyuni](https://user...


    Package docker compose v2 by hennevogel

    Package [compose](https://github.com/docker/com...


    containerizing MicroOS Desktop components (reloaded) by fcrozat

    [comment]: # (Please use the project descriptio...


    WASM support for cloud native buildpacks by atgracey

    [comment]: # (Please use the project descriptio...


    Bring to life the Studio Kiosk in Frankenstrasse by digitaltomm

    [comment]: # (Please use the project descriptio...


    Migrate from Raspbian to openSUSE aarch64 for motion detection camera by tjyrinki_suse

    Project Description

    I have had a hobby pr...