I'm running a simple home mirror, but I managed to get into a situation where I have to use a bunch of custom excludes. I think we should be able to offer what people want nowadays.\

To my knowledge, hotstuff and other pre-configured rsync modules are no longer maintained and advised not to use. This leads people to maintain their own exclusion list for the rsync.

So I'm trying to improve my current experience at home and reduce the effort for others.

Project Description

I'd like to offer people with limited storage a super-easy way how to deploy a home mirror.

Ideally, Leap Micro + preconfigured combustion script. Boot and pull a pre-configured mirror container via cockpit (unless done in the combustion script) (160GB hotstuff, 320GB, etc ...). Done in like 5-8 minutes and already serving.

That's the easier part^

The more tricky part will be that nobody really touched are pre-configured rsync modules hotstuff etc for quite some time.

So refreshment of our pre-configured hotstuff, exposing it on github.com/openSUSEm, updating docs, and advertising is the next goal. Also, make a pre-recorded demo similar to the one I've done for Micro.

Trigger Ish recently configured a mirror for the Mauritius region and he struggled with quota issues. That only confirmed that I'm not alone who sees the issue. I shared with him my exclusion list and crontab line and he was happy. I think we can do a better job and nobody should be using these exclusion lists that require maintenance.

Goal for this Hackweek

Avoid using exclude list while being on a budget. Offer up-to-date pre-configured options. Offer simple solutions via containers for (not-only) people with rasp pi or similar home NAS.

  • rsync modules for mirrors with quota up2date in git (currently located on pontifex)
  • auto-deployment of mirror config to pontifex? (bonus, otherwise manual pull via request to admin@opensuse.org)
  • Set of container images (per offered rsync module)
  • Update docs (see resources) and publish a demo

Resources

https://en.opensuse.org/openSUSE:Mirror_howto

https://www.youtube.com/watch?v=ft8UVx9elKc

Looking for hackers with the skills:

hotstuff mirror opensuse rsync leapmicro containers raspberrypi

This project is part of:

Hack Week 21 Hack Week 23

Activity

  • over 2 years ago: tjyrinki_suse liked this project.
  • over 2 years ago: avicenzi joined this project.
  • over 2 years ago: pdostal joined this project.
  • over 2 years ago: pdostal liked this project.
  • over 2 years ago: Pharaoh_Atem liked this project.
  • over 2 years ago: Ishwon joined this project.
  • over 2 years ago: lkocman started this project.
  • over 2 years ago: lkocman added keyword "raspberrypi" to this project.
  • over 2 years ago: lkocman added keyword "hotstuff" to this project.
  • over 2 years ago: lkocman added keyword "mirror" to this project.
  • over 2 years ago: lkocman added keyword "opensuse" to this project.
  • over 2 years ago: lkocman added keyword "rsync" to this project.
  • over 2 years ago: lkocman added keyword "leapmicro" to this project.
  • over 2 years ago: lkocman added keyword "containers" to this project.
  • over 2 years ago: lkocman originated this project.

  • Comments

    • lrupp
      over 2 years ago by lrupp | Reply

      Really love to see this happening! Just one note: if possible, please provide an RPM package that can easily be installed and updated on the download mirror (pontifex). Single scripts, deployed at some time on a machine and not maintained any longer are a way we really should avoid. I had/have this experience especially with pontifex too often in the past....

      • lkocman
        over 2 years ago by lkocman | Reply

        Will keep that in mind. Yes I also think we want github and rpm deployment.

    • pjessen
      over 2 years ago by pjessen | Reply

      I can't remember when, but I hardwired the hotstuff modules, after I realised no one had the insight to fix the knapsack stuff. I try to maintain the other rsync modules, but I don't spend much time on it. Wrt this project, I say go for it, as long as it doesn't mean any extra ongoing maintenance.

      • lkocman
        over 2 years ago by lkocman | Reply

        Absolutely the goal is to simplify work. There will be some effort initially (refresh itself, initial bugfixes and rework deployment). That's what the hackweek is about.

    • lkocman
      over 2 years ago by lkocman | Reply

      Initial drop of the current configuration (previously was under svn) https://github.com/openSUSE/opensuse-hotstuff (openSUSE Heroes can commit)

      Very simple rpm utilizing obs_scm and explicit listing of individual files https://build.opensuse.org/request/show/975806 If infra team accepts this I'd like to start deploying via update from the infra repo.

      Next step is refresh of the files :-)

    • lkocman
      over 2 years ago by lkocman | Reply

      Perhaps a related project https://github.com/Firstyear/opensuse-proxy-cache

    • simotek
      over 2 years ago by simotek | Reply

      The cache @firstyear wrote (https://github.com/Firstyear/opensuse-proxy-cache) might be more useful for alot of people rather then syncing as a fully populated mirror it fetches packages etc on demand the first time they are requested and caches them for future use. A number of people in Australia are already using it with decent success, I have it running in a container on one of my machines and the rest point to that as there main repo. It means you don't have to download the whole mirror just the stuff you actually use.

    • avicenzi
      over 2 years ago by avicenzi | Reply

      Hi, just fix hotstuff modules, other problems are solved in https://github.com/opensuse-brasil/mirror-images (I think).

      I still need to clean up and upload my swarm/compose files and create some k8s files.

      I've been running my mirror for probably over a year now on containers, no issues, it works really well.

      My mirror current runs on this repo https://github.com/alexandrevicenzi/opensuse-mirror-docker/, it has some flaws.

      I would really love if modules just included versions for example:

      • Leap 15.4 oss/non-oss and updates
      • TW oss/non-oss and updates

      In my mirror I have current (15.4), current -1 (15.3) and TW, no ports, no debugs, no sources. Just Leap 15.4 with updates uses about 75 GB, that's really good even if you want to host at home. If you add 15.3, and TW that goes up to 700 GB, and if you add debug, source, ports, that goes over 2 TB. Would be nice if it was easier to filter architectures, I have zero interest in hosting IBM/PPC packages, that's often mainframe, and most people would not use openSUSE TW in a mainframe.

      I don't really like the idea that I need to update my include/exclude file every new release, if we can avoid that with hotstuff, that is better. It's easier to maintain modules on our side, than expect many to update their configs every new release.

    • avicenzi
      over 2 years ago by avicenzi | Reply

      I just had an idea, maybe a silly one, but what if we had a service where the user chooses what he wants to mirror, and they get a custom url that is plain text, and with this url, they can just wget before they run rsync in cron to get their desired include/exclude list.

      I would like sometime like that, and to be honest, not that difficult to be achieved. I would be willing, or I may implement that myself, just for fun.

    • lkocman
      over 2 years ago by lkocman | Reply

      I'm spending initial 2-3 days on the https://hackweek.opensuse.org/21/projects/opensuse-build-supported-by-suse-it

      I'll be looking into this on Wednesday.

      I did get an access to MySQL database for mirrorcache from Andrii, so scripting this should be possible

    • lkocman
      about 1 year ago by lkocman | Reply

      Oky I've spent some extra time on this yesterday. So I'm making it part of this year's hackweek again :-)

      Refreshed https://build.opensuse.org/package/show/home:lkocman:Images/opensuse-mirror I plan to rework it to use exclusively exclude lists and offer user a decent choice.

      next task would be to have a convenient hack with openSUSE-repos to use my local mirror on my local network only. I was thinking of repo priority and passing url to service fiel as a zypper variable.

    Similar Projects

    Digital art wallpapers for openSUSE Leap and Tumbleweed by lkocman

    Description

    We've enrolled set of new wallpapers to both Leap 16 and Tumbleweed as part of https://news.opensuse.org/2024/10/26/leap-tw-get-makeovers/

    We've previewed digital art wallpapers which were not part of the initial drop. I'd like to spend time on hackweek to finialize my current Taipei (mountains) and Mauritius digital art wallpapers.

    Goals

    Finalize existing two digital art wallpapers for Leap and Tumbleweed https://github.com/openSUSE/branding/issues/155 Make them available as part of leap16 dir in https://github.com/openSUSE/wallpapers and update (This makes is available to Tumbleweed users as well). Update https://build.opensuse.org/package/show/X11:common:Factory/wallpapers-openSUSE-extra && Leap:16.0 && Factory.

    Resources

    https://github.com/openSUSE/branding/issues/155 The mauritius draft can be found in https://github.com/lkocman/geo-wallpapers


    Update Haskell ecosystem in Tumbleweed to GHC-9.10.x by psimons

    Description

    We are currently at GHC-9.8.x, which a bit old. So I'd like to take a shot at the latest version of the compiler, GHC-9.10.x. This is gonna be interesting because the new version requires major updates to all kinds of libraries and base packages, which typically means patching lots of packages to make them build again.

    Goals

    Have working builds of GHC-9.10.x and the required Haskell packages in 'devel:languages:haskell` so that we can compile:

    • git-annex
    • pandoc
    • xmonad
    • cabal-install

    Resources

    • https://build.opensuse.org/project/show/devel:languages:haskell/
    • https://github.com/opensuse-haskell/configuration/
    • #discuss-haskell
    • https://www.twitch.tv/peti343


    New openSUSE-welcome by lkocman

    Project Description

    Let's revisit our existing openSUSE welcome app.

    My goal was to show Leap 16 in a new coat. Welcome app adds to the first time use experience. We've recently added donation button to our existing welcome.

    Some things that I recently wanted to address were EOL and possibly upgrade notification.

    I've already done some experiments with mint welcome app, but not sure if it's better than the existing one.

    There is also a PR to rework existing app https://github.com/openSUSE/openSUSE-welcome/pull/36 (this should be considered as an option too)

    Goal for this Hackweek

    New welcome app, possibly with EOL notification for Leap.

    1) Welcome application(s) with (rebrand changes) maintained under github.com/openSUSE

    2) Application is submitted to openSUSE:Factory && openSUSE:Leap:16.0

    3) Updated needles in openQA (probably post hackweek)

    Resources

    Reddit discussion about the best welcome app out there.

    Github repo for the current welcome app.


    New migration tool for Leap by lkocman

    Update

    I will call a meeting with other interested people at 11:00 CET https://meet.opensuse.org/migrationtool

    Description

    SLES 16 plans to have no yast tool in it. Leap 16 might keep some bits, however, we need a new tool for Leap to SLES migration, as this was previously handled by a yast2-migration-sle

    Goals

    A tool able to migrate Leap 16 to SLES 16, I would like to cover also other scenarios within openSUSE, as in many cases users would have to edit repository files manually.

    • Leap -> Leap n+1 (minor and major version updates)
    • Leap -> SLES docs
    • Leap -> Tumbleweed
    • Leap -> Slowroll
    • Leap Micro -> Leap Micro n+1 (minor and major version updates)
    • Leap Micro -> MicroOS

    Hackweek 24 update

    Marcela and I were working on the project from Brno coworking as well as finalizing pieces after the hackweek. We've tested several migration scenarios and it works. But it needs further polishing and testing.

    Projected was renamed to opensuse-migration-tool and was submitted to devel project https://build.opensuse.org/requests/1227281

    Repository

    https://github.com/openSUSE/opensuse-migration-tool

    Out of scope is any migration to an immutable system. I know Richard already has some tool for that.

    Resources

    Tracker for yast stack reduction code-o-o/leap/features#173 YaST stack reduction


    Create openSUSE images for Arm/RISC-V boards by avicenzi

    Project Description

    Create openSUSE images (or test generic EFI images) for Arm and/or RISC-V boards that are not yet supported.

    Goal for this Hackweek

    Create bootable images of Tumbleweed for SBCs that currently have no images available or are untested.

    Consider generic EFI images where possible, as some boards can hold a bootloader.

    Document in the openSUSE Wiki how to flash and use the image for a given board.

    Boards that I have around and there are no images:

    • Rock 3B
    • Nano PC T3 Plus
    • Lichee RV D1
    • StartFive VisionFive (has some image needs testing)

    Hack Week 22

    Hack Week 21

    Resources


    Improve Development Environment on Uyuni by mbussolotto

    Description

    Currently create a dev environment on Uyuni might be complicated. The steps are:

    • add the correct repo
    • download packages
    • configure your IDE (checkstyle, format rules, sonarlint....)
    • setup debug environment
    • ...

    The current doc can be improved: some information are hard to be find out, some others are completely missing.

    Dev Container might solve this situation.

    Goals

    Uyuni development in no time:

    • using VSCode:
      • setting.json should contains all settings (for all languages in Uyuni, with all checkstyle rules etc...)
      • dev container should contains all dependencies
      • setup debug environment
    • implement a GitHub Workspace solution
    • re-write documentation

    Lots of pieces are already implemented: we need to connect them in a consistent solution.

    Resources

    • https://github.com/uyuni-project/uyuni/wiki


    SUSE AI Meets the Game Board by moio

    Use tabletopgames.ai’s open source TAG and PyTAG frameworks to apply Statistical Forward Planning and Deep Reinforcement Learning to two board games of our own design. On an all-green, all-open source, all-AWS stack!
    A chameleon playing chess in a train car, as a metaphor of SUSE AI applied to games


    Results: Infrastructure Achievements

    We successfully built and automated a containerized stack to support our AI experiments. This included:

    A screenshot of k9s and nvtop showing PyTAG running in Kubernetes with GPU acceleration

    ./deploy.sh and voilà - Kubernetes running PyTAG (k9s, above) with GPU acceleration (nvtop, below)

    Results: Game Design Insights

    Our project focused on modeling and analyzing two card games of our own design within the TAG framework:

    • Game Modeling: We implemented models for Dario's "Bamboo" and Silvio's "Totoro" and "R3" games, enabling AI agents to play thousands of games ...in minutes!
    • AI-driven optimization: By analyzing statistical data on moves, strategies, and outcomes, we iteratively tweaked the game mechanics and rules to achieve better balance and player engagement.
    • Advanced analytics: Leveraging AI agents with Monte Carlo Tree Search (MCTS) and random action selection, we compared performance metrics to identify optimal strategies and uncover opportunities for game refinement .

    Cards from the three games

    A family picture of our card games in progress. From the top: Bamboo, Totoro, R3

    Results: Learning, Collaboration, and Innovation

    Beyond technical accomplishments, the project showcased innovative approaches to coding, learning, and teamwork:

    • "Trio programming" with AI assistance: Our "trio programming" approach—two developers and GitHub Copilot—was a standout success, especially in handling slightly-repetitive but not-quite-exactly-copypaste tasks. Java as a language tends to be verbose and we found it to be fitting particularly well.
    • AI tools for reporting and documentation: We extensively used AI chatbots to streamline writing and reporting. (Including writing this report! ...but this note was added manually during edit!)
    • GPU compute expertise: Overcoming challenges with CUDA drivers and cloud infrastructure deepened our understanding of GPU-accelerated workloads in the open-source ecosystem.
    • Game design as a learning platform: By blending AI techniques with creative game design, we learned not only about AI strategies but also about making games fun, engaging, and balanced.

    Last but not least we had a lot of fun! ...and this was definitely not a chatbot generated line!

    The Context: AI + Board Games


    ClusterOps - Easily install and manage your personal kubernetes cluster by andreabenini

    Description

    ClusterOps is a Kubernetes installer and operator designed to streamline the initial configuration and ongoing maintenance of kubernetes clusters. The focus of this project is primarily on personal or local installations. However, the goal is to expand its use to encompass all installations of Kubernetes for local development purposes.
    It simplifies cluster management by automating tasks and providing just one user-friendly YAML-based configuration config.yml.

    Overview

    • Simplified Configuration: Define your desired cluster state in a simple YAML file, and ClusterOps will handle the rest.
    • Automated Setup: Automates initial cluster configuration, including network settings, storage provisioning, special requirements (for example GPUs) and essential components installation.
    • Ongoing Maintenance: Performs routine maintenance tasks such as upgrades, security updates, and resource monitoring.
    • Extensibility: Easily extend functionality with custom plugins and configurations.
    • Self-Healing: Detects and recovers from common cluster issues, ensuring stability, idempotence and reliability. Same operation can be performed multiple times without changing the result.
    • Discreet: It works only on what it knows, if you are manually configuring parts of your kubernetes and this configuration does not interfere with it you can happily continue to work on several parts and use this tool only for what is needed.

    Features

    • distribution and engine independence. Install your favorite kubernetes engine with your package manager, execute one script and you'll have a complete working environment at your disposal.
    • Basic config approach. One single config.yml file with configuration requirements (add/remove features): human readable, plain and simple. All fancy configs managed automatically (ingress, balancers, services, proxy, ...).
    • Local Builtin ContainerHub. The default installation provides a fully configured ContainerHub available locally along with the kubernetes installation. This configuration allows the user to build, upload and deploy custom container images as they were provided from external sources. Internet public sources are still available but local development can be kept in this localhost server. Builtin ClusterOps operator will be fetched from this ContainerHub registry too.
    • Kubernetes official dashboard installed as a plugin, others planned too (k9s for example).
    • Kubevirt plugin installed and properly configured. Unleash the power of classic virtualization (KVM+QEMU) on top of Kubernetes and manage your entire system from there, libvirtd and virsh libs are required.
    • One operator to rule them all. The installation script configures your machine automatically during installation and adds one kubernetes operator to manage your local cluster. From there the operator takes care of the cluster on your behalf.
    • Clean installation and removal. Just test it, when you are done just use the same program to uninstall everything without leaving configs (or pods) behind.

    Planned features (Wishlist / TODOs)

    • Containerized Data Importer (CDI). Persistent storage management add-on for Kubernetes to provide a declarative way of building and importing Virtual Machine Disks on PVCs for


    Technical talks at universities by agamez

    Description

    This project aims to empower the next generation of tech professionals by offering hands-on workshops on containerization and Kubernetes, with a strong focus on open-source technologies. By providing practical experience with these cutting-edge tools and fostering a deep understanding of open-source principles, we aim to bridge the gap between academia and industry.

    For now, the scope is limited to Spanish universities, since we already have the contacts and have started some conversations.

    Goals

    • Technical Skill Development: equip students with the fundamental knowledge and skills to build, deploy, and manage containerized applications using open-source tools like Kubernetes.
    • Open-Source Mindset: foster a passion for open-source software, encouraging students to contribute to open-source projects and collaborate with the global developer community.
    • Career Readiness: prepare students for industry-relevant roles by exposing them to real-world use cases, best practices, and open-source in companies.

    Resources

    • Instructors: experienced open-source professionals with deep knowledge of containerization and Kubernetes.
    • SUSE Expertise: leverage SUSE's expertise in open-source technologies to provide insights into industry trends and best practices.


    ADS-B receiver with MicroOS by epaolantonio

    I would like to put one of my spare Raspberry Pis to good use, and what better way to see what flies above my head at any time? add-emoji

    There are various ready-to-use distros already set-up to provide feeder data to platforms like Flightradar24, ADS-B Exchange, FlightAware etc... The goal here would be to do it using MicroOS as a base and containerized decoding of ADS-B data (via tools like dump1090) and web frontend (tar1090).

    Goals

    • Create a working receiver using MicroOS as a base, and containers based on Tumbleweed
    • Make it easy to install
    • Optimize for maximum laziness (i.e. it should take care of itself with minimum intervention)

    Resources

    • 1x Small Board Computer capable of running MicroOS
    • 1x RTL2832U DVB-T dongle
    • 1x MicroSD card
    • https://github.com/antirez/dump1090
    • https://github.com/flightaware/dump1090 (dump1090 fork by FlightAware)
    • https://github.com/wiedehopf/tar1090

    Project status (2024-11-22)

    So I'd say that I'm pretty satisfied with how it turned out. I've packaged readsb (as a replacement for dump1090), tar1090, tar1090-db and mlat-client (not used yet).

    Current status:

    • Able to set-up a working receiver using combustion+ignition (web app based on Fuel Ignition)
    • Able to feed to various feeds using the Beast protocol (Airplanes.live, ADSB.fi, ADSB.lol, ADSBExchange.com, Flyitalyadsb.com, Planespotters.net)
    • Able to feed to Flightradar24 (initial-setup available but NOT tested! I've only tested using a key I already had)
    • Local web interface (tar1090) to easily visualize the results
    • Cockpit pre-configured to ease maintenance

    What's missing:

    • MLAT (Multilateration) support. I've packaged mlat-client already, but I have to wire it up
    • FlightAware support

    Give it a go at https://g7.github.io/adsbreceiver/ !

    Project links