Orcas are amazing animals. They are playful, intelligent, great swimmers, and very social. They also love to play with their food, hunting down their prey with advanced strategies - understanding where its prey hides, how it will try to escape, and how to overcome those tactics - and having a lot of fun doing so, before relentlessly tearing it apart, killing it, and eat it. Not necessarily in that order. Oh, and they have the right color scheme.
This forces their prey to also improve and adapt more advanced strategies and tactics. In this arms race, both sides evolve and improve: the evolutionary pressure has made cephalopods highly intelligent, adaptable, and resilient. Unfortunately (for them), they are still very tasty. So we should exert more evolutionary pressure on individuals to help them stay alive as a species.
The most promiment example of this is Netflix's chaos monkey. However, that is very heavily focused on Amazon cloud services. The Ceph project also has Teuthology; but that's mainly checking whether Ceph remembers all the tricks it has been taught. And CBT, which measures how fast it can swim while static. CeTune helps it swim faster. All are needed and provide valuable insights, but too tame; Ceph is not afraid enough of them.
A large distributed Ceph cluster will always be "in transition"; something fails, it's being rebalanced, nodes are being added, removed, ... all the while the clients are expecting it to deliver service.
We need a stress test harness for Ceph that one can point at an existing Ceph cluster, and that will understand the failure domains (OSD trees, nodes, NIC connections, ...) and inject faults until it eventually breaks. (All the while measuring the performance to see if the cluster is still within it's SLAs.)
You could think of this as a form of black-/gray-box testing at the system level. We don't really need to know a lot about Ceph's internals; we only know the high level architecture so we can group the components into failure domains and see how many errors we should be able to inject without failure. And once we heal the error, watch while - or rather, if - Ceph properly recovers.
Customers also don't care if it's Ceph crashing and not recovering, or if the specific workload has triggered a bug in some other part of the kernel. Thus, we need to holistically test at the system level.
Goals: - Make Ceph more robust in the face of faults; - Improve Ceph recovery; - Increase customer confidence in their deployed clusters; - Improve supportability of production clusters by forcing developers to look into failure scenarios more frequently.
Possible errors to inject: - killing daemons, - SIGSTOP (simulates hangs), - inducing kernel panics, - network outages on the front-end or back-end, - invoking random network latency and bottlenecks, - out of memory errors, - CPU overload, - corrupting data on disk, - Full cluster outage and reboot (think power outage), - ...
There are several states of the cluster to trigger:
baseline ("sunny weather"): establish a performance baseline while everything actually works. (While this is never really the case in production, it is the goal of performance under adverse conditions.)
"lightly" degraded - the system must be able to cope with a single fault in one of its failure domains, all the while providing service within the high-end range of its SLAs. Also, if this error is healed, the system should fully recover.
"heavily" degraded - the system should be able to cope with a single fault in several of its failure domains, all the while providing services within its SLAs. Also, if this error is healed, the system should fully recover. (This is harder than the previous case due to unexpected interdependencies.)
"crashed": if the faults in any of its failure domains exceed the available redundancy, it would be expected that the system indeed stops providing service. However, it must do so cleanly. And for many of these scenarios, it would still be expected that the system is capable of automatically recovery once the faults have healed.
"byzantine" faults: if the faults injected have corrupted more than a certain threshold of the persistently stored data, the data can be considered lost beyond hope. (Think split brain, etc.) For faults that are within the design spec, this state should never occur, even if the system had crashed; it must refuse service before reaching this state. Dependable systems also must fail gracefully ("safely") and detect this state ("scrub") and refuse service as appropriate.
While this can be run in a lab, it should actually be possible to run Orca against a production cluster as part of its on-going evaluation or pre-production certification. It may even be possible to run Teuthology while Orca is running(?), one of these days.
Basic loop: - discover topology (may be manually configured in the beginning) - Start load generator - Audit cluster health - Induce a new fault - Watch cluster state - Heal faults (possibly, unless we want to next induce one in a different failure domain) - Watch whether it heals as expected - Repeat ;-)
- Runs should be repeatable if provided with the same (random) seed and list of allowed tests.
- It must also be possible to specify a list of tests and timing explicitly.
- Configure list of tests/blacklists of tests for specific environments
- Fault inducers configurable
- Audits configurable
Number of max faults per failure domain and in total to be configurable, of course
Can this be done within Teuthology?
Can this leverage any of the Pacemaker CTS work?
Flag and abort the run if the state the cluster is in is worse than we anticipated. e.g., if we think we induced a lightly degraded cluster, but service actually went down. Or if we healed all faults and triggered a restart, and the system does not recover within a reasonable timeout.
We need to minimize false positives, otherwise it'll require just as much as overhead to sort through as Teuthology.
First step is to design the requirements a bit better and then decide where to implement this. I don't want to randomly start a new project, but also not shoehorn it into an existing project if it's not a good fit.
- Trello board for requirements/use cases? taiga.io project? ;-)
I think that's about enough for a quick draft ;-)
This project is part of:
Hack Week 14
Activity
Comments
Similar Projects
Hack on isotest-ng - a rust port of isotovideo (os-autoinst aka testrunner of openQA) by szarate
Description
Some time ago, I managed to convince ByteOtter to hack something that resembles isotovideo but in Rust, not because I believe that Perl is dead, but more because there are certain limitations in the perl code (how it was written), and its always hard to add new functionalities when they are about implementing a new backend, or fixing bugs (Along with people complaining that Perl is dead, and that they don't like it)
In reality, I wanted to see if this could be done, and ByteOtter proved that it could be, while doing an amazing job at hacking a vnc console, and helping me understand better what RuPerl needs to work.
I plan to keep working on this for the next few years, and while I don't aim for feature completion or replacing isotovideo tih isotest-ng (name in progress), I do plan to be able to use it on a daily basis, using specialized tooling with interfaces, instead of reimplementing everything in the backend
Todo
- Add
maketargets for testability, e.g "spawn qemu and type" - Add image search matching algorithm
- Add a Null test distribution provider
- Add a Perl Test Distribution Provider
- Fix unittests https://github.com/os-autoinst/isotest-ng/issues/5
- Research OpenTofu how to add new hypervisors/baremetal to OpenTofu
- Add an interface to openQA cli
Goals
- Implement at least one of the above, prepare proposals for GSoC
- Boot a system via it's BMC
Resources
See https://github.com/os-autoinst/isotest-ng
Drag Race - comparative performance testing for pull requests by balanza
Description
«Sophia, a backend developer, submitted a pull request with optimizations for a critical database query. Once she pushed her code, an automated load test ran, comparing her query against the main branch. Moments later, she saw a new comment automatically added to her PR: the comparison results showed reduced execution time and improved efficiency. Smiling, Sophia messaged her team, “Performance gains confirmed!”»
Goals
- To have a convenient and ergonomic framework to describe test scenarios, including environment and seed;
- to compare results from different tests
- to have a GitHub action that executes such tests on a CI environment
Resources
The MVP will be built on top of Preevy and K6.
Testing and adding GNU/Linux distributions on Uyuni by juliogonzalezgil
Join the Gitter channel! https://gitter.im/uyuni-project/hackweek
Uyuni is a configuration and infrastructure management tool that saves you time and headaches when you have to manage and update tens, hundreds or even thousands of machines. It also manages configuration, can run audits, build image containers, monitor and much more!
Currently there are a few distributions that are completely untested on Uyuni or SUSE Manager (AFAIK) or just not tested since a long time, and could be interesting knowing how hard would be working with them and, if possible, fix whatever is broken.
For newcomers, the easiest distributions are those based on DEB or RPM packages. Distributions with other package formats are doable, but will require adapting the Python and Java code to be able to sync and analyze such packages (and if salt does not support those packages, it will need changes as well). So if you want a distribution with other packages, make sure you are comfortable handling such changes.
No developer experience? No worries! We had non-developers contributors in the past, and we are ready to help as long as you are willing to learn. If you don't want to code at all, you can also help us preparing the documentation after someone else has the initial code ready, or you could also help with testing :-)
The idea is testing Salt and Salt-ssh clients, but NOT traditional clients, which are deprecated.
To consider that a distribution has basic support, we should cover at least (points 3-6 are to be tested for both salt minions and salt ssh minions):
- Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)
- Onboarding (salt minion from UI, salt minion from bootstrap scritp, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator)
- Package management (install, remove, update...)
- Patching
- Applying any basic salt state (including a formula)
- Salt remote commands
- Bonus point: Java part for product identification, and monitoring enablement
- Bonus point: sumaform enablement (https://github.com/uyuni-project/sumaform)
- Bonus point: Documentation (https://github.com/uyuni-project/uyuni-docs)
- Bonus point: testsuite enablement (https://github.com/uyuni-project/uyuni/tree/master/testsuite)
If something is breaking: we can try to fix it, but the main idea is research how supported it is right now. Beyond that it's up to each project member how much to hack :-)
- If you don't have knowledge about some of the steps: ask the team
- If you still don't know what to do: switch to another distribution and keep testing.
This card is for EVERYONE, not just developers. Seriously! We had people from other teams helping that were not developers, and added support for Debian and new SUSE Linux Enterprise and openSUSE Leap versions :-)
Pending
FUSS
FUSS is a complete GNU/Linux solution (server, client and desktop/standalone) based on Debian for managing an educational network.
https://fuss.bz.it/
Seems to be a Debian 12 derivative, so adding it could be quite easy.
[ ]Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)[ ]Onboarding (salt minion from UI, salt minion from bootstrap scritp, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator)[ ]Package management (install, remove, update...)[ ]Patching (if patch information is available, could require writing some code to parse it, but IIRC we have support for Ubuntu already)[ ]Applying any basic salt state (including a formula)[ ]Salt remote commands[ ]Bonus point: Java part for product identification, and monitoring enablement
Make more sense of openQA test results using AI by livdywan
Description
AI has the potential to help with something many of us spend a lot of time doing which is making sense of openQA logs when a job fails.
User Story
Allison Average has a puzzled look on their face while staring at log files that seem to make little sense. Is this a known issue, something completely new or maybe related to infrastructure changes?
Goals
- Leverage a chat interface to help Allison
- Create a model from scratch based on data from openQA
- Proof of concept for automated analysis of openQA test results
Bonus
- Use AI to suggest solutions to merge conflicts
- This would need a merge conflict editor that can suggest solving the conflict
- Use image recognition for needles
Resources
Timeline
Day 1
- Conversing with open-webui to teach me how to create a model based on openQA test results
- Asking for example code using TensorFlow in Python
- Discussing log files to explore what to analyze
- Drafting a new project called Testimony (based on Implementing a containerized Python action) - the project name was also suggested by the assistant
Day 2
- Using NotebookLLM (Gemini) to produce conversational versions of blog posts
- Researching the possibility of creating a project logo with AI
- Asking open-webui, persons with prior experience and conducting a web search for advice
Highlights
- I briefly tested compared models to see if they would make me more productive. Between llama, gemma and mistral there was no amazing difference in the results for my case.
- Convincing the chat interface to produce code specific to my use case required very explicit instructions.
- Asking for advice on how to use open-webui itself better was frustratingly unfruitful both in trivial and more advanced regards.
- Documentation on source materials used by LLM's and tools for this purpose seems virtually non-existent - specifically if a logo can be generated based on particular licenses
Outcomes
- Chat interface-supported development is providing good starting points and open-webui being open source is more flexible than Gemini. Although currently some fancy features such as grounding and generated podcasts are missing.
- Allison still has to be very experienced with openQA to use a chat interface for test review. Publicly available system prompts would make that easier, though.
Automated Test Report reviewer by oscar-barrios
Description
In SUMA/Uyuni team we spend a lot of time reviewing test reports, analyzing each of the test cases failing, checking if the test is a flaky test, checking logs, etc.
Goals
Speed up the review by automating some parts through AI, in a way that we can consume some summary of that report that could be meaningful for the reviewer.
Resources
No idea about the resources yet, but we will make use of:
- HTML/JSON Report (text + screenshots)
- The Test Suite Status GithHub board (via API)
- The environment tested (via SSH)
- The test framework code (via files)
Yearly Quality Engineering Ask me Anything - AMA for not-engineering by szarate
Goal
Get a closer look at how developers work on the Engineering team (R & D) of SUSE, and close the collaboration gap between GSI and Engineering
Why?
Santiago can go over different development workflows, and can do a deepdive into how Quality Engineering works (think of my QE Team, the advocates for your customers), The idea of this session is to help open the doors to opportunities for collaboration, and broaden our understanding of SUSE as a whole.
Objectives
- Give $audience a small window on how to get some questions answered either on the spot or within days of how some things at engineering are done
- Give Santiago Zarate from Quality Engineering a look into how $audience sees the engineering departments, and find out possibilities of further collaboration
How?
By running an "Ask me Anything" session, which is a format of a kind of open Q & A session, where participants ask the host multiple questions.
How to make it happen?
I'm happy to help joining a call or we can do it async (online/in person is more fun). Ping me over email-slack and lets make the magic happen!. Doesn't need to be during hackweek, but we gotta kickstart the idea during hackweek ;)
Rules
The rules are simple, the more questions the more fun it will be; while this will be only a window into engineering, it can also be the place to help all of us get to a similar level of understanding of the processes that are behind our respective areas of the organization.
Dynamics
The host will be monitoring the questions on some pre-agreed page, and try to answer to the best of their knowledge, if a question is too difficult or the host doesn't have the answer, he will do his best to provide an answer at a later date.
Atendees are encouraged to add questions beforehand; in the case there aren't any, we would be looking at how Quality Engineering tests new products or performs regression tests
Agenda
- Introduction of Santiago Zarate, Product Owner of Quality Engineering Core team
- Introduction of the Group/Team/Persons interested
- Ice breaker
- AMA time! Add your questions $PAGE
- Looking at QE Workflows: How is
- A maintenance update being tested before being released to our customers
- Products in development are tested before making it generally available
- Engineering Opportunity Board
Run local LLMs with Ollama and explore possible integrations with Uyuni by PSuarezHernandez
Description
Using Ollama you can easily run different LLM models in your local computer. This project is about exploring Ollama, testing different LLMs and try to fine tune them. Also, explore potential ways of integration with Uyuni.
Goals
- Explore Ollama
- Test different models
- Fine tuning
- Explore possible integration in Uyuni
Resources
- https://ollama.com/
- https://huggingface.co/
- https://apeatling.com/articles/part-2-building-your-training-data-for-fine-tuning/
ClusterOps - Easily install and manage your personal kubernetes cluster by andreabenini
Description
ClusterOps is a Kubernetes installer and operator designed to streamline the initial configuration
and ongoing maintenance of kubernetes clusters. The focus of this project is primarily on personal
or local installations. However, the goal is to expand its use to encompass all installations of
Kubernetes for local development purposes.
It simplifies cluster management by automating tasks and providing just one user-friendly YAML-based
configuration config.yml.
Overview
- Simplified Configuration: Define your desired cluster state in a simple YAML file, and ClusterOps will handle the rest.
- Automated Setup: Automates initial cluster configuration, including network settings, storage provisioning, special requirements (for example GPUs) and essential components installation.
- Ongoing Maintenance: Performs routine maintenance tasks such as upgrades, security updates, and resource monitoring.
- Extensibility: Easily extend functionality with custom plugins and configurations.
- Self-Healing: Detects and recovers from common cluster issues, ensuring stability, idempotence and reliability. Same operation can be performed multiple times without changing the result.
- Discreet: It works only on what it knows, if you are manually configuring parts of your kubernetes and this configuration does not interfere with it you can happily continue to work on several parts and use this tool only for what is needed.
Features
- distribution and engine independence. Install your favorite kubernetes engine with your package
manager, execute one script and you'll have a complete working environment at your disposal.
- Basic config approach. One single
config.ymlfile with configuration requirements (add/remove features): human readable, plain and simple. All fancy configs managed automatically (ingress, balancers, services, proxy, ...). - Local Builtin ContainerHub. The default installation provides a fully configured ContainerHub available locally along with the kubernetes installation. This configuration allows the user to build, upload and deploy custom container images as they were provided from external sources. Internet public sources are still available but local development can be kept in this localhost server. Builtin ClusterOps operator will be fetched from this ContainerHub registry too.
- Kubernetes official dashboard installed as a plugin, others planned too (k9s for example).
- Kubevirt plugin installed and properly configured. Unleash the power of classic virtualization (KVM+QEMU) on top of Kubernetes and manage your entire system from there, libvirtd and virsh libs are required.
- One operator to rule them all. The installation script configures your machine automatically during installation and adds one kubernetes operator to manage your local cluster. From there the operator takes care of the cluster on your behalf.
- Clean installation and removal. Just test it, when you are done just use the same program to uninstall everything without leaving configs (or pods) behind.
Planned features (Wishlist / TODOs)
- Containerized Data Importer (CDI). Persistent storage management add-on for Kubernetes to provide a declarative way of building and importing Virtual Machine Disks on PVCs for
Team Hedgehogs' Data Observability Dashboard by gsamardzhiev
Description
This project aims to develop a comprehensive Data Observability Dashboard that provides r insights into key aspects of data quality and reliability. The dashboard will track:
Data Freshness: Monitor when data was last updated and flag potential delays.
Data Volume: Track table row counts to detect unexpected surges or drops in data.
Data Distribution: Analyze data for null values, outliers, and anomalies to ensure accuracy.
Data Schema: Track schema changes over time to prevent breaking changes.
The dashboard's aim is to support historical tracking to support proactive data management and enhance data trust across the data function.
Goals
Although the final goal is to create a power bi dashboard that we are able to monitor, our goals is to 1. Create the necessary tables that track the relevant metadata about our current data 2. Automate the process so it runs in a timely manner
Resources
AWS Redshift; AWS Glue, Airflow, Python, SQL
Why Hedgehogs?
Because we like them.
Testing and adding GNU/Linux distributions on Uyuni by juliogonzalezgil
Join the Gitter channel! https://gitter.im/uyuni-project/hackweek
Uyuni is a configuration and infrastructure management tool that saves you time and headaches when you have to manage and update tens, hundreds or even thousands of machines. It also manages configuration, can run audits, build image containers, monitor and much more!
Currently there are a few distributions that are completely untested on Uyuni or SUSE Manager (AFAIK) or just not tested since a long time, and could be interesting knowing how hard would be working with them and, if possible, fix whatever is broken.
For newcomers, the easiest distributions are those based on DEB or RPM packages. Distributions with other package formats are doable, but will require adapting the Python and Java code to be able to sync and analyze such packages (and if salt does not support those packages, it will need changes as well). So if you want a distribution with other packages, make sure you are comfortable handling such changes.
No developer experience? No worries! We had non-developers contributors in the past, and we are ready to help as long as you are willing to learn. If you don't want to code at all, you can also help us preparing the documentation after someone else has the initial code ready, or you could also help with testing :-)
The idea is testing Salt and Salt-ssh clients, but NOT traditional clients, which are deprecated.
To consider that a distribution has basic support, we should cover at least (points 3-6 are to be tested for both salt minions and salt ssh minions):
- Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)
- Onboarding (salt minion from UI, salt minion from bootstrap scritp, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator)
- Package management (install, remove, update...)
- Patching
- Applying any basic salt state (including a formula)
- Salt remote commands
- Bonus point: Java part for product identification, and monitoring enablement
- Bonus point: sumaform enablement (https://github.com/uyuni-project/sumaform)
- Bonus point: Documentation (https://github.com/uyuni-project/uyuni-docs)
- Bonus point: testsuite enablement (https://github.com/uyuni-project/uyuni/tree/master/testsuite)
If something is breaking: we can try to fix it, but the main idea is research how supported it is right now. Beyond that it's up to each project member how much to hack :-)
- If you don't have knowledge about some of the steps: ask the team
- If you still don't know what to do: switch to another distribution and keep testing.
This card is for EVERYONE, not just developers. Seriously! We had people from other teams helping that were not developers, and added support for Debian and new SUSE Linux Enterprise and openSUSE Leap versions :-)
Pending
FUSS
FUSS is a complete GNU/Linux solution (server, client and desktop/standalone) based on Debian for managing an educational network.
https://fuss.bz.it/
Seems to be a Debian 12 derivative, so adding it could be quite easy.
[ ]Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)[ ]Onboarding (salt minion from UI, salt minion from bootstrap scritp, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator)[ ]Package management (install, remove, update...)[ ]Patching (if patch information is available, could require writing some code to parse it, but IIRC we have support for Ubuntu already)[ ]Applying any basic salt state (including a formula)[ ]Salt remote commands[ ]Bonus point: Java part for product identification, and monitoring enablement
SUSE AI Meets the Game Board by moio
Use tabletopgames.ai’s open source TAG and PyTAG frameworks to apply Statistical Forward Planning and Deep Reinforcement Learning to two board games of our own design. On an all-green, all-open source, all-AWS stack!
AI + Board Games
Board games have long been fertile ground for AI innovation, pushing the boundaries of capabilities such as strategy, adaptability, and real-time decision-making - from Deep Blue's chess mastery to AlphaZero’s domination of Go. Games aren’t just fun: they’re complex, dynamic problems that often mirror real-world challenges, making them interesting from an engineering perspective.
As avid board gamers, aspiring board game designers, and engineers with careers in open source infrastructure, we’re excited to dive into the latest AI techniques first-hand.
Our goal is to develop an all-open-source, all-green AWS-based stack powered by some serious hardware to drive our board game experiments forward!
Project Goals
Set Up the Stack:
- Install and configure the TAG and PyTAG frameworks on SUSE Linux Enterprise Base Container Images.
- Integrate with the SUSE AI stack for GPU-accelerated training on AWS.
- Validate a sample GPU-accelerated PyTAG workload on SUSE AI.
- Ensure the setup is entirely repeatable with Terraform and configuration scripts, documenting results along the way.
Design and Implement AI Agents:
- Develop AI agents for the two board games, incorporating Statistical Forward Planning and Deep Reinforcement Learning techniques.
- Fine-tune model parameters to optimize game-playing performance.
- Document the advantages and limitations of each technique.
Test, Analyze, and Refine:
- Conduct AI vs. AI and AI vs. human matches to evaluate agent strategies and performance.
- Record insights, document learning outcomes, and refine models based on real-world gameplay.
Technical Stack
- Frameworks: TAG and PyTAG for AI agent development
- Platform: SUSE AI
- Tools: AWS for high-performance GPU acceleration
Why This Project Matters
This project not only deepens our understanding of AI techniques by doing but also showcases the power and flexibility of SUSE’s open-source infrastructure for supporting high-level AI projects. By building on an all-open-source stack, we aim to create a pathway for other developers and AI enthusiasts to explore, experiment, and deploy their own innovative projects within the open-source space.
Our Motivation
We believe hands-on experimentation is the best teacher.
Combining our engineering backgrounds with our passion for board games, we’ll explore AI in a way that’s both challenging and creatively rewarding. Our ultimate goal? To hack an AI agent that’s as strategic and adaptable as a real human opponent (if not better!) — and to leverage it to design even better games... for humans to play!