I once had a bad dream.

I started good, a sunny day. I had just fixed an issue and push it to my fork, in order to create a Pull Request. I was happy. It felt awesome to have found a fix so elegant. Two lines of code.

But then, something happened. A cloud appear in the sky and partially hide the sun. github triggered the end to end test suite. You could see the waiting icon, you could almost hear the engines starting, ... a lightning and a thunder appeared in the sky, the sky turned dark and the e2e test started in our private jenkins server.

1 hour passed by... 2 hours .... the e2e tests still running ... 4 hours, not even 50%. Finally, I had to go, get some dinner, get some rest, so I decided to look at it the next day.

Sleep was not good. Dreaming the tests would fail and the deadline be missed .... I couldn't sleep no more so I wake up early in the morning, before sunrise. The computer still opened was lightning up the room. Out, the storm turned out to be a heavy rain. And the tests did not passed! 4 hours and half after starting them, so half an hour after leaving for dinner, and there was a fail test.

Finding out what went wrong took 2 hours...it was not even related to the code, but an infrastructure issue. A timeout when connecting to a mirror which was being restarted when the tests run. Apparently we hit a maintenance window.

Damn! So let's just "restart" the tests. This time, I decided to put an alarm 4 hours after and switch to another task, just to avoid the anxiety of the previous evening looking at the results. 4 hours passed by, and the tests are good. It is not raining anymore, a git of sun is filtering throw the window, hope is in the air. 4 more hours, and the tests still have not failed. Going to dinner and bed again and putting the alarm early in the morning. First thing in the morning I looked at the tests ... running again, feels good. All tests have passed and there is only one final "cleanup" state. Shower, breakfeast, sun is shining again!

Finally, all is green, so let's go and push the merge button ... oh oh ... clouds hide the sun again ... where is the merge button?? no way, there is a conflict with the code in master and I can't merge!!! I can't merge!!! shocked, I looked into the history ... John merged a PR overnight that was touching the same file ... hopeless I start crying on my desk...

Scared?

This is not so different on what could happen if we were running the whole e2e test suite in the SUSE Manager (uyuni) Pull Requests. However, not running any e2e tests has also a bad consequence. We find the issues after code has been merged in master, and then we spend days looking at what could have caused this, reverting commits, and starting again.

However, if we could run a subset of the e2e tests, we could shorten the time it takes and so run them at the PR level, so no broken code could get merged.

The thing is , how do we select which tests to run?

This project is about using Machine Learning to do predictive test selection, thus selecting which tests to run based on the history of previous test runs.

Inspired by: https://engineering.fb.com/2018/11/21/developer-tools/predictive-test-selection/

Looking for hackers with the skills:

ml ci qa ai

This project is part of:

Hack Week 20

Activity

  • over 3 years ago: PSuarezHernandez liked this project.
  • over 3 years ago: llansky3 liked this project.
  • over 3 years ago: ories liked this project.
  • over 3 years ago: j_renner liked this project.
  • over 3 years ago: RDiasMateus liked this project.
  • almost 4 years ago: pagarcia liked this project.
  • almost 4 years ago: jordimassaguerpla added keyword "ci" to this project.
  • almost 4 years ago: jordimassaguerpla added keyword "qa" to this project.
  • almost 4 years ago: jordimassaguerpla added keyword "ai" to this project.
  • almost 4 years ago: jordimassaguerpla added keyword "ml" to this project.
  • almost 4 years ago: jordimassaguerpla originated this project.

  • Comments

    • llansky3
      over 3 years ago by llansky3 | Reply

      This could have some wider potential - clever picking of tests to shorten the time yet maximize the findings (and learning that in time based on test execution history to predict). I can see how that saves tons of money in cases where test on a real industrial machines need to be executed (e.g. cost of running engine on a test bed is 5-10k$ per day so there is huge difference if you need to run for 1 or 3 days)

    Similar Projects

    COOTWbot by ngetahun

    Project Description

    At SCC, we have a rotating task of COOTW (Commanding Office of the Week). This task involves responding to customer requests from jira and slack help channels, monitoring production systems and doing small chores. Usually, we have documentation to help the COOTW answer questions and quickly find fixes. Most of these are distributed across github, trello and SUSE Support documentation. The aim of this project is to explore the magic of LLMs and create a conversational bot.

    Goal for this Hackweek

    • Build data ingestion Data source:
      • SUSE KB docs
      • scc github docs
      • scc trello knowledge board
    • Test out new RAG architecture

    • https://gitlab.suse.de/ngetahun/cootwbot


    Drag Race - comparative performance testing for pull requests by balanza

    Description

    «Sophia, a backend developer, submitted a pull request with optimizations for a critical database query. Once she pushed her code, an automated load test ran, comparing her query against the main branch. Moments later, she saw a new comment automatically added to her PR: the comparison results showed reduced execution time and improved efficiency. Smiling, Sophia messaged her team, “Performance gains confirmed!”»

    Goals

    • To have a convenient and ergonomic framework to describe test scenarios, including environment and seed;
    • to compare results from different tests
    • to have a GitHub action that executes such tests on a CI environment

    Resources

    The MVP will be built on top of Preevy and K6.


    Yearly Quality Engineering Ask me Anything - AMA for not-engineering by szarate

    Goal

    Get a closer look at how developers work on the Engineering team (R & D) of SUSE, and close the collaboration gap between GSI and Engineering

    Why?

    Santiago can go over different development workflows, and can do a deepdive into how Quality Engineering works (think of my QE Team, the advocates for your customers), The idea of this session is to help open the doors to opportunities for collaboration, and broaden our understanding of SUSE as a whole.

    Objectives

    • Give $audience a small window on how to get some questions answered either on the spot or within days of how some things at engineering are done
    • Give Santiago Zarate from Quality Engineering a look into how $audience sees the engineering departments, and find out possibilities of further collaboration

    How?

    By running an "Ask me Anything" session, which is a format of a kind of open Q & A session, where participants ask the host multiple questions.

    How to make it happen?

    I'm happy to help joining a call or we can do it async (online/in person is more fun). Ping me over email-slack and lets make the magic happen!. Doesn't need to be during hackweek, but we gotta kickstart the idea during hackweek ;)

    Rules

    The rules are simple, the more questions the more fun it will be; while this will be only a window into engineering, it can also be the place to help all of us get to a similar level of understanding of the processes that are behind our respective areas of the organization.

    Dynamics

    The host will be monitoring the questions on some pre-agreed page, and try to answer to the best of their knowledge, if a question is too difficult or the host doesn't have the answer, he will do his best to provide an answer at a later date.

    Atendees are encouraged to add questions beforehand; in the case there aren't any, we would be looking at how Quality Engineering tests new products or performs regression tests

    Agenda

    • Introduction of Santiago Zarate, Product Owner of Quality Engineering Core team
    • Introduction of the Group/Team/Persons interested
    • Ice breaker
    • AMA time! Add your questions $PAGE
    • Looking at QE Workflows: How is
      • A maintenance update being tested before being released to our customers
      • Products in development are tested before making it generally available
    • Engineering Opportunity Board


    Learn how to integrate Elixir and Phoenix Liveview with LLMs by ninopaparo

    Description

    Learn how to integrate Elixir and Phoenix Liveview with LLMs by building an application that can provide answers to user queries based on a corpus of custom-trained data.

    Goals

    Develop an Elixir application via the Phoenix framework that:

    • Employs Retrieval Augmented Generation (RAG) techniques
    • Supports the integration and utilization of various Large Language Models (LLMs).
    • Is designed with extensibility and adaptability in mind to accommodate future enhancements and modifications.

    Resources

    • https://elixir-lang.org/
    • https://www.phoenixframework.org/
    • https://github.com/elixir-nx/bumblebee
    • https://ollama.com/


    Automated Test Report reviewer by oscar-barrios

    Description

    In SUMA/Uyuni team we spend a lot of time reviewing test reports, analyzing each of the test cases failing, checking if the test is a flaky test, checking logs, etc.

    Goals

    Speed up the review by automating some parts through AI, in a way that we can consume some summary of that report that could be meaningful for the reviewer.

    Resources

    No idea about the resources yet, but we will make use of:

    • HTML/JSON Report (text + screenshots)
    • The Test Suite Status GithHub board (via API)
    • The environment tested (via SSH)
    • The test framework code (via files)


    SUSE AI Meets the Game Board by moio

    Use tabletopgames.ai’s open source TAG and PyTAG frameworks to apply Statistical Forward Planning and Deep Reinforcement Learning to two board games of our own design. On an all-green, all-open source, all-AWS stack!
    A chameleon playing chess in a train car, as a metaphor of SUSE AI applied to games


    AI + Board Games

    Board games have long been fertile ground for AI innovation, pushing the boundaries of capabilities such as strategy, adaptability, and real-time decision-making - from Deep Blue's chess mastery to AlphaZero’s domination of Go. Games aren’t just fun: they’re complex, dynamic problems that often mirror real-world challenges, making them interesting from an engineering perspective.

    As avid board gamers, aspiring board game designers, and engineers with careers in open source infrastructure, we’re excited to dive into the latest AI techniques first-hand.

    Our goal is to develop an all-open-source, all-green AWS-based stack powered by some serious hardware to drive our board game experiments forward!


    Project Goals

    1. Set Up the Stack:

      • Install and configure the TAG and PyTAG frameworks on SUSE Linux Enterprise Base Container Images.
      • Integrate with the SUSE AI stack for GPU-accelerated training on AWS.
      • Validate a sample GPU-accelerated PyTAG workload on SUSE AI.
      • Ensure the setup is entirely repeatable with Terraform and configuration scripts, documenting results along the way.
    2. Design and Implement AI Agents:

      • Develop AI agents for the two board games, incorporating Statistical Forward Planning and Deep Reinforcement Learning techniques.
      • Fine-tune model parameters to optimize game-playing performance.
      • Document the advantages and limitations of each technique.
    3. Test, Analyze, and Refine:

      • Conduct AI vs. AI and AI vs. human matches to evaluate agent strategies and performance.
      • Record insights, document learning outcomes, and refine models based on real-world gameplay.

    Technical Stack

    • Frameworks: TAG and PyTAG for AI agent development
    • Platform: SUSE AI
    • Tools: AWS for high-performance GPU acceleration

    Why This Project Matters

    This project not only deepens our understanding of AI techniques by doing but also showcases the power and flexibility of SUSE’s open-source infrastructure for supporting high-level AI projects. By building on an all-open-source stack, we aim to create a pathway for other developers and AI enthusiasts to explore, experiment, and deploy their own innovative projects within the open-source space.


    Our Motivation

    We believe hands-on experimentation is the best teacher.

    Combining our engineering backgrounds with our passion for board games, we’ll explore AI in a way that’s both challenging and creatively rewarding. Our ultimate goal? To hack an AI agent that’s as strategic and adaptable as a real human opponent (if not better!) — and to leverage it to design even better games... for humans to play!


    ghostwrAIter - a local AI assisted tool for helping with support cases by paolodepa

    Description

    This project is meant to fight the loneliness of the support team members, providing them an AI assistant (hopefully) capable of scraping supportconfigs in a RAG fashion, trying to answer specific questions.

    Goals

    • Setup an Ollama backend, spinning one (or more??) code-focused LLMs selected by license, performance and quality of the results between:
    • Setup a Web UI for it, choosing an easily extensible and customizable option between:
    • Extend the solution in order to be able to:
      • Add ZIU/Concord shared folders to its RAG context
      • Add BZ cases, splitted in comments to its RAG context
        • A plus would be to login using the IDP portal to ghostwrAIter itself and use the same credentials to query BZ
      • Add specific packages picking them from IBS repos
        • A plus would be to login using the IDP portal to ghostwrAIter itself and use the same credentials to query IBS
        • A plus would be to desume the packages of interest and the right channel and version to be picked from the added BZ cases


    AI for product management by a_jaeger

    Description

    Learn about AI and how it can help myself

    What are the jobs that a PM does where AI can help - and how?

    Goals

    • Investigate how AI can help with different tasks
    • Check out different AI tools, which one is best for which job
    • Summarize learning

    Resources

    • Reading some blog posts by PMs that looked into it
    • Popular and less popular AI tools