openQA Package Testing
a project by RBrownSUSE
Updated over 5 years ago. 13 hacker ♥️. 1 follower.

openQA has a well earned reputation as a 'full system' testing tool, able to test a system end-to-end from the operating system to it's applications on a number of different platforms and architectures, including VM's & Bare Metal.

But one area of weakness is it's usefulness as a testing tool for developers or packagers. openQA can easily test a package once it's INSIDE a distribution, but how do you test that package BEFORE submitting it to the distribution?

openSUSE/SLE has the concept of 'staging', where we utilise OBS and openQA to basically create 'what-if' builds's for the whole OS where we can do basic validation tests to ensure a proposed check-in doesn't break the OS or it's basic functionality, but staging doesn't test the package itself

This hackweek project will experiment with solutions to this problem.

Concepts to be investigated include

  • Providing reference disk images for package-specific tests for each OS (SLE, Tumbleweed, Leap)
  • Injecting test code into openQA without needing it to be submitted via github. This might include openQA tests being stored as part of the OBS package.
  • Loading package-specific tests in openQA by feeding openQA variables over its REST API
  • Booting these reference images to either a console or X11 as required by the test
  • Testing and providing the results in a meaningful way to the Developer

I will try to find solutions to the above which will work in the following contexts

  • a Developer working 'packageless' with just a git repo and a local compiler (unlikely, but still want to think about it)
  • a Developer working on a package in their OBS Home/Devel project
  • Release Management/Devel Project reviewers of an OBS submit request wanting to see the results of Package Testing BEFORE deciding whether or not the package can be checked-in or forwarded to staging

Join this project

Looking for hackers with the skills:

perl openqa obs

This project is part of:

Hack Week 15

Activity

  • over 5 years ago: tjyrinki_suse liked this project.
  • over 5 years ago: cdywan liked this project.
  • almost 8 years ago: tbechtold liked this project.
  • almost 8 years ago: pgeorgiadis liked this project.
  • almost 8 years ago: szarate liked this project.
  • almost 8 years ago: bchou liked this project.
  • almost 8 years ago: okurz liked this project.
  • almost 8 years ago: pluskalm liked this project.
  • almost 8 years ago: osukup liked this project.
  • almost 8 years ago: diegoakechi liked this project.
  • almost 8 years ago: Knurpht liked this project.
  • almost 8 years ago: aaronluna75 liked this project.
  • almost 8 years ago: RBrownSUSE added keyword "openqa" to this project.
  • almost 8 years ago: RBrownSUSE added keyword "obs" to this project.
  • almost 8 years ago: RBrownSUSE added keyword "perl" to this project.
  • almost 8 years ago: RBrownSUSE liked this project.
  • almost 8 years ago: RBrownSUSE started this project.
  • almost 8 years ago: RBrownSUSE originated this project.


    • Comments

    • RBrownSUSE
      almost 8 years ago by RBrownSUSE | Reply

      Part 2 - Autogenerate X11 tests from Desktop files in packages - including booting the image for the first time, running the app for the first time, and autoneedling for the first time

    • tbechtold
      almost 8 years ago by tbechtold | Reply

      Maybe interessting is the system Debian/Ubuntu uses called autopkgtest. See also https://ci.debian.net/

      • pgeorgiadis
        almost 8 years ago by pgeorgiadis | Reply

        I watched the relevant talk at FOSDEM and it seems really great the fact that they are running all the upstream testsuites plus the fact that they trigger these testsuites against the package's reverse dependencies... (!) But autopkgtest it's tighten only to Debian, which means we can't just take it and use it as it is. We have to come up with our own equivalent system. But the most important part here is that the developers/maintainer has to take care about these upstream tests, so IMHO there has to be an agreement that the src rpm should include the tests into them, and build service has to have the resources + the knowledge on how to run them; then either reject or forward the update up to openQA.

    Similar Projects

    Create object oriented API for perl's YAML::XS module, with YAML 1.2 Support by tinita

    Description

    YAML::XS is a binding to libyaml and already quite old, but the most popular YAML module for perl. There are two main issues:

    • It uses global package variables to influence behaviour.
    • It didn't implement the loading of types like numbers and booleans according to the YAML spec (neither 1.1 nor 1.2).

    Goals

    Create a new interface which works object oriented. Currently YAML::XS exports a list of functions.

    • The new API will allow to create a YAML::XS object containing configuration influencing the behaviour of loading and dumping.
      • It keeps the libyaml parser and emitter structs in memory, so repeated calls can save the creation of those structs
    • It will by default implement the YAML 1.2 Core Schema, so it is compatible to other YAML processors in perl and in other languages
    • If I have time, I would like to add the merge << key feature as an option. We could then use it in openQA as a replacement for YAML::PP to be faster.

    I already created a proof of concept with a minimal functionality some weeks before this HackWeek.

    Resources

    Make more sense of openQA test results using AI by livdywan

    Description

    AI has the potential to help with something many of us spend a lot of time doing which is making sense of openQA logs when a job fails.

    User Story

    Allison Average has a puzzled look on their face while staring at log files that seem to make little sense. Is this a known issue, something completely new or maybe related to infrastructure changes?

    Goals

    • Leverage a chat interface to help Allison
    • Create a model from scratch based on data from openQA
    • Proof of concept for automated analysis of openQA test results

    Bonus

    • Use AI to suggest solutions to merge conflicts
      • This would need a merge conflict editor that can suggest solving the conflict
    • Use image recognition for needles

    Resources

    Timeline

    Day 1

    • Conversing with open-webui to teach me how to create a model based on openQA test results

    Day 2

    Highlights

    • I briefly tested compared models to see if they would make me more productive. Between llama, gemma and mistral there was no amazing difference in the results for my case.
    • Convincing the chat interface to produce code specific to my use case required very explicit instructions.
    • Asking for advice on how to use open-webui itself better was frustratingly unfruitful both in trivial and more advanced regards.
    • Documentation on source materials used by LLM's and tools for this purpose seems virtually non-existent - specifically if a logo can be generated based on particular licenses

    Outcomes

    • Chat interface-supported development is providing good starting points and open-webui being open source is more flexible than Gemini. Although currently some fancy features such as grounding and generated podcasts are missing.
    • Allison still has to be very experienced with openQA to use a chat interface for test review. Publicly available system prompts would make that easier, though.

    Setup a new openQA on more powerful server by JNa

    Description

    • currently local openQA storage is insufficient

    Goals

    -Migrate to more powerful machine

    Resources

    -Service Rainbow

    Learn obs/ibs sync tool by xlai

    Description

    Once images/repo are built from IBS/OBS, there is a tool to sync the image from IBS/OBS to openqa asset directory and trigger openqa jobs accordingly.

    Goals

    Check how the tool is implemented, and be capable to add/modify our needed images/repo in future by ourselves.

    Resources

    • https://github.com/os-autoinst/openqa-trigger-from-obs
    • https://gitlab.suse.de/openqa/openqa-trigger-from-ibs-plugin/-/tree/master?ref_type=heads

    Enhance UV openQA helper script by mdonis

    Description

    A couple months ago an UV openQA helper script was created to help/automate the searching phase inside openQA for a given MU to test. The script searches inside all our openQA job groups (qam-sle) related with a given MU and generates an output suitable to add (copy & paste) inside the update log.

    This is still a WIP and could use some enhancements.

    Goals

    • Move script from bash to python: this would be useful in case we want to include this into MTUI in the future. The script will be separate from MTUI for now. The idea is to have this as a CLI tool using the click library or something similar.
    • Add option to look for jobs in other sections inside aggregated updates: right now, when looking for regression tests under aggregated updates for a given MU, the script only looks inside the Core MU job group. This is where most of the regression tests we need are located, but some MUs have their regression tests under the YaST/Containers/Security MU job groups. We should keep the Core MU group as a default, but add an option to be able to look into other job groups under aggregated updates.
    • Remove the -a option: this option is used to indicate the update ID and is mandatory right now. This is a bit weird and goes against posix stardards. It was developed this way in order to avoid using positional parameters. This problem should be fixed if we move the script to python.

    Some other ideas to consider:

    • Look into the QAM dashboard API. This has more info on each MU, could use this to link general openQA build results, whether the related RR is approved or not, etc
    • Make it easier to see if there's regression tests for a package in an openQA test build. Check if there's a possibility to search for tests that have the package name in them inside each testsuite.
    • Unit testing?

    More ideas TBD

    Resources

    https://github.com/os-autoinst/scripts/blob/master/openqa-search-maintenance-core-jobs

    https://confluence.suse.com/display/maintenanceqa/Guide+on+how+to+test+Updates

    Post-Hackweek update

    All major features were implemented. Unit tests are still in progress, and project will be moved to the SUSE github org once everything's done. https://github.com/mjdonis/oqa-search

    Hack on isotest-ng - a rust port of isotovideo (os-autoinst aka testrunner of openQA) by szarate

    Description

    Some time ago, I managed to convince ByteOtter to hack something that resembles isotovideo but in Rust, not because I believe that Perl is dead, but more because there are certain limitations in the perl code (how it was written), and its always hard to add new functionalities when they are about implementing a new backend, or fixing bugs (Along with people complaining that Perl is dead, and that they don't like it)

    In reality, I wanted to see if this could be done, and ByteOtter proved that it could be, while doing an amazing job at hacking a vnc console, and helping me understand better what RuPerl needs to work.

    I plan to keep working on this for the next few years, and while I don't aim for feature completion or replacing isotovideo tih isotest-ng (name in progress), I do plan to be able to use it on a daily basis, using specialized tooling with interfaces, instead of reimplementing everything in the backend

    Todo

    • Add make targets for testability, e.g "spawn qemu and type"
    • Add image search matching algorithm
    • Add a Null test distribution provider
    • Add a Perl Test Distribution Provider
    • Fix unittests https://github.com/os-autoinst/isotest-ng/issues/5
    • Research OpenTofu how to add new hypervisors/baremetal to OpenTofu
    • Add an interface to openQA cli

    Goals

    • Implement at least one of the above, prepare proposals for GSoC
    • Boot a system via it's BMC

    Resources

    See https://github.com/os-autoinst/isotest-ng

    Explore the integration between OBS and GitHub by pdostal

    Project Description

    The goals:

    1) When GitHub pull request is created or modified the OBS project will be forked and the build results reported back to GitHub. 2) When new version of the GitHub project will be published the OBS will redownload the source and rebuild the project.

    Goal for this Hackweek

    Do as much as possible, blog about it and maybe use it another existing project.

    Resources

    Git CI to automate the creation of product definition by gyribeiro

    Description

    Automate the creation of product definition

    Goals

    Create a Git CI that will:

    • automatically be triggered once a change (commit) in package list is done.
    • run tool responsible to update product definition based on the changes in package list
    • test the updated product definition in OBS
    • submit a pull request updating the product definition in the repository

    NOTE: this Git CI may also be triggered manually

    Resources

    • https://docs.gitlab.com/ee/ci/
    • https://openbuildservice.org/2021/05/31/scm-integration/
    • https://github.com/openSUSE/openSUSE-release-tools

    Automation of ABI compatibility checks by ateixeira

    Description

    ABI compatibility checks could be further automated by using the OBS API to download built RPMs and using existing tools to analyze ABI compatibility between the libraries contained in those packages. This project aims to explore these possibilities and figure out a way to make ABI checks as painless and fast as possible for package maintainers.

    Resources

    https://github.com/openSUSE/abi-compliance-checker

    https://github.com/lvc/abi-compliance-checker

    https://sourceware.org/libabigail/

    Testing and adding GNU/Linux distributions on Uyuni by juliogonzalezgil

    Join the Gitter channel! https://gitter.im/uyuni-project/hackweek

    Uyuni is a configuration and infrastructure management tool that saves you time and headaches when you have to manage and update tens, hundreds or even thousands of machines. It also manages configuration, can run audits, build image containers, monitor and much more!

    Currently there are a few distributions that are completely untested on Uyuni or SUSE Manager (AFAIK) or just not tested since a long time, and could be interesting knowing how hard would be working with them and, if possible, fix whatever is broken.

    For newcomers, the easiest distributions are those based on DEB or RPM packages. Distributions with other package formats are doable, but will require adapting the Python and Java code to be able to sync and analyze such packages (and if salt does not support those packages, it will need changes as well). So if you want a distribution with other packages, make sure you are comfortable handling such changes.

    No developer experience? No worries! We had non-developers contributors in the past, and we are ready to help as long as you are willing to learn. If you don't want to code at all, you can also help us preparing the documentation after someone else has the initial code ready, or you could also help with testing :-)

    The idea is testing Salt and Salt-ssh clients, but NOT traditional clients, which are deprecated.

    To consider that a distribution has basic support, we should cover at least (points 3-6 are to be tested for both salt minions and salt ssh minions):

    1. Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)
    2. Onboarding (salt minion from UI, salt minion from bootstrap scritp, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator)
    3. Package management (install, remove, update...)
    4. Patching
    5. Applying any basic salt state (including a formula)
    6. Salt remote commands
    7. Bonus point: Java part for product identification, and monitoring enablement
    8. Bonus point: sumaform enablement (https://github.com/uyuni-project/sumaform)
    9. Bonus point: Documentation (https://github.com/uyuni-project/uyuni-docs)
    10. Bonus point: testsuite enablement (https://github.com/uyuni-project/uyuni/tree/master/testsuite)

    If something is breaking: we can try to fix it, but the main idea is research how supported it is right now. Beyond that it's up to each project member how much to hack :-)

    • If you don't have knowledge about some of the steps: ask the team
    • If you still don't know what to do: switch to another distribution and keep testing.

    This card is for EVERYONE, not just developers. Seriously! We had people from other teams helping that were not developers, and added support for Debian and new SUSE Linux Enterprise and openSUSE Leap versions :-)

    Pending

    FUSS

    FUSS is a complete GNU/Linux solution (server, client and desktop/standalone) based on Debian for managing an educational network.

    https://fuss.bz.it/

    Seems to be a Debian 12 derivative, so adding it could be quite easy.

    • [W] Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)
    • [W] Onboarding (salt minion from UI, salt minion from bootstrap script, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator) --> Working for all 3 options (salt minion UI, salt minion bootstrap script and salt-ssh minion from the UI).
    • [W] Package management (install, remove, update...) --> Installing a new package works, needs to test the rest.
    • [I] Patching (if patch information is available, could require writing some code to parse it, but IIRC we have support for Ubuntu already). No patches detected. Do we support patches for Debian at all?
    • [W] Applying any basic salt state (including a formula)
    • [W] Salt remote commands
    • [ ] Bonus point: Java part for product identification, and monitoring enablement

    Research openqa-trigger-from-obs and openqa-trigger-from-ibs-plugin by qwang

    Description

    openqa-trigger-from-obs project is a framework that OSD is using it to automatically sync the defined images and repositories from OBS/IBS to its assets for testing. This framework very likely will be used for the synchronize to each location's openqa include openqa.qa2.suse.asia Beijing local procy scc scc-proxy.suse.asia(although it's not a MUST to our testing) it's now rewriting requests to openqa.qa2.suse.asia instead of openqa.suse.de, the assets/repo should be consistent the format Beijing local openQA is maintaining an own script but still need many manually activities when new build comes, and not consistent to OSD, that will request many test code change due to CC network change

    Goals

    Research this framework in case it will be re-used for Beijing local openQA, and will need to be setup and maintained by ourselves

    Resources

    https://github.com/os-autoinst/openqa-trigger-from-obs/tree/master https://gitlab.suse.de/openqa/openqa-trigger-from-ibs-plugin

    beijing :rainbow machine