openQA Package Testing
a project by RBrownSUSE
Updated almost 6 years ago. 13 hacker ♥️. 1 follower.

openQA has a well earned reputation as a 'full system' testing tool, able to test a system end-to-end from the operating system to it's applications on a number of different platforms and architectures, including VM's & Bare Metal.

But one area of weakness is it's usefulness as a testing tool for developers or packagers. openQA can easily test a package once it's INSIDE a distribution, but how do you test that package BEFORE submitting it to the distribution?

openSUSE/SLE has the concept of 'staging', where we utilise OBS and openQA to basically create 'what-if' builds's for the whole OS where we can do basic validation tests to ensure a proposed check-in doesn't break the OS or it's basic functionality, but staging doesn't test the package itself

This hackweek project will experiment with solutions to this problem.

Concepts to be investigated include

  • Providing reference disk images for package-specific tests for each OS (SLE, Tumbleweed, Leap)
  • Injecting test code into openQA without needing it to be submitted via github. This might include openQA tests being stored as part of the OBS package.
  • Loading package-specific tests in openQA by feeding openQA variables over its REST API
  • Booting these reference images to either a console or X11 as required by the test
  • Testing and providing the results in a meaningful way to the Developer

I will try to find solutions to the above which will work in the following contexts

  • a Developer working 'packageless' with just a git repo and a local compiler (unlikely, but still want to think about it)
  • a Developer working on a package in their OBS Home/Devel project
  • Release Management/Devel Project reviewers of an OBS submit request wanting to see the results of Package Testing BEFORE deciding whether or not the package can be checked-in or forwarded to staging

Join this project

Looking for hackers with the skills:

perl openqa obs

This project is part of:

Hack Week 15

Activity

  • almost 6 years ago: tjyrinki_suse liked this project.
  • almost 6 years ago: cdywan liked this project.
  • about 8 years ago: tbechtold liked this project.
  • about 8 years ago: pgeorgiadis liked this project.
  • about 8 years ago: szarate liked this project.
  • about 8 years ago: bchou liked this project.
  • about 8 years ago: okurz liked this project.
  • about 8 years ago: pluskalm liked this project.
  • about 8 years ago: osukup liked this project.
  • about 8 years ago: diegoakechi liked this project.
  • about 8 years ago: Knurpht liked this project.
  • about 8 years ago: aaronluna75 liked this project.
  • about 8 years ago: RBrownSUSE added keyword "openqa" to this project.
  • about 8 years ago: RBrownSUSE added keyword "obs" to this project.
  • about 8 years ago: RBrownSUSE added keyword "perl" to this project.
  • about 8 years ago: RBrownSUSE liked this project.
  • about 8 years ago: RBrownSUSE started this project.
  • about 8 years ago: RBrownSUSE originated this project.


    • Comments

    • RBrownSUSE
      about 8 years ago by RBrownSUSE | Reply

      Part 2 - Autogenerate X11 tests from Desktop files in packages - including booting the image for the first time, running the app for the first time, and autoneedling for the first time

    • tbechtold
      about 8 years ago by tbechtold | Reply

      Maybe interessting is the system Debian/Ubuntu uses called autopkgtest. See also https://ci.debian.net/

      • pgeorgiadis
        about 8 years ago by pgeorgiadis | Reply

        I watched the relevant talk at FOSDEM and it seems really great the fact that they are running all the upstream testsuites plus the fact that they trigger these testsuites against the package's reverse dependencies... (!) But autopkgtest it's tighten only to Debian, which means we can't just take it and use it as it is. We have to come up with our own equivalent system. But the most important part here is that the developers/maintainer has to take care about these upstream tests, so IMHO there has to be an agreement that the src rpm should include the tests into them, and build service has to have the resources + the knowledge on how to run them; then either reject or forward the update up to openQA.

    Similar Projects

    Create object oriented API for perl's YAML::XS module, with YAML 1.2 Support by tinita

    Description

    YAML::XS is a binding to libyaml and already quite old, but the most popular YAML module for perl. There are two main issues:

    • It uses global package variables to influence behaviour.
    • It didn't implement the loading of types like numbers and booleans according to the YAML spec (neither 1.1 nor 1.2).

    Goals

    Create a new interface which works object oriented. Currently YAML::XS exports a list of functions.

    • The new API will allow to create a YAML::XS object containing configuration influencing the behaviour of loading and dumping.
      • It keeps the libyaml parser and emitter structs in memory, so repeated calls can save the creation of those structs
    • It will by default implement the YAML 1.2 Core Schema, so it is compatible to other YAML processors in perl and in other languages
    • If I have time, I would like to add the merge << key feature as an option. We could then use it in openQA as a replacement for YAML::PP to be faster.

    I already created a proof of concept with a minimal functionality some weeks before this HackWeek.

    Resources

    Hack on isotest-ng - a rust port of isotovideo (os-autoinst aka testrunner of openQA) by szarate

    Description

    Some time ago, I managed to convince ByteOtter to hack something that resembles isotovideo but in Rust, not because I believe that Perl is dead, but more because there are certain limitations in the perl code (how it was written), and its always hard to add new functionalities when they are about implementing a new backend, or fixing bugs (Along with people complaining that Perl is dead, and that they don't like it)

    In reality, I wanted to see if this could be done, and ByteOtter proved that it could be, while doing an amazing job at hacking a vnc console, and helping me understand better what RuPerl needs to work.

    I plan to keep working on this for the next few years, and while I don't aim for feature completion or replacing isotovideo tih isotest-ng (name in progress), I do plan to be able to use it on a daily basis, using specialized tooling with interfaces, instead of reimplementing everything in the backend

    Todo

    • Add make targets for testability, e.g "spawn qemu and type"
    • Add image search matching algorithm
    • Add a Null test distribution provider
    • Add a Perl Test Distribution Provider
    • Fix unittests https://github.com/os-autoinst/isotest-ng/issues/5
    • Research OpenTofu how to add new hypervisors/baremetal to OpenTofu
    • Add an interface to openQA cli

    Goals

    • Implement at least one of the above, prepare proposals for GSoC
    • Boot a system via it's BMC

    Resources

    See https://github.com/os-autoinst/isotest-ng

    Enhance UV openQA helper script by mdonis

    Description

    A couple months ago an UV openQA helper script was created to help/automate the searching phase inside openQA for a given MU to test. The script searches inside all our openQA job groups (qam-sle) related with a given MU and generates an output suitable to add (copy & paste) inside the update log.

    This is still a WIP and could use some enhancements.

    Goals

    • Move script from bash to python: this would be useful in case we want to include this into MTUI in the future. The script will be separate from MTUI for now. The idea is to have this as a CLI tool using the click library or something similar.
    • Add option to look for jobs in other sections inside aggregated updates: right now, when looking for regression tests under aggregated updates for a given MU, the script only looks inside the Core MU job group. This is where most of the regression tests we need are located, but some MUs have their regression tests under the YaST/Containers/Security MU job groups. We should keep the Core MU group as a default, but add an option to be able to look into other job groups under aggregated updates.
    • Remove the -a option: this option is used to indicate the update ID and is mandatory right now. This is a bit weird and goes against posix stardards. It was developed this way in order to avoid using positional parameters. This problem should be fixed if we move the script to python.

    Some other ideas to consider:

    • Look into the QAM dashboard API. This has more info on each MU, could use this to link general openQA build results, whether the related RR is approved or not, etc
    • Make it easier to see if there's regression tests for a package in an openQA test build. Check if there's a possibility to search for tests that have the package name in them inside each testsuite.
    • Unit testing?

    More ideas TBD

    Resources

    https://github.com/os-autoinst/scripts/blob/master/openqa-search-maintenance-core-jobs

    https://confluence.suse.com/display/maintenanceqa/Guide+on+how+to+test+Updates

    Post-Hackweek update

    All major features were implemented. Unit tests are still in progress, and project will be moved to the SUSE github org once everything's done. https://github.com/mjdonis/oqa-search

    Learn obs/ibs sync tool by xlai

    Description

    Once images/repo are built from IBS/OBS, there is a tool to sync the image from IBS/OBS to openqa asset directory and trigger openqa jobs accordingly.

    Goals

    Check how the tool is implemented, and be capable to add/modify our needed images/repo in future by ourselves.

    Resources

    • https://github.com/os-autoinst/openqa-trigger-from-obs
    • https://gitlab.suse.de/openqa/openqa-trigger-from-ibs-plugin/-/tree/master?ref_type=heads

    New features in openqa-trigger-from-obs for openQA by jlausuch

    Description

    Implement new features in openqa-trigger-from-obs to make xml more flexible.

    Goals

    One of the features to be implemented: - Possibility to define "VERSION" and "ARCH" variables per flavor instead of global.

    Resources

    https://github.com/os-autoinst/openqa-trigger-from-obs

    OpenQA Golang api client by hilchev

    Description

    I would like to make a simple cli tool to communicate with the OpenQA API

    Goals

    • OpenQA has a ton of information that is hard to get via the UI. A tool like this would make my life easier :)
    • Would potentially make it easier in the future to make UI changes without Perl.
    • Improve my Golang skills

    Resources

    • https://go.dev/doc/
    • https://openqa.opensuse.org/api

    Explore the integration between OBS and GitHub by pdostal

    Project Description

    The goals:

    1) When GitHub pull request is created or modified the OBS project will be forked and the build results reported back to GitHub. 2) When new version of the GitHub project will be published the OBS will redownload the source and rebuild the project.

    Goal for this Hackweek

    Do as much as possible, blog about it and maybe use it another existing project.

    Resources

    Git CI to automate the creation of product definition by gyribeiro

    Description

    Automate the creation of product definition

    Goals

    Create a Git CI that will:

    • automatically be triggered once a change (commit) in package list is done.
    • run tool responsible to update product definition based on the changes in package list
    • test the updated product definition in OBS
    • submit a pull request updating the product definition in the repository

    NOTE: this Git CI may also be triggered manually

    Resources

    • https://docs.gitlab.com/ee/ci/
    • https://openbuildservice.org/2021/05/31/scm-integration/
    • https://github.com/openSUSE/openSUSE-release-tools

    Learn obs/ibs sync tool by xlai

    Description

    Once images/repo are built from IBS/OBS, there is a tool to sync the image from IBS/OBS to openqa asset directory and trigger openqa jobs accordingly.

    Goals

    Check how the tool is implemented, and be capable to add/modify our needed images/repo in future by ourselves.

    Resources

    • https://github.com/os-autoinst/openqa-trigger-from-obs
    • https://gitlab.suse.de/openqa/openqa-trigger-from-ibs-plugin/-/tree/master?ref_type=heads

    New features in openqa-trigger-from-obs for openQA by jlausuch

    Description

    Implement new features in openqa-trigger-from-obs to make xml more flexible.

    Goals

    One of the features to be implemented: - Possibility to define "VERSION" and "ARCH" variables per flavor instead of global.

    Resources

    https://github.com/os-autoinst/openqa-trigger-from-obs

    Fix RSpec tests in order to replace the ruby-ldap rubygem in OBS by enavarro_suse

    Description

    "LDAP mode is not official supported by OBS!". See: config/options.yml.example#L100-L102

    However, there is an RSpec file which tests LDAP mode in OBS. These tests use the ruby-ldap rubygem, mocking the results returned by a LDAP server.

    The ruby-ldap rubygem seems no longer maintaned, and also prevents from updating to a more recent Ruby version. A good alternative is to replace it with the net-ldap rubygem.

    Before replacing the ruby-ldap rubygem, we should modify the tests so the don't mock the responses of a LDAP server. Instead, we should modify the tests and run them against a real LDAP server.

    Goals

    Goals of this project:

    • Modify the RSpec tests and run them against a real LDAP server
    • Replace the net-ldap rubygem with the ruby-ldap rubygem

    Achieving the above mentioned goals will:

    • Permit upgrading OBS from Ruby 3.1 to Ruby 3.2
    • Make a step towards officially supporting LDAP in OBS.

    Resources