YaST Integration Tests Using Cucumber
a project by lslezak
Updated over 3 years ago. 4 hacker ♥️.

Currently we use openQA for the the YaST integration tests. It runs YaST in a VM and controls it via emulating keyboard input. The result is checked by comparing the screenshots.

This approach has several disadvantages:

  • Comparing the screenshots is very fragile, any trivial change in the UI (label, font, color, widget position, spacing, ...) completely breaks the tests.
  • The tests cannot be written by an unskilled person, writing the tests is not trivial, you need to know the Perl library used for running the tests and some tricks how to control the YaST UI.
  • If you want to run the tests locally you need to deploy the complete openQA server (or have access to a shared instance)

This project is about finding and trying an alternative approach.

I found a cucumber-cpp project which allows writing cucumber style tests for a C++ application. Originally Cucumber was designed for Ruby programs, but it allows using the cucumber wire protocol for communication with any non-Ruby program. The communication uses the JSON data format over a TCP or Unix socket.

For the step definitions it would be nice to use similar format like the selenium-cucumber extension.

The example test could look like:

Background:
  Given the "/etc/sysconfig/foo" file contains line "FOO=yes"
  And I run "yast2 foo" command.

# use widgets id, does not break after changing the label
# but needs the internal knowledge
Scenario: Changing the value
  Given the checkbox with id ":value" is checked
  Then I uncheck the checkbox having id ":value"
  When I click on button having id ":finish"
  Then the "/etc/sysconfig/foo" file contains line "FOO=no"

# use widget label - easier to write the test for non-developers
# but breaks after changing the label
Scenario: Aborting the module and keeping the old value
  Given the checkbox with label "Enable foo feature" is checked
  Then I uncheck the checkbox having label "Enable foo feature"
  When I click on button having label "Abort"
  Then the "/etc/sysconfig/foo" file contains line "FOO=yes"

The cucumber-cpp even contains an example for a simple Qt based application, see a test example and the related step definition.

However, the cucumber-cpp implementation expects that the tested object can be represented as a single C++ object. That's actually not true for YaST as the architecture is quite complex. The UI and the Ruby interpreter are loaded as independent plugins.

Maybe in the end we won't be able to use the cucumber-cpp library directly but write our own implementation for the wire protocol...

The initial implementation would be focused on running the tests in installed system, but as the cucumber wire protocol uses a TCP port for communication it should not be hard to enable this feature during installation and send the test commands from the openQA from outside...

Results

I successfully implemented a prototype for the integration tests, it allows testing YaST in installed system, during installation and it can be used even for plain libyui applications outside YaST.

The code is still more or less a proof of concept, the are still some issues or missing features but it shows that it is possible to go this way...

Installed System

Adding a new repository in installed system:

Adding Repository

Installation

This is a patched openSUSE Leap 42.2 installation running in a VirtualBox virtual machine:

openSUSE Leap 42.2 Installation

Libyui Test

Here is a test for the SelectionBox2.cc libyui example. (A standalone C++ application, not connected to YaST at all.)

Libyui Example

More Details

See more details in my blog post and in the lslezak/cucumber-yast GitHub repository.

Join this project

Looking for hackers with the skills:

yast yastui testing bdd cucumber c++

This project is part of:

Hack Week 15

Activity

  • almost 9 years ago: locilka liked this project.
  • almost 9 years ago: mvidner liked this project.
  • almost 9 years ago: dmaiocchi liked this project.
  • almost 9 years ago: dwaas liked this project.
  • almost 9 years ago: lslezak started this project.
  • almost 9 years ago: lslezak added keyword "yast" to this project.
  • almost 9 years ago: lslezak added keyword "yastui" to this project.
  • almost 9 years ago: lslezak added keyword "testing" to this project.
  • almost 9 years ago: lslezak added keyword "bdd" to this project.
  • almost 9 years ago: lslezak added keyword "cucumber" to this project.
  • almost 9 years ago: lslezak added keyword "c++" to this project.
  • almost 9 years ago: lslezak originated this project.


    • Comments

    • okurz
      almost 9 years ago by okurz | Reply

      Interesting project idea.

      As I am one of the openQA contributors and openQA is one of the main tools we rely on during QA work on the SLE products I am interested in the results and finding. But also I would like to comment on your initial statements:

      > Comparing the screenshots is very fragile, any trivial change in the UI (label, font, color, widget position, spacing, ...) completely breaks the tests

      That is by design because openQA with os-autoinst is used for system testing combined with UI regression testing. If you do not care that your label/font/color changes without you realizing then needles can be created in a very robust to just look for the details you want to see, use a low matching level to match regardless of font changes or use OCR to read the text without caring at all about how it looks.

      > The tests cannot be written by an unskilled person, writing the tests is not trivial, you need to know the Perl library used for running the tests and some tricks how to control the YaST UI

      You need to know the test API but isn't that true for any other approach as well?

      > If you want to run the tests locally you need to deploy the complete openQA server (or have access to a shared instance)

      If openQA can install and run openQA within openQA (see here) then it shouldn't be too hard. But that is a point that openQA can improve on, granted :-)

    • lslezak
      almost 9 years ago by lslezak | Reply

      1) Comparing screenshots is not bad in general, openQA + os-autoinst is a generic solution which is able to test things which are not testable otherwise (e.g. you can test the grub boot menu). Or in some cases you really want to test the screenshot to make sure the dialog is still well readable (imagine a bug in the Qt stylesheet). But this generic solution has some disadvantages as mentioned above. Changing the font is not common in YaST, but on the other hand we quite often fine tune the dialog layout (better spacing, alignment...) which does not change the behavior but still requires extra work for the openQA team.

      2) Sure, every testing framework requires some skills and practice. Here I rather meant "non-developer" or "non-programmer". The test example above could be written even by such persons. I'm not saying that this is the best way, but IMHO makes sense in some cases. And I'd like to see how that approach would work for YaST. We have relatively low test coverage in YaST, if we allow easy contribution for people outside the team (or from the community) then this might help to improve the current situation.

      3) The point is that we run rake test:unit for unit tests, that's trivial and does not need any setup (besides some rake package installed in the system). If we could simply run rake test:integration locally then it would be great. Of course, we would need to somehow identify or tag potentially dangerous operations, you do not want the YaST partitioner cleaned up your disk... ;-)

    • bear454
      almost 9 years ago by bear454 | Reply

      Suggestion: extend OpenQA to accept tests written in Cucumber, using the Cucumber-Perl module, and some training-wheels like OpenQA steps.

    • lslezak
      over 8 years ago by lslezak | Reply

      JFYI: I have added the "Results" section into the description to summarize my work done during the Hackweek.

    Similar Projects

    openSUSE on ZoL from OpenZFS project by jkohoutek

    Idea is to have SUSE system with OpenZFS as root FS.

    Why ZFS

    Ways in which ZFS is better than BTRFS

    Main goal

    Have OpenZFS as install option in the installer and utilize zedenv Boot Environment Manager for SUSE updates install

    Goals

    • synergy of ZFS with dracut, so snapshots are correctly added to the grub
    • synergy of zedenv with zypper
      • before every update snapshot is created
      • when new kernel or other package which requires reboot is about to be installed, the update will be processed to the new boot environment snapshot and grub configuration changed to boot to this new one
    • integrate Root on ZFS as install option to the YaST
    • configure Kiwi for the ZFS install images

    Completed goals

    • prepare ZFS pool compatible with openSUSE installation ✓
    • install openSUSE with root on ZFS ✓
    • boot to the prepared and installed system ✓

    Current progress

    Resources:

    openSUSE on ZoL from OpenZFS project by jkohoutek

    Idea is to have SUSE system with OpenZFS as root FS.

    Why ZFS

    Ways in which ZFS is better than BTRFS

    Main goal

    Have OpenZFS as install option in the installer and utilize zedenv Boot Environment Manager for SUSE updates install

    Goals

    • synergy of ZFS with dracut, so snapshots are correctly added to the grub
    • synergy of zedenv with zypper
      • before every update snapshot is created
      • when new kernel or other package which requires reboot is about to be installed, the update will be processed to the new boot environment snapshot and grub configuration changed to boot to this new one
    • integrate Root on ZFS as install option to the YaST
    • configure Kiwi for the ZFS install images

    Completed goals

    • prepare ZFS pool compatible with openSUSE installation ✓
    • install openSUSE with root on ZFS ✓
    • boot to the prepared and installed system ✓

    Current progress

    Resources:

    openQA tests needles elaboration using AI image recognition by mdati

    Description

    In the openQA test framework, to identify the status of a target SUT image, a screenshots of GUI or CLI-terminal images, the needles framework scans the many pictures in its repository, having associated a given set of tags (strings), selecting specific smaller parts of each available image. For the needles management actually we need to keep stored many screenshots, variants of GUI and CLI-terminal images, eachone accompanied by a dedicated set of data references (json).

    A smarter framework, using image recognition based on AI or other image elaborations tools, nowadays widely available, could improve the matching process and hopefully reduce time and errors, during the images verification and detection process.

    Goals

    Main scope of this idea is to match a non-text status of a running openQA test, an image of a shell console or application-GUI screenshot, using less time and resources and with less errors in data preparation and use, than the actual openQA needles framework; that is:

    • having a given SUT (system under test) GUI or CLI-terminal screenshot, with a local distribution of pixels or text commands related to a running test status,
    • we want to identify a desired target, e.g. a screen image status or data/commands context,
      • based on AI/ML-pretrained archives containing object or other proper elaboration tools,
      • possibly able to identify also object not present in the archive, i.e. by means of AI/ML mechanisms.
    • the matching result should be then adapted to continue working in the openQA test, likewise and in place of the same result that would have been produced by the original openQA needles framework.
    • We expect an improvement of the matching-time(less time), reliability of the expected result(less error) and simplification of archive maintenance in adding/removing objects(smaller DB and less actions).

    Hackweek step

    POC:

    • study the available tools
    • prepare a plan for the process to build
    • write and build a draft application
    • prepare the data archive from a subset of needles
    • initialize/pre-train the base archive
    • select a screenshot from the subset, removing/changing some part
    • run the POC application
    • expect the image type is identified in a good %.

    Resources

    first step of this project is quite identification of useful resources for the scope; some possibilities are:

    • SUSE AI and other ML tools (i.e. Tensorflow)
    • Tools able to manage images
    • RPA test tools (like i.e. Robot framework)
    • other.

    Testing and adding GNU/Linux distributions on Uyuni by juliogonzalezgil

    Join the Gitter channel! https://gitter.im/uyuni-project/hackweek

    Uyuni is a configuration and infrastructure management tool that saves you time and headaches when you have to manage and update tens, hundreds or even thousands of machines. It also manages configuration, can run audits, build image containers, monitor and much more!

    Currently there are a few distributions that are completely untested on Uyuni or SUSE Manager (AFAIK) or just not tested since a long time, and could be interesting knowing how hard would be working with them and, if possible, fix whatever is broken.

    For newcomers, the easiest distributions are those based on DEB or RPM packages. Distributions with other package formats are doable, but will require adapting the Python and Java code to be able to sync and analyze such packages (and if salt does not support those packages, it will need changes as well). So if you want a distribution with other packages, make sure you are comfortable handling such changes.

    No developer experience? No worries! We had non-developers contributors in the past, and we are ready to help as long as you are willing to learn. If you don't want to code at all, you can also help us preparing the documentation after someone else has the initial code ready, or you could also help with testing :-)

    The idea is testing Salt and Salt-ssh clients, but NOT traditional clients, which are deprecated.

    To consider that a distribution has basic support, we should cover at least (points 3-6 are to be tested for both salt minions and salt ssh minions):

    1. Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)
    2. Onboarding (salt minion from UI, salt minion from bootstrap scritp, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator)
    3. Package management (install, remove, update...)
    4. Patching
    5. Applying any basic salt state (including a formula)
    6. Salt remote commands
    7. Bonus point: Java part for product identification, and monitoring enablement
    8. Bonus point: sumaform enablement (https://github.com/uyuni-project/sumaform)
    9. Bonus point: Documentation (https://github.com/uyuni-project/uyuni-docs)
    10. Bonus point: testsuite enablement (https://github.com/uyuni-project/uyuni/tree/master/testsuite)

    If something is breaking: we can try to fix it, but the main idea is research how supported it is right now. Beyond that it's up to each project member how much to hack :-)

    • If you don't have knowledge about some of the steps: ask the team
    • If you still don't know what to do: switch to another distribution and keep testing.

    This card is for EVERYONE, not just developers. Seriously! We had people from other teams helping that were not developers, and added support for Debian and new SUSE Linux Enterprise and openSUSE Leap versions :-)

    Pending

    Debian 13

    The new version of the beloved Debian GNU/Linux OS

    Seems to be a Debian 12 derivative, so adding it could be quite easy.

    • [ ] Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)
    • W] Onboarding (salt minion from UI, salt minion from bootstrap script, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator)
    • [ ] Package management (install, remove, update...)
    • [ ] Patching (if patch information is available, could require writing some code to parse it, but IIRC we have support for Ubuntu already). Probably not for Debian as IIRC we don't support patches yet.
    • [ ] Applying any basic salt state (including a formula)
    • [ ] Salt remote commands
    • [ ] Bonus point: Java part for product identification, and monitoring enablement
    • [ ] Bonus point: sumaform enablement (https://github.com/uyuni-project/sumaform)
    • [ ] Bonus point: Documentation (https://github.com/uyuni-project/uyuni-docs)

    Multimachine on-prem test with opentofu, ansible and Robot Framework by apappas

    Description

    A long time ago I explored using the Robot Framework for testing. A big deficiency over our openQA setup is that bringing up and configuring the connection to a test machine is out of scope.

    Nowadays we have a way¹ to deploy SUTs outside openqa, but we only use if for cloud tests in conjuction with openqa. Using knowledge gained from that project I am going to try to create a test scenario that replicates an openqa test but this time including the deployment and setup of the SUT.

    Goals

    Create a simple multimachine test scenario with the support server and SUT all created by the robot framework.

    Resources

    1. https://github.com/SUSE/qe-sap-deployment
    2. terraform-libvirt-provider

    Testing and adding GNU/Linux distributions on Uyuni by juliogonzalezgil

    Join the Gitter channel! https://gitter.im/uyuni-project/hackweek

    Uyuni is a configuration and infrastructure management tool that saves you time and headaches when you have to manage and update tens, hundreds or even thousands of machines. It also manages configuration, can run audits, build image containers, monitor and much more!

    Currently there are a few distributions that are completely untested on Uyuni or SUSE Manager (AFAIK) or just not tested since a long time, and could be interesting knowing how hard would be working with them and, if possible, fix whatever is broken.

    For newcomers, the easiest distributions are those based on DEB or RPM packages. Distributions with other package formats are doable, but will require adapting the Python and Java code to be able to sync and analyze such packages (and if salt does not support those packages, it will need changes as well). So if you want a distribution with other packages, make sure you are comfortable handling such changes.

    No developer experience? No worries! We had non-developers contributors in the past, and we are ready to help as long as you are willing to learn. If you don't want to code at all, you can also help us preparing the documentation after someone else has the initial code ready, or you could also help with testing :-)

    The idea is testing Salt and Salt-ssh clients, but NOT traditional clients, which are deprecated.

    To consider that a distribution has basic support, we should cover at least (points 3-6 are to be tested for both salt minions and salt ssh minions):

    1. Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)
    2. Onboarding (salt minion from UI, salt minion from bootstrap scritp, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator)
    3. Package management (install, remove, update...)
    4. Patching
    5. Applying any basic salt state (including a formula)
    6. Salt remote commands
    7. Bonus point: Java part for product identification, and monitoring enablement
    8. Bonus point: sumaform enablement (https://github.com/uyuni-project/sumaform)
    9. Bonus point: Documentation (https://github.com/uyuni-project/uyuni-docs)
    10. Bonus point: testsuite enablement (https://github.com/uyuni-project/uyuni/tree/master/testsuite)

    If something is breaking: we can try to fix it, but the main idea is research how supported it is right now. Beyond that it's up to each project member how much to hack :-)

    • If you don't have knowledge about some of the steps: ask the team
    • If you still don't know what to do: switch to another distribution and keep testing.

    This card is for EVERYONE, not just developers. Seriously! We had people from other teams helping that were not developers, and added support for Debian and new SUSE Linux Enterprise and openSUSE Leap versions :-)

    Pending

    Debian 13

    The new version of the beloved Debian GNU/Linux OS

    Seems to be a Debian 12 derivative, so adding it could be quite easy.

    • [ ] Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)
    • W] Onboarding (salt minion from UI, salt minion from bootstrap script, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator)
    • [ ] Package management (install, remove, update...)
    • [ ] Patching (if patch information is available, could require writing some code to parse it, but IIRC we have support for Ubuntu already). Probably not for Debian as IIRC we don't support patches yet.
    • [ ] Applying any basic salt state (including a formula)
    • [ ] Salt remote commands
    • [ ] Bonus point: Java part for product identification, and monitoring enablement
    • [ ] Bonus point: sumaform enablement (https://github.com/uyuni-project/sumaform)
    • [ ] Bonus point: Documentation (https://github.com/uyuni-project/uyuni-docs)

    Move Uyuni Test Framework from Selenium to Playwright + AI by oscar-barrios

    Description

    This project aims to migrate the existing Uyuni Test Framework from Selenium to Playwright. The move will improve the stability, speed, and maintainability of our end-to-end tests by leveraging Playwright's modern features. We'll be rewriting the current Selenium code in Ruby to Playwright code in TypeScript, which includes updating the test framework runner, step definitions, and configurations. This is also necessary because we're moving from Cucumber Ruby to CucumberJS.

    If you're still curious about the AI in the title, it was just a way to grab your attention. Thanks for your understanding.

    Goals

    • Migrate Core tests including Onboarding of clients
    • Improve test reliabillity: Measure and confirm a significant reduction of flakynes.
    • Implement a robust framework: Establish a well-structured and reusable Playwright test framework using the CucumberJS

    Resources

    Smart lighting with Pico 2 by jmodak

    Description

    I am trying to create a smart-lighting project with a Raspberry Pi Pico that reacts to a movie's visuals and audio that involves combining two distinct functions: ambient screen lighting(visual response) and sound-reactive lighting(audio response)

    Goals

    • Visuals: Capturing the screen's colour requires an external device to analyse screen content and send colour data to the MCU via serial communication.
    • Audio: A sound sensor module connected directly to the Pico that can detect sound volume.
    • Pico 2W: The MCU receives data fro, both inputs and controls an LED strip.

    Resources

    • Raspberry Pi Pico 2 W
    • RGB LED strip
    • Sound detecting sensor
    • Power supply
    • breadboard and wires

    Port some classic game to Linux by MDoucha

    Let's pick some old classic game, reverse engineer the data formats and game rules and write an open source engine for it from scratch. Some games from 1990s are simple enough that we could have a playable prototype by the end of the week.

    Write which games you'd like to hack on in the comments. Don't forget to check e.g. on Open Source Game Clones, Github and SourceForge whether the game is ported already.

    Hack Week 25 - TBD

    It's time to pick a game for the upcoming Hack Week. Discuss in the comments what game you'd like to hack!

    Hack Week 24 - Master of Orion II: Battle at Antares & Chaos Overlords

    Work on Master of Orion II continues but we can hack more than one game. Chaos Overlords is a dystopian, lighthearted, cyberpunk turn-based strategy game originally released in 1996 for Windows 95 and Mac OS. The player takes on the role of a Chaos Overlord, attempting to control a city. Gameplay involves hiring mercenary gangs and deploying them on an 8-by-8 grid of city sectors to generate income, occupy sectors and take over the city.

    How to ~~install & play~~ observe the decompilation progress:

    • Clone the Git repository
    • A playable reimplementation does not exist yet, but when it does, it will be linked in the repository mentioned above.

    Further work needed:

    • Analyze the remaining unknown data structures, most of which are related to the AI.
    • Decompile the AI completely. The strong AI is part of the appeal of the game. It cannot be left out.
    • Reimplement the game.

    Hack Week 20, 21, 22 & 23 - Master of Orion II: Battle at Antares

    Master of Orion II is one of the greatest turn-based 4X games of the 1990s. Explore the galaxy, colonize planets, research new technologies, fight space monsters and alien empires and in the end, become the ruler of the galaxy one way or another.

    How to install & play:

    • Clone the Git repository
    • Run ./bootstrap; ./configure; make && make install
    • Copy all *.LBX files from the original Master of Orion II to the installation data directory (/usr/local/share/openorion2 by default)
    • Run openorion2

    Further work needed:

    • Analyze the rest of the original savegame format and a few remaining data files.
    • Implement most of the game. The open source engine currently supports only loading saved games from the original version and viewing the galaxy map, fleet management and list of known planets.

    Hack Week 19 - Signus: The Artifact Wars

    Signus is a Czech turn-based strategy game similar to Panzer General or Battle Isle series. Originally published in 1998 and open-sourced by the original developers in 2003.

    How to install & play:

    • Clone the Git repository
    • Run ./bootstrap; ./configure; make && make install in both signus and signus-data directories.
    • Run signus

    Further work needed: