Currently we use openQA for the the YaST integration tests. It runs YaST in a VM and controls it via emulating keyboard input. The result is checked by comparing the screenshots.
This approach has several disadvantages:
- Comparing the screenshots is very fragile, any trivial change in the UI (label, font, color, widget position, spacing, ...) completely breaks the tests.
- The tests cannot be written by an unskilled person, writing the tests is not trivial, you need to know the Perl library used for running the tests and some tricks how to control the YaST UI.
- If you want to run the tests locally you need to deploy the complete openQA server (or have access to a shared instance)
This project is about finding and trying an alternative approach.
I found a cucumber-cpp project which allows writing cucumber style tests for a C++ application. Originally Cucumber was designed for Ruby programs, but it allows using the cucumber wire protocol for communication with any non-Ruby program. The communication uses the JSON data format over a TCP or Unix socket.
For the step definitions it would be nice to use similar format like the selenium-cucumber extension.
The example test could look like:
Background: Given the "/etc/sysconfig/foo" file contains line "FOO=yes" And I run "yast2 foo" command. # use widgets id, does not break after changing the label # but needs the internal knowledge Scenario: Changing the value Given the checkbox with id ":value" is checked Then I uncheck the checkbox having id ":value" When I click on button having id ":finish" Then the "/etc/sysconfig/foo" file contains line "FOO=no" # use widget label - easier to write the test for non-developers # but breaks after changing the label Scenario: Aborting the module and keeping the old value Given the checkbox with label "Enable foo feature" is checked Then I uncheck the checkbox having label "Enable foo feature" When I click on button having label "Abort" Then the "/etc/sysconfig/foo" file contains line "FOO=yes"
However, the cucumber-cpp implementation expects that the tested object can be represented as a single C++ object. That's actually not true for YaST as the architecture is quite complex. The UI and the Ruby interpreter are loaded as independent plugins.
Maybe in the end we won't be able to use the cucumber-cpp library directly but write our own implementation for the wire protocol...
The initial implementation would be focused on running the tests in installed system, but as the cucumber wire protocol uses a TCP port for communication it should not be hard to enable this feature during installation and send the test commands from the openQA from outside...
I successfully implemented a prototype for the integration tests, it allows testing YaST in installed system, during installation and it can be used even for plain libyui applications outside YaST.
The code is still more or less a proof of concept, the are still some issues or missing features but it shows that it is possible to go this way...
Adding a new repository in installed system:
This is a patched openSUSE Leap 42.2 installation running in a VirtualBox virtual machine:
Here is a test for the SelectionBox2.cc libyui example. (A standalone C++ application, not connected to YaST at all.)
This project is part of:
Hack Week 15