Currently we use openQA for the the YaST integration tests. It runs YaST in a VM and controls it via emulating keyboard input. The result is checked by comparing the screenshots.
This approach has several disadvantages:
- Comparing the screenshots is very fragile, any trivial change in the UI (label, font, color, widget position, spacing, ...) completely breaks the tests.
- The tests cannot be written by an unskilled person, writing the tests is not trivial, you need to know the Perl library used for running the tests and some tricks how to control the YaST UI.
- If you want to run the tests locally you need to deploy the complete openQA server (or have access to a shared instance)
This project is about finding and trying an alternative approach.
I found a cucumber-cpp project which allows writing cucumber style tests for a C++ application. Originally Cucumber was designed for Ruby programs, but it allows using the cucumber wire protocol for communication with any non-Ruby program. The communication uses the JSON data format over a TCP or Unix socket.
For the step definitions it would be nice to use similar format like the selenium-cucumber extension.
The example test could look like:
Background:
Given the "/etc/sysconfig/foo" file contains line "FOO=yes"
And I run "yast2 foo" command.
# use widgets id, does not break after changing the label
# but needs the internal knowledge
Scenario: Changing the value
Given the checkbox with id ":value" is checked
Then I uncheck the checkbox having id ":value"
When I click on button having id ":finish"
Then the "/etc/sysconfig/foo" file contains line "FOO=no"
# use widget label - easier to write the test for non-developers
# but breaks after changing the label
Scenario: Aborting the module and keeping the old value
Given the checkbox with label "Enable foo feature" is checked
Then I uncheck the checkbox having label "Enable foo feature"
When I click on button having label "Abort"
Then the "/etc/sysconfig/foo" file contains line "FOO=yes"
The cucumber-cpp even contains an example for a simple Qt based application, see a test example and the related step definition.
However, the cucumber-cpp implementation expects that the tested object can be represented as a single C++ object. That's actually not true for YaST as the architecture is quite complex. The UI and the Ruby interpreter are loaded as independent plugins.
Maybe in the end we won't be able to use the cucumber-cpp library directly but write our own implementation for the wire protocol...
The initial implementation would be focused on running the tests in installed system, but as the cucumber wire protocol uses a TCP port for communication it should not be hard to enable this feature during installation and send the test commands from the openQA from outside...
Results
I successfully implemented a prototype for the integration tests, it allows testing YaST in installed system, during installation and it can be used even for plain libyui applications outside YaST.
The code is still more or less a proof of concept, the are still some issues or missing features but it shows that it is possible to go this way...
Installed System
Adding a new repository in installed system:
Installation
This is a patched openSUSE Leap 42.2 installation running in a VirtualBox virtual machine:
Libyui Test
Here is a test for the SelectionBox2.cc libyui example. (A standalone C++ application, not connected to YaST at all.)
More Details
See more details in my blog post and in the lslezak/cucumber-yast GitHub repository.
This project is part of:
Hack Week 15
Activity
Comments
-
over 7 years ago by okurz | Reply
Interesting project idea.
As I am one of the openQA contributors and openQA is one of the main tools we rely on during QA work on the SLE products I am interested in the results and finding. But also I would like to comment on your initial statements:
Comparing the screenshots is very fragile, any trivial change in the UI (label, font, color, widget position, spacing, ...) completely breaks the tests
That is by design because openQA with os-autoinst is used for system testing combined with UI regression testing. If you do not care that your label/font/color changes without you realizing then needles can be created in a very robust to just look for the details you want to see, use a low matching level to match regardless of font changes or use OCR to read the text without caring at all about how it looks.
The tests cannot be written by an unskilled person, writing the tests is not trivial, you need to know the Perl library used for running the tests and some tricks how to control the YaST UI
You need to know the test API but isn't that true for any other approach as well?
If you want to run the tests locally you need to deploy the complete openQA server (or have access to a shared instance)
If openQA can install and run openQA within openQA (see here) then it shouldn't be too hard. But that is a point that openQA can improve on, granted :-)
-
over 7 years ago by lslezak | Reply
1) Comparing screenshots is not bad in general, openQA + os-autoinst is a generic solution which is able to test things which are not testable otherwise (e.g. you can test the grub boot menu). Or in some cases you really want to test the screenshot to make sure the dialog is still well readable (imagine a bug in the Qt stylesheet). But this generic solution has some disadvantages as mentioned above. Changing the font is not common in YaST, but on the other hand we quite often fine tune the dialog layout (better spacing, alignment...) which does not change the behavior but still requires extra work for the openQA team.
2) Sure, every testing framework requires some skills and practice. Here I rather meant "non-developer" or "non-programmer". The test example above could be written even by such persons. I'm not saying that this is the best way, but IMHO makes sense in some cases. And I'd like to see how that approach would work for YaST. We have relatively low test coverage in YaST, if we allow easy contribution for people outside the team (or from the community) then this might help to improve the current situation.
3) The point is that we run
rake test:unit
for unit tests, that's trivial and does not need any setup (besides some rake package installed in the system). If we could simply runrake test:integration
locally then it would be great. Of course, we would need to somehow identify or tag potentially dangerous operations, you do not want the YaST partitioner cleaned up your disk... ;-) -
over 7 years ago by bear454 | Reply
Suggestion: extend OpenQA to accept tests written in Cucumber, using the Cucumber-Perl module, and some training-wheels like OpenQA steps.
Similar Projects