YaST Integration Tests Using Cucumber
Currently we use openQA for the the YaST integration tests. It runs YaST in a VM and controls it via emulating keyboard input. The result is checked by comparing the screenshots.
This approach has several disadvantages:
- Comparing the screenshots is very fragile, any trivial change in the UI (label, font, color, widget position, spacing, ...) completely breaks the tests.
- The tests cannot be written by an unskilled person, writing the tests is not trivial, you need to know the Perl library used for running the tests and some tricks how to control the YaST UI.
- If you want to run the tests locally you need to deploy the complete openQA server (or have access to a shared instance)
This project is about finding and trying an alternative approach.
I found a cucumber-cpp project which allows writing cucumber style tests for a C++ application. Originally Cucumber was designed for Ruby programs, but it allows using the cucumber wire protocol for communication with any non-Ruby program. The communication uses the JSON data format over a TCP or Unix socket.
For the step definitions it would be nice to use similar format like the selenium-cucumber extension.
The example test could look like:
Background:
Given the "/etc/sysconfig/foo" file contains line "FOO=yes"
And I run "yast2 foo" command.
# use widgets id, does not break after changing the label
# but needs the internal knowledge
Scenario: Changing the value
Given the checkbox with id ":value" is checked
Then I uncheck the checkbox having id ":value"
When I click on button having id ":finish"
Then the "/etc/sysconfig/foo" file contains line "FOO=no"
# use widget label - easier to write the test for non-developers
# but breaks after changing the label
Scenario: Aborting the module and keeping the old value
Given the checkbox with label "Enable foo feature" is checked
Then I uncheck the checkbox having label "Enable foo feature"
When I click on button having label "Abort"
Then the "/etc/sysconfig/foo" file contains line "FOO=yes"
The cucumber-cpp even contains an example for a simple Qt based application, see a test example and the related step definition.
However, the cucumber-cpp implementation expects that the tested object can be represented as a single C++ object. That's actually not true for YaST as the architecture is quite complex. The UI and the Ruby interpreter are loaded as independent plugins.
Maybe in the end we won't be able to use the cucumber-cpp library directly but write our own implementation for the wire protocol...
The initial implementation would be focused on running the tests in installed system, but as the cucumber wire protocol uses a TCP port for communication it should not be hard to enable this feature during installation and send the test commands from the openQA from outside...
Results
I successfully implemented a prototype for the integration tests, it allows testing YaST in installed system, during installation and it can be used even for plain libyui applications outside YaST.
The code is still more or less a proof of concept, the are still some issues or missing features but it shows that it is possible to go this way...
Installed System
Adding a new repository in installed system:
Installation
This is a patched openSUSE Leap 42.2 installation running in a VirtualBox virtual machine:
Libyui Test
Here is a test for the SelectionBox2.cc libyui example. (A standalone C++ application, not connected to YaST at all.)
More Details
See more details in my blog post and in the lslezak/cucumber-yast GitHub repository.
This project is part of:
Hack Week 15
Activity
Comments
-
almost 8 years ago by okurz | Reply
Interesting project idea.
As I am one of the openQA contributors and openQA is one of the main tools we rely on during QA work on the SLE products I am interested in the results and finding. But also I would like to comment on your initial statements:
> Comparing the screenshots is very fragile, any trivial change in the UI (label, font, color, widget position, spacing, ...) completely breaks the tests
That is by design because openQA with os-autoinst is used for system testing combined with UI regression testing. If you do not care that your label/font/color changes without you realizing then needles can be created in a very robust to just look for the details you want to see, use a low matching level to match regardless of font changes or use OCR to read the text without caring at all about how it looks.
> The tests cannot be written by an unskilled person, writing the tests is not trivial, you need to know the Perl library used for running the tests and some tricks how to control the YaST UI
You need to know the test API but isn't that true for any other approach as well?
> If you want to run the tests locally you need to deploy the complete openQA server (or have access to a shared instance)
If openQA can install and run openQA within openQA (see here) then it shouldn't be too hard. But that is a point that openQA can improve on, granted :-)
-
almost 8 years ago by lslezak | Reply
1) Comparing screenshots is not bad in general, openQA + os-autoinst is a generic solution which is able to test things which are not testable otherwise (e.g. you can test the grub boot menu). Or in some cases you really want to test the screenshot to make sure the dialog is still well readable (imagine a bug in the Qt stylesheet). But this generic solution has some disadvantages as mentioned above. Changing the font is not common in YaST, but on the other hand we quite often fine tune the dialog layout (better spacing, alignment...) which does not change the behavior but still requires extra work for the openQA team.
2) Sure, every testing framework requires some skills and practice. Here I rather meant "non-developer" or "non-programmer". The test example above could be written even by such persons. I'm not saying that this is the best way, but IMHO makes sense in some cases. And I'd like to see how that approach would work for YaST. We have relatively low test coverage in YaST, if we allow easy contribution for people outside the team (or from the community) then this might help to improve the current situation.
3) The point is that we run
rake test:unitfor unit tests, that's trivial and does not need any setup (besides some rake package installed in the system). If we could simply runrake test:integrationlocally then it would be great. Of course, we would need to somehow identify or tag potentially dangerous operations, you do not want the YaST partitioner cleaned up your disk... ;-) -
almost 8 years ago by bear454 | Reply
Suggestion: extend OpenQA to accept tests written in Cucumber, using the Cucumber-Perl module, and some training-wheels like OpenQA steps.
-
Similar Projects
Hack on isotest-ng - a rust port of isotovideo (os-autoinst aka testrunner of openQA) by szarate
Description
Some time ago, I managed to convince ByteOtter to hack something that resembles isotovideo but in Rust, not because I believe that Perl is dead, but more because there are certain limitations in the perl code (how it was written), and its always hard to add new functionalities when they are about implementing a new backend, or fixing bugs (Along with people complaining that Perl is dead, and that they don't like it)
In reality, I wanted to see if this could be done, and ByteOtter proved that it could be, while doing an amazing job at hacking a vnc console, and helping me understand better what RuPerl needs to work.
I plan to keep working on this for the next few years, and while I don't aim for feature completion or replacing isotovideo tih isotest-ng (name in progress), I do plan to be able to use it on a daily basis, using specialized tooling with interfaces, instead of reimplementing everything in the backend
Todo
- Add
maketargets for testability, e.g "spawn qemu and type" - Add image search matching algorithm
- Add a Null test distribution provider
- Add a Perl Test Distribution Provider
- Fix unittests https://github.com/os-autoinst/isotest-ng/issues/5
- Research OpenTofu how to add new hypervisors/baremetal to OpenTofu
- Add an interface to openQA cli
Goals
- Implement at least one of the above, prepare proposals for GSoC
- Boot a system via it's BMC
Resources
See https://github.com/os-autoinst/isotest-ng
Yearly Quality Engineering Ask me Anything - AMA for not-engineering by szarate
Goal
Get a closer look at how developers work on the Engineering team (R & D) of SUSE, and close the collaboration gap between GSI and Engineering
Why?
Santiago can go over different development workflows, and can do a deepdive into how Quality Engineering works (think of my QE Team, the advocates for your customers), The idea of this session is to help open the doors to opportunities for collaboration, and broaden our understanding of SUSE as a whole.
Objectives
- Give $audience a small window on how to get some questions answered either on the spot or within days of how some things at engineering are done
- Give Santiago Zarate from Quality Engineering a look into how $audience sees the engineering departments, and find out possibilities of further collaboration
How?
By running an "Ask me Anything" session, which is a format of a kind of open Q & A session, where participants ask the host multiple questions.
How to make it happen?
I'm happy to help joining a call or we can do it async (online/in person is more fun). Ping me over email-slack and lets make the magic happen!. Doesn't need to be during hackweek, but we gotta kickstart the idea during hackweek ;)
Rules
The rules are simple, the more questions the more fun it will be; while this will be only a window into engineering, it can also be the place to help all of us get to a similar level of understanding of the processes that are behind our respective areas of the organization.
Dynamics
The host will be monitoring the questions on some pre-agreed page, and try to answer to the best of their knowledge, if a question is too difficult or the host doesn't have the answer, he will do his best to provide an answer at a later date.
Atendees are encouraged to add questions beforehand; in the case there aren't any, we would be looking at how Quality Engineering tests new products or performs regression tests
Agenda
- Introduction of Santiago Zarate, Product Owner of Quality Engineering Core team
- Introduction of the Group/Team/Persons interested
- Ice breaker
- AMA time! Add your questions $PAGE
- Looking at QE Workflows: How is
- A maintenance update being tested before being released to our customers
- Products in development are tested before making it generally available
- Engineering Opportunity Board
Automated Test Report reviewer by oscar-barrios
Description
In SUMA/Uyuni team we spend a lot of time reviewing test reports, analyzing each of the test cases failing, checking if the test is a flaky test, checking logs, etc.
Goals
Speed up the review by automating some parts through AI, in a way that we can consume some summary of that report that could be meaningful for the reviewer.
Resources
No idea about the resources yet, but we will make use of:
- HTML/JSON Report (text + screenshots)
- The Test Suite Status GithHub board (via API)
- The environment tested (via SSH)
- The test framework code (via files)
Make more sense of openQA test results using AI by livdywan
Description
AI has the potential to help with something many of us spend a lot of time doing which is making sense of openQA logs when a job fails.
User Story
Allison Average has a puzzled look on their face while staring at log files that seem to make little sense. Is this a known issue, something completely new or maybe related to infrastructure changes?
Goals
- Leverage a chat interface to help Allison
- Create a model from scratch based on data from openQA
- Proof of concept for automated analysis of openQA test results
Bonus
- Use AI to suggest solutions to merge conflicts
- This would need a merge conflict editor that can suggest solving the conflict
- Use image recognition for needles
Resources
Timeline
Day 1
- Conversing with open-webui to teach me how to create a model based on openQA test results
- Asking for example code using TensorFlow in Python
- Discussing log files to explore what to analyze
- Drafting a new project called Testimony (based on Implementing a containerized Python action) - the project name was also suggested by the assistant
Day 2
- Using NotebookLLM (Gemini) to produce conversational versions of blog posts
- Researching the possibility of creating a project logo with AI
- Asking open-webui, persons with prior experience and conducting a web search for advice
Highlights
- I briefly tested compared models to see if they would make me more productive. Between llama, gemma and mistral there was no amazing difference in the results for my case.
- Convincing the chat interface to produce code specific to my use case required very explicit instructions.
- Asking for advice on how to use open-webui itself better was frustratingly unfruitful both in trivial and more advanced regards.
- Documentation on source materials used by LLM's and tools for this purpose seems virtually non-existent - specifically if a logo can be generated based on particular licenses
Outcomes
- Chat interface-supported development is providing good starting points and open-webui being open source is more flexible than Gemini. Although currently some fancy features such as grounding and generated podcasts are missing.
- Allison still has to be very experienced with openQA to use a chat interface for test review. Publicly available system prompts would make that easier, though.
Testing and adding GNU/Linux distributions on Uyuni by juliogonzalezgil
Join the Gitter channel! https://gitter.im/uyuni-project/hackweek
Uyuni is a configuration and infrastructure management tool that saves you time and headaches when you have to manage and update tens, hundreds or even thousands of machines. It also manages configuration, can run audits, build image containers, monitor and much more!
Currently there are a few distributions that are completely untested on Uyuni or SUSE Manager (AFAIK) or just not tested since a long time, and could be interesting knowing how hard would be working with them and, if possible, fix whatever is broken.
For newcomers, the easiest distributions are those based on DEB or RPM packages. Distributions with other package formats are doable, but will require adapting the Python and Java code to be able to sync and analyze such packages (and if salt does not support those packages, it will need changes as well). So if you want a distribution with other packages, make sure you are comfortable handling such changes.
No developer experience? No worries! We had non-developers contributors in the past, and we are ready to help as long as you are willing to learn. If you don't want to code at all, you can also help us preparing the documentation after someone else has the initial code ready, or you could also help with testing :-)
The idea is testing Salt and Salt-ssh clients, but NOT traditional clients, which are deprecated.
To consider that a distribution has basic support, we should cover at least (points 3-6 are to be tested for both salt minions and salt ssh minions):
- Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)
- Onboarding (salt minion from UI, salt minion from bootstrap scritp, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator)
- Package management (install, remove, update...)
- Patching
- Applying any basic salt state (including a formula)
- Salt remote commands
- Bonus point: Java part for product identification, and monitoring enablement
- Bonus point: sumaform enablement (https://github.com/uyuni-project/sumaform)
- Bonus point: Documentation (https://github.com/uyuni-project/uyuni-docs)
- Bonus point: testsuite enablement (https://github.com/uyuni-project/uyuni/tree/master/testsuite)
If something is breaking: we can try to fix it, but the main idea is research how supported it is right now. Beyond that it's up to each project member how much to hack :-)
- If you don't have knowledge about some of the steps: ask the team
- If you still don't know what to do: switch to another distribution and keep testing.
This card is for EVERYONE, not just developers. Seriously! We had people from other teams helping that were not developers, and added support for Debian and new SUSE Linux Enterprise and openSUSE Leap versions :-)
Pending
FUSS
FUSS is a complete GNU/Linux solution (server, client and desktop/standalone) based on Debian for managing an educational network.
https://fuss.bz.it/
Seems to be a Debian 12 derivative, so adding it could be quite easy.
[W]Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)[W]Onboarding (salt minion from UI, salt minion from bootstrap script, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator) --> Working for all 3 options (salt minion UI, salt minion bootstrap script and salt-ssh minion from the UI).[W]Package management (install, remove, update...) --> Installing a new package works, needs to test the rest.[I]Patching (if patch information is available, could require writing some code to parse it, but IIRC we have support for Ubuntu already). No patches detected. Do we support patches for Debian at all?[W]Applying any basic salt state (including a formula)[W]Salt remote commands[ ]Bonus point: Java part for product identification, and monitoring enablement
Testing and adding GNU/Linux distributions on Uyuni by juliogonzalezgil
Join the Gitter channel! https://gitter.im/uyuni-project/hackweek
Uyuni is a configuration and infrastructure management tool that saves you time and headaches when you have to manage and update tens, hundreds or even thousands of machines. It also manages configuration, can run audits, build image containers, monitor and much more!
Currently there are a few distributions that are completely untested on Uyuni or SUSE Manager (AFAIK) or just not tested since a long time, and could be interesting knowing how hard would be working with them and, if possible, fix whatever is broken.
For newcomers, the easiest distributions are those based on DEB or RPM packages. Distributions with other package formats are doable, but will require adapting the Python and Java code to be able to sync and analyze such packages (and if salt does not support those packages, it will need changes as well). So if you want a distribution with other packages, make sure you are comfortable handling such changes.
No developer experience? No worries! We had non-developers contributors in the past, and we are ready to help as long as you are willing to learn. If you don't want to code at all, you can also help us preparing the documentation after someone else has the initial code ready, or you could also help with testing :-)
The idea is testing Salt and Salt-ssh clients, but NOT traditional clients, which are deprecated.
To consider that a distribution has basic support, we should cover at least (points 3-6 are to be tested for both salt minions and salt ssh minions):
- Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)
- Onboarding (salt minion from UI, salt minion from bootstrap scritp, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator)
- Package management (install, remove, update...)
- Patching
- Applying any basic salt state (including a formula)
- Salt remote commands
- Bonus point: Java part for product identification, and monitoring enablement
- Bonus point: sumaform enablement (https://github.com/uyuni-project/sumaform)
- Bonus point: Documentation (https://github.com/uyuni-project/uyuni-docs)
- Bonus point: testsuite enablement (https://github.com/uyuni-project/uyuni/tree/master/testsuite)
If something is breaking: we can try to fix it, but the main idea is research how supported it is right now. Beyond that it's up to each project member how much to hack :-)
- If you don't have knowledge about some of the steps: ask the team
- If you still don't know what to do: switch to another distribution and keep testing.
This card is for EVERYONE, not just developers. Seriously! We had people from other teams helping that were not developers, and added support for Debian and new SUSE Linux Enterprise and openSUSE Leap versions :-)
Pending
FUSS
FUSS is a complete GNU/Linux solution (server, client and desktop/standalone) based on Debian for managing an educational network.
https://fuss.bz.it/
Seems to be a Debian 12 derivative, so adding it could be quite easy.
[W]Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)[W]Onboarding (salt minion from UI, salt minion from bootstrap script, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator) --> Working for all 3 options (salt minion UI, salt minion bootstrap script and salt-ssh minion from the UI).[W]Package management (install, remove, update...) --> Installing a new package works, needs to test the rest.[I]Patching (if patch information is available, could require writing some code to parse it, but IIRC we have support for Ubuntu already). No patches detected. Do we support patches for Debian at all?[W]Applying any basic salt state (including a formula)[W]Salt remote commands[ ]Bonus point: Java part for product identification, and monitoring enablement
YQPkg - Bringing the Single Package Selection Back to Life by shundhammer
tl;dr
Rip out the high-level YQPackageSelector widget from YaST and make it a standalone Qt program without any YaST dependencies.
See section "Result" at the bottom for the current status after the hack week.
Current Status
See the development status issue at the GitHub repo.
tl;dr: It's usable now with all the key features.
It does real package installation / removal / update with reasonable user feedback.
The Past and the Present
We used to have and still have a powerful software selection with the YaST sw_single module (and the YaST patterns counterpart): You can select software down to the package level, you can easily select one of many available package versions, you can select entire patterns - or just view them and pick individual packages from patterns.
You can search packages based on name, description, "requires" or "provides" level, and many more things.
The Future
YaST is on its way out, to be replaced by the new Agama installer and Cockpit for system administration. Those tools can do many things, but fine-grained package selection is not among them. And there are also no other Open Source tools available for that purpose that even come close to the YaST package selection.
Many aspects of YaST have become obsolete over the years; many subsystems now come with a good default configuration, or they can configure themselves automatically. Just think about sound or X11 configuration; when did you last need to touch them?
For others, the desktops bring their own tools (e.g. printers), or there are FOSS configuration tools (NetworkManager, BlueMan). Most YaST modules are no longer needed, and for many others there is a replacement in tools like Cockpit.
But no longer having a powerful fine-grained package selection like in YaST sw_single will hurt. Big time. At least until there is an adequate replacement, many users will want to keep it.
The Idea
YaST sw_single always revolved around a powerful high-level widget on the abstract UI level. Libyui has low-level widgets like YPushButton, YCheckBox, YInputField, more advanced ones like YTable, YTree; and some few very high-level ones like YPackageSelector and YPatternSelector that do the whole package selection thing alone, working just on the libzypp level and changing the status of packages or patterns there.
For the YaST Qt UI, the YQPackageSelector / YQPatternSelector widgets work purely on the Qt and libzypp level; no other YaST infrastructure involved, in particular no Ruby (or formerly YCP) interpreter, no libyui-level widgets, no bindings between Qt / C++ and Ruby / YaST-core, nothing. So it's not too hard to rip all that part out of YaST and create a standalone program from it.
For the NCurses UI, the NCPackageSelector / NCPatternSelector create a lot of libyui widgets (inheriting YWidget / NCWidget) and use a lot of libyui calls to glue them together; and all that of course still needs a lot of YaST / libyui / libyui-ncurses infrastructure. So NCurses is out of scope here.
Preparatory Work: Initializing the Package Subsystem
To see if this is feasible at all, the existing UI examples needed some fixing to check what is needed on that level. That was the make-or-break decision: Would it be realistically possible to set the needed environment in libzypp up (without being stranded in the middle of that task alone at the end of the hack week)?
Yes, it is: That part is already working:
https://github.com/yast/yast-ycp-ui-bindings/pull/71
RISC-V emulator in GLSL capable of running Linux by favogt
Description
There are already numerous ways to run Linux and some programs through emulation in a web browser (e.g. x86 and riscv64 on https://bellard.org/jslinux/), but none use WebGL/WebGPU to run the emulation on the GPU.
I already made a PoC of an AArch64 (64-bit Arm) emulator in OpenCL which is unfortunately hindered by a multitude of OpenCL compiler bugs on all platforms (Intel with beignet or the new compute runtime and AMD with Mesa Clover and rusticl). With more widespread and thus less broken GLSL vs. OpenCL and the less complex implementation requirements for RV32 (especially 32bit integers instead of 64bit), that should not be a major problem anymore.
Goals
Write an RISC-V system emulator in GLSL that is capable of booting Linux and run some userspace programs interactively. Ideally it is small enough to work on online test platforms like Shaderoo with a custom texture that contains bootstrap code, kernel and initrd.
Minimum:
riscv32 without FPU (RV32 IMA) and MMU (µClinux), running Linux in M-mode and userspace in U-mode.
Stretch goals:
FPU support, S-Mode support with MMU, SMP. Custom web frontend with more possibilities for I/O (disk image, network?).
Resources
RISC-V ISA Specifications
Shaderoo
OpenGL 4.5 Quick Reference Card
Result as of Hackweek 2024
WebGL turned out to be insufficient, it only supports OpenGL ES 3.0 but imageLoad/imageStore needs ES 3.1. So we switched directions and had to write a native C++ host for the shaders.
As of Hackweek Friday, the kernel attempts to boot and outputs messages, but panics due to missing memory regions.
Since then, some bugs were fixed and enough hardware emulation implemented, so that now Linux boots with framebuffer support and it's possible to log in and run programs!
The repo with a demo video is available at https://github.com/Vogtinator/risky-v
New KDE Plasma notification app/applet by apappas
Description
My memory is terrible so I depend a lot on notifications to carry me through the workday. As a plasma user I am ok with the current applet, but I don't love it. It is too small for the centrality it has in my day. Also I dislike how you can not go back to notifications you have dismissed
Goals
Develop a plasma app that * must gather notifications without disrupting the existing notification app * must offer the ablity to refer to dismissed/archived/seen notification up to some defined point in the past * must allow deletion of notifications
Port some classic game to Linux by MDoucha
Let's pick some old classic game, reverse engineer the data formats and game rules and write an open source engine for it from scratch. Some games from 1990s are simple enough that we could have a playable prototype by the end of the week.
Write which games you'd like to hack on in the comments. Don't forget to check e.g. on Open Source Game Clones, Github and SourceForge whether the game is ported already.
Hack Week 24 - Master of Orion II: Battle at Antares & Chaos Overlords
Work on Master of Orion II continues but we can hack more than one game. Chaos Overlords is a dystopian, lighthearted, cyberpunk turn-based strategy game originally released in 1996 for Windows 95 and Mac OS. The player takes on the role of a Chaos Overlord, attempting to control a city. Gameplay involves hiring mercenary gangs and deploying them on an 8-by-8 grid of city sectors to generate income, occupy sectors and take over the city.
How to ~~install & play~~ observe the decompilation progress:
- Clone the Git repository
- A playable reimplementation does not exist yet, but when it does, it will be linked in the repository mentioned above.
Further work needed:
- Analyze the remaining unknown data structures, most of which are related to the AI.
- Decompile the AI completely. The strong AI is part of the appeal of the game. It cannot be left out.
- Reimplement the game.
Hack Week 20, 21, 22 & 23 - Master of Orion II: Battle at Antares
Master of Orion II is one of the greatest turn-based 4X games of the 1990s. Explore the galaxy, colonize planets, research new technologies, fight space monsters and alien empires and in the end, become the ruler of the galaxy one way or another.
How to install & play:
- Clone the Git repository
- Run
./bootstrap; ./configure; make && make install - Copy all *.LBX files from the original Master of Orion II to the installation data directory (
/usr/local/share/openorion2by default) - Run
openorion2
Further work needed:
- Analyze the rest of the original savegame format and a few remaining data files.
- Implement most of the game. The open source engine currently supports only loading saved games from the original version and viewing the galaxy map, fleet management and list of known planets.
Hack Week 19 - Signus: The Artifact Wars
Signus is a Czech turn-based strategy game similar to Panzer General or Battle Isle series. Originally published in 1998 and open-sourced by the original developers in 2003.
How to install & play:
- Clone the Git repository
- Run
./bootstrap; ./configure; make && make installin bothsignusandsignus-datadirectories. - Run
signus
Further work needed:
- Create openSUSE package
- Implement full support for original game data (the open source version uses slightly different data file contents but original game data can be converted using a script).