Motivation / Problem

Right now, our QA team tests YaST with OpenQA very much based on keyboard shortcuts: Set the keyboard focus to the "User Name" input field with Alt-U, enter a user name, [Tab] to get to the next field, enter more text, finally Alt-N to activate the "Next" button.

This works quite well in general, but it happens that those keyboard shortcuts change when some other widget is added to the dialog that might compete with one of those keyboard shortcuts that is already taken: An input field "Name" might request the same Alt-N keyboard shortcut as the "Next" button. This might also happen only under certain circumstances, for example on a per-product basis where some product needs more user input and thus more widgets.

The YaST UI always had an automated keyboard shortcut conflict resolver to sort this out so within the same dialog, those shortcuts are always unique. But that might mean that the keyboard shortcut that an OpenQA test case relies on now does something different, thus making the test fail (sometimes even in very subtle ways that are hard to debug).

For one thing, that keyboard shortcut conflict resolver is a very useful thing to have for the user, so we wouldn't like to give it up for the sake of easier testing; for another, the behaviour would still change (albeit in slightly different ways) when we would deactivate that conflict resolver.

Approach for a Solution

So here is another approach: Use the widget IDs that YaST already uses internally and set the keyboard focus to a widget with that ID or activate a button with that ID: For example, the "Next" button internally uses an ID "next", "Name" typically uses "name" etc., independent of the target language the user chose for the user interface (in German, it would be a "Weiter" button, but it still uses "next" as its ID).

Basically, it would work like this:

  • Press a special key combination to prepare for sending a widget ID; say, Ctrl-Alt-Shift-I
  • Send the widget ID: "next", then [Return]
  • YaST activates a widget if it's a button or a similar widget, or just sets the keyboard focus to it otherwise

Feature Request

Fate #324098

Affected Packages

Further Reading

YaST UI

UI Examples

YaST UI Examples

(Ruby examples despite the misleading package name)

Check out that repo, go to the examples/ subdirectory and start an example like this:

/usr/lib/YaST2/bin/y2base ./HelloWorld.rb qt

(for the Qt (graphical) UI)

/usr/lib/YaST2/bin/y2base ./HelloWorld.rb ncurses

(for the NCurses (text based) UI)

Most comprehensive example: Events.rb

/usr/lib/YaST2/bin/y2base ./Events.rb qt

Watch the YaST log in another shell window with

tail -F ~/.y2log

C++ UI Examples

https://github.com/libyui/libyui/tree/master/examples

Looking for hackers with the skills:

yastui openqa libyui-ncurses libyui-qt

This project is part of:

Hack Week 16

Activity

  • about 7 years ago: riafarov liked this project.
  • about 7 years ago: ancorgs liked this project.
  • about 7 years ago: okurz liked this project.
  • about 7 years ago: teclator liked this project.
  • about 7 years ago: shundhammer liked this project.
  • about 7 years ago: shundhammer added keyword "libyui-qt" to this project.
  • about 7 years ago: shundhammer added keyword "yastui" to this project.
  • about 7 years ago: shundhammer added keyword "openqa" to this project.
  • about 7 years ago: shundhammer added keyword "libyui-ncurses" to this project.
  • about 7 years ago: riafarov joined this project.
  • about 7 years ago: shundhammer started this project.
  • about 7 years ago: shundhammer originated this project.

  • Comments

    • shundhammer
      about 7 years ago by shundhammer | Reply

      Implementation Ideas

      The Simplistic Approach: Use a Pop-Up Dialog

      Extend the key event handler in the Qt UI so it catches the special key combination (Ctrl-Shift-Alt-I or whatever), then open a simple Qt pop-up dialog to let the user enter the widget ID and rely on the standard Qt behaviour that [Return] closes that dialog.

      Pro

      • Simple
      • Easy to test
      • It's obvious what's wrong when the process gets stuck during testing - there is still an open pop-up on the screen

      Con

      • The pop-up will easily (and very likely) get in the way of OpenQA's needles (screenshots); OpenQA would have to make sure the pop-up is closed and the screen redrawn to show the original dialog (the dialog under test) so the current screenshot can be compared against the expected "needle".

    • shundhammer
      about 7 years ago by shundhammer | Reply

      The Behind-the-Scenes Approach: Send Widget IDs via a Named Pipe (or Socket)

      This would create a named pipe in, say, /tmp/YaST/widget-ids where the test application (OpenQA) would write the widget ID, and the YaST UI would read from that named pipe in parallel to keyboard and mouse input (Qt does provide this with the QSocketNotifier class).

      Pro

      • No pop-up dialog that might get in the way; the screen (the dialog under test) remains unchanged.

      Con

      • Synchronisation with keyboard input will become a problem; if the test application fills the keyboard buffer very quickly with some widget IDs sent to that named pipe in between. When should the widget IDs be used? There will be race conditions for sure.
      • Security implications: We have to make sure that this named pipe does not open a security hole. It must be writable by root only, and YaST has to check for that and not use it if it is not. Also, it needs to be in a separate directory below /tmp/ because of the sticky bit, and that subdirectory also needs to be accessible only by root. Remember to double-check with the security team.

      TO DO

      • Figure out who creates the directory and the named pipe and when
      • Find a strategy what should happen if there are multiple instances of YaST running; maybe also put a YaST-pid file there and check if that process is still alive? (send a signal #0 to that process)
      • Clean up after YaST / the test ends

    • shundhammer
      about 7 years ago by shundhammer | Reply

      The Combined Approach: Key Combo and a Named Pipe

      Use the keyboard to tell YaST to read one widget ID from a named pipe: Ctrl-Shift-Alt-I, then write the widget ID to the named pipe followed by a newline.

      Pro

      • Also no pop-up dialog that might get in the way
      • No synchronization problems with keyboard buffer vs. named pipe

      Con

      • More complicated
      • Need clear logging (y2log) to make it possible to debug situations where a test gets stuck somewhere in between: log something like "received Ctrl-Shift-Alt-I; reading /tmp/YaST/ids", and when finished "Read widget ID 'next' from /tmp/YaST/ids".
      • Same security implications as with the named pipe approach

      TO DO

      Same as with the named pipe approach

    • shundhammer
      about 7 years ago by shundhammer | Reply

      Needed Tools in the Ecosystem

      • The YDialogSpy needs to be extended to always show a widget's ID, preferably as the first item under its properties (even though it's not strictly speaking a property).
      • Maybe add another special key combination that pops up a dialog showing only the widgets with IDs (i.e. not the complete widget tree, not all the layouts and spacings and alignment widgets) along with their "debug labels"

    • lslezak
      about 7 years ago by lslezak | Reply

      Interesting idea, but it solves only half of the usual openQA issue. One problem is sending the input, the other problem is checking the output (the UI state). I have proposed another solution (REST API) here. It additionally solves the other problem.

      Moreover I already have a working proof of concept with HTTP + JSON (read-only so far, just dumps the current dialog in JSON response).

    • shundhammer
      about 7 years ago by shundhammer | Reply

      @lslezak: I discussed several approaches with people from our QA team in the past, and they seem to be very reluctant to use things that they consider too intrusive; very much like "don't fratzernize with the enemy" (the application under test). I can understand those reservations, yet I also see the benefits of checking widget content on the logical level rather than just comparing pixels in screenshots. Both have their pros and cons.

      The approach in this project might be considerably easier for them to accept since it's not that far away from what they already do: Send key events and then compare screenshots. Only this time it's no longer just plain and dumb key events, they come with some semantics: The widget ID, not just a keyboard shortcut that can (and does) change for lots of reasons.

      IMHO we can easily and we should do both: Giving QA the feature to send widget IDs and keep using their current way of testing with screenshots, and in parallel offer deeper introspection with checking actual internal widget state. Both have their merits, depending on what exactly a test wants to achieve.

    • shundhammer
      about 7 years ago by shundhammer | Reply

      Seel also the reladed project here.

    • lslezak
      about 7 years ago by lslezak | Reply

      I see, this project is simpler so it might be easier to accept and integrate it into the current openQA. We can implement both and see how it evolves later...

    • okurz
      about 7 years ago by okurz | Reply

      I think a special keyboard shortcut + widget id is a nice approach for a potentially problematic button so I am looking forward to the results of this hackweek project. I doubt we want to use it in all occasions though as it will be slower (more keypresses to send) and less obvious but it could be more robust of course.

      Regarding "don't fratzernize with the enemy" I would rather phrase it differently: Test as the user does. What we do with openQA is using exactly the interface that the user has available and will most likely use. That is for example shortcuts where it makes sense, e.g. I assume everybody can agree that "alt-n" for the next button is something that should be expected to work. In other cases we can click on buttons identified by "needle matching" where multiple needles can be created and it's easier to maintain then changing screenshots by versions/products/distributions. We can also do send_key_until_needlematch, e.g. "press tab until you reach ", where "" is also identified by needles. Pressing a shortcut is just the fastest way because it is just one specific key press rather than comparing screen and then clicking or hitting keys multiple times until the right button is reached. I think all examples where we had this problem of changing hotkeys so far were mainly because of the following problems: Test writers do not know that YaST has this (awesome) hotkey conflict resolver. So we just need shundhammer to help us understand how it's working and he is doing a great job in doing so already :)

    • coolo
      about 7 years ago by coolo | Reply

      I don't see how a named pipe could work at all - openQA does not run in the SUT. openQA accesses the system remotely - through VNC or doing a ssh installation.

      What can work is a special spy for special cases - to create the popup, enter something (I think QA engineers would prefer the label text instead of the widget ID) and then popup closes and action is triggered. This wouldn't interfere with screen matching at all.

    • coolo
      about 7 years ago by coolo | Reply

      I don't see how a named pipe could work at all - openQA does not run in the SUT. openQA accesses the system remotely - through VNC or doing a ssh installation.

      What can work is a special spy for special cases - to create the popup, enter something (I think QA engineers would prefer the label text instead of the widget ID) and then popup closes and action is triggered. This wouldn't interfere with screen matching at all.

      • shundhammer
        about 7 years ago by shundhammer | Reply

        Coolo, good point; I hadn't thought about that.

        That would leaves us with an alternative approach I had considered, but I wanted to keep it simple: Use a socket. For testing, nc (netcat) would do just nicely to feed it.

    • shundhammer
      about 7 years ago by shundhammer | Reply

      Since I had intended to try the popup solution first anyway as a very simple proof of concept, this would be one version, maybe even one to stay as an alternative to a socket connection. Both would feed the same internal queue anyway.

      As for widget labels: This is also very fragile; they might change due to proofreading or changed product manager preferences about wording, so widget IDs are definitely the prefered way. But as outlined here, we need some better tools to get the widget IDs; one is always displaying them in the YDialogSpy. A popup-based solution might use a combo box to provide some suggestions which should be obvious enough, maybe even with part of the widget label or whatever; we'll need to play with that to get an impression what it feels like.

    • shundhammer
      about 7 years ago by shundhammer | Reply

      Result

      Implemented the popup-based version for both the Qt and the NCurses UI. So far, this seems to work well.

    • riafarov
      about 7 years ago by riafarov | Reply

      Conclusion and future steps

      Defined goal was reached, pop-up approached was implemented and integrated with openqa to prove the concept. Here is link to the branch with changes: os-autoinst-distri-opensuse During the project I've learned more about yast gui internals. This functionality can be used in the tests with constantly changing keyboard hotkeys in first place.

      After seeing this possibility to have more stable tests using yast gui internals, we can use advantage of it and stop relying on needles and keyboard shortcuts. After that I was introduced to YCP functionality which is there for a long time and already has a recorder and a player. Documentation for the Yast Programming language can be found here. This functionality can be used for testing and already contains the functionality to read widget properties and set them, as well as interact with them simulating user input.

      YCP macro can be executed when application is started by simply calling: /usr/lib/YaST2/bin/y2base ./HelloWorld.rb qt --macro macro.ycp Or, it can be also called during the run using ctrl-alt-shift-p key combination and then selecting ycp file.

      This part makes it possible to operate on lower level and hence make automated tests more stable and scalable.

      Pros

      • Usage of widget IDs allows to write tests which are resistant to fonts and layout changes
      • Usage of YCP allows to avoid syncronization issues (when some control is not yet shown, but we try to act on it)
      • YCP allows to avoid send keys problems we experience in openQA
      • Reduced execution time as we are able to set controls property to certain value instead of typing
      • Macro recorder allows to partly reduce test code development
      • Tests developed in this way can be easily executed by developers and significantly reduce feedback cycle
      • Can be combined with needle matching approach
      • Easy migration to the solution (e.g. only newly or unstable developed tests are developed using YCP)

      Cons

      • Learning curve for test developers
      • Missing functionality and requires further development (e.g. not possible to verify values in the table or select row with certain cell)
      • Unstable behavior if used with installer (unfortunately due to some bugs installer may crash during macro execution, some trivial actions like pressing button may not work if change installers page, etc.)
      • Missing openQA integration (even though it's easy to run such a test, and log errors in y2log file, the results won't be parsed and displayed in friendly way on the dashboard)
      • Works only with Yast GUI, so usage scope is limited

      All the disadvantages are mainly caused by the fact that YCP is not widely used, but that can be solved with reasonable time investment. In order to prove benefits of the solution we could implement one of unstable tests using YCP and run them in parallel to calculate time spent on maintenance of both solutions.

    • shundhammer
      about 7 years ago by shundhammer | Reply

      Packages

      Popup-based solution implemented when pressing Shift-F6 in these packages:

      • libyui-3.4.0
      • libyui-qt-2.48.0
      • libyui-ncurses-2.49.0

    Similar Projects

    Make more sense of openQA test results using AI by livdywan

    Description

    AI has the potential to help with something many of us spend a lot of time doing which is making sense of openQA logs when a job fails.

    User Story

    Allison Average has a puzzled look on their face while staring at log files that seem to make little sense. Is this a known issue, something completely new or maybe related to infrastructure changes?

    Goals

    • Leverage a chat interface to help Allison
    • Create a model from scratch based on data from openQA
    • Proof of concept for automated analysis of openQA test results

    Bonus

    • Use AI to suggest solutions to merge conflicts
      • This would need a merge conflict editor that can suggest solving the conflict
    • Use image recognition for needles

    Resources

    Timeline

    Day 1

    • Conversing with open-webui to teach me how to create a model based on openQA test results

    Day 2

    Highlights

    • I briefly tested compared models to see if they would make me more productive. Between llama, gemma and mistral there was no amazing difference in the results for my case.
    • Convincing the chat interface to produce code specific to my use case required very explicit instructions.
    • Asking for advice on how to use open-webui itself better was frustratingly unfruitful both in trivial and more advanced regards.
    • Documentation on source materials used by LLM's and tools for this purpose seems virtually non-existent - specifically if a logo can be generated based on particular licenses

    Outcomes

    • Chat interface-supported development is providing good starting points and open-webui being open source is more flexible than Gemini. Although currently some fancy features such as grounding and generated podcasts are missing.
    • Allison still has to be very experienced with openQA to use a chat interface for test review. Publicly available system prompts would make that easier, though.


    OpenQA Golang api client by hilchev

    Description

    I would like to make a simple cli tool to communicate with the OpenQA API

    Goals

    • OpenQA has a ton of information that is hard to get via the UI. A tool like this would make my life easier :)
    • Would potentially make it easier in the future to make UI changes without Perl.
    • Improve my Golang skills

    Resources

    • https://go.dev/doc/
    • https://openqa.opensuse.org/api


    Learn obs/ibs sync tool by xlai

    Description

    Once images/repo are built from IBS/OBS, there is a tool to sync the image from IBS/OBS to openqa asset directory and trigger openqa jobs accordingly.

    Goals

    Check how the tool is implemented, and be capable to add/modify our needed images/repo in future by ourselves.

    Resources

    • https://github.com/os-autoinst/openqa-trigger-from-obs
    • https://gitlab.suse.de/openqa/openqa-trigger-from-ibs-plugin/-/tree/master?ref_type=heads


    Enhance UV openQA helper script by mdonis

    Description

    A couple months ago an UV openQA helper script was created to help/automate the searching phase inside openQA for a given MU to test. The script searches inside all our openQA job groups (qam-sle) related with a given MU and generates an output suitable to add (copy & paste) inside the update log.

    This is still a WIP and could use some enhancements.

    Goals

    • Move script from bash to python: this would be useful in case we want to include this into MTUI in the future. The script will be separate from MTUI for now. The idea is to have this as a CLI tool using the click library or something similar.
    • Add option to look for jobs in other sections inside aggregated updates: right now, when looking for regression tests under aggregated updates for a given MU, the script only looks inside the Core MU job group. This is where most of the regression tests we need are located, but some MUs have their regression tests under the YaST/Containers/Security MU job groups. We should keep the Core MU group as a default, but add an option to be able to look into other job groups under aggregated updates.
    • Remove the -a option: this option is used to indicate the update ID and is mandatory right now. This is a bit weird and goes against posix stardards. It was developed this way in order to avoid using positional parameters. This problem should be fixed if we move the script to python.

    Some other ideas to consider:

    • Look into the QAM dashboard API. This has more info on each MU, could use this to link general openQA build results, whether the related RR is approved or not, etc
    • Make it easier to see if there's regression tests for a package in an openQA test build. Check if there's a possibility to search for tests that have the package name in them inside each testsuite.
    • Unit testing?

    More ideas TBD

    Resources

    https://github.com/os-autoinst/scripts/blob/master/openqa-search-maintenance-core-jobs

    https://confluence.suse.com/display/maintenanceqa/Guide+on+how+to+test+Updates

    Post-Hackweek update

    All major features were implemented. Unit tests are still in progress, and project will be moved to the SUSE github org once everything's done. https://github.com/mjdonis/oqa-search


    New features in openqa-trigger-from-obs for openQA by jlausuch

    Description

    Implement new features in openqa-trigger-from-obs to make xml more flexible.

    Goals

    One of the features to be implemented: - Possibility to define "VERSION" and "ARCH" variables per flavor instead of global.

    Resources

    https://github.com/os-autoinst/openqa-trigger-from-obs