Testing GUIs is hard: unit tests require a tremendous amount of mocking and often don't capture the exact user input anyway, integration tests on the other hand are difficult to setup as they often require a dedicated infrastructure with virtual machines.

libuitest should become something like the image recognition part of openQA: it receives a video or image stream from some source and searches for matches (e.g. button found, sound played). It should also be able to send clicks, key presses, strings, etc. back via an abstract API to a backend.

It should be compatible to openQA's needles, but the main way how to write tests should be done via a more generic API: `

help_menu = find_menu("Help", search_area_percent=((0, 0), (100, 0)))

help_menu.click()

about_my_program_button = find_menu("About Foo", search_area_percent=(help_menu.location, (100, 0)))

about_my_program_button.click()

window = find_window("About Foo")

about_foo_text = get_text_from_area(window.area)

assert(about_foo_text == f"This is Foo version {env.FOO_VERSION}")

The image recognition part should be handled by OpenCV and the ocr via tesseract. It would be also nice to use some simple machine learning so that libuitest could be trained to find buttons, menus, windows, etc. independently of the used GUI and theme (or at least robustly if you tell it which theme & GUI you are using).

Join this project Leave this project

Looking for hackers with the skills:

machinelearning testing image_recognition opencv tesseract

This project is part of:

Hack Week 19

Activity

almost 6 years ago: Pharaoh_Atem liked this project.

almost 6 years ago: mwilck liked this project.

almost 6 years ago: dancermak added keyword "machinelearning" to this project.

almost 6 years ago: dancermak added keyword "testing" to this project.

almost 6 years ago: dancermak added keyword "image_recognition" to this project.

almost 6 years ago: dancermak added keyword "opencv" to this project.

almost 6 years ago: dancermak added keyword "tesseract" to this project.

almost 6 years ago: dancermak started this project.

almost 6 years ago: dancermak originated this project.

Comments

Be the first to comment!

Similar Projects

machinelearning

Song Search with CLAP by gcolangiuli

Description

Contrastive Language-Audio Pretraining (CLAP) is an open-source library that enables the training of a neural network on both Audio and Text descriptions, making it possible to search for Audio using a Text input. Several pre-trained models for song search are already available on huggingface

Goals

Evaluate how CLAP can be used for song searching and determine which types of queries yield the best results by developing a Minimum Viable Product (MVP) in Python. Based on the results of this MVP, future steps could include:

Music Tagging;
Free text search;
Integration with an LLM (for example, with MCP or the OpenAI API) for music suggestions based on your own library.

The code for this project will be entirely written using AI to better explore and demonstrate AI capabilities.

Resources

CLAP: The main model being researched;
huggingface: Pre-trained models for CLAP;
Free Music Archive: Creative Commons songs that can be used for testing;
Colab: To be used as the development environment;
hw25-song-search: Github repo of the project.

testing

openQA tests needles elaboration using AI image recognition by mdati

Description

In the openQA test framework, to identify the status of a target SUT image, a screenshots of GUI or CLI-terminal images, the needles framework scans the many pictures in its repository, having associated a given set of tags (strings), selecting specific smaller parts of each available image. For the needles management actually we need to keep stored many screenshots, variants of GUI and CLI-terminal images, eachone accompanied by a dedicated set of data references (json).

A smarter framework, using image recognition based on AI or other image elaborations tools, nowadays widely available, could improve the matching process and hopefully reduce time and errors, during the images verification and detection process.

Goals

Main scope of this idea is to match a "graphical" image of the console or GUI status of a running openQA test, an image of a shell console or application-GUI screenshot, using less time and resources and with less errors in data preparation and use, than the actual openQA needles framework; that is:

having a given SUT (system under test) GUI or CLI-terminal screenshot, with a local distribution of pixels or text commands related to a running test status,
we want to identify a desired target, e.g. a screen image status or data/commands context,
- based on AI/ML-pretrained archives containing object or other proper elaboration tools,
- possibly able to identify also object not present in the archive, i.e. by means of AI/ML mechanisms.
the matching result should be then adapted to continue working in the openQA test, likewise and in place of the same result that would have been produced by the original openQA needles framework.
We expect an improvement of the matching-time(less time), reliability of the expected result(less error) and simplification of archive maintenance in adding/removing objects(smaller DB and less actions).

Hackweek step

POC:

study the available tools
prepare a plan for the process to build
write and build a draft application
prepare the data archive from a subset of needles
initialize/pre-train the base archive
select a screenshot from the subset, removing/changing some part
run the POC application
expect the image type is identified in a good %.

Resources

first step of this project is quite identification of useful resources for the scope; some possibilities are:

SUSE AI and other ML tools (i.e. Tensorflow)
Tools able to manage images
RPA test tools (like i.e. Robot framework)
other.

Multimachine on-prem test with opentofu, ansible and Robot Framework by apappas

Description

A long time ago I explored using the Robot Framework for testing. A big deficiency over our openQA setup is that bringing up and configuring the connection to a test machine is out of scope.

Nowadays we have a way¹ to deploy SUTs outside openqa, but we only use if for cloud tests in conjuction with openqa. Using knowledge gained from that project I am going to try to create a test scenario that replicates an openqa test but this time including the deployment and setup of the SUT.

Goals

Create a simple multimachine test scenario with the support server and SUT all created by the robot framework.

Resources

https://github.com/SUSE/qe-sap-deployment
terraform-libvirt-provider

Testing and adding GNU/Linux distributions on Uyuni by juliogonzalezgil

Join the Gitter channel! https://gitter.im/uyuni-project/hackweek

Uyuni is a configuration and infrastructure management tool that saves you time and headaches when you have to manage and update tens, hundreds or even thousands of machines. It also manages configuration, can run audits, build image containers, monitor and much more!

Currently there are a few distributions that are completely untested on Uyuni or SUSE Manager (AFAIK) or just not tested since a long time, and could be interesting knowing how hard would be working with them and, if possible, fix whatever is broken.

For newcomers, the easiest distributions are those based on DEB or RPM packages. Distributions with other package formats are doable, but will require adapting the Python and Java code to be able to sync and analyze such packages (and if salt does not support those packages, it will need changes as well). So if you want a distribution with other packages, make sure you are comfortable handling such changes.

No developer experience? No worries! We had non-developers contributors in the past, and we are ready to help as long as you are willing to learn. If you don't want to code at all, you can also help us preparing the documentation after someone else has the initial code ready, or you could also help with testing :-)

The idea is testing Salt and Salt-ssh clients, but NOT traditional clients, which are deprecated.

To consider that a distribution has basic support, we should cover at least (points 3-6 are to be tested for both salt minions and salt ssh minions):

Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)
Onboarding (salt minion from UI, salt minion from bootstrap scritp, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator)
Package management (install, remove, update...)
Patching
Applying any basic salt state (including a formula)
Salt remote commands
Bonus point: Java part for product identification, and monitoring enablement
Bonus point: sumaform enablement (https://github.com/uyuni-project/sumaform)
Bonus point: Documentation (https://github.com/uyuni-project/uyuni-docs)
Bonus point: testsuite enablement (https://github.com/uyuni-project/uyuni/tree/master/testsuite)

If something is breaking: we can try to fix it, but the main idea is research how supported it is right now. Beyond that it's up to each project member how much to hack :-)

If you don't have knowledge about some of the steps: ask the team
If you still don't know what to do: switch to another distribution and keep testing.

This card is for EVERYONE, not just developers. Seriously! We had people from other teams helping that were not developers, and added support for Debian and new SUSE Linux Enterprise and openSUSE Leap versions :-)

Pending

Debian 13

The new version of the beloved Debian GNU/Linux OS

Seems to be a Debian 12 derivative, so adding it could be quite easy.

[ ] Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)
W] Onboarding (salt minion from UI, salt minion from bootstrap script, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator)
[ ] Package management (install, remove, update...)
[ ] Patching (if patch information is available, could require writing some code to parse it, but IIRC we have support for Ubuntu already). Probably not for Debian as IIRC we don't support patches yet.
[ ] Applying any basic salt state (including a formula)
[ ] Salt remote commands
[ ] Bonus point: Java part for product identification, and monitoring enablement
[ ] Bonus point: sumaform enablement (https://github.com/uyuni-project/sumaform)
[ ] Bonus point: Documentation (https://github.com/uyuni-project/uyuni-docs)

image_recognition

openQA tests needles elaboration using AI image recognition by mdati

Description

Goals

having a given SUT (system under test) GUI or CLI-terminal screenshot, with a local distribution of pixels or text commands related to a running test status,
we want to identify a desired target, e.g. a screen image status or data/commands context,
- based on AI/ML-pretrained archives containing object or other proper elaboration tools,
- possibly able to identify also object not present in the archive, i.e. by means of AI/ML mechanisms.
the matching result should be then adapted to continue working in the openQA test, likewise and in place of the same result that would have been produced by the original openQA needles framework.
We expect an improvement of the matching-time(less time), reliability of the expected result(less error) and simplification of archive maintenance in adding/removing objects(smaller DB and less actions).

Hackweek step

POC:

study the available tools
prepare a plan for the process to build
write and build a draft application
prepare the data archive from a subset of needles
initialize/pre-train the base archive
select a screenshot from the subset, removing/changing some part
run the POC application
expect the image type is identified in a good %.

Resources

first step of this project is quite identification of useful resources for the scope; some possibilities are:

SUSE AI and other ML tools (i.e. Tensorflow)
Tools able to manage images
RPA test tools (like i.e. Robot framework)
other.