Testing GUIs is hard: unit tests require a tremendous amount of mocking and often don't capture the exact user input anyway, integration tests on the other hand are difficult to setup as they often require a dedicated infrastructure with virtual machines.

libuitest should become something like the image recognition part of openQA: it receives a video or image stream from some source and searches for matches (e.g. button found, sound played). It should also be able to send clicks, key presses, strings, etc. back via an abstract API to a backend.

It should be compatible to openQA's needles, but the main way how to write tests should be done via a more generic API: `

help_menu = find_menu("Help", search_area_percent=((0, 0), (100, 0)))

help_menu.click()

about_my_program_button = find_menu("About Foo", search_area_percent=(help_menu.location, (100, 0)))

about_my_program_button.click()

window = find_window("About Foo")

about_foo_text = get_text_from_area(window.area)

assert(about_foo_text == f"This is Foo version {env.FOO_VERSION}")

The image recognition part should be handled by OpenCV and the ocr via tesseract. It would be also nice to use some simple machine learning so that libuitest could be trained to find buttons, menus, windows, etc. independently of the used GUI and theme (or at least robustly if you tell it which theme & GUI you are using).

Join this project Leave this project

Looking for hackers with the skills:

machinelearning testing image_recognition opencv tesseract

This project is part of:

Hack Week 19

Activity

almost 5 years ago: Pharaoh_Atem liked this project.

almost 5 years ago: mwilck liked this project.

almost 5 years ago: dancermak added keyword "machinelearning" to this project.

almost 5 years ago: dancermak added keyword "testing" to this project.

almost 5 years ago: dancermak added keyword "image_recognition" to this project.

almost 5 years ago: dancermak added keyword "opencv" to this project.

almost 5 years ago: dancermak added keyword "tesseract" to this project.

almost 5 years ago: dancermak started this project.

almost 5 years ago: dancermak originated this project.

Comments

Be the first to comment!

Similar Projects

machinelearning

FamilyTrip Planner: A Personalized Travel Planning Platform for Families by pherranz

Description

FamilyTrip Planner is an innovative travel planning application designed to optimize travel experiences for families with children. By integrating APIs for flights, accommodations, and local activities, the app generates complete itineraries tailored to each family’s unique interests and needs. Recommendations are based on customizable parameters such as destination, trip duration, children’s ages, and personal preferences. FamilyTrip Planner not only simplifies the travel planning process but also offers a comprehensive, personalized experience for families.

Goals

This project aims to: - Create a user-friendly platform that assists families in planning complete trips, from flight and accommodation options to recommended family-friendly activities. - Provide intelligent, personalized travel itineraries using artificial intelligence to enhance travel enjoyment and minimize time and cost. - Serve as an educational project for exploring Go programming and artificial intelligence, with the goal of building proficiency in both.

Resources

To develop FamilyTrip Planner, the project will leverage: - APIs such as Skyscanner, Google Places, and TripAdvisor to source real-time information on flights, accommodations, and activities. - Go programming language to manage data integration, API connections, and backend development. - Basic machine learning libraries to implement AI-driven itinerary suggestions tailored to family needs and preferences.

testing

Hack on isotest-ng - a rust port of isotovideo (os-autoinst aka testrunner of openQA) by szarate

Description

Some time ago, I managed to convince ByteOtter to hack something that resembles isotovideo but in Rust, not because I believe that Perl is dead, but more because there are certain limitations in the perl code (how it was written), and its always hard to add new functionalities when they are about implementing a new backend, or fixing bugs (Along with people complaining that Perl is dead, and that they don't like it)

In reality, I wanted to see if this could be done, and ByteOtter proved that it could be, while doing an amazing job at hacking a vnc console, and helping me understand better what RuPerl needs to work.

I plan to keep working on this for the next few years, and while I don't aim for feature completion or replacing isotovideo tih isotest-ng (name in progress), I do plan to be able to use it on a daily basis, using specialized tooling with interfaces, instead of reimplementing everything in the backend

Todo

Add make targets for testability, e.g "spawn qemu and type"
Add image search matching algorithm
Add a Null test distribution provider
Add a Perl Test Distribution Provider
Fix unittests https://github.com/os-autoinst/isotest-ng/issues/5
Research OpenTofu how to add new hypervisors/baremetal to OpenTofu
Add an interface to openQA cli

Goals

Implement at least one of the above, prepare proposals for GSoC
Boot a system via it's BMC

Resources

See https://github.com/os-autoinst/isotest-ng

Testing and adding GNU/Linux distributions on Uyuni by juliogonzalezgil

Join the Gitter channel! https://gitter.im/uyuni-project/hackweek

Uyuni is a configuration and infrastructure management tool that saves you time and headaches when you have to manage and update tens, hundreds or even thousands of machines. It also manages configuration, can run audits, build image containers, monitor and much more!

Currently there are a few distributions that are completely untested on Uyuni or SUSE Manager (AFAIK) or just not tested since a long time, and could be interesting knowing how hard would be working with them and, if possible, fix whatever is broken.

For newcomers, the easiest distributions are those based on DEB or RPM packages. Distributions with other package formats are doable, but will require adapting the Python and Java code to be able to sync and analyze such packages (and if salt does not support those packages, it will need changes as well). So if you want a distribution with other packages, make sure you are comfortable handling such changes.

No developer experience? No worries! We had non-developers contributors in the past, and we are ready to help as long as you are willing to learn. If you don't want to code at all, you can also help us preparing the documentation after someone else has the initial code ready, or you could also help with testing :-)

The idea is testing Salt and Salt-ssh clients, but NOT traditional clients, which are deprecated.

To consider that a distribution has basic support, we should cover at least (points 3-6 are to be tested for both salt minions and salt ssh minions):

Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)
Onboarding (salt minion from UI, salt minion from bootstrap scritp, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator)
Package management (install, remove, update...)
Patching
Applying any basic salt state (including a formula)
Salt remote commands
Bonus point: Java part for product identification, and monitoring enablement
Bonus point: sumaform enablement (https://github.com/uyuni-project/sumaform)
Bonus point: Documentation (https://github.com/uyuni-project/uyuni-docs)
Bonus point: testsuite enablement (https://github.com/uyuni-project/uyuni/tree/master/testsuite)

If something is breaking: we can try to fix it, but the main idea is research how supported it is right now. Beyond that it's up to each project member how much to hack :-)

If you don't have knowledge about some of the steps: ask the team
If you still don't know what to do: switch to another distribution and keep testing.

This card is for EVERYONE, not just developers. Seriously! We had people from other teams helping that were not developers, and added support for Debian and new SUSE Linux Enterprise and openSUSE Leap versions :-)

Pending

FUSS

FUSS is a complete GNU/Linux solution (server, client and desktop/standalone) based on Debian for managing an educational network.

https://fuss.bz.it/

Seems to be a Debian 12 derivative, so adding it could be quite easy.

[W] Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)
[W] Onboarding (salt minion from UI, salt minion from bootstrap script, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator) --> Working for all 3 options (salt minion UI, salt minion bootstrap script and salt-ssh minion from the UI).
[W] Package management (install, remove, update...) --> Installing a new package works, needs to test the rest.
[I] Patching (if patch information is available, could require writing some code to parse it, but IIRC we have support for Ubuntu already). No patches detected. Do we support patches for Debian at all?
[W] Applying any basic salt state (including a formula)
[W] Salt remote commands
[ ] Bonus point: Java part for product identification, and monitoring enablement

Yearly Quality Engineering Ask me Anything - AMA for not-engineering by szarate

Goal

Get a closer look at how developers work on the Engineering team (R & D) of SUSE, and close the collaboration gap between GSI and Engineering

Why?

Santiago can go over different development workflows, and can do a deepdive into how Quality Engineering works (think of my QE Team, the advocates for your customers), The idea of this session is to help open the doors to opportunities for collaboration, and broaden our understanding of SUSE as a whole.

Objectives

Give $audience a small window on how to get some questions answered either on the spot or within days of how some things at engineering are done
Give Santiago Zarate from Quality Engineering a look into how $audience sees the engineering departments, and find out possibilities of further collaboration

How?

By running an "Ask me Anything" session, which is a format of a kind of open Q & A session, where participants ask the host multiple questions.

How to make it happen?

I'm happy to help joining a call or we can do it async (online/in person is more fun). Ping me over email-slack and lets make the magic happen!. Doesn't need to be during hackweek, but we gotta kickstart the idea during hackweek ;)

Rules

The rules are simple, the more questions the more fun it will be; while this will be only a window into engineering, it can also be the place to help all of us get to a similar level of understanding of the processes that are behind our respective areas of the organization.

Dynamics

The host will be monitoring the questions on some pre-agreed page, and try to answer to the best of their knowledge, if a question is too difficult or the host doesn't have the answer, he will do his best to provide an answer at a later date.

Atendees are encouraged to add questions beforehand; in the case there aren't any, we would be looking at how Quality Engineering tests new products or performs regression tests

Agenda

Introduction of Santiago Zarate, Product Owner of Quality Engineering Core team
Introduction of the Group/Team/Persons interested
Ice breaker
AMA time! Add your questions $PAGE
Looking at QE Workflows: How is
- A maintenance update being tested before being released to our customers
- Products in development are tested before making it generally available
Engineering Opportunity Board

Automated Test Report reviewer by oscar-barrios

Description

In SUMA/Uyuni team we spend a lot of time reviewing test reports, analyzing each of the test cases failing, checking if the test is a flaky test, checking logs, etc.

Goals

Speed up the review by automating some parts through AI, in a way that we can consume some summary of that report that could be meaningful for the reviewer.

Resources

No idea about the resources yet, but we will make use of:

HTML/JSON Report (text + screenshots)
The Test Suite Status GithHub board (via API)
The environment tested (via SSH)
The test framework code (via files)

Drag Race - comparative performance testing for pull requests by balanza

Description

«Sophia, a backend developer, submitted a pull request with optimizations for a critical database query. Once she pushed her code, an automated load test ran, comparing her query against the main branch. Moments later, she saw a new comment automatically added to her PR: the comparison results showed reduced execution time and improved efficiency. Smiling, Sophia messaged her team, “Performance gains confirmed!”»

Goals

To have a convenient and ergonomic framework to describe test scenarios, including environment and seed;
to compare results from different tests
to have a GitHub action that executes such tests on a CI environment

Resources

The MVP will be built on top of Preevy and K6.