an invention by oscar-barrios
Project Description
Flaky tests: Those that have not a consistent result, and sometimes are failing and others are passing.
When we review a Test Report in a Pull Request, we would like to identify which of the test features or scenarios we are reviewing has been failing intermittently in the past, which will facilitate a comparison of results between our PR and the main branch.
To address that we want to:
Have a list of flaky test cases
- We can use the cards in the view https://github.com/orgs/SUSE/projects/23/views/7 (We can do API calls with graphql for that)
- We can try to use some algorithm based on the results stored in Prometheus
Inject Cucumber Tags into the .feature file (feature or scenario levels)
- In Runtime (not committing the change)
- That will allow us to show it in the HTML Test report later in a transparent way
Goal for this Hackweek
Show, in the HTML Test Report, a new Cucumber Tag "flaky_test" in those scenarios that the team identified as Flaky Tests.
Looking for hackers with the skills:
This project is part of:
Hack Week 23
Activity
Comments
-
about 2 years ago by oscar-barrios | Reply
The main code can be found in this PR: https://github.com/uyuni-project/uyuni/pull/7832 Additionally, there is a small change in our CI pipelines, in order to call a new Rake task defined in the PR above.
Similar Projects
Testing and adding GNU/Linux distributions on Uyuni by juliogonzalezgil
Join the Gitter channel! https://gitter.im/uyuni-project/hackweek
Uyuni is a configuration and infrastructure management tool that saves you time and headaches when you have to manage and update tens, hundreds or even thousands of machines. It also manages configuration, can run audits, build image containers, monitor and much more!
Currently there are a few distributions that are completely untested on Uyuni or SUSE Manager (AFAIK) or just not tested since a long time, and could be interesting knowing how hard would be working with them and, if possible, fix whatever is broken.
For newcomers, the easiest distributions are those based on DEB or RPM packages. Distributions with other package formats are doable, but will require adapting the Python and Java code to be able to sync and analyze such packages (and if salt does not support those packages, it will need changes as well). So if you want a distribution with other packages, make sure you are comfortable handling such changes.
No developer experience? No worries! We had non-developers contributors in the past, and we are ready to help as long as you are willing to learn. If you don't want to code at all, you can also help us preparing the documentation after someone else has the initial code ready, or you could also help with testing :-)
The idea is testing Salt and Salt-ssh clients, but NOT traditional clients, which are deprecated.
To consider that a distribution has basic support, we should cover at least (points 3-6 are to be tested for both salt minions and salt ssh minions):
- Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)
- Onboarding (salt minion from UI, salt minion from bootstrap scritp, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator)
- Package management (install, remove, update...)
- Patching
- Applying any basic salt state (including a formula)
- Salt remote commands
- Bonus point: Java part for product identification, and monitoring enablement
- Bonus point: sumaform enablement (https://github.com/uyuni-project/sumaform)
- Bonus point: Documentation (https://github.com/uyuni-project/uyuni-docs)
- Bonus point: testsuite enablement (https://github.com/uyuni-project/uyuni/tree/master/testsuite)
If something is breaking: we can try to fix it, but the main idea is research how supported it is right now. Beyond that it's up to each project member how much to hack :-)
- If you don't have knowledge about some of the steps: ask the team
- If you still don't know what to do: switch to another distribution and keep testing.
This card is for EVERYONE, not just developers. Seriously! We had people from other teams helping that were not developers, and added support for Debian and new SUSE Linux Enterprise and openSUSE Leap versions :-)
Pending
Debian 13
The new version of the beloved Debian GNU/Linux OS
Seems to be a Debian 12 derivative, so adding it could be quite easy.
[ ]Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)W]Onboarding (salt minion from UI, salt minion from bootstrap script, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator)[ ]Package management (install, remove, update...)[ ]Patching (if patch information is available, could require writing some code to parse it, but IIRC we have support for Ubuntu already). Probably not for Debian as IIRC we don't support patches yet.[ ]Applying any basic salt state (including a formula)[ ]Salt remote commands[ ]Bonus point: Java part for product identification, and monitoring enablement[ ]Bonus point: sumaform enablement (https://github.com/uyuni-project/sumaform)[ ]Bonus point: Documentation (https://github.com/uyuni-project/uyuni-docs)
Move Uyuni Test Framework from Selenium to Playwright + AI by oscar-barrios
Description
This project aims to migrate the existing Uyuni Test Framework from Selenium to Playwright. The move will improve the stability, speed, and maintainability of our end-to-end tests by leveraging Playwright's modern features. We'll be rewriting the current Selenium code in Ruby to Playwright code in TypeScript, which includes updating the test framework runner, step definitions, and configurations. This is also necessary because we're moving from Cucumber Ruby to CucumberJS.
If you're still curious about the AI in the title, it was just a way to grab your attention. Thanks for your understanding.
Goals
- Migrate Core tests including Onboarding of clients
- Improve test reliabillity: Measure and confirm a significant reduction of flakynes.
- Implement a robust framework: Establish a well-structured and reusable Playwright test framework using the CucumberJS
Resources
- Existing Uyuni Test Framework (Cucumber Ruby + Capybara + Selenium)
- My Template for CucumberJS + Playwright in TypeScript
- Started Hackweek Project
Testing and adding GNU/Linux distributions on Uyuni by juliogonzalezgil
Join the Gitter channel! https://gitter.im/uyuni-project/hackweek
Uyuni is a configuration and infrastructure management tool that saves you time and headaches when you have to manage and update tens, hundreds or even thousands of machines. It also manages configuration, can run audits, build image containers, monitor and much more!
Currently there are a few distributions that are completely untested on Uyuni or SUSE Manager (AFAIK) or just not tested since a long time, and could be interesting knowing how hard would be working with them and, if possible, fix whatever is broken.
For newcomers, the easiest distributions are those based on DEB or RPM packages. Distributions with other package formats are doable, but will require adapting the Python and Java code to be able to sync and analyze such packages (and if salt does not support those packages, it will need changes as well). So if you want a distribution with other packages, make sure you are comfortable handling such changes.
No developer experience? No worries! We had non-developers contributors in the past, and we are ready to help as long as you are willing to learn. If you don't want to code at all, you can also help us preparing the documentation after someone else has the initial code ready, or you could also help with testing :-)
The idea is testing Salt and Salt-ssh clients, but NOT traditional clients, which are deprecated.
To consider that a distribution has basic support, we should cover at least (points 3-6 are to be tested for both salt minions and salt ssh minions):
- Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)
- Onboarding (salt minion from UI, salt minion from bootstrap scritp, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator)
- Package management (install, remove, update...)
- Patching
- Applying any basic salt state (including a formula)
- Salt remote commands
- Bonus point: Java part for product identification, and monitoring enablement
- Bonus point: sumaform enablement (https://github.com/uyuni-project/sumaform)
- Bonus point: Documentation (https://github.com/uyuni-project/uyuni-docs)
- Bonus point: testsuite enablement (https://github.com/uyuni-project/uyuni/tree/master/testsuite)
If something is breaking: we can try to fix it, but the main idea is research how supported it is right now. Beyond that it's up to each project member how much to hack :-)
- If you don't have knowledge about some of the steps: ask the team
- If you still don't know what to do: switch to another distribution and keep testing.
This card is for EVERYONE, not just developers. Seriously! We had people from other teams helping that were not developers, and added support for Debian and new SUSE Linux Enterprise and openSUSE Leap versions :-)
Pending
Debian 13
The new version of the beloved Debian GNU/Linux OS
Seems to be a Debian 12 derivative, so adding it could be quite easy.
[ ]Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)W]Onboarding (salt minion from UI, salt minion from bootstrap script, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator)[ ]Package management (install, remove, update...)[ ]Patching (if patch information is available, could require writing some code to parse it, but IIRC we have support for Ubuntu already). Probably not for Debian as IIRC we don't support patches yet.[ ]Applying any basic salt state (including a formula)[ ]Salt remote commands[ ]Bonus point: Java part for product identification, and monitoring enablement[ ]Bonus point: sumaform enablement (https://github.com/uyuni-project/sumaform)[ ]Bonus point: Documentation (https://github.com/uyuni-project/uyuni-docs)