SUSE Hack Week: Collect flaky test cases identified by the team in a GitHub board and highlight them in the Test report

Project Description

Flaky tests: Those that have not a consistent result, and sometimes are failing and others are passing.

When we review a Test Report in a Pull Request, we would like to identify which of the test features or scenarios we are reviewing has been failing intermittently in the past, which will facilitate a comparison of results between our PR and the main branch.

To address that we want to:

Have a list of flaky test cases
- We can use the cards in the view https://github.com/orgs/SUSE/projects/23/views/7 (We can do API calls with graphql for that)
- We can try to use some algorithm based on the results stored in Prometheus
Inject Cucumber Tags into the .feature file (feature or scenario levels)
- In Runtime (not committing the change)
- That will allow us to show it in the HTML Test report later in a transparent way

Goal for this Hackweek

Show, in the HTML Test Report, a new Cucumber Tag "flaky_test" in those scenarios that the team identified as Flaky Tests.

Looking for hackers with the skills:

github graphql cucumber susemanager

This project is part of:

Hack Week 23

Activity

10 months ago: jordimassaguerpla liked this project.

11 months ago: oscar-barrios started this project.

11 months ago: oscar-barrios added keyword "susemanager" to this project.

11 months ago: oscar-barrios added keyword "github" to this project.

11 months ago: oscar-barrios added keyword "graphql" to this project.

11 months ago: oscar-barrios added keyword "cucumber" to this project.

11 months ago: oscar-barrios originated this project.

Comments

10 months ago by oscar-barrios | Reply

The main code can be found in this PR: https://github.com/uyuni-project/uyuni/pull/7832 Additionally, there is a small change in our CI pipelines, in order to call a new Rake task defined in the PR above.

Similar Projects

This project is one of its kind!