Collect flaky test cases identified by the team in a GitHub board and highlight them in the Test report
an invention by oscar-barrios
Updated over 1 year ago. 1 hackers ♥️. 1 follower.

Project Description

Flaky tests: Those that have not a consistent result, and sometimes are failing and others are passing.

When we review a Test Report in a Pull Request, we would like to identify which of the test features or scenarios we are reviewing has been failing intermittently in the past, which will facilitate a comparison of results between our PR and the main branch.

To address that we want to:

  • Have a list of flaky test cases

    • We can use the cards in the view https://github.com/orgs/SUSE/projects/23/views/7 (We can do API calls with graphql for that)
    • We can try to use some algorithm based on the results stored in Prometheus

  • Inject Cucumber Tags into the .feature file (feature or scenario levels)

    • In Runtime (not committing the change)
    • That will allow us to show it in the HTML Test report later in a transparent way

Goal for this Hackweek

Show, in the HTML Test Report, a new Cucumber Tag "flaky_test" in those scenarios that the team identified as Flaky Tests.

Looking for hackers with the skills:

github graphql cucumber susemanager

This project is part of:

Hack Week 23

Activity

  • over 1 year ago: jordimassaguerpla liked this project.
  • over 1 year ago: oscar-barrios started this project.
  • over 1 year ago: oscar-barrios added keyword "susemanager" to this project.
  • over 1 year ago: oscar-barrios added keyword "github" to this project.
  • over 1 year ago: oscar-barrios added keyword "graphql" to this project.
  • over 1 year ago: oscar-barrios added keyword "cucumber" to this project.
  • over 1 year ago: oscar-barrios originated this project.


    • Comments

    • oscar-barrios
      over 1 year ago by oscar-barrios | Reply

      The main code can be found in this PR: https://github.com/uyuni-project/uyuni/pull/7832 Additionally, there is a small change in our CI pipelines, in order to call a new Rake task defined in the PR above.

    Similar Projects

    This project is one of its kind!