Collect flaky test cases identified by the team in a GitHub board and highlight them in the Test report
an invention by oscar-barrios
Updated almost 2 years ago. 1 hackers ♥️. 1 follower.

Project Description

Flaky tests: Those that have not a consistent result, and sometimes are failing and others are passing.

When we review a Test Report in a Pull Request, we would like to identify which of the test features or scenarios we are reviewing has been failing intermittently in the past, which will facilitate a comparison of results between our PR and the main branch.

To address that we want to:

  • Have a list of flaky test cases

    • We can use the cards in the view https://github.com/orgs/SUSE/projects/23/views/7 (We can do API calls with graphql for that)
    • We can try to use some algorithm based on the results stored in Prometheus

  • Inject Cucumber Tags into the .feature file (feature or scenario levels)

    • In Runtime (not committing the change)
    • That will allow us to show it in the HTML Test report later in a transparent way

Goal for this Hackweek

Show, in the HTML Test Report, a new Cucumber Tag "flaky_test" in those scenarios that the team identified as Flaky Tests.

Looking for hackers with the skills:

github graphql cucumber susemanager

This project is part of:

Hack Week 23

Activity

  • almost 2 years ago: jordimassaguerpla liked this project.
  • almost 2 years ago: oscar-barrios started this project.
  • almost 2 years ago: oscar-barrios added keyword "susemanager" to this project.
  • almost 2 years ago: oscar-barrios added keyword "github" to this project.
  • almost 2 years ago: oscar-barrios added keyword "graphql" to this project.
  • almost 2 years ago: oscar-barrios added keyword "cucumber" to this project.
  • almost 2 years ago: oscar-barrios originated this project.


    • Comments

    • oscar-barrios
      almost 2 years ago by oscar-barrios | Reply

      The main code can be found in this PR: https://github.com/uyuni-project/uyuni/pull/7832 Additionally, there is a small change in our CI pipelines, in order to call a new Rake task defined in the PR above.

    Similar Projects

    Move Uyuni Test Framework from Selenium to Playwright + AI by oscar-barrios

    Description

    This project aims to migrate the existing Uyuni Test Framework from Selenium to Playwright. The move will improve the stability, speed, and maintainability of our end-to-end tests by leveraging Playwright's modern features. We'll be rewriting the current Selenium code in Ruby to Playwright code in TypeScript, which includes updating the test framework runner, step definitions, and configurations. This is also necessary because we're moving from Cucumber Ruby to CucumberJS.

    If you're still curious about the AI in the title, it was just a way to grab your attention. Thanks for your understanding.

    Goals

    • Migrate Core tests including Onboarding of clients
    • Improve test reliabillity: Measure and confirm a significant reduction of flakynes.
    • Implement a robust framework: Establish a well-structured and reusable Playwright test framework using the CucumberJS

    Resources