Project Description

At that moment in order to validate that a branch doesn't introduce bugs to the main branch, we have a set of unit tests running per PR. But those tests only cover methods, classes, and in few cases some integration tests with the database.

We want to increase the test coverage in our Pull Request, proving feedback about end-to-end tests, so real user stories tested from a customer point of view for our User Interfaces (WebUI, XMLRPC API, or Command Tools).

This faster feedback will be immediately translated into fewer bugs introduced into our main branch, reducing the time of CI test report reviews, reducing time fixing those due to context switching.

Goal for this Hackweek

• Increase test coverage and reduce CI issues, by introducing End-to-End tests​ in our Pull Requests
• Smart testing, identifying functional areas of risk (at least giving support to manually select the areas to be tested)

Steps

• Create a Jenkins pipeline to:
  • Provide a list of checkboxes describing testing functional areas, so the user selects which tests prefer to run.
  • Having parameters to set the PR number and the repository
  • Have a pool of 6 slots, to run jobs in parallel
  • Have a pool of 6 test environments (domain names) and three hypervisors, so we can deploy and run multiple jobs
  • A logic to giving a free slot, assign a test environment to use.
  • Dynamically create an OBS namespace for this PR number
  • Push the build of the PR branch
  • Monitor the OBS builds until it is completed
  • Deploy a test environment with Sumaform using the OBS repositories for this PR
  • Run the selected tests
  • Generate an HTML Report and temporary store logs
• If time permits, give a suggested list of functional areas depending on the changes made in the PR

Wiki

Developer doc

Design