Currently, the SUMA test-suite takes about 6 hours to complete, often fails in the first tests, which set up the environment for the rest of the tests, those tests are what we named "core features".

To solve this problem we had planned to move from Jenkins Job to a Jenkins Pipeline, having stages to split the test suite into core features, initialize clients, secondary features. So,if one stage fails, the rest of the stages will not be executed.

During the hackweek, we want to finish this idea and add a very important bonus, parallelization, which will involve the reorganization and revision of some tests so there are no conflicts between them. We will use the parallel_tests framework that supports Cucumber in Ruby.

We will also work on the merge of the reports obtained by the different stages and processes in parallel, to have a unique test suite report.

Current results are promising, the Jenkins pipeline is working, report merge properly, the time of the whole test suite was reduced considerately (more than 2 hours), I still working on it. As some tests running in parallel can collide and must have a deeper review.

Order of run in the pipeline:

• Deploy
• Core features (Consecutive)
• Initialize clients (Parallel)
• Secondary features which start/stop/delete things on the environment (Consecutive)
• Secondary features (Parallel)
• Generate the report in HTML, merging all reports from each stage, including a link in the Jenkins execution

Links:

• New Jenkins Pipeline
• Refactor of jenkins-runner.sh to be able to run separate commands for each stage
• Jenkins Pipeline (Under development) for SUMA 3.2
• New Rake tasks
• Split and re-organization of run_sets

WIP Pull Requests:

• https://github.com/SUSE/susemanager-ci/pull/47
• https://gitlab.suse.de/galaxy/sumaform-test-runner/merge_requests/148
• https://github.com/SUSE/spacewalk/pull/8202