SUSE Hack Week: TumbleSLE - Applying Tumbleweed Logic to the SLE codebase for more efficient testing & development

Right now internal SLE development is still organised & structured around the concept of 'Milestones'. Schedules are defined, deadlines are set, and off we go making Alpha 1, 2, 3, Betas 1, 2, 3, RC's, and so on.

Meanwhile, QA has evolved, and with openQA and other automated tooling we are increasingly testing SLE in a more agile, rolling model, testing every single build as soon as it's produced by OBS, and just paying extra attention to the Milestones with additional manual testing.

But this mismatch is starting to cause some problems. QA more and more are filing bugs in new builds which were introduced since the latest Milestone. Developers cannot use that Milestone to reproduce the bug. Developers can get the latest build, but that takes time and might have introduced new issues which prohibit testing, because there is no 'release gating' to ensure that the latest SLE builds are at least a usable level of quality (they normally are, but there is no guarantees).

This is a particular problem during heavy development periods, where the SLE codebase is often as fast or maybe even a little faster moving than Tumbleweed.

However, this is a problem any openSUSEr knows well.. openSUSE Factory used to be like that, and the solution was Tumbleweed, prohibiting human use of Factory and using openQA not only as an indicator of quality but tying it to an explicit release process so before Repositories and ISOs are published as Tumbleweed.

This Hackweek project will experiment with the concept of adopting & adapting Tumbleweed style release tooling and processes to the SLE codebase. If all goes according to plan, we might end up with a constantly usable repository and set of ISOs that QA, Developers, and potentially maybe even one day external Beta testers can use as a reliable 'last known good' SLE development version, regardless of the age of the latest SLE milestones or the status of the absolutely latest SLE builds.

Looking for hackers with the skills:

python perl

This project is part of:

Hack Week 14

Activity

over 8 years ago: cseader liked this project.

over 8 years ago: okurz liked this project.

over 8 years ago: okurz joined this project.

over 8 years ago: benediktg liked this project.

over 8 years ago: fcrozat liked this project.

over 8 years ago: digitaltomm liked this project.

over 8 years ago: sebchlad liked this project.

over 8 years ago: dmaiocchi liked this project.

over 8 years ago: osukup liked this project.

over 8 years ago: evshmarnev liked this project.

over 8 years ago: RBrownSUSE liked this project.

over 8 years ago: RBrownSUSE added keyword "python" to this project.

over 8 years ago: RBrownSUSE added keyword "perl" to this project.

over 8 years ago: RBrownSUSE started this project.

over 8 years ago: RBrownSUSE originated this project.

Comments

over 8 years ago by okurz | Reply

What was achieved

Webpage of TumbleSLE is live on tumblesle.qa

TumbleSLE is following the scheme on our shirts for current hack week, "putting the pieces together". You all certainly know Tumbleweed, the rolling distribution of openSUSE, and you should know SLE, the SUSE Linux Enterprise family. TumbleSLE is currently the always latest good build of SLES based on openQA test results.

How does it work

The script tumblesle-release governing the automatic release process is available from okurz/openqa_review extending my python package "openqa-review" which is already used to support the daily openQA review of SLE products on o.s.d. The script is continuously looking for new builds of SLES and comparing it against the last released TumbleSLE version. If a new build is found that does not introduce any regressions (new tests failed) this build is released as the new version of TumbleSLE.

Richard setup a beautiful webpage on tumblesle.qa which is updated automatically whenever the corresponding gitlab repo is changed.

Technical details of system administration

TumbleSLE is stored and made available on a virtual machine in QA departement with name tumblesle.qa.suse.de. The automatic webpage update on changes in the gitlab repo is done by a cronjob of user wwwrun on this machine. The script tumblesle-release is running as user tumblesle continuously. A cronjob of tumblesle is just making sure the script is still running and restarting if not.

over 8 years ago by okurz | Reply

The jekyll sources are updated by /srv/jekyll-source/.git/hooks/post-merge whenever the jekyll sources change. rbrown added reading the build number into the webpage generation process by adding an appropriate template call. I call tumblesle-release now from a wrapper script with all the parameters in /home/tumblesle/bin/tumblesle-release-server-12sp2-x86_64 which calls as a post-release-hook update_jekyll.

Content of /home/tumblesle/bin/tumblesle-release-server-12sp2-x86_64:

flock -n /tmp/tumblesle_release.lock -c "tumblesle-release --openqa-host http://openqa.suse.de --group-id 25 --product 'Server' --src openqa:/var/lib/openqa/factory/ --match '*SP2*Server*x86_64*' --match-hdd 'SLES-12-SP2-x86_64*' -vvvv --post-release-hook /home/tumblesle/bin/update_jekyll" 2>&1 >> /var/log/tumblesle/tumblesle-release.log

Content of /home/tumblesle/bin/update_jekyll:

sudo -u wwwrun /srv/jekyll-source/.git/hooks/post-merge

Added with visudo:

tumblesle ALL=(wwwrun) NOPASSWD: /srv/jekyll-source/.git/hooks/post-merge

Now, can I better save all those system setup instructions as salt states? ;-)