an invention by lslezak
Use Jenkins as a Travis replacement for YaST
Introduction
Currently we use Travis for testing commits and pull requests at GitHub.
Unfortunately there are many disadvantages which require additional work or limit us what we can run in a Travis job.
Travis Disadvantages
- Travis is sometimes very loaded, that usually happens between 15:00 - 19:00 CET. (See the Active Linux Builds graph at the status page.) At that time range starting a build might take about 15 minutes or even more. If a build fails because of some trivial mistake you need to push a new commit and wait another 15 minutes...
- No caching, every build starts from scratch (e.g. GCC, YaST or Ruby packages need to be downloaded and installed for each build again)
- The biggest disadvantage is that the Travis builders use Ubuntu 12.04 LTS as the base system, this makes a lot of troubles:
- We need to build YaST packages (and it's dependencies like libzypp, libyui, libstorage, snapper, hwinfo...) for Ubuntu 12.04. We do it in the YaST:Head:Travis OBS project. The porting required a lot of work, but fortunately thanks to the OBS support the maintenance is low.
- Ubuntu 12.04 is very very old (3 years!, that's too much in the Linux world...), we need to update some basic system components to more recent versions, namely GCC (to compile the projects which use the new C++ features), Ruby (there is obsolete version 1.9)
- We need to upgrade the GCC compiler one more because snapper does not build currenly
- Old Ubunutu also means that updating the system will get more difficult in the future (e.g. backporting a newer GCC or Ruby)
- We cannot run all checks in Travis as it's a completely different environment, e.g. we cannot test building RPM packages
- A test might fail in Ubuntu, but pass in openSUSE (or the other way round), that means the build result is not 100% reliable
- Debugging Travis issues is difficult, you have to install an Ubuntu VM machine and try it there (and even that might not help, I came across a failure which happened in Travis but I was not able to reproduce it in Ubuntu 12.04 VM)
Travis Advantages
- It is hosted and maintained by the Travis CI company, that means no maintenance is needed from our side.
- Nice integration with GitHub, it automatically reports the build status back to GitHub.
Jenkins Advantages
- The workers can run on openSUSE, we can easily use the usual openSUSE tools for building (like
osc) - No need for Ubuntu port, no need for backporting new packages
- We can easily cache needed packages, the builds should be much faster
- We can easily add more workers to build more repositories or commits in parallel
- We can use more powerful workers (more CPU cores, more memory)
Solution
We already use the openSUSE Jenkins instance for building and submitting YaST packages. I found out that there are several Jenkins plugins which could be used to replace Travis with Jenkins:
- The GitHub plugin - handlers commit hooks sent by GitHub, reports build status back
- The GitHub Pull Request Builder - handles pull request hooks sent by GitHub (similar to the previous plugin, but builds the merged result)
- The Embeddable build status plugin - provides a build status badge (similar to Travis one)
Preliminary Work
I already tried the plugins in a testing Jenkins instance to not break the public Jenkins node. It builds the lslezak/yast-yast2 GitHub project. You can see the Jenkins build results in the branch overview page or in a testing pull request at GitHub. The details icons link to the Jenkins instance back.
Possible Problems
- We need to properly handle the branches (i.e. use the SLE12, openSUSE or Factory build target)
- Coveralls support? - The question is whether Coveralls can work outside Travis, this needs to be tested.
The Goal of this Project
- Setup the Jenkins plugins at https://ci.opensuse.org - start with the libyui project - it is a smaller project than YaST and some parts are not built in a CI at all so we can improve it
- Define what will be executed during the build (defined in Jenkins or in the Git repository like in
.travis.ymlfor Travis?) - Provide a HowTo document describing how to setup the Jenkins plugins so Jenkins can be used as a Travis replacement in other project (even outside the YaST scope)
- If the setup for libyui project works well start then converting the YaST modules one by one
This project is part of:
Hack Week 12
Activity
Comments
-
over 9 years ago by adrianSuSE | Reply
Do you plan to chain the builds in jenkins better? The biggest advantage of Travis-ci.org is that everybody's code out there is getting tested when they create pull request. But we do not want to run their code in house in an unprotected environement (means just chroot or VM's with open network).
-
over 9 years ago by lslezak | Reply
The GitHub Pull Request Builder plugin has a nice solution for this.
By default building PRs is disabled, you can configure it to
- allow all repository members to run PR builds automatically
- whitelist specific users
- or use PR comments - the plugin watches the PR comments and if any user with push permissions writes comment "test this please" the plugin starts the build.
See the plugin page for more details, there are some more "commands" supported.
The workflow for the contributions coming from outside is to do a review of the PR and if it is OK (no malicious code found) write the "test this please" comment to trigger the build. The contributions from the team should be allowed to build automatically to make it easier.
If your project does not receive a dozen of PRs a day this approach should work well. For too many PRs it would be probably better to setup building in a protected VM (no network, reset VM after every build, etc..) and build all PRs unconditionaly...
-
Similar Projects
Drag Race - comparative performance testing for pull requests by balanza
Description
«Sophia, a backend developer, submitted a pull request with optimizations for a critical database query. Once she pushed her code, an automated load test ran, comparing her query against the main branch. Moments later, she saw a new comment automatically added to her PR: the comparison results showed reduced execution time and improved efficiency. Smiling, Sophia messaged her team, “Performance gains confirmed!”»
Goals
- To have a convenient and ergonomic framework to describe test scenarios, including environment and seed;
- to compare results from different tests
- to have a GitHub action that executes such tests on a CI environment
Resources
The MVP will be built on top of Preevy and K6.