Project Description
Everything we do in SUSE requires a certain amount of energy. This energy has a cost and it causes also a certain amount of CO2 emissions. In particular, as Kernel QA team, we run Kernel testing pretty often causing energy consumption that could be saved by introducing optimizations inside the LTP testing.
In this project we use a new parallel execution implementation, in order to talk about how software creation process can save energy and CO2 emissions inside a SW company.
Goal for this Hackweek
We want to answer the following questions:
- How many tests can run in parallel?
- How much energy we save per LTP execution in a virtualized system such as openQA?
- Can we improve the parallelization model to save more energy?
Resources
- runltp-ng: https://github.com/linux-test-project/runltp-ng/
- runltp-ng with parallelization support: https://github.com/acerv/runltp-ng/tree/parallel_coroutines
Jan 31
I had some issues with the runltp-ng
parallel execution, due to the choice of moving UI thread in the coroutines Thread. Tests took +30% time to complete with previous code, but now UI thread is working back again. Created a script to check how many parallel executions we have for all testing suites.
``` Suite: can Total tests: 3 Parallelizable tests: 2
Suite: cap_bounds Total tests: 1 Parallelizable tests: 0
Suite: commands Total tests: 37 Parallelizable tests: 0
Suite: connectors Total tests: 1 Parallelizable tests: 0
Suite: containers Total tests: 86 Parallelizable tests: 0
Suite: controllers Total tests: 346 Parallelizable tests: 1
Suite: cpuhotplug Total tests: 6 Parallelizable tests: 0
Suite: crashme Total tests: 4 Parallelizable tests: 0
Suite: crypto Total tests: 10 Parallelizable tests: 6
Suite: cve Total tests: 77 Parallelizable tests: 5
Suite: dio Total tests: 30 Parallelizable tests: 0
Suite: dmathreaddiotest Total tests: 7 Parallelizable tests: 0
Suite: fcntl-locktests Total tests: 1 Parallelizable tests: 0
Suite: filecaps Total tests: 1 Parallelizable tests: 0
Suite: fs Total tests: 68 Parallelizable tests: 0
Suite: fs_bind Total tests: 95 Parallelizable tests: 0
Suite: fspermssimple Total tests: 18 Parallelizable tests: 0
Suite: fs_readonly Total tests: 55 Parallelizable tests: 0
Suite: fsx Total tests: 1 Parallelizable tests: 0
Suite: hugetlb Total tests: 50 Parallelizable tests: 0
Suite: hyperthreading Total tests: 2 Parallelizable tests: 0
Suite: ima Total tests: 9 Parallelizable tests: 0
Suite: input Total tests: 6 Parallelizable tests: 0
Suite: io Total tests: 2 Parallelizable tests: 1
Suite: ipc Total tests: 8 Parallelizable tests: 0
Suite: irq Total tests: 1 Parallelizable tests: 1
Suite: kernel_misc Total tests: 16 Parallelizable tests: 0
Suite: kvm Total tests: 1 Parallelizable tests: 0
Suite: ltp-aio-stress Total tests: 54 Parallelizable tests: 0
Suite: ltp-aiodio.part1 Total tests: 140 Parallelizable tests: 0
Suite: ltp-aiodio.part2 Total tests: 83 Parallelizable tests: 0
Suite: ltp-aiodio.part3 Total tests: 48 Parallelizable tests: 0
Suite: ltp-aiodio.part4 Total tests: 57 Parallelizable tests: 0
Suite: math Total tests: 10 Parallelizable tests: 0
Suite: mm Total tests: 75 Parallelizable tests: 2
Suite: net.features Total tests: 62 Parallelizable tests: 0
Suite: net.ipv6 Total tests: 11 Parallelizable tests: 0
Suite: net.ipv6_lib Total tests: 6 Parallelizable tests: 2
Suite: net.multicast Total tests: 4 Parallelizable tests: 0
Suite: net.nfs Total tests: 84 Parallelizable tests: 0
Suite: net.rpc_tests Total tests: 51 Parallelizable tests: 0
Suite: net.sctp Total tests: 41 Parallelizable tests: 0
Suite: net.tcp_cmds Total tests: 21 Parallelizable tests: 0
Suite: net.tirpc_tests Total tests: 41 Parallelizable tests: 0
Suite: net_stress.appl Total tests: 10 Parallelizable tests: 0
Suite: netstress.brokenip Total tests: 11 Parallelizable tests: 0
Suite: net_stress.interface Total tests: 25 Parallelizable tests: 0
Suite: netstress.ipsecdccp Total tests: 104 Parallelizable tests: 0
Suite: netstress.ipsecicmp Total tests: 86 Parallelizable tests: 0
Suite: netstress.ipsecsctp Total tests: 104 Parallelizable tests: 0
Suite: netstress.ipsectcp Total tests: 104 Parallelizable tests: 0
Suite: netstress.ipsecudp Total tests: 106 Parallelizable tests: 0
Suite: net_stress.multicast Total tests: 24 Parallelizable tests: 0
Suite: net_stress.route Total tests: 14 Parallelizable tests: 0
Suite: nptl Total tests: 1 Parallelizable tests: 0
Suite: numa Total tests: 20 Parallelizable tests: 2
Suite: powermanagementtests Total tests: 5 Parallelizable tests: 0
Suite: powermanagementtests_exclusive Total tests: 5 Parallelizable tests: 0
Suite: pty Total tests: 9 Parallelizable tests: 1
Suite: s390x_tests Total tests: 1 Parallelizable tests: 0
Suite: sched Total tests: 11 Parallelizable tests: 0
Suite: scsi_debug.part1 Total tests: 140 Parallelizable tests: 0
Suite: securebits Total tests: 3 Parallelizable tests: 0
Suite: smack Total tests: 10 Parallelizable tests: 0
Suite: smoketest Total tests: 15 Parallelizable tests: 5
Suite: staging Total tests: 1 Parallelizable tests: 0
Suite: syscalls Total tests: 1384 Parallelizable tests: 526
Suite: syscalls-ipc Total tests: 61 Parallelizable tests: 26
Suite: tpm_tools Total tests: 12 Parallelizable tests: 0
Suite: tracing Total tests: 9 Parallelizable tests: 0
Suite: uevent Total tests: 3 Parallelizable tests: 0
Suite: watchqueue Total tests: 9 Parallelizable tests: 9
Total tests: 4017 Parallelizable tests: 589
14.66% of the tests are parallelizable ```
Feb 1
Added a new option runltp-ng --force-parallel
to force parallelization even if it's not enabled by tests, but using it causes application crashes, especially for more important suites such as syscalls
or syscalls-ipc
. Not a good idea to use it.
In general, I run a few suites collecting times we need to complete them. It seems the current rule selecting tests for parallel execution is not smart enough and most of the selected tests just end in a seconds or less. This will reflect on time results, where important testing suites, such as syscalls
, will end up just a few minutes before the normal execution. We can do probably better on that side by optimizing the rule, which is currently implemented here.
``` Qemu: Distro: Tumbleweed Kernel: 6.1.8-1-default SMP: 16 RAM: 2GB
syscalls: tests: 1384 parallel: 526 (38% of the tests)
16 workers: 31m 54s
1 worker: 36m 18s
syscalls-ipc: tests: 61 parallel: 26 (42.62% of the tests)
16 workers: 2m 4s
1 worker: 2m 7s
mm: tests: 75 parallel: 2 (42.62% of the tests)
16 workers: 8m 2s
1 worker: 8m 10s
cve: tests: 77 parallel: 5 (6.49% of the tests)
16 workers: 29m 53s
1 worker: 29m 57s
```
02-03 Feb
I focused more on syscalls
testing suites, since it's the most important
suite that can be easily parallelized. All power consumption measurements have
been taken using powerstat -a -R -d 0 1 3600
command, bringing data from
the start of the testing suite execution until the end. All stats have been
taken using my own laptop, since I wasn't able to access openQA workers
physically. Also, to improve measurements, it would be better to have an
external device for measuring power consumption. All tests run inside a Qemu
instance. According with openQA stats, syscalls
has been executed 35 times
in the last month (Jan 2023), so we take this value into account.
Environment
``` Laptop: Model: Lenovo T14s Gen 1 CPU: AMD Ryzen 7 PRO 4750U Memory: 16GB DDR4 Hard disk: NVMe SSD
Qemu:
CPUs: 16
RAM: 4096MB
```
Data
CO2 emission per kWh -> W = 0.244kg CO2/kWh (5% uncertainty)
Avg idle consumption -> I = 2.50 W
Cost energy in germany -> P = 0.534 $/kWh
syscalls exec per month -> R = 35
Normal execution
execution time: T1 = 38m 57s = 2337s
energy consumption: E1 = 9 Wh
monthly consumption: C1 = 35 * 9 = 0.315 kWh
Parallel execution (16 workers)
execution time: T2 = 35m 22s = 2122s -> 10% less
energy consumption: E2 = 10 Wh
monthly consumption: C2 = 35 * 10 = 0.350 kWh
Results
As we notice, there's a small difference between parallelization and normal execution, but overall it's so small that it won't particularly affect CO2 emissions or costs. In particular, in one year we have:
diff: D = (0.315 - 0.350) * 12 = +0.42 kWh
cost: C = D * P = -0.42 * 0.534 = +0.224 $
emissions: C02 = D * W = -0.42 * 0.244 = +0.102 kg
Considering that servers might consume a bit more energy during the execution, we might have bigger values, but still pretty small. The reason is that during parallelization we use more power to run many tests in parallel.
Optimizations
At the end, we can see that in terms of costs or emissions, we don't have a big impact, but in terms of time we still can have a significant impact in one year. We have the possibility to realease openQA workers in a faster way and to complete also other jobs a bit faster. And that of course will have an impact on production, energy consumption and emissions. By taking into account our data, we can say that in one year we will save:
(T1 - T2) * R * 12 = (2337 - 2122) * 35 * 12 ~25 hours
If we are able to introduce a smarter rule to select tests which can run in parallel, the amount of saved time per year might significantly increase. Also, we still have 332 syscalls tests (about 24%) using old API which can't run in parallel nowadays.
This project is part of:
Hack Week 22
Activity
Comments
Similar Projects
RISC-V emulator in GLSL capable of running Linux by favogt
Description
There are already numerous ways to run Linux and some programs through emulation in a web browser (e.g. x86 and riscv64 on https://bellard.org/jslinux/), but none use WebGL/WebGPU to run the emulation on the GPU.
I already made a PoC of an AArch64 (64-bit Arm) emulator in OpenCL which is unfortunately hindered by a multitude of OpenCL compiler bugs on all platforms (Intel with beignet or the new compute runtime and AMD with Mesa Clover and rusticl). With more widespread and thus less broken GLSL vs. OpenCL and the less complex implementation requirements for RV32 (especially 32bit integers instead of 64bit), that should not be a major problem anymore.
Goals
Write an RISC-V system emulator in GLSL that is capable of booting Linux and run some userspace programs interactively. Ideally it is small enough to work on online test platforms like Shaderoo with a custom texture that contains bootstrap code, kernel and initrd.
Minimum:
riscv32 without FPU (RV32 IMA) and MMU (µClinux), running Linux in M-mode and userspace in U-mode.
Stretch goals:
FPU support, S-Mode support with MMU, SMP. Custom web frontend with more possibilities for I/O (disk image, network?).
Resources
RISC-V ISA Specifications
Shaderoo
OpenGL 4.5 Quick Reference Card
Improve various phones kernel mainline support (Qualcomm, Exynos, MediaTek) by pvorel
Similar to previous hackweeks ( https://hackweek.opensuse.org/projects/improve-qualcomm-soc-msm8994-slash-msm8992-kernel-mainline-support, https://hackweek.opensuse.org/projects/test-mainline-kernel-on-an-older-qualcomm-soc-msm89xx-explore-mainline-kernel-qualcomm-mainlining) try to improve kernel mainline support of various phones.
Officially Become a Kernel Hacker! by m.crivellari
Description
My studies as well my spare time are dedicated to the Linux Kernel. Currently I'm focusing on interrupts on x86_64, but my interests are not restricted to one specific topic, for now.
I also "played" a little bit with kernel modules (ie lantern, a toy packet analyzer) and I've added a new syscall in order read from a task A, the memory of a task B.
Maybe this will be a good chance to...
Goals
- create my first kernel patch
Resources
- https://www.kernel.org/doc/html/latest/process/submitting-patches.html
- https://git-send-email.io/ (mentioned also in the kernel doc)
- https://javiercarrascocruz.github.io/kernel-contributor-1
Achivements
- found while working on a bug, this is the 1st patch: cifs: avoid deadlocks while updating iface
Modernize ocfs2 by goldwynr
Ocfs2 has gone into a stage of neglect and disrepair. Modernize the code to generate enough interest.
Goals: * Change the mount sequence to use fscontext * Move from using bufferhead to bio/folios * Use iomap * Run it through xfstests
Hacking on sched_ext by flonnegren
Description
Sched_ext upstream has some interesting issues open for grabs:
Goals
Send patches to sched_ext upstream
Also set up perfetto to trace some of the example schedulers.
Resources
https://github.com/sched-ext/scx
Automated Test Report reviewer by oscar-barrios
Description
In SUMA/Uyuni team we spend a lot of time reviewing test reports, analyzing each of the test cases failing, checking if the test is a flaky test, checking logs, etc.
Goals
Speed up the review by automating some parts through AI, in a way that we can consume some summary of that report that could be meaningful for the reviewer.
Resources
No idea about the resources yet, but we will make use of:
- HTML/JSON Report (text + screenshots)
- The Test Suite Status GithHub board (via API)
- The environment tested (via SSH)
- The test framework code (via files)
Yearly Quality Engineering Ask me Anything - AMA for not-engineering by szarate
Goal
Get a closer look at how developers work on the Engineering team (R & D) of SUSE, and close the collaboration gap between GSI and Engineering
Why?
Santiago can go over different development workflows, and can do a deepdive into how Quality Engineering works (think of my QE Team, the advocates for your customers), The idea of this session is to help open the doors to opportunities for collaboration, and broaden our understanding of SUSE as a whole.
Objectives
- Give $audience a small window on how to get some questions answered either on the spot or within days of how some things at engineering are done
- Give Santiago Zarate from Quality Engineering a look into how $audience sees the engineering departments, and find out possibilities of further collaboration
How?
By running an "Ask me Anything" session, which is a format of a kind of open Q & A session, where participants ask the host multiple questions.
How to make it happen?
I'm happy to help joining a call or we can do it async (online/in person is more fun). Ping me over email-slack and lets make the magic happen!. Doesn't need to be during hackweek, but we gotta kickstart the idea during hackweek ;)
Rules
The rules are simple, the more questions the more fun it will be; while this will be only a window into engineering, it can also be the place to help all of us get to a similar level of understanding of the processes that are behind our respective areas of the organization.
Dynamics
The host will be monitoring the questions on some pre-agreed page, and try to answer to the best of their knowledge, if a question is too difficult or the host doesn't have the answer, he will do his best to provide an answer at a later date.
Atendees are encouraged to add questions beforehand; in the case there aren't any, we would be looking at how Quality Engineering tests new products or performs regression tests
Agenda
- Introduction of Santiago Zarate, Product Owner of Quality Engineering Core team
- Introduction of the Group/Team/Persons interested
- Ice breaker
- AMA time! Add your questions $PAGE
- Looking at QE Workflows: How is
- A maintenance update being tested before being released to our customers
- Products in development are tested before making it generally available
- Engineering Opportunity Board
Testing and adding GNU/Linux distributions on Uyuni by juliogonzalezgil
Join the Gitter channel! https://gitter.im/uyuni-project/hackweek
Uyuni is a configuration and infrastructure management tool that saves you time and headaches when you have to manage and update tens, hundreds or even thousands of machines. It also manages configuration, can run audits, build image containers, monitor and much more!
Currently there are a few distributions that are completely untested on Uyuni or SUSE Manager (AFAIK) or just not tested since a long time, and could be interesting knowing how hard would be working with them and, if possible, fix whatever is broken.
For newcomers, the easiest distributions are those based on DEB or RPM packages. Distributions with other package formats are doable, but will require adapting the Python and Java code to be able to sync and analyze such packages (and if salt does not support those packages, it will need changes as well). So if you want a distribution with other packages, make sure you are comfortable handling such changes.
No developer experience? No worries! We had non-developers contributors in the past, and we are ready to help as long as you are willing to learn. If you don't want to code at all, you can also help us preparing the documentation after someone else has the initial code ready, or you could also help with testing :-)
The idea is testing Salt and Salt-ssh clients, but NOT traditional clients, which are deprecated.
To consider that a distribution has basic support, we should cover at least (points 3-6 are to be tested for both salt minions and salt ssh minions):
- Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)
- Onboarding (salt minion from UI, salt minion from bootstrap scritp, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator)
- Package management (install, remove, update...)
- Patching
- Applying any basic salt state (including a formula)
- Salt remote commands
- Bonus point: Java part for product identification, and monitoring enablement
- Bonus point: sumaform enablement (https://github.com/uyuni-project/sumaform)
- Bonus point: Documentation (https://github.com/uyuni-project/uyuni-docs)
- Bonus point: testsuite enablement (https://github.com/uyuni-project/uyuni/tree/master/testsuite)
If something is breaking: we can try to fix it, but the main idea is research how supported it is right now. Beyond that it's up to each project member how much to hack :-)
- If you don't have knowledge about some of the steps: ask the team
- If you still don't know what to do: switch to another distribution and keep testing.
This card is for EVERYONE, not just developers. Seriously! We had people from other teams helping that were not developers, and added support for Debian and new SUSE Linux Enterprise and openSUSE Leap versions :-)
Pending
FUSS
FUSS is a complete GNU/Linux solution (server, client and desktop/standalone) based on Debian for managing an educational network.
https://fuss.bz.it/
Seems to be a Debian 12 derivative, so adding it could be quite easy.
[ ]
Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)[ ]
Onboarding (salt minion from UI, salt minion from bootstrap scritp, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator)[ ]
Package management (install, remove, update...)[ ]
Patching (if patch information is available, could require writing some code to parse it, but IIRC we have support for Ubuntu already)[ ]
Applying any basic salt state (including a formula)[ ]
Salt remote commands[ ]
Bonus point: Java part for product identification, and monitoring enablement
Hack on isotest-ng - a rust port of isotovideo (os-autoinst aka testrunner of openQA) by szarate
Description
Some time ago, I managed to convince ByteOtter to hack something that resembles isotovideo but in Rust, not because I believe that Perl is dead, but more because there are certain limitations in the perl code (how it was written), and its always hard to add new functionalities when they are about implementing a new backend, or fixing bugs (Along with people complaining that Perl is dead, and that they don't like it)
In reality, I wanted to see if this could be done, and ByteOtter proved that it could be, while doing an amazing job at hacking a vnc console, and helping me understand better what RuPerl needs to work.
I plan to keep working on this for the next few years, and while I don't aim for feature completion or replacing isotovideo tih isotest-ng (name in progress), I do plan to be able to use it on a daily basis, using specialized tooling with interfaces, instead of reimplementing everything in the backend
Todo
- Add
make
targets for testability, e.g "spawn qemu and type" - Add image search matching algorithm
- Add a Null test distribution provider
- Add a Perl Test Distribution Provider
- Fix unittests https://github.com/os-autoinst/isotest-ng/issues/5
- Research OpenTofu how to add new hypervisors/baremetal to OpenTofu
- Add an interface to openQA cli
Goals
- Implement at least one of the above, prepare proposals for GSoC
- Boot a system via it's BMC
Resources
See https://github.com/os-autoinst/isotest-ng
Drag Race - comparative performance testing for pull requests by balanza
Description
«Sophia, a backend developer, submitted a pull request with optimizations for a critical database query. Once she pushed her code, an automated load test ran, comparing her query against the main branch. Moments later, she saw a new comment automatically added to her PR: the comparison results showed reduced execution time and improved efficiency. Smiling, Sophia messaged her team, “Performance gains confirmed!”»
Goals
- To have a convenient and ergonomic framework to describe test scenarios, including environment and seed;
- to compare results from different tests
- to have a GitHub action that executes such tests on a CI environment
Resources
The MVP will be built on top of Preevy and K6.