motivation

Test automation can be fun and openQA is good at it but it has limitations. This project is meant to find out where the limits are for applications with lots of things moving on the screen, e.g. a game. Completely unrelated, web based tools become the norm for many users, e.g. when working from a chromebook or similar environments. The idea here is to try to use only web based tools and not force users to use a terminal which can be frightening :) As a third requirement: Some people are not feeling comfortable using openQA "because tests are written in Perl". This is not necessarily true because tests are rather written using a simplistic test API and only a small subset of Perl knowledge is necessary, however, we still want to support other entry points so there is Python support added in https://github.com/os-autoinst/os-autoinst/pull/1558 . To find out what this new way is capable of this project should use only that Python support and find out any restrictions.

goals

• G1: New automated test available as PR in github.com/os-autoinst/os-autoinst-distri-opensuse/
• G2: Evaluation about what used web tools and their limitations is written down
• G3: Evaluation about the Python support in os-autoinst test API is written down

execution

• Select an application to automate, e.g. https://play0ad.com/
• Research about available web based tools that can be used to develop openQA tests
• Use only what is provided by https://github.com/os-autoinst/os-autoinst/pull/1558 either after it's merged or on another instance connected to openqa.opensuse.org (e.g. cloud instance running that code?)

current status and findings

starting situation

I had applied for the closed Beta for "github codespaces", something like "Virtual Studio Code in the cloud", but so far was not granted access. Means I need to find other solutions.

I did not manage yet to have the python backend for os-autoinst in https://github.com/os-autoinst/os-autoinst/pull/1558 merged as surprisingly tests failed even though I am pretty sure this worked some months ago. As an alternative last week I did https://github.com/os-autoinst/openQA/pull/3804 to allow a custom openQA worker engine so that for example I an experiment with the mentioned python support on production openQA workers while the pull request is still open.

day 1

To provide a custom openQA worker engine with python support I changed https://build.opensuse.org/projects/devel:openQA:TestGithub:OPR-1558/meta to publish packages and add a container including the changes directly

```

…

x86_64

```

and in the project config from https://build.opensuse.org/projects/home:okurz:container/prjconf

``` %if "%_repository" == "containers" Type: docker Repotype: none Patterntype: none %endif

Prefer: openSUSE-release-appliance-docker ```

then branched https://build.opensuse.org/package/show/home:okurz:container/isotovideo-qemu-x86 to https://build.opensuse.org/package/show/devel:openQA:TestGithub:OPR-1558/isotovideo-qemu-x86-python . All using the OBS web UI as using the terminal is to be avoided in this hackweek project ;)

Now I can find the container image on https://registry.opensuse.org/cgi-bin/cooverview?srch_term=project%3D%5Edevel%3AopenQA%3ATestGithub%3AOPR-1558+container%3D.*

Now to be used like in http://open.qa/docs/#personalaccess_token I created a new API key+secret pair in https://openqa.opensuse.org/api_keys and added a new secret variable on https://github.com/okurz/os-autoinst-distri-opensuse/settings/secrets/actions/new name OPENQA_O3_USER_KEY_SECRET with value okurz:: with and filled from the key and secret from https://openqa.opensuse.org/api_keys . Now I can trigger a test job on every push like the following (untested example):

curl -u $OPENQA_O3_USER_KEY_SECRET -X POST https://openqa.opensuse.org/api/v1/jobs …

So that this can be done by github actions for every push to a branch in my fork of the openSUSE openQA tests I created a branch "hackweek_20"

https://github.com/okurz/os-autoinst-distri-opensuse/tree/hackweek_20

where I should merge the content from current upstream master and then add instructions with the above curl command to trigger a job for a to-be-written test, e.g. for 0ad. Also, at best the trigger with curl can already include a command line to use a customized openQA worker engine following https://github.com/os-autoinst/openQA/pull/3804/files , e.g.

curl … ISOTOVIDEO="podman run --pull=always --rm -it registry.opensuse.org/devel/openqa/testgithub/opr-1558/containers/isotovideo:qemu-x86 /usr/bin/isotovideo -d" CASEDIR=https://github.com/okurz/os-autoinst-distri-opensuse/tree/hackweek_20

As a further web-based IDE I envision something like https://github.com/cdr/deploy-code-server/blob/main/guides/railway.md using https://railway.app/ and https://github.com/cdr/code-server with alternatives https://github.com/cdr/deploy-code-server

day 2

Included the above proposal in https://github.com/okurz/os-autoinst-distri-opensuse/blob/hackweek_20/.github/workflows/openqa.yml:

``` name: openqa

yamllint disable-line rule:truthy

on: [push, pullrequest] jobs: openqa: runs-on: ubuntu-latest name: "Trigger an openQA test with custom new test together with a custom os-autoinst backend" container: image: curlimages/curl steps: - name: Trigger openQA test # see https://stackoverflow.com/a/58034787 regarding the repo and branch variables # hdd variable needs to be updated when old images are pruned run: | curl -u ${{ secrets.OPENQAO3USERKEYSECRET }} -X POST \ -H "Content-Type: application/json" \ -d '{"ARCH": "x8664", "FLAVOR": "test", "VERSION": "Tumbleweed", "MACHINE": "64bit", \ "TEST": "okurzhackweek200ad", \ "ISOTOVIDEO": "podman run --pull=always --rm -it registry.opensuse.org/devel/openqa/testgithub/opr-1558/containers/isotovideo:qemu-x86 /usr/bin/isotovideo -d", \ "CASEDIR": "https://github.com/${{ github.repository }}#${GITHUBREF#refs/heads/}", \ "HDD1": "opensuse-Tumbleweed-x8664-20210320-Tumbleweed@64bit.qcow2", \ "SCHEDULE": "tests/boot/boottodesktop,tests/games/0ad" \ }' \ https://openqa.opensuse.org/api/v1/jobs ```

with so far using the normal github web editor. This nicely triggers github actions which execute rather fast (8s) but fails in https://github.com/okurz/os-autoinst-distri-opensuse/runs/2177870127?checksuitefocus=true#step:3:18 with {"error":"TEST field mandatory"}. So JSON mode does not seem to work as I expected. Let's try url-encoded way for curl:

curl -u ${{ secrets.OPENQA_O3_USER_KEY_SECRET }} \ -d 'arch=x86_64&flavor=test&version=Tumbleweed&machine=64bit&test=okurz_hackweek_20_0ad&isotovideo="podman run --pull=always --rm -it registry.opensuse.org/devel/openqa/testgithub/opr-1558/containers/isotovideo:qemu-x86 /usr/bin/isotovideo -d"&casedir=https://github.com/${{ github.repository }}#${GITHUB_REF#refs/heads/}&hdd_1=opensuse-Tumbleweed-x86_64-20210320-Tumbleweed@64bit.qcow2&schedule="tests/boot/boot_to_desktop,tests/games/0ad" \ https://openqa.opensuse.org/api/v1/jobs

This seems to work better: https://github.com/okurz/os-autoinst-distri-opensuse/runs/2178370806?checksuitefocus=true triggered https://openqa.opensuse.org/tests/1678248

The job is incomplete missing "podman" on workers which of course so far is not available. For testing purposes I installed podman on openqaworker7.o.o. Likely I need to add it to apparmor profiles, but let's see it first. So adding the test parameter worker_class=openqaworker7 and getting https://openqa.opensuse.org/tests/1678249 with still "No such file or directory" as if the parameter quoting has gone wrong.

Seems like curl can handle the parameter with just the spaces and without additional quotes ending up in https://openqa.opensuse.org/tests/1678251 which ends up in "Permission denied" for podman as expected due to apparmor. So I did aa-genprof /usr/share/openqa/script/worker to include podman and multiple reruns to come up with a complete profile and now we have in https://openqa.opensuse.org/tests/1678261#details

Error: please update to v2.0.1 or later: outdated conmon version

Well, that's interesting :) rpm -qa conmon reveals conmon-2.0.20-lp152.4.3.1.x86_64, maybe again apparmor? Calling aa-complain usr.share.openqa.script.worker and retriggering yields what I can also confirm when running sudo -u _openqa-worker podman run --rm -it mini/base cat /etc/os-release on openqaworker7.o.o:

Error: error creating runtime static files directory: mkdir /var/lib/empty/.local: permission denied

Well, /var/lib/empty is the actual home directory of the user _openqa-worker and it's correct that the user is denied writing there. https://github.com/containers/podman/blob/master/troubleshooting.md mentions the possibility to specify a different path for which again the best would be a config file in the home directory of the user. Trying to specify a different working directory with:

sudo -u _openqa-worker sh -c 'HOME=/tmp; podman run --rm -it mini/base cat /etc/os-release'

this yields:

ERRO[0000] cannot find UID/GID for user _openqa-worker: No subuid ranges found for user "_openqa-worker" in /etc/subuid - check rootless mode in man pages. Completed short name "mini/base" with unqualified-search registries (origin: /etc/containers/registries.conf) Trying to pull registry.opensuse.org/mini/base:latest... name unknown Trying to pull docker.io/mini/base:latest... Getting image source signatures Copying blob d0cfd5f0b761 done Copying blob 5f4e1487b379 done Copying blob 2aecc7e1714b done Copying blob ebe21ca6e618 done Copying config 81d4bcc7ba done Writing manifest to image destination Storing signatures Error processing tar file(exit status 1): potentially insufficient UIDs or GIDs available in user namespace (requested 0:42 for /etc/shadow): Check /etc/subuid and /etc/subgid: lchown /etc/shadow: invalid argument

I think I have seen this before. I assume rootless podman on openSUSE Leap 15.2 standard packages is not that easy. Ok, going with sudo podman, ending up in https://openqa.opensuse.org/tests/1678263 with

[2021-03-23T20:51:58.0353 CET] [debug] +++ worker notes +++ Trying to pull registry.opensuse.org/devel/openqa/testgithub/opr-1558/containers/isotovideo:qemu-x86... Getting image source signatures Copying blob sha256:0c0b1fc21e7c891fd7642e4f8a4e624a677c212a09553ea655deae26561d0046 Copying blob sha256:81605b3fa72198ecbd994128ebb2651ba4658f73f4b5be9bcfb3f9aed1df0105 Copying config sha256:4d0d2f262c72b58efb8f3795a5ca30c185025c78c98f0fb34b63e4b7e5eb983c Writing manifest to image destination Storing signatures Error: error creating network namespace for container d9e41dac8a7078fba899988c9fa784dc2d851cf5951e3a588306a7fdd26ef714: failed to create namespace: open /proc/14057/task/14116/ns/net: permission denied [2021-03-23T20:52:30.0581 CET] [info] Isotovideo exit status: 126

For now switching on again apparmor

Side-story of the day: For the whole day I have used a Microsoft Surface Go Tablet with pen and keyboard. Works nice so far :)

day 3

So, for the story of "web-based development tools" I thought: "What would be a possibility to run git rebase --interactive without needing a local console?" and then I found https://github.com/GoogleCloudPlatform/cloud-shell-tutorials using https://console.cloud.google.com/cloudshell . There is also http://shell.azure.com/ . First I followed the tutorial covering Azure as I never worked with Azure. But the cloud shell (nice that one can select bash as well as PowerShell) needs persistent storage which costs something. For now I am exploring free options although the cost is rather minimal and I have "170 $" available in my Azure trial account and I am sure the cost for me or my employer SUSE for persistent storage in a personal cloud shell option would be negligible.

So over to https://shell.cloud.google.com . Seems like it has persistent storage without cost as I cloned a small repo from github in the cloudshell and after closing that and reopening on another computer the data was still there! https://gist.github.com/manekinekko/65b4654fb1aac991d8b804803b71d96a#open-a-github-repository-in-the-cloud-shell is a mini-tutorial how one can add a link to a specific git project to immediately clone within a google cloudshell all behind a button that can be put into the web. I imagine this is something we could use in various projects, e.g. a "try out the openQA client here" link?

According to https://cloud.google.com/build/docs/access-private-github-repos managing secrets incurs cost. Of course I don't want to save my password in cleartext in this environment. So following https://codemilltech.com/cloudshell-git/ and http://blog.jeffscudder.com/2018/06/developing-in-github-using-cloud-shell.html I setup a dedicated github authentication token to be used instead of a password when authenticating against github.

As additional permissions I only selected "workflows" to be allowed for this token. Then within a google cloud shell session I did

git config --global --unset credential.helper git config --global credential.helper store git config --global user.name "Oliver Kurz" git config --global user.email "okurz@suse.de" git clone https://github.com/origin/os-autoinst-distri-opensuse.git opensuse git remote add okurz https://github.com/okurz/os-autoinst-distri-opensuse.git git remote update git checkout -b hackweek_20 okurz/hackweek_20 git pull --rebase origin master git push okurz -f

which asked me for username ("okurz") and password, where I entered the token. All worked nicely so far :) This again triggered a new github action https://github.com/okurz/os-autoinst-distri-opensuse/runs/2184932071?checksuitefocus=true which in turn triggered an openQA job: https://openqa.opensuse.org/tests/1678764

With this one can find a rapid development cycle that feels efficient enough, either using the editor component or just the bare shell, e.g. using vim and git command line tools. So I continued with the actual openQA test. First, I thought I might get something running using the perl test backend and the first problem I hit was that the test module file name "0ad" is causing a problem because perl wants to define a package with the same name but package names are only allowed to be non-numerical :D After every change I commit and push and refresh https://openqa.opensuse.org/tests/latest?test=okurzhackweek20_0ad# in a second browser tab.

By the way, I learned by clicking on the three-dot menu on top right of Cloud Shell that there is a weekly usage limit of 50h. Oh, and docker works as well, no podman though: docker run --rm -it mini/base

As 0ad is not in openSUSE:Factory I tried to install from the "games" repository but the installation fails on missing dependencies about libiuuc. Might be transient but anyway it's easier to select something that is part of Tumbleweed already. So switching to http://simutrans.com/ . After minor changes I have https://openqa.opensuse.org/tests/latest?test=okurzhackweek20_simutrans with e.g. the specific job https://openqa.opensuse.org/tests/1678927 that fails in https://openqa.opensuse.org/tests/1678927#step/simutrans/21 with "no pak set found". I guess by default we do not follow recommends. zypper -n in --recommends simutrans also does not pull in recommends it seems. Likely we do not have a pak in Factory at all.

I also learned that one can run a web service on port 8080, then click the "Web Preview" button in top right and have an immediate access to a web service. So one can for example use commands like

docker run -p 8080:3000 -ti my/service_on_port_3000

then click "Web Preview" and "Preview on port 8080" and a browser tab opens pointing to that service

The cloud shell session is not persistent but the filesystem content seems to be. With that I can quickly reinstantiate something like an os-autoinst development environment:

echo 'sudo apt-get install -y libssh2-1-dev fftw-dev libfftw3-dev libtheora-dev libsndfile-dev libopencv-dev cpanminus && cd ~/os-autoinst && cpanm --installdeps --with-feature=test --with-feature=devel --with-recommends . && cpanm Carton && export PATH=~/perl5/bin:$PATH && export PERL5LIB=~/perl5/lib/perl5:$PERL5LIB && carton install && mkdir -p build.cmake && (cd build.cmake && cmake .. && make && sudo make symlinks) && carton exec -- prove -I. t/04-testapi-python.t' > ./test && bash ./test

only that cpanm does not seem to install dependencies for the "test" phase and such.

Switching to "wesnoth":

https://openqa.opensuse.org/tests/latest?test=okurzhackweek20_wesnoth

Recorded a video showing the develop+test cycle using Google Cloud Shell, github actions and openQA: https://www.youtube.com/watch?v=KFWLN1gGOnI

day 4

I am a bit limited in my time this week due to "sick child care" at home. However, would I could do: As the markdown rendering on hackweek.suse.com seems to be … special at times, especially code and URL rendering, I was exploring alternatives. Following https://kalikiana.gitlab.io/post/2021-02-12-setup-gitlab-pages-blog-with-hugo/ from my fellow team colleague I tried hugo on gitlab in https://gitlab.com/okurz/hugo/-/tree/master but failed to publish anything on https://okurz.gitlab.io/ so far but experimented with https://gitpod.io which can be directly started from gitlab on a repo, nice. This is apparently using the same editor component as google Cloud Shell based on Visual Studio Code.

cdywan and mkittler joined me in https://play.wa.suse.net/ for a nice mini-mob and we learned something about perl+python together :) I managed to run all os-autoinst tests within the Google Cloud Shell so also qemu runs fine. With this I could update https://github.com/os-autoinst/os-autoinst/pull/1558 for the os-autoinst change "Add python binding/import/wrapping code for testapi".´As the build of os-autoinst as well as docker runs nicely (fast download, fast build, qemu works) within Google Cloud Shell I tried out to run a test module with isotovideo and surprisingly managed to trigger a segfault :D

Trying to run a custom isotovideo container works like:

docker run --rm -it --pull=always -v ~/opensuse:/opt/opensuse registry.opensuse.org/home/okurz/container/os-autoinst-python/containers/isotovideo:qemu-x86-python /usr/bin/isotovideo -d casedir=/opt/opensuse schedule=tests/games/wesnoth.py

shows that the container does not include the custom os-autoinst that it should have. I wonder if I should create another project and add that as repo.

verdict

During this hack week project due to some capacity constraints I focussed on the "web based development" part. I found that a development environment based on Google Cloud Shell can be very efficient and offers all what is commonly needed for the work where I feel the need for a terminal. Multiple alternatives exist. Also gitpod with its browser extension https://www.gitpod.io/extension-activation/ looks very good. A short video how the gitpod browser extension can be used together with github: https://www.youtube.com/watch?v=7edHele0iow . I can recommend any of the above solutions to anyone needing a terminal for free-software work from time to time from anywhere :) openQA support for python based test modules exists in https://github.com/os-autoinst/os-autoinst/pull/1558 . I deliberately kept the PR open to try out another feature to run a custom backend, including an os-autoinst version in a container. In theory that approach works but so far I have not yet found a convincing way how to control necessary permissions. This is something I would like to explore further and properly think about but I think we will find a convincing way. Then I failed to include my custom os-autoinst version in a container image built by OBS. For the actual openQA test after trying out for 0ad and simutrans I settled for wesnoth which so far is very simple, both in python code that I am not exactly sure if it will work as well as in stable perl code. .

̧