Comments

• 

  about 7 years ago by okurz | Reply

  Interesting project idea.

  As I am one of the openQA contributors and openQA is one of the main tools we rely on during QA work on the SLE products I am interested in the results and finding. But also I would like to comment on your initial statements:

  Comparing the screenshots is very fragile, any trivial change in the UI (label, font, color, widget position, spacing, ...) completely breaks the tests

  That is by design because openQA with os-autoinst is used for system testing combined with UI regression testing. If you do not care that your label/font/color changes without you realizing then needles can be created in a very robust to just look for the details you want to see, use a low matching level to match regardless of font changes or use OCR to read the text without caring at all about how it looks.

  The tests cannot be written by an unskilled person, writing the tests is not trivial, you need to know the Perl library used for running the tests and some tricks how to control the YaST UI

  You need to know the test API but isn't that true for any other approach as well?

  If you want to run the tests locally you need to deploy the complete openQA server (or have access to a shared instance)

  If openQA can install and run openQA within openQA (see here) then it shouldn't be too hard. But that is a point that openQA can improve on, granted :-)

×
Edit Comment 2214
# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

* Use asterisks
* for lists

This is an [example link](http://example.com/)

This is an ![example image](http://paste.opensuse.org/view/raw/68957446)

This is a user link @hans

This is a project link hw#some-cool-title
More Complex Markdown Help
Formatting Help
• Edit
• Preview
Interesting project idea. As I am one of the openQA contributors and openQA is one of the main tools we rely on during QA work on the SLE products I am interested in the results and finding. But also I would like to comment on your initial statements: > Comparing the screenshots is very fragile, any trivial change in the UI (label, font, color, widget position, spacing, ...) completely breaks the tests That is by design because openQA with os-autoinst is used for system testing combined with UI regression testing. If you do not care that your label/font/color changes without you realizing then needles can be created in a very robust to just look for the details you want to see, use a low matching level to match regardless of font changes or use OCR to read the text without caring at all about how it looks. > The tests cannot be written by an unskilled person, writing the tests is not trivial, you need to know the Perl library used for running the tests and some tricks how to control the YaST UI You need to know the [test API](http://open.qa/api/testapi/) but isn't that true for any other approach as well? > If you want to run the tests locally you need to deploy the complete openQA server (or have access to a shared instance) If openQA can install and run openQA within openQA (see [here](https://openqa.opensuse.org/tests/latest?arch=x86_64&flavor=dev&test=install_on_tw&distri=openQA&machine=64bit)) then it shouldn't be too hard. But that is a point that openQA can improve on, granted :-)
 
×
Reply to okurz
Interesting project idea.
As I am one of the openQA contributors and openQA is one of the main tools we rely on during QA work on the SLE products I am interested in the results and finding. But also I would like to comment on your initial statements:
Comparing the screenshots is very fragile, any trivial change in the UI (label, font, color, widget position, spacing, ...) completely breaks the tests
That is by design because openQA with os-autoinst is used for system testing combined with UI regression testing. If you do not care that your label/font/color changes without you realizing then needles can be created in a very robust to just look for the details you want to see, use a low matching level to match regardless of font changes or use OCR to read the text without caring at all about how it looks.
The tests cannot be written by an unskilled person, writing the tests is not trivial, you need to know the Perl library used for running the tests and some tricks how to control the YaST UI
You need to know the test API but isn't that true for any other approach as well?
If you want to run the tests locally you need to deploy the complete openQA server (or have access to a shared instance)
If openQA can install and run openQA within openQA (see here) then it shouldn't be too hard. But that is a point that openQA can improve on, granted :-)
---
# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

* Use asterisks
* for lists

This is an [example link](http://example.com/)

This is an ![example image](http://paste.opensuse.org/view/raw/68957446)

This is a user link @hans

This is a project link hw#some-cool-title
More Complex Markdown Help
Formatting Help
• Edit
• Preview
 
• 

  about 7 years ago by lslezak | Reply

  1) Comparing screenshots is not bad in general, openQA + os-autoinst is a generic solution which is able to test things which are not testable otherwise (e.g. you can test the grub boot menu). Or in some cases you really want to test the screenshot to make sure the dialog is still well readable (imagine a bug in the Qt stylesheet). But this generic solution has some disadvantages as mentioned above. Changing the font is not common in YaST, but on the other hand we quite often fine tune the dialog layout (better spacing, alignment...) which does not change the behavior but still requires extra work for the openQA team.

  2) Sure, every testing framework requires some skills and practice. Here I rather meant "non-developer" or "non-programmer". The test example above could be written even by such persons. I'm not saying that this is the best way, but IMHO makes sense in some cases. And I'd like to see how that approach would work for YaST. We have relatively low test coverage in YaST, if we allow easy contribution for people outside the team (or from the community) then this might help to improve the current situation.

  3) The point is that we run rake test:unit for unit tests, that's trivial and does not need any setup (besides some rake package installed in the system). If we could simply run rake test:integration locally then it would be great. Of course, we would need to somehow identify or tag potentially dangerous operations, you do not want the YaST partitioner cleaned up your disk... ;-)

×
Edit Comment 2224
# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

* Use asterisks
* for lists

This is an [example link](http://example.com/)

This is an ![example image](http://paste.opensuse.org/view/raw/68957446)

This is a user link @hans

This is a project link hw#some-cool-title
More Complex Markdown Help
Formatting Help
• Edit
• Preview
1) Comparing screenshots is not bad in general, openQA + os-autoinst is a generic solution which is able to test things which are not testable otherwise (e.g. you can test the grub boot menu). Or in some cases you really want to test the screenshot to make sure the dialog is still well readable (imagine a bug in the Qt stylesheet). But this generic solution has some disadvantages as mentioned above. Changing the font is not common in YaST, but on the other hand we quite often fine tune the dialog layout (better spacing, alignment...) which does not change the behavior but still requires extra work for the openQA team. 2) Sure, every testing framework requires some skills and practice. Here I rather meant "non-developer" or "non-programmer". The test example above could be written even by such persons. I'm not saying that this is the best way, but IMHO makes sense in some cases. And I'd like to see how that approach would work for YaST. We have relatively low test coverage in YaST, if we allow easy contribution for people outside the team (or from the community) then this might help to improve the current situation. 3) The point is that we run `rake test:unit` for unit tests, that's trivial and does not need any setup (besides some rake package installed in the system). If we could simply run `rake test:integration` locally then it would be great. Of course, we would need to somehow identify or tag potentially dangerous operations, you do not want the YaST partitioner cleaned up your disk... ;-)
 
×
Reply to lslezak
1) Comparing screenshots is not bad in general, openQA + os-autoinst is a generic solution which is able to test things which are not testable otherwise (e.g. you can test the grub boot menu). Or in some cases you really want to test the screenshot to make sure the dialog is still well readable (imagine a bug in the Qt stylesheet). But this generic solution has some disadvantages as mentioned above. Changing the font is not common in YaST, but on the other hand we quite often fine tune the dialog layout (better spacing, alignment...) which does not change the behavior but still requires extra work for the openQA team.
2) Sure, every testing framework requires some skills and practice. Here I rather meant "non-developer" or "non-programmer". The test example above could be written even by such persons. I'm not saying that this is the best way, but IMHO makes sense in some cases. And I'd like to see how that approach would work for YaST. We have relatively low test coverage in YaST, if we allow easy contribution for people outside the team (or from the community) then this might help to improve the current situation.
3) The point is that we run rake test:unit for unit tests, that's trivial and does not need any setup (besides some rake package installed in the system). If we could simply run rake test:integration locally then it would be great. Of course, we would need to somehow identify or tag potentially dangerous operations, you do not want the YaST partitioner cleaned up your disk... ;-)
---
# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

* Use asterisks
* for lists

This is an [example link](http://example.com/)

This is an ![example image](http://paste.opensuse.org/view/raw/68957446)

This is a user link @hans

This is a project link hw#some-cool-title
More Complex Markdown Help
Formatting Help
• Edit
• Preview
 
• 

  about 7 years ago by bear454 | Reply

  Suggestion: extend OpenQA to accept tests written in Cucumber, using the Cucumber-Perl module, and some training-wheels like OpenQA steps.

×
Edit Comment 2240
# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

* Use asterisks
* for lists

This is an [example link](http://example.com/)

This is an ![example image](http://paste.opensuse.org/view/raw/68957446)

This is a user link @hans

This is a project link hw#some-cool-title
More Complex Markdown Help
Formatting Help
• Edit
• Preview
Suggestion: extend OpenQA to accept tests written in Cucumber, using the [Cucumber-Perl module](https://github.com/pjlsergeant/test-bdd-cucumber-perl), and some training-wheels like OpenQA steps.
 
×
Reply to bear454
Suggestion: extend OpenQA to accept tests written in Cucumber, using the Cucumber-Perl module, and some training-wheels like OpenQA steps.
---
# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

* Use asterisks
* for lists

This is an [example link](http://example.com/)

This is an ![example image](http://paste.opensuse.org/view/raw/68957446)

This is a user link @hans

This is a project link hw#some-cool-title
More Complex Markdown Help
Formatting Help
• Edit
• Preview
 
• 

  about 7 years ago by lslezak | Reply

  JFYI: I have added the "Results" section into the description to summarize my work done during the Hackweek.

×
Edit Comment 2588
# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

* Use asterisks
* for lists

This is an [example link](http://example.com/)

This is an ![example image](http://paste.opensuse.org/view/raw/68957446)

This is a user link @hans

This is a project link hw#some-cool-title
More Complex Markdown Help
Formatting Help
• Edit
• Preview
JFYI: I have added the "Results" section into the description to summarize my work done during the Hackweek.
 
×
Reply to lslezak
JFYI: I have added the "Results" section into the description to summarize my work done during the Hackweek.
---
# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

* Use asterisks
* for lists

This is an [example link](http://example.com/)

This is an ![example image](http://paste.opensuse.org/view/raw/68957446)

This is a user link @hans

This is a project link hw#some-cool-title
More Complex Markdown Help
Formatting Help
• Edit
• Preview