The title of this project is inspired from the must-see video
John Hughes - Testing the Hard Stuff and Staying Sane
I would like to play with QuickCheck to generate tests and possibly find bugs automatically.
I chose Wicked as the tested software, but it could be anything else.
My hope is to stay away from Haskell as much as possible as I have no real interest in learning Yet Another Programming Language. Ideally, I would reuse as much as possible from the existing Ruby testsuite for Wicked and stick to Ruby. There are projects like RubyCheck that are ports of QuickCheck to Ruby. But well, if I have to learn Haskell as part of this project, I would do it.
Feel welcome to join the project.
Looking for hackers with the skills:
This project is part of:
Hack Week 13
Activity
Comments
-
-
almost 10 years ago by e_bischoff | Reply
Yes, apart from the choice of examples (Wicked in my case), it is more or less the same project.
And as a matter of fact, I don't want to start with Wicked straight ahead. The plan was to implement in a first step more or less the same example as in the video : it would be an external C program that manages a queue (of integers, why not). I hope to do as many bugs as they do in the video :-) so RubyCheck has at least something to find.
I would use Rantly too to generate the testcase integers. So yeah, a lot of similarities. Thanks for pointing that.
-
almost 10 years ago by e_bischoff | Reply
I just realized that Rantly implements QuickCheck too, it is not only for generating random data.
So there are three out there: Rantly, RushCheck, RubyCheck . Not sure yet which one to take, that will be part of the exploration.
-
almost 10 years ago by e_bischoff | Reply
Nope, rantly does not implement QuickCheck, despite what it says. It is just a very convenient random input generator.
Getting back to RubyCheck as initially planned.
-
almost 10 years ago by e_bischoff | Reply
RubyCheck too does no test failure reduction. Rantly at least has "shrinking", but it is shrinking of data types (strings, integers...), not a sequence of test cases... Do I have to map manually a sequence of test cases to such a data type?
I also discovered Theft. Continuing exploration...
-
almost 10 years ago by e_bischoff | Reply
I ussceeded reducing (shrinking) with Rantly. Starting to save my experiments in my gitlab home project.
-
almost 10 years ago by e_bischoff | Reply
Hackweek finished. I did not make it in time testing Wicked with that way.
I wrote a report with all my finding in the gitlab project:
https://gitlab.suse.de/ebischoff/generate_tests/blob/master/hackweek.txt
-
almost 10 years ago by e_bischoff | Reply
I would need 3 days to finish the hackweek project and do its application to wicked.
-
over 9 years ago by e_bischoff | Reply
I'll consider the project as finished - Rantly has been improved, with my patches accepted upstream, and I played with the tool enough to know quite well his strengths and weaknesses.
If I want to test wicked with it, it could be a separate hackweek project.
Similar Projects
Testing and adding GNU/Linux distributions on Uyuni by juliogonzalezgil
Join the Gitter channel! https://gitter.im/uyuni-project/hackweek
Uyuni is a configuration and infrastructure management tool that saves you time and headaches when you have to manage and update tens, hundreds or even thousands of machines. It also manages configuration, can run audits, build image containers, monitor and much more!
Currently there are a few distributions that are completely untested on Uyuni or SUSE Manager (AFAIK) or just not tested since a long time, and could be interesting knowing how hard would be working with them and, if possible, fix whatever is broken.
For newcomers, the easiest distributions are those based on DEB or RPM packages. Distributions with other package formats are doable, but will require adapting the Python and Java code to be able to sync and analyze such packages (and if salt does not support those packages, it will need changes as well). So if you want a distribution with other packages, make sure you are comfortable handling such changes.
No developer experience? No worries! We had non-developers contributors in the past, and we are ready to help as long as you are willing to learn. If you don't want to code at all, you can also help us preparing the documentation after someone else has the initial code ready, or you could also help with testing :-)
The idea is testing Salt and Salt-ssh clients, but NOT traditional clients, which are deprecated.
To consider that a distribution has basic support, we should cover at least (points 3-6 are to be tested for both salt minions and salt ssh minions):
- Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)
- Onboarding (salt minion from UI, salt minion from bootstrap scritp, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator)
- Package management (install, remove, update...)
- Patching
- Applying any basic salt state (including a formula)
- Salt remote commands
- Bonus point: Java part for product identification, and monitoring enablement
- Bonus point: sumaform enablement (https://github.com/uyuni-project/sumaform)
- Bonus point: Documentation (https://github.com/uyuni-project/uyuni-docs)
- Bonus point: testsuite enablement (https://github.com/uyuni-project/uyuni/tree/master/testsuite)
If something is breaking: we can try to fix it, but the main idea is research how supported it is right now. Beyond that it's up to each project member how much to hack :-)
- If you don't have knowledge about some of the steps: ask the team
- If you still don't know what to do: switch to another distribution and keep testing.
This card is for EVERYONE, not just developers. Seriously! We had people from other teams helping that were not developers, and added support for Debian and new SUSE Linux Enterprise and openSUSE Leap versions :-)
Pending
In progress
FUSS
FUSS is a complete GNU/Linux solution (server, client and desktop/standalone) based on Debian for managing an educational network.
https://fuss.bz.it/
Seems to be a Debian 12 derivative, so adding it could be quite easy.
[W]Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)[W]Onboarding (salt minion from UI, salt minion from bootstrap script, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator) --> Working for all 3 options (salt minion UI, salt minion bootstrap script and salt-ssh minion from the UI).[W]Package management (install, remove, update...) --> Installing a new package works, needs to test the rest.[I]Patching (if patch information is available, could require writing some code to parse it, but IIRC we have support for Ubuntu already). No patches detected. Do we support patches for Debian at all?[W]Applying any basic salt state (including a formula)