[haiku] [RFC] QA Test Management
- From: Richie Nyhus-Smith <richienyhus@xxxxxxxxx>
- To: haiku@xxxxxxxxxxxxx
Hi all,
What is everyone's thoughts on having a manual QA test management tool for
Haiku?
I think that for a project the size and scope of Haiku, there really is a
need for some type of manual test management.
I have included below an overview of test management and our possible
options.
Cheers,
Richard.
------
There have been discussions in the past about test management, but nothing
substantial due to the lack of knowledge of our options. Unfortunately I
have found, after a fair bit of research, that there really are not that
many free options for Haiku (for manual testing).
This is primarily because there are not that many open source test
management tools out there that are not designed by lunatics (i.e.
testlink); have not been abandoned 10 years ago or do not require closed
source binary blobs to run (thus not really being open source).
Manual testing gives end users the ability to help out through
crowdsourcing regression tracking and defect verification. Basicly, they
can help let devopers know that something that was meant to be working, now
is not working and something that was reported or thought to not be
working, now is working.
This is done by writing a kind of step by step howto guide on a feature
that allows the tester to report if they can or can't follow each step.
There is normally space in the test management tool to give the tester
context (expected results) and prerequisites (i.e. you need a .mp4 file
from the demopack) and space for the tester to comment on each step and/or
each testsuite.
Automated (non-unit) testing tools like Cucumber(Ruby), Lettuce(Python)
etc, are similar, but do not have the context that manual testing has and
also requires the author to at least have some rundermentory programing
skills. We would only have to invest in buildbot - trac integration to give
us access to Lettuce/Python-Behave testing, if we want this type of testing
as well.
-------
There are a couple open source options for us. One is to borrow the Drupal
based QA tool from Ubuntu [1] and install that as part of a intranet style
backend website separate from our main Drupal website. Another option is to
use a trac plugin [2] that adds test management features to trac, although
we might have to make it a bit more lightweight to fit our needs.
Both squashtest [3] and tarantula [4] are open source web apps which are
still actively developed, although they less intuitive than some of their
commercial competitors (But still way better than testlink).
There is the option of someone starting a project from scratch for Haiku,
which would only have a limited amount of open source competitors or there
is the option of taking on one of the abandoned open source tools [5][6][7].
The commercial webapp PolarionQA is free for opensource projects, however I
am not sure how integrated it is to the larger PolarionALM tool. [8]
Gurock Testrail is another commercial tool that is highly intuitive, has a
python api client that could be connected to buildbot's release management
system [9] and it already has great trac integration [10]. Its website
states that open source projects should fill out a form or email them to
find out how to get free licensing, however form is redirecting to their
main page.
The test management system by Zephyr that hooks into Atlassian cloud is
free for open source projects [11] and has the same UI and api as
Bitbucket. It can be self-hosted if desired and there are two other
competing tools using the same framework [12][13], but not quite up to the
standard of Zephyr.
[1] https://launchpad.net/ubuntu-qa-website
[2] http://trac-hacks.org/wiki/TestManagerForTracPlugin
[3] http://www.squashtest.org
[4] http://www.testiatarantula.com
[5] http://sourceforge.net/projects/qualityspy/
[6] http://endeavour-mgmt.sourceforge.net
[7] http://admc.io/testmill/
[8] http://www.polarion.com/products/alm/index.php
[9] https://github.com/gurock/testrail-api
[10] http://www.gurock.com/testrail/trac-test-management.i.html
[11] https://marketplace.atlassian.com/plugins/com.thed.zephyr.je#cloud
[12]
https://marketplace.atlassian.com/plugins/com.intenso.jira.plugins.suiTest
[13] https://marketplace.atlassian.com/plugins/com.xpandit.plugins.xray
Other related posts:
