[haiku] Re: R1/beta1: Here We Go (at last)

  • From: Dennis Catt <dcatt.haiku@xxxxxxx>
  • To: haiku@xxxxxxxxxxxxx
  • Date: Sat, 25 Aug 2018 10:37:56 -0400

Adrien Destugues wrote:

On Thu, Aug 23, 2018 at 09:38:45PM -0400, DCatt wrote:
I have over the years thought about what it would take to get a more
official QA presence established for the Haiku project, but I think I
struggle with how it would fit into the scheme of things.  It would
obviously have to not get in the way so-to-speak.  I think the hardest part
is how would QA fit into the overall process.  QA folks usually like to
formulate test cycles for planned releases, providing feedback to the
developers in hopes to get nasty show-stopping bugs fixed and eventually
sign off something worth shipping to production.  However, maybe that's not
the way you do it with the Haiku project.  QA could be more transparent and
maybe just sample nightly builds on a scheduled interval and treat them as
unofficial RCs and perform regression testing to validate the reliable state
of Haiku and report back to the developers and let them decide what is
shippable concerning a more official release.

As a developer, I value QA a lot and I think you should, indeed, get in
the way. Don't be so shy :)

If you think we need a few more weeks to get the release through QA,
just say so and get things running. The developers job kind of stops at
providing nightly builds. We are a bit clueless about what happens then
and what makes the difference between a nightly and a release (well, I
do see it, but the constant pressure for "let's just tag random nighly
as betas" from devs and users shows me not everyone is aware).

So, it's up to you as the current QA team (yes, it is a small team yet,
but I think a few people are ready to help), to set up your
requirements. How much time do you need to test a release? Can you
involve developers and users in the process? Do you need any specific

We have not done any real release yet (just alphas and betas). The betas
would be a great time for experimenting with QA, I'd say. We can start
with beta1, or, if there is too much things to setup and it would delay
by several months or so, instead prepare this for beta2.


With the B1 release being so close and with people clamoring for such a release, I don't think a QA team being assembled overnight and prepping a test plan for a regression testing cycle could happen quick enough. We could take the exploratory testing route and use the Haiku user guide as a map (of sorts) to coordinate a test plan and strategy and have folks provide testing reports (in a standard format) that are small and easy to complete and provides feedback on what was testing, what probable bugs they found, etc...

I think longterm it would be great to get an in-house test case management system stood up and some QA type folks like myself can start documenting manual test cases using the Haiku user guide as the base of requirements to test against. TestLink <http://www.testlink.org/> is open source and free to use and does integrate with Trac AFAIK, so seems like the best candidate as it has plenty of bells and whistles since it's been around a long time and is quite mature. I'm not sure if the Haiku project has the IT resources for hosting such a solution.

I think it could be possible to work towards a more formal QA test plan and strategy targeting B2 if we could get a test management solution that could help us execute and report on testing cycles that delivers value to the project. We could also maybe take advantage of doing this grunt work preparing for B2 during GCI for the students who help out with the QA related tasks where we could port existing test cases as well as document news ones and prep for a formal test cycle targeting B2.

I'll see if I can get some folks interested in joining the QA ranks and maybe we can assemble a more official team. We have an old haiku-qa <https://www.freelists.org/archive/haiku-qa/> mailing list as well as an IRC channel called #haiku-qa (on freenode). So we have some official ways to communicate. I think the mailing list has been inactivated, so maybe we can reactivate it and add the #haiku-qa IRC channel to the site and get the word out :)


Other related posts: