[openbeos] Re: Quality Assurance

On 2005-04-15 at 04:18:35 [-0400], Niels Reedijk wrote:
> On 4/14/05, Axel Dörfler <axeld@xxxxxxxxxxxxxxxx> wrote:
> > While I agree with you that QA is a critical point, I don't see how we 
> > could realize a system you are proposing - if we had QA people that 
> > qualified, they would probably be coding themselves.
> 
> I don't know if this is entirely true. There's a difference between testers 
> and QA people. Let's face it: testers have the most boring job in the world. 
> Repeatedly breaking things can be very boring, and as Joel (on software) once 
> said, they will probably come and go. That's why I'm making a difference 
> between testing and a Quality Assurance person. The latter distills bug 
> reports, provides user-developer communication and make sure builds will roll 
> out as good as possible. While this job isn't as steady as doing development, 
> it does provide a lower entry point (you get to know the code and the team), 
> and it might even promote new developers entering the project.

Speaking as a person who has both written and received bug reports, the more 
complete and helpful the bug reports, the more productive developers are. 
Personally, while I feel a little shame about creating bugs, I preferred to 
avoid dealing with bug reports because the vast majority of the time it was an 
issue of the bug reporter not understandingor a configuration issue far more 
than an actual bug. People in place to help us to document and verify the bugs 
would be very helpful in the future.
 
> > And while alpha/beta releases might not always be a good replacement for 
> > inhouse testing, it's pretty important to us, as we probably can't buy all 
> > the hardware that is out there (and even test them in all possible 
> > configurations). That's a special attribute of OS and driver development, 
> > though.
> 
> Well, naturally that's true. I don't imagine we can catch every bug in 
> smaller testing releases, but it would filter out the common pitfalls. What 
> if Haiku R1-alpha1 has a bug that crashes Tracker as soon as you move more 
> than two files to another disk. This sort of things are bugs that will be 
> very likely to be noticed, since this is a young OSS project, people will 
> actually file bug reports, and boom, you've got
> 30 duplicates. These issues can be controlled a little more by doing
> in-house testing beforehand.

I think that this is, to a large degree, a matter of PR. Instead of saying "it 
builds and I used it for 5 minutes on my machine; off to SF to make it an 
Alpha", it needs to be carefully considered what to call an Alpha.
 
> > What I could imagine, though, is something you also mentioned in your text 
> > more or less: that a QA team will try to reproduce the bug, or resolve any 
> > issues in a bug report. Then they could either acknowledge the bug, or 
> > remove it. The developers would only get to the acknowledged and complete 
> > bug reports, so that they can start fix it right away. That would lower the 
> > requirements for the QA team quite a lot, and would still help the 
> > development process as well.
> 
> Okay, so we do have a common vision here.  To be honest, it might be wise to 
> start assembling one or two people who try to set up the procedures. But the 
> trick is to find a balance between 'making rules' and how it works in 
> practise. I'm also not sure about how and when to start. I guess it might be 
> a good idea for a few people to come together and work things out (for 
> example, how can the UI of bugzilla be improved, what will be the exact 
> procedures, etc.) These are all issues that are already playing (read: how 
> often bug reporting tools are discussed, etc.), but it's not something I can 
> decide (I don't have the power, nor the exact knowledge).
> 
> I'm hoping for a constructive discussion.

Personally, I would focus mostly on adding developer value. Level 1 is the 
Build Factory. Level 2 might be an easy way to see what Unit Tests exist and 
which ones work ATM. Level 3 might be further than that...

Other related posts: