[openbeos] Quality Assurance
- From: Niels Reedijk <niels.reedijk@xxxxxxxxx>
- To: Haiku Mailinglist <openbeos@xxxxxxxxxxxxx>
- Date: Thu, 14 Apr 2005 15:45:17 +0200
Hello guys,
I've always been intrigued by quality assurance and the lack of it in
Open Source Software. At the current rate of developments we are
propelling towards the first generation of alpha and beta builds, and
man, will these get attention. The downside of this, is that the
project isn't ready to handle that yet. I've made several observations
on the issue, and I've posted these on my web-log (
http://nielx.blogspot.com ). With the many discussions surrounding bug
trackers and source control management, I think it is time to start a
good discussion on these issues. I'm not implying that my writeup
contains the ultimate answers, neither do I think that my solution is
the ultimate one, but I guess it is time to open a dialog about the
issue, to get to a point where we can assure a certain level of
quality.
I've pasted the entry at the end of this message, open your mouths!
Niels
[ nicer font at http://nielx.blogspot.com ]
In this document I'd like to set out a few thoughts regarding Quality
Assurance. As a somewhat active participant in Open Source projects, I
have received a reasonable amount of observations regarding quality
assurance and it's practises in Open Source software. I must admit,
thought, that I don't have any professional experience in quality
assurance, or software development in general. However, that doesn't
belittle the fact that the Haiku project is at a certain point in its
life where a good road to QA should be chosen.
In order to work with the observations and ideas, we should be first
set out the definition of quality assurance. My view is that quality
assurance is the ongoing process by which the quality of an end
product is constantly monitored and controlled. Software quality
assurance is no different, perhaps with the only problem being that
software in some ways will always be 'broken' (or buggy) - the trick
is to have it working for as many people as possible.
This simple definition reveals the ultimate problem which Open Source
software is having: QA is a process - meaning it needs constant
attention and care. But the open software projects are generally
catered by developers. The consensus seems to be that whoever
contributes, gets something to say about the project. The fact that
developers provide the most substantial part of code, leaves them to
say things about what happens next. The ultimate problem is twofold.
First of all, most developers code what they need/require (if it works
for them, it's okay). The other side of the coin is that there comes a
point where the developer:user ratio becames greatly disturbed, so
that developers will be overwhelmed with bug reports to an extend they
can only ignore them. Furthermore, a developer might not have all the
patience and care needed to provide a descent amount of QA.
Mozilla
When you carefully look at most Open Source projects, you see that
they do recognise the need for QA. Most obvious leader in OSS QA is
the mozilla project. This is probably most due to the fact that it
stems from Netscape - a company with a dubious QA record, but
nevertheless a company. Companies have commercial interests to
protect, so they usually employ some kind of QA mechanism. The
policies of Netscape seem to have been flowing into the Mozilla
project.
The implementation of the Mozilla project is two-fold. First of all,
they have a system called 'tinderboxes'. A tinderbox is merely a
server that's continually building and running unit tests on the
latest code from CVS. The boxes report their status back to a central
server which neatly publishes all data on a web page. The nice thing
is that everybody is responsible for their own changes. If you commit
something that breaks on other hardware, it's your job to make sure
the repository get fixed. So this is actually an ongoing process to
ensure a constantly working tree (= a more constant quality).
The second main tool is their custom developed bug-tracker 'Bugzilla'.
Bugzilla is the number one choice for bug tracking. The interesting
thing is that it has a stable code base and it is well maintained. The
good things about it are the power it has, and the workflow it
imposes. Naturally, there are some issues that lower the fun a little.
First of all, it's a usuability nightmare for the users. Second of
all, over the years it has evolved (with the Mozilla project) to a new
level where it functions more like a project tracker tool, instead of
merely a bug tracker.
I might need to illustrate that point. Over the years (ever since
Netscape was taken over by AOL and has taken its hands from the
Mozilla project) the use of Bugzilla has changed. Instead of a bug
tracker for users to report their bugs in, it has moved over to be
more like a large container of all patches that need to be tested.
This means that the emphasis on the users has started to disappear.
This is reflected in the fact that they don't usually assign QA
contacts to bugs anymore. Also a lot of features in Bugzilla have
turned it into helping developers and complicating the user interface
even more. That said, bugzilla still offers great features for bug
tracking, it just needs a tweaked UI (which can be done quite easily).
It's not the tools; it's the use
Even though the Mozilla projects seem to have tools to properly
perform QA projects, they still lack something. First of all, you will
find many bugs in their database that seem to be forgotten, or even
never been looked at. Second of all, the project still believes that
it will find most issues by issueing alpha and beta builds. I would
argue that issueing builds won't have the influence on quality we
expect, and I intend to prove that claim.
But first on the first issue. One of the reasons to use a bugtracker
is to lower the entrypoint for bug reports, and for developers to keep
track of the reported bugs (instead of using e-mail to do that).
However, you won't prevent the problem mentioned earlier: at a certain
moment in time, the user:developer ratio changes to such an extend the
developer will still have too much bug reports, and the sad thing is
that nothing happens with it anyway. This leads to the point that
employing a bug tracker to improve communication with the users is a
myth.
The second issue, posting builds in order to get feedback to improve
the product, is also overhyped. The common idea is that 'more eyes'
will improve the general quality. While in fact this may be
theoretically true (or not), it's at least the case that when someone
is a developer, betas and alphas are usually employed by users to
check out the 'cool new stuff'. Users are generally not developers,
and most probably won't report bugs. The ones that do, will often
publish low-quality (useless) bug reports that in no way can be
reproduced. Add this up to the previous point, and you get no
improvement in Quality Assurance whatsoever.
When you add all this up, you'll understand that Quality Assurance
might not be a job for developers. Thus an alternative way of keeping
QA up to par in Open Source needs to be devised.
Solving the problems
In order to try to work around the common pitfalls of Open Source
Quality Asurance, I will present some ideas/concepts that may be able
to counter the problem.
Just as a reminder: QA is a continual process that checks the code of
the project. The first thing that implies, is that at every point in
time, the code in the subversion repository should be checked, and
regular builds should be provided so that people will be able to keep
an eye on what's working and what's not. Second of all, regular unit
testing should be designed in such a way that they would be able to
check the most basic failures. In effect, this probably means a system
like the tinderboxes. This would also allow to have continual build
checks on multiple systems (haiku, R5, R5-bone, zeta) and multiple
compilers (gcc 3).
The second part of the solution is a bit different. It requires a
specific QA team whose task it is to provide all the work developers
might not have the time for. In an ideal world, every part of the
codebase should have a QA manager that does three things. First of
all, he/she is a regular user of the code, and preferably continually
stress tests the part. He/she keeps an open communication with the
developer(s) of that part, which implies a healthy relationship.
Second of all, the QA manager will be the one that is involved with
the whole process of bug reporting and resolution. He checks every bug
that enters his realm, tries to reproduce it, and works with the
reporter to make it easily reproducible for the developer(s). After a
fix is made, the QA manager tries to verify the fix, together with the
user, which then cleanly resolves the issue. This approach has several
advantages: the developers are released of a lot of work regarding bug
reports, the users have a larger chance for a response and a fewer
number of bug reports will enter the realm of obscurity. The final
advantage of a QA team would be that it would create a lot of openings
for non-coders to participate. Many times one hears remarks on the
mailing lists from 'lurkers' who just don't know what to do. This
approach generates a lot of valuable 'jobs'.
A third thing is needed to improve OSS QA. This is what I would call
the 'openly closed beta build'. This would mean that a select amount
of people would be invited to a special forum or mailing list, would
get a special build of the software to test, and more importantly,
would get the opportunity to have a good influence on the project. By
creating small test groups of enthousiasts with many different
configurations, you create a perfect environment to do intense
testing. The developers would participate directly with the users, but
because there are a few of these users, that communication is
possible. The selection would also find enthousiast testers, who will
probably find a lot more bugs that you would with regular betas. Even
if that's not the case, the general quality of the beta bug reports
would be better than in another fashion. When you finally do release a
beta build in the open (which would attract a larger audience), you
will have ironed out the most obvious bugs, and most certainly also a
lot more obscure ones.
Getting to a conclusion
All these solutions may seem a bit strange to OSS developers. It
requires dropping the whole idea of 'whom codes, controlls the
project'. This attitude will have to change to get this idea working.
With a young open source project (like the Haiku project) this
mentality change is probably a reacheble one. Without many rigid
structures commonly found in older OSS projects, and an open area for
discussion, we might reach a world where Quality Assurance is an
integral part of the equation. Furthermore, even the older OSS
projects seem to be walking into a different direction (albeit a bit
slower). The KDE Quality team seems to be a vast new force to improve
the general Quality of the code. GNOME has the bug-squat. These teams
both try to get to a world which I'm promoting as well. And that's the
next challenge of getting Open Source Software to the desktop:
promoting a higher quality standard.
Other related posts:
