[haiku-development] Rant about quality and testing
- From: Stephan Assmus <superstippi@xxxxxx>
- To: haiku-development@xxxxxxxxxxxxx
- Date: Sun, 27 Feb 2011 14:23:04 +0100
Hi all,
I would like to get something off my chest which is on my mind lately
about the quality of some changes to Haiku and about the testing (or
lack thereof) that seems to be performed.
It's dangerous to write such an email, for two reasons: Firstly I know
that we are all in it for the fun, and I don't want to spoil the fun for
anyone. But I figure I will have to mention some example changes in this
mail, since without them you probably don't know what I am talking about
and why I find it justified to rant about it. However I don't want the
people responsible for the examples to feel bad. Secondly, I have been
responsible for similar changes myself in the past, and I will probably
make such mistakes again. Therefor, take what follows with a grain of
salt and only as a reminder of what might be improved. I believe it can
be beneficial to read such a rant once in a while, it may raise the
quality of changes, at least for a while. :-D
We can basically work on two different types of things in Haiku. It's
either something new or broken anyway, no one is seriously using it and
depending on it. The second type are components that are crucial to the
work flows each of us use to get stuff done right in Haiku.
Over the last couple of days, I've just fixed or looked at too many
problems in Haiku which are simply caused by lack of testing before
committing them into the source tree, and which affect components that
are crucial. I have very little time for Haiku, and when I do find some
time on the weekend, I don't want to spend it running into one problem
after the other, in things that worked perfectly fine before.
Today it went something like this: I wanted to look at a problem in
AboutSystem with the building of the translated Haiku license string.
Basically, the translated name for the license is inserted broken. Now,
this means someone didn't test their changes. Either it was a
contributor of a patch. Then the lack of testing began there, and
continued at the committer of the patch who didn't test it properly
either and just trusted the contributor. When I wanted to log the bug
from WebPositive in Haiku, I could not log into Trac. "Protocol not
supported". HTTPS is not supported in WebPositive anymore, I suspect
because of the very recent update of libcurl. Again, something was
replaced that worked perfectly fine before, without the necessary
testing. WebPositive is a tool that I suspect many are using when they
use Haiku. So extra care has to be taken when changing it or components
that it depends on. Especially so when you make a commit for no obvious
benefit. Then I wanted to look at the AboutSystem problem myself, but
now Pe does not open, because there is a problem with a missing libpcre.
Again, some package was changed without proper testing prior to the
commit. That just did for today and now I am writing this mail.
Yesterday I fixed a problem in libtracker's StatusWindow. Someone
committed a fix for a Coverty issue, but actually changed more at the
time and removed some locking and unlocking. Again, without doing any
testing. The bug caused by this went unnoticed for some time, because
most people don't try to pause and unpause their Tracker jobs. Consider
the time this has caused the project: Alex Wilson was irritated and did
some debugging, since he thought he caused the problem. Phillippe, I
think it was, figured out the responsible commit by looking it up, I
looked at the problem yet again and committed the fix (mostly just
restoring what was removed in error).
Another example is the mail_daemon. Clemens developed the changes in a
branch, which was the absolutely right thing to do when working on a
component that many people may be using and relying on. Then he asked
whether he should finally merge his changes back to trunk. I was in
favor of this, since I assumed he developed the new mail_daemon to a
point where it worked better than before, and he couldn't see any big
problems on his side and with his workflow, so he needed the additional
exposure. IMHO, the changes were not ready to be merged, since there
were numerous problems still present that should have been addressed in
the branch, since it was easy to see them during testing. I think I
remember one comment from Clemens mentioning that he only tested in a VM
so far and didn't really use the new mail_daemon himself (after the merge).
Another exmaple are the networking changes that frequently broke DHCP.
Some of them could not be revealed by testing, since they only showed up
on certain hardware, but the most recent breakage was caused by an
endianess issues that would have shown up with testing (I believe).
So, this is a rant mostly meant as a reminder that the quality of your
work does affect other people. Those people may have very little time
for Haiku only, and they don't want to spend it on problems that to them
look like problems that could have been avoided with minimal testing. If
you have the time to work on something, do take the extra time to do the
proper testing. Even if you get only half as many changes committed that
way, it just causes less frustration for everyone and sucks up less time
in the aftermath.
Best regards,
-Stephan
Other related posts:
