[gmpi] Re: [OT] listening tests (Was: Where are we?: 7.1.2/Untanglingissues)

on 6/18/03 10:41 AM, Paul Davis at paul@xxxxxxxxxxxxxxxxxxxxx wrote:

>> structure and in listening tests that has a real impact on the sound (the 64
>> bit implementation is smoother and other weird subjective kind of words). So
> 
> do you use double-blind testing techniques for your listening tests? i
> have almost never seen them used by the audio industry, which instead
> seems to prefer so-called "A/B comparisons". no other testing domains
> that i am familiar with dares to use this technique, since it has been
> shown over and over again to suffer from fatal bias. i would be
> impressed if you do in fact use it, but you'd be the first audio
> software or hardware company i've come across that claimed to do
> so. even food tasting uses at least single-blind these days, and often
> double blind. ditto for cosmetics, smell, color, and fabric texture.

We use single blind for most testing, and double blind for some testing.
Often we do unblinded A/B testing during development. If we see a trend in
the unblinded A/B testing we follow it up with a single or double blinded
test to see if the perceived difference maintains itself. This is *VERY*
instructive! There have been differences that our test subjects would swear
to that they absolutely could not identify blind. I would say that unblinded
A/B tests are only useful to determine what things are worth measuring blind
(blind tests take more time to set up and as a result are more expensive --
if you cannot perceive a difference unblinded it is probably not worth doing
the blinded test because people tend to perceive differences). Anyway, the
Float64 internal stuff passed blind testing. And it is measurable.

Anyway -- unblinded A/B test are not revealing. Use at least single blind if
you want to get a reasonable result. Double blind is quite expensive, and
probably not appropriate for ad-hoc testing in development. You just need to
be careful to not let the subject know what to expect -- set it up, leave
the room, and send the subject in....

Best regards,


B.J. Buchalter

Metric Halo
M/S 601 - Building 8
Castle Point Campus
Castle Point, NY 12511-0601 USA
tel +1 845 831-8600
fax +1 603 250-2451

If you haven't heard ChannelStrip yet, you don't know what you're missing!

Check out SpectraFoo, ChannelStrip and Mobile I/O at http://www.mhlabs.com/

Download a 12 day demo from <http://www.mhlabs.com/demo/>




----------------------------------------------------------------------
Generalized Music Plugin Interface (GMPI) public discussion list
Participation in this list is contingent upon your abiding by the
following rules:  Please stay on topic.  You are responsible for your own
words.  Please respect your fellow subscribers.  Please do not
redistribute anyone else's words without their permission.

Archive: http://www.freelists.org/archives/gmpi
Email gmpi-request@xxxxxxxxxxxxx w/ subject "unsubscribe" to unsubscribe

Other related posts: