[accessibleimage] Re: Molyneaux's question rephrased

Dear Lisa,

> I was wondering to understand a circle, square etc through
> sound does it also need experience as in sight?

No, no prior *visual* experience is needed, but one probably
needs some prior sensory exposure as a "reference", and touch
is an excellent reference to have a good grasp of what makes
up a circle or square. Suppose we have a cardboard circle and
square for tactile reference, you can easily imagine how they
would sound under The vOICe mapping (assuming bright objects
on a dark background for ease of discussion). The vOICe scans
any view from left to right while associating height with
pitch, and brightness with loudness. Now if you gently slide
the side of your hand from left to right over a hand-sized
cardboard circle (or the top of your coffee mug), you will
feel on the left side after first contact one edge going up
and at the same time another edge going down starting from
the point of first contact. This continues until you are at
the top/bottom of the circle, after which the top curve bends
downward and the bottom curve bends upward such that they meet
again at the far right of the circle. Quite lengthy to put in
words, but done in just a second or so in the actual exercise.
So you have a curve going up and down for the top half of the
circle, and simultaneously a curve going down and up for the
bottom half of the circle. Similarly, The vOICe will for an
open circle sound two tones, one going up and down in pitch for
the top half of the circle, and a simultaneously sounding tone
going down and up in pitch for the bottom half of the circle.
With a filled circle it is the same, but you have all the
pitches in between as well, giving a noise-like "swoosh" sound
that similarly gradually broadens its pitch range during the
left-to-right scan until halfway the circle, after which the
pitch range narrows again until the sound vanishes at the far
right. All of this is open to rational understanding as also
indicated by Will's post. With the filled square, assuming it
is oriented upright, one has a left edge that comes in suddenly
in its entirety, after which top and bottom edge stay at constant
height until the right edge suddenly stops the left-to-right
scan experience again. Same thing with The vOICe: you get a
noise burst with constant pitch range that starts and stops
suddenly as left and right edge are "touched" during the left
to right scan. All of this happens in a second or so, allowing
short term memory to quickly build some mental "picture" of
the overall view content without being too sequential. This
does not take away that a *lot* of practice and experience is
needed in learning to deal with more complex and realistic
views, with trees and buildings and such - objects that are
generally also too large or distant to explore by touch alone.

> If so do you think there is the similar progress as in  learning visual
> information that is one starts off with picture books, moving on up to
> understanding maps, graphs etc?

I'm not sure. In a way, I think sighted children first learn
the very complex vision as needed to deal with their cluttered
visual environment even before they get to picture books with
simplified images. Same thing with learning a language: babies
do not get grammar lessons first (although parents will speak
simplified sentences to support learning by example). With
adults it can be somewhat the other way around, taking grammar
lessons plus memorizing vocabulary lists in mastering a new
language before getting fully immersed in the foreign language.
So with learning-to-see it could also depend a lot on whether
you are a young child or an adult to decide what works best.
I believe some immersion should come soon, because without that
it is hard to get a grasp of the effects of visual perspective
as you move around. A higher level of interaction is needed
for that than can be offered through (static) sample images.
The interaction is also needed to calibrate your "vision" with
your body movements, e.g., in grasping that coffee mug from the
table based on its visual appearance. Blind users of The vOICe
who switch cameras also need to "recalibrate" because their new
camera may show items as slightly smaller or larger than the
other camera, causing them to wrongly judge distances and sizes
for some days (apart from possible nausea as their brain tries
to deal with this temporary mismatch between what their brain
expects from seeing-with-sound and what their touch tells them
about the physical reality).

BTW, good points Will. One of the exciting challenges now is
to find out to what extent the *rational* understanding that
indeed lets you use attributes for identification can become
"automatic" / "subconscious" / "intuitive" through extensive
experience, and beyond that, to what extent this might also
give rise to the qualia of vision (e.g., to be judged by late
blind people who have their memories of vision for reference).

Best wishes,

Peter Meijer


Seeing with Sound - The vOICe
http://www.seeingwithsound.com


Lisa Yayla wrote: 
Dear Peter,

I liked your rephrasing of Molyneux's problem.

I was wondering to understand a circle, square etc through sound does it
also need experience as in sight? Visual information graphics need
experience to understand. Graphs and diagrams for a lot of people take
time to understand.
If so do you think there is the similar progress as in  learning visual
information that is one starts off with picture books, moving on up to
understanding maps, graphs etc?

Best,
Lisa


Other related posts: