[blindza] Fw: Device lets blind people 'see' with sounds
- From: "Jacob Kruger" <jacobk@xxxxxxxxxxxx>
- To: "BlindZA" <blindza@xxxxxxxxxxxxx>
- Date: Thu, 11 Dec 2008 19:43:02 +0200
From Peter Meijer from SeeingWithSound.
Long, but very interesting article.
Jacob Kruger
Blind Biker
Skype: BlindZA
'...Fate had broken his body, but not his spirit...'
----- Original Message -----
Hi All,
Appended is an excerpt from the lecture by Colin Blakemore
at the Sense Annual Lecture on Thursday 6 November 2008.
Note that at the end of his lecture he refers to use of
The vOICe and the research in the group of Petra Stoerig
in Germany, in saying
I’m actually involved with the research group in Germany
at the moment, trying to think about how we can use this
sort of modifiability to help to restore some kind of
visual experience system for blind people, delivered
through other senses, particularly through hearing. So
we’re working with a system for converting visual images
into sounds, and to give away some of the unpublished
results, we’re already showing that with training, people
can learn to induce activity in the visual parts of their
brains as a result of listening to sounds that are
representing pictures that are being scanned. So there’s
a hope that people might be able to learn to recognise
images through the sound system by making use of the
visual parts of their brain.
Best wishes,
Peter Meijer
Seeing with Sound - The vOICe
http://www.seeingwithsound.com/winvoice.htm
Colin Blakemore, November 6, 2008:
[...]
So again, just to remind you of how things are arranged. The whole of the
back
of our heads, maybe one sixth of our entire nervous system is devoted to
seeing,
and I calculated that seeing consumes about 4% of all the food energy we
eat.
The brain is a very hungry organ in terms of metabolism, and vision is a
very
big part of the brain. So probably about 4% of what you eat is used to see.
You can ask the question, what would happen in a blind person, where the
blindness is not caused by brain damage, but by a defect in the eye, one of
the
many ocular causes of blindness?
What happens to this vast area of the brain which is now deprived of its
normal
input completely? Does it just sit there for the rest of life, doing
nothing?
I’m going to describe briefly, some experiments on blind volunteers who
helped
me and my research group by being scanned, to try to find out whether the
visual
parts of their brain was still doing things even though they were blind. We
found a very interesting group of people who had a condition called
synesthesia,
which some of you might have heard of. It’s a very fashionable topic
amongst
psychologists and neuroscientists these days. It is a condition in which
people
have extrasensory experiences. Now I don’t mean extrasensory like an
X-files
sort of experience. These are people who when they look at a particular
thing
or hear a particular thing, see it or hear it as we do, but in addition,
they
get other sensations. So they might see flashes of colour when they hear a
Beethoven symphony, or when they read the newspaper, they might see the
black
and white letters as if they’re in colour. Those are quite common forms of
synesthesia.
Some people claim that they taste their friends, so when a particular friend
appears, they have a particular taste in their mouths, or smell, and it’s
nothing to do with the person. So it’s a very bizarre condition. It’s
partly
inherited and runs in families. Anyway, we studied a number of blind people
who
have been blind for many years, most of them for more than two decades, who
had
been synesthetic and had had extra visual sensations before they became
blind.
They had all become blind after the age of about two or three, and their
synesthesia had persisted even after they were blind. So they still had
visual
sensations in association with other things. So this chap, for instance,
J.F.,
a very talented computer programmer whom we studied, had been blind for 30
years, and he saw colours associated with words about time, days of the
week,
and months of the year.
Also, he had colour associations with musical instruments or military
arrangements of regiments, so each of them had their own colour. Bizarre
colour
experiences. So here was his description of the way in which he saw the
days of
the week. So the present week, he saw on the right-hand side, Monday,
Tuesday,
Wednesday, Thursday, Friday, in these different colours, blobs of colour,
and
last week was on the other side, arranged this way.
So he saw time arranged in front of him as patches of colour. This would
have
been dismissed as a very bizarre form of hysteria by most people 40, 50
years
ago, but the fact is, that if you look in his brain whilst he’s getting
these
experiences, you can quite clearly see the activity which is causing these
sensations. So here we are, looking in the top four panels here, at the
activity in this man’s brain, first of all, whilst he’s just imagining
coloured
things, and what happens is this causes activity in a particular region of
his
brain known to be associated with colour sensation in normal people but he’s
blind, but he’s able to turn on his own visual system from inside when he’s
imagining things. And when he’s listening to words that produce sensations
of
colour, lo and behold, the same areas become active, you can see very strong
activity in areas that are known to be concerned with analysing colour in
normal-sighted people.
Compare that with a very closely matched non-synesthetic, blind person with
no
activity at all in the visual regions. So the visual cortex can remain
functional. It can be stimulated from within to give blind people visual
experiences. It’s something that really surprised me, which I’m sure many
of
you will confirm, is actually well-developed. People have become blind
later in
life, after a few years or more, of vision, are still capable of having very
vivid visual experiences, in their mind’s eye so to speak. So we might
conclude
that we see because there is a particular combination of activity going on
in
our brains as we look at a scene, and the sum of all that determines what we
see.
Let’s consider then what happens in this area I’ve already described, MT, in
the
human brain that responds to movement. When it’s damaged, we can’t see
movement
anymore. It’s clearly responsible for our perceptual sensation of motion.
In
this beautiful picture produced in Queen Square in the Functional Imaging
Laboratories in London, you can see the reconstructed view of a living
person’s
brain produced from Magnetic Resonance Imaging, and superimposed on top of
it,
this red blob is the region of this person’s brain in the right hemisphere
of
the brain, the left side of the brain, which was most active when this
person
saw moving patterns rather than static patterns. That is the region, MT,
which
is just on the side. But it’s reasonable to ask, I think, what exactly does
this area respond to, what does it compute, if you think in terms of the
analogy
of a computer, because movement of the retinal image inside our eye can be
produced in many different ways?
If we just hold our eyes stationary and look at the visual scene and
something
moves across it, then obviously the image moves across the back of our eye.
But
we could produce exactly the same movement of the retinal image by moving
our
eye across a static object. So how does the brain disentangle movements
produced by the outside world and movements produced by your own actions?
That
was a question that one of my graduate students, Loredana Santoro asked.
She
asked sighted people to lie in the scanner and to look at this very simple
display, just a cross and stripes. And she asked them to look at the cross
all
the time. So imagine in this situation though, looking at the cross, the
stripes are moving backwards and forwards, the image is moving on their
retina,
they see the stripes moving, so not surprisingly, when she looked at the
activity in their brain, there was enormous activity in the motion-sensitive
area. That’s fine. The retinal image is moving, they see the outside
stripes
moving, and the brain is active. Contrast that with this situation.
Remember
they’re told to watch the little cross.
Now in this case, the cross moves and the stripes are stationary on the
screen
in front of them. But their eye is moving following, tracking backwards and
forwards, following the cross. If you think about it, the image of the
stripes
on their eye is moving on their retina in exactly the same way as that it
was in
the condition, but in this case, they are causing the image to move by their
own
eye movements, whereas in the other case, the world was moving. Now in this
case, you did not see the stripes moving; your eye was tracking, but you don’t
see the stripes moving of course, because they’re stationary out there. So
what
happens in the brain? Remarkably, there is little or no activity. So the
retina is being stimulated in just the same way, but the brain is somehow
able
to tell the difference between real movement and self-induced movement. And
here’s the real pinching experiment. Again, if you look at the little
cross,
track it with your eyes, it moves and so do the stripes. Now just think
about
what’s happening on your retina. Your eye is moving with the stripes, so
the
stripes are actually stationary on your retina all the time.
They’re always in the same place. But you see them moving as you track
them,
because you know they’re moving, your eyes following them. So then, nothing
is
moving on the retina, but you are perceiving the movement. And what does
the
area of the brain do? It responds. So this part of the brain is able to
perform a little algebraic computation, taking into account your eye
movements,
taking into account the retinal image movements, and what it comes up with
is
activity which relates to what you’re actually seeing and perceiving; not
relating to what’s on your eye at all. The task of the brain is not to tell
you
about your retinal image; it is to tell you about the outside world. And it
has
to do clever computations and calculations like this to do it. So we see
what
the visual areas of our brains compute. They’re acting like computers,
analysing the image. Then how do they deal with something like this?
This is a well-known painting by Salvador Dali, and it has two titles. It’s
called the Bust of Voltaire, because there is a funny image in the middle
here,
this is the two eyes and the chin, standing on the stand, it looks rather
like a
real Bust of Voltaire which is in the Louvre actually. There you can see it
even better if I make it smaller. But it’s also called The Slave Market,
because if you look at it closely, there are these little characters here
with
their funny smocks and dresses on. You can see it in two different ways.
It’s
very cleverly painted. It’s an ambiguous picture that you see in two
different
ways. You never see it simultaneously as both; your brain flips between one
and
the other, again proving that what you are seeing is not your retinal image
which is the same all the time.
You’re seeing what your brain interprets from the retinal image. In that
case,
what happens if you look at something like this? Do you remember the two
areas
that I described to faces and to objects? What are they doing when you’re
looking at this very famous vase/face illusion, which flips, I think you’ll
agree, from being either two silhouette faces or a white vase? Every few
seconds, it changes from faces to vase and nothing is happening on your
retina.
Your interpretation is changing. What’s happening in your brain? So Tim
Andrews gave people lying in the scanner two buttons to press. They had to
press one when they were seeing it as faces and the other one when they were
seeing the vases, and the computer tracked what was happening in their brain
as
they were doing this and what we found was that the activity shifts
backwards
and forwards between these two regions, directly in line with their
conscious
perception. The image on their retina is not changing at all, but the
activity
shifts as they change their perceptual state. A very direct relationship
between what’s happening in the brain and what they see, and it’s based on
the
expectation of the analysis of what might be there. The brain is looking
for
meaning in the retinal image. And when the meaning is ambiguous, it sees
both
things, flipping between them, never simultaneously.
Finally then, let’s go back to the question of what happens in the visual
parts
of the brain of a blind person. This is work that was done by an American
grad
student of mine, Manu Goyal. When he came to visit me as a Rhodes scholar,
we
saw some scientific publications on blind people which showed intriguingly
that
when blind people touch a Braille or any richly textured surface with their
fingertips, blood flow to the back of the brain increased much more than in
normal people. That’s the visual parts of the brain. So this led the
researchers to speculate that perhaps the visual cortex is somehow being
taken
over to help blind people to interpret other senses. So we decided to try
and
test that very specifically. First of all, we did this experiment, the
results
of which I think are remarkable. It was a surprise to me. We simply asked
people to lie in the scanner and imagine that they were looking at a static
visual scene, actually patterned curtains closed over a window. Then we
asked
them to imagine that they were opening and closing, and then that were
opened
and there was a face looking at them through the window. And to our
amazement,
all these volunteers who had been blind for at least 20 years, had
absolutely no
problem in doing this, in conjuring up images in their mind’s eye, and what’s
more, when we looked in their brain, we found that the brain activity was
exactly what we would expect from the experiences that they themselves were
generating. So here, we are looking at activity in the brain when they were
imagining moving patterns, and the activities here, in area MT, that visual
movement area. It’s just the same in sighted people and in people who
became
blind later in life; not in congenitally blind individuals, I have to say
that.
They find it impossible, not surprisingly, to know what you mean when you
talk
about imagining a visual scene. And equally, if you asked them to imagine
faces, here’s what normal-sighted people do, they turn on that
face-selective
area and so do late-onset blind people. So blind people are able to control
their own visual cortex’s to give them internally generated, mind’s eye
visual
experiences. So we then asked them to do the crucial experiment. We put
the
people in the scanner, blind, or blindfolded in the case of sighted people,
and
we put into their hands, without them knowing what we were going to put into
their hands, these two objects, a doll’s head which felt like a face
instantly
to anybody who felt it including the sighted people, or a mutilated doll’s
head
that didn’t feel like a face but had all the same features on the surface.
And we compared the responses of the brain to these two things, either
static or
moving, so we could see which parts of the brain were more strongly
stimulated
by a moving stimulus on the skin and which was more stimulated by a face on
the
skin, and the results are very clear. Moving stimulus on the skin strongly
activates the visual movement parts of the brain in people with late-onset
blindness, but not in sighted people and not in the congenitally blind. It’s
as
if the touch system is borrowing the parts of the visual system to help it
with
the task of analysing the movement on the skin. And equally the face, if
you
contrast the activity produced by the doll’s head with that produced by the
non-doll’s head, then there is strong activity in that one tiny area of the
face
area in blind people, but not in sighted people and not in the congenitally
blind. So I think that’s pretty good evidence that the unused machinery of
the
visual system, once it has been programmed by vision early in life, it
persists
forever through life and can be used by the other senses and taken over by
the
other senses. Remember the experiments on the cats reared in striped worlds
which showed that vision early in life is crucial for setting up the system.
So just to conclude, I won’t say that we understand exactly how vision
works; we
certainly don’t, but we’re getting a much richer picture of the way in which
the
computations that are going on are very sophisticated computational
calculations
that are going on are analysing different bits of the visual scene, and
somehow
we put it together to see the whole thing. And it’s remarkably modifiable,
plastic. I’m actually involved with the research group in Germany at the
moment, trying to think about how we can use this sort of modifiability to
help
to restore some kind of visual experience system for blind people, delivered
through other senses, particularly through hearing. So we’re working with a
system for converting visual images into sounds, and to give away some of
the
unpublished results, we’re already showing that with training, people can
learn
to induce activity in the visual parts of their brains as a result of
listening
to sounds that are representing pictures that are being scanned. So there’s
a
hope that people might be able to learn to recognise images through the
sound
system by making use of the visual parts of their brain. Thank you.
(Transcript prepared by Stephen McCarthy of www.sense.org.uk)
----------
To send a message to the list, send any message to blindza@xxxxxxxxxxxxx
----------
To unsubscribe from this list, send a message to blindza-request@xxxxxxxxxxxxx
with 'unsubscribe' in the subject line
---
The 'homepage' for this list is at http://www.blindza.co.za
Other related posts:
- » [blindza] Fw: Device lets blind people 'see' with sounds - Jacob Kruger
