[accessibleimage] Response 2: Molyneaux's question rephrased

Here is a chapter from the very interesting book,
Another Eyesight:
Multi-Sensory Design in Context, which also substantiates the idea of the
primary importance of touch. It is from  CHAPTER 2.26

"Visual Ears, Aural Eyes: A Cross Sensory Approach to Design Education"
by Eric Somers

Educating the Visual and Aural Designer

Educators in the visual and aural arts often misunderstand the education
required for applied designers in contrast to that for fine artists. While
fine artists often base their work on self expression, playful novelty, or
conscious copying from an established tradition, the applied artist is
usually in the position of designing a structure which communicates
information to meet the needs of an external client. Thus training for the
designer, whether a designer of sound or image, must include learning to
think of visual and aural structures as solutions to design problems.
Recently I have come to realize that an effective way to teach this kind of
thinking is to ask students of sound design to analyze works of visual art
then translate those structures and symbols into aural equivalents and,
conversely, to ask visual design students to analyze sound compositions and
use those structures as organizing principles for visual designs. Mladen
Milicevic [5] has called this process "semiotic transformation."

The Function of the Designer

There is a great tendency to confuse the role of the designer with that of
either the fine artist or the stylist. The term "designer" is often applied
to the stylist. Most properly thought of, the information designer is a
problem solver or information "architect" who takes ideas or information
developed by a client and puts it into a form which will communicate
effectively. A graphic designer takes text and images which communicate
information about a business, for example, and creates a publication which
presents the information clearly and interestingly to the reader. The
stylist, though often confused with the designer, usually takes a design
already produced by an original designer and re-shapes or decorates it to
make it more novel or individualistic. Thus a creator of "designer" jeans
does not really design a piece of clothing which functions more effectively
than the original Levi Strauss product of a century ago, but creates a more
novel variation of the original design. Such a "designer" is most properly
called a "stylist." The fine artist may develop his or her work in a variety
of ways. Some artists consciously copy the tradition of a certain genre of
art. Others treat art as a form of free self-expression and create works
meaningful to themselves, but which sometimes are not meaningful to others
without explanation. Others may try to challenge the viewers' or listeners'
expectations or perceptions with art forms that are perplexing or shocking.

Visual and Aural Imagination

Clearly the designer of a visual or aural structure must understand the
information to be communicated then develop a concept for an effective
presentation of that information. Ideally this should he done by
understanding the nature of perception in the viewer/listener and developing
an idea about how the information to he communicated could he designed for
successful perception.

Often art education and design practice fail in meeting this ideal. Too
often design is copying of a look, a sound, indeed a "style," and applying
it to a problem which may or may not he suited to the solution. One only has
to look at the many copy-cat confusing web sites to see this phenomenon, or
listen to much popular music. Yet real design progress requires original
conception, not ornamentation or augmentation.

The Integration of the Senses

There is an inherent integration of the human senses as noted by Sergei
Eisenstein [6]. Erich M. von Hornhostel [20], Oliver Sacks [18] and others.
The experience of one sense can, at a minimum, remind one of a sensation of
another sense. For example, when most people hear someone talking on the
radio, they develop a mental picture of the appearance of that person.
William Gaver [7] has noted that in everyday perception of sound in the
environment people tend to visualize a sound by associating it with the
source they believe produced it. In the most extreme form of sense
integration, synesthesia, a person exposed to stimulation of one sense
actually experiences the physical sensations of another. Richard Cytowic [4]
has documented cases in which people actually see colours when hearing music
and feel shapes when tasting food. But true synesthesia is a neurological
condition. My pedagogy is an attempt to make use of the inherent "everyday"
integration of the senses (in Gaver's sense) in order to allow sense
experience of one form to promote creative thinking in another.

Thinking Visually and Aurally

The idea of an artist studying inure than one art form in order to derive
first principles is not new. The painter Wassily Kandinsky wrote "One art
must learn how another uses its method, so that its own means may than he
used according to the same fundamental principles, but in its own medium.-
He added that "The natural result of this work is a comparison of the
elements of one art with those of another. Music is found to be the best
teacher." [10] Though perception and creative thinking are certainly
related, there is a vast difference between the value placed on visual
experience and on aural perception.

Scholars studying the anthropology of communication, especially Edmund
Carpenter [3], Marshall McLuhan [14], Walter Ong [16] and David Berlo [2]
have noted that literate cultures tend to emphasize sight over the other
senses. Western societies, especially, have tended to privilege the sense of
sight since the widespread adoption of literacy made possible by the
printing press. In such cultures people equate seeing with thinking and use
visual terms as a kind of code to describe understanding of thought: "I see
your point," "Can you make yourself more clear'?" "Your logic is fuzzy," "I
get the picture;' "Seeing is believing. etc. Cultures which do not equate
seeing with thinking would have difficulty decoding such statements in a
meaningful way since they do not share the same code.

Cultures which do not have a strong tradition of literacy develop an aural
culture with its own particular characteristics and code. In these cultures
terms related to thinking often get expressed in aural terms. For example,
"I hear you" is often used to mean "I understand you:' Though people from
most literate cultures would rather go deaf than blind, people from oral
cultures feel deafness to be a greater hardship than blindness. Non-literate
cultures tend to believe something heard over something seen. Even Plato
expressed skepticism and distrust of the then-new medium of writing,
believing it to be inferior to word-of-mouth communication. (17] Speech in
non-literate cultures tends to make use of several aural and visual elements
of expression: the meaning of the words chosen ("diction"), the vocal
inflections of the person saying the words, and the body movements, facial
positions and gestures of the speaker. Speakers in non-literate cultures,
including children in all cultures who have not yet learned to read, tend to
use many inflections and gestures. But as people become educated in literate
cultures they are often taught to "modulate" their vocal inflections, stand
still as they talk, and not use gestures. Thus speech becomes reduced to the
single element which can be coded by writing or printing: the meaning of the
words themselves. [3, 14, 16]

David Berlo [2] and Tony Schwartz [19] remind us that meanings do not reside
in words, images or sounds, but in the cultural experience of each
reader-viewer-listener. Thus perceptions bring to mind meanings which may
not be the same for each recipient of the same act of communication. The
complexity of meaning is apt to be greater than the symbol used for
communication. Schwartz writes "A listener or viewer brings far more
information to the communication event than a communicator can put into his
program, commercial or message." [19] Maribeth Back [1] has taken a similar
approach to sound design, letting a well chosen sound evoke a deeper
cultural meaning. Miroslav Malik, [12] a Czech-Canadian designer of
multimedia exhibits for worlds fairs and other multicultural festivals,
often used a "mosaic" design approach in which many different images are
presented at once drawn from the cultural backgrounds of different cultures.
Viewers from each culture are then able to decode at least part of the
message. The ability for a sound, word or image to evoke a complex thought
allows for very powerful communication with a minimum of means or a narrow
channel. But such associations can also lead to "stereotyping" in which a
symbol always brings to mind a single instance of what could otherwise be a
multidimensional thought. These symbols become icons that are not analyzed
for the information they contain but which simply redirect the perceiver to
his/her own stored meaning. Indeed, some people try to find symbolic
associations in certain kinds of messages where none are intended. A common
example is the preoccupation naive art viewers often have trying to figure
out what a non-representational work of art is "supposed to be."

Context and Imagination

Designer Steve McCallion relates an assignment a professor of his gave in
architecture school: "to create an innovative office tower. He insisted that
we not reference outside resources - no books, no photographs... The result?
A depressingly familiar display of foam core and hot glue." 113] Given a
lack of context, designers often resort to the stereotypical thinking
mentioned above. They tend to recall a stored impression and make it an
"original" design by changing small details. They become stylists. By
greatly changing the context of a design problem it is much more likely that
original designs, not "styles," will result. Thus if one asks a class simply
to design a "car" most will create a stylistic variation of the typical car.
If however, one stipulates that the car has to he able to cross water and
also drive up sand dunes, then the discontinuity of the new context will
more likely produce innovative thinking. The architect John M. Johansen has
produced some highly original experimental designs-the kind McCallion may
have been seeking from his students by setting for himself some imaginative
recontextualizing problems: a structure which hangs between twin skyscraper
towers and is supported by them; a theatre which uses magnetic levitation to
move people and objects; buildings which "adapt" themselves to changing
environmental conditions similar to adaptation in biological forms, etc. [9]

Discontinuity as an Agent of Contextual Change

Eric Hoffer [8], Peter Drucker [5] and others have discussed how
discontinuity leads to innovation. Immigrants to the U.S. have a better
than-average record of success as inventors, business entrepreneurs, and
artists. The change of social context leads these immigrants to creative
thinking more easily than someone brought up within the culture. Dealing
with the discontinuity of change seems to produce creative thinking.
Discontinuities can he caused by various forces. Many young people in the
1960s became disillusioned with certain social structures and political
policies in the U.S. and sought innovative changes. Writers and painters
often travel to consciously experience discontinuities which might awaken
them to new ideas and images. In my teaching of visual and aural design I
have produced discontinuity as a stimulant to creative thinking by asking
students to "re-frame" their visual perceptions in aural terms and to
re-frame their aural perceptions in visual terms. This process of semiotic
transformation involves the conversion of the symbols and structure of one
sensory medium to those of another. It is an attempt to re-integrate the
senses so that auditory experience will play a role in visualization and
visual experience will play a role in aural thinking.

Semiotic Transformation as a Stimulant to Design Thinking

There is a tradition of using art of one form as "inspiration" for another.
Some teachers of fine art play music in the classroom and ask students to
"paint what they feel." A similar approach is used by some writing teachers.
In music there is the tradition of "program music" which is intended to
bring to mind a specific experience. Thus in Beethoven's Sixth Symphony we
hear passages intended to remind us of a storm, of villagers playing, etc.
Lest we miss the intent Beethoven thoughtfully annotated the score. The
pedagogy being presented here is not related to any of those strategies. Its
purpose is not to evoke shared cultural memory of certain images or
experiences, nor is it intended to focus the student on his or her own
"feelings." It is analytical and structural in nature. The student is not
asked to "interpret" a sound or image, but to understand the nature of its
elements and how they are related, then create an artwork in another medium
which uses elements perceived to be of a similar nature arranged in a
similar structure. What sounds are similar to certain pictorial elements,
and what pictorial representations would be good "translations" of sound
elements, are decisions the student designer must make. The result then is
not self expression (the domain of the fine artist) but analytical problem
solving (the domain of the true designer).

Sound Composition Based on Sonification of Visual Experience

In teaching a course in sound composition and musique concrete a few years
ago I faced a problem similar to that of Steve McCallion's professor (see
above). When I asked students to make sound compositions from "found sounds"
recorded by them using a portable DAT (digital-audio tape) recorder, I
tended to either get compositions that were more or less random collections
of sounds or else imitations of non-concrete music. Without a structural
framework the students floundered. Since most of these students were media
studies majors with no background in music composition, they could not rely
on their understanding of the design of music to design these sound
compositions. But each student had taken a foundations course in two
dimensional visual design. As an experiment I took a abstract painting from
the wall in my office and asked the sound design students each to create a
sound composition equivalent to the visual work of art. I explained that the
students were to study the visual art. break it down into separate visual
elements, then find sounds equivalent to each element. Finally, they were to
study the arrangement, or composition, of the elements on the canvas and
arrange their sound elements so they seemed equivalent in time and sonic
space to the arrangement in visual space of the visual elements.

This process of semiotic transformation greatly improved the structures of
the student sound compositions. In the following semesters we repeated the
assignment successfully using visual works by a number of artists including
Ann Wilson, Elsworth Kelly, Mark Rothko, Cy Twombley, and Jackson Pollack.
Most recently I expanded the technique by showing two works of visual art
and asking students to create a composition which begins with the form of
the first visual design and gradually develops into a piece having the form
of the second visual composition. In all cases the visual designs presented
are non-representational in order to keep the assignment focused on form
rather than on outside references. The results can be summarized as follows:

1. By limiting the composition to elements re-mapped from the visual model,
students used fewer different sounds and made better use of repetition and
variation.

2. Greater structural unity resulted from their "seeing" in the visual model
how to think of the sound composition as a structure having multiple
overlapping elements, not just a string of linear events.

3. In having to do the semiotic transformations, the students improved their
analytical skills by learning how to break down visual and aural structures
into individual elements.

4. By re-framing their visual experience aurally, the process of composition
became focused in the brain, with the tools of the sound studio more a means
of execution and less the means of composition itself.

Visual Design Based on Visualization of Sound Experience

With the success of the sound composition project, I tested the possibility
of reversing the process described above: to use sound composition as a
pedagogical aid in teaching visual design. When presenting a student with a
visual design problem one often finds that student solutions tend to copy
other designs the student has seen. Indeed, some teachers actually show
worked-out solutions as models. But this tends, in my view, to promote an
emphasis on styling rather than design. I once joked that it would he better
to teach visual design by radio (so students would have nothing visual to
copy), but then realized that using sound might provide a way to train
students to come up with original visualizations. Basing visual art on sound
is not entirely a new concept in teaching art and design. In addition to the
"listen and paint what you feel" exercises noted above, some major artists
have experimented with the visualization of music. Paul Klee, in his
Notebooks [11], describes exercises in basing visual art on musical rhythm
and structure. He includes an analysis of a three part passage by Bach, but
most of his analysis seems to have been made from the musical score rather
than from the sound itself. For my teaching experiment, I designed
electro-acoustic sound clips produced by computer sound synthesis. As with
the visual art using in my sound composition exercises, the sounds were
non-representational. Again, too, the task was analytical: to find elements
in sound that can he represented by equivalent visual structures. Students
were trained to make the transformations a step at a time. Small, short
sounds were produced and students asked to produce visualizations of them.
Then longer sound passages were presented and students were given time to
create drawing which represented both individual sound elements and the
relationship of these elements to each other. In the most advanced stage
four minute compositions were played and students were asked to create color
paper cut-out collages representing the sound piece.

Several results were noted:

1. There was sufficient formal consistency between both the elements and
overall compositions to indicate that the semiotic transformation from the
sound events was genuine. An example can he seen in the three drawings
below, each a different student representation of two plucked string notes,
one louder than the other:

Figure 1: Student sketches by Bill Miller, Frank Corridori and Sophie Phan

2. Though nothing was ever said to the students about how the sounds were
produced, many of the visual images showed a striking understanding from the
aural perception alone. The examples below are based on a sound produced by
modulation. The visual reproductions are clear representations of structures
where one element controls (or modulates) another:

Figure 2: Student sketches by Jay Diesing, Ashley Benner and Frank Corridor'

3. The students developed a remarkable ability to articulate their reasons
for making visual choices based on their sound perceptions, thus supporting
the analytical purpose of this pedagogy.

4. Many of the student visual compositions were interesting and original
visual designs completely pleasing on a purely visual level.

5. Many of the students felt the experience helped them to understand the
design process better.

Conclusion

With the specialization inherent in much of modern society, it is often easy
to forget a past where an educated person made simultaneous contributions to
several fields including visual art, theatre, music, the natural sciences
and philosophy. In such a context the idea of using one's ears to aid visual
thinking, or the reverse, would probably not have seemed unusual. Today,
when someone asks "what do you do?" they sometimes seem unsatisfied by any
answer that can't "peg" you to a certain narrow area of specialty.

Sometimes specialization is necessary. One probably cannot be a noted
concert violinist and an accomplished oil painter simultaneously because of
the amount of time it takes to study and practice the technique of each art.
But with the aid of modern technology, especially the computer, one can
design sonic information structures and visual ones and can perform tasks
that are both technical and artistic. With the visual-aural demands of a
multimedia culture, it is not unreasonable for a young person to want to
understand the underlying principles of both aural and visual design.

The experiments I have done and described above show that semiotic
transformation can be a useful tool of design education. Thus the
multi-sensory pedagogy described in this paper seems to be an appropriate
direction for design educators to consider in the 21st century.

NOTE: This article first appeared, in slightly different form, in the
proceedings of the 1998 International Conference on Auditory Display (ICAD
'98) under the title of "A Pedagogy of Creative Thinking based on
Sonification of Visual Structures and Visualization of Aural Structures."

References

1.  Back M. "Micro-Narratives in Sound Design: Context, Character, and
Caricature in Waveform Manipulation." In: Proceedings of the International
Conference on Auditory Display, 1996.

2.  Berlo D. The Process of Communication: An Introduction to Theory and
Practice. Holt, Rinehart, Winston, New York 1960.

3.  Carpenter E. Oh, What a Blow That Phantom Gave Me!. Holt, Rinehart and
Winston, New York, 1972.

4.  Cytowic R. The Man Who Tasted Shapes. Tarcher/Putnam, New York, 1993.

5.  Drucker P. The Age of Discontinuity: Guidelines to Our Changing Society.
Harper and Row, New York, 1969.

6.  Eisenstein S. Leyda J (cd and trans) Film Sense. Harcourt Brace
Jovanovich, 1942.

7.  Gaver W. "Using and Creating Auditory Icons." In: Kramer G. (ed)
Auditory Display: Sonification, Audification, and Auditory Interfaces.
Addison Wesley, Reading MA, 1994.

8.  Hoffer E. The Ordeal of Change. Harper and Row, New York, 1963.

9.  Johansen J. A Life in the Continuum of Modern Architecture. 13 Area
Edizioni, Milan, 1995.

10.  Kandinsky W. Concerning the Spiritual in Art. George Wittenborn, New
York, 1947,

11. Klee P., Spitler J. (ed), Maheim R. (trans) Paul Klee Notebooks Volume
1: The Thinking Eye. The Overlook Press, Woodstock New York, 1992.

12.  Malik, M. Private conversation with the author (though some references
to his work can he found in various reviews of the Czech pavilion at Expo
'67 held in Montreal).

13.  McCallion S, Muoio A (ed.) from a profile in "They Have a Better Idea
... Do You?" Fast Company, August-September 1997.

14.  McLuhan M. The Gutenberg Galaxy.

University of Toronto Press, Toronto, 1963.

15.  Milicevic M. Private conversation with the author after hearing a
presentation of preliminary findings from experiments with the pedagogy
presented.

16.  Ong W. Orality and Literacy, Methuan, London and New York. 1982.

17.  Plato, Phaedrus. also discussed by Ong (see above).

18.  Sacks O. Seeing Voices: A Journey Into the World of the Deaf.
HarperCollins, New York 1990.

19.  Schwartz T. The Responsive Chord. Anchor Press/ Doubleday, Garden City
(NJ) and New York, 1973.

20.  Von Hornhostel E. "The Unity of the Senses." In Psyche Vol. VII No. 4,
April 1927.

Eric Somers is Professor of Design and Communication at Dutchess Community
College of the State University of New York, where he served as Chair of the
Department of Visual and Performing Arts for 15 years. He began his career
as a fine arts television producer which triggered a life long interest in
the relationship of sound to image. He currently maintains a production
service, The Sandbook Studio, which specializes in visual and aural
documentation of the fine arts and in presenting workshops related to high
end sound and image capture and reproduction. Professor Somers has served as
Chair of the New York section of the Audio Engineering Society (AES),
President of the International Community for Auditory Display (ICAD), and
President of the Museum for Preservation of Illustrative Art.

----- Original Message ----- 
From: "Lisa Yayla" <fnugg@xxxxxxxxx>
To: <accessibleimage@xxxxxxxxxxxxx>; "Art Beyond Sight Theory and Research"
<art_beyond_sight_theory_and_research@xxxxxxxxxx>
Sent: Sunday, September 03, 2006 6:29 AM
Subject: [accessibleimage] Molyneaux's question rephrased


Hi,
Following are some questions I have about touch and sight. Would
appreciate any feedback, thoughts

In the paper "Recovery from Early Blindness" by Richard Gregory he
describes a man, S.B, gaining vision at the age of 51. Shortly after the
operation he draws pictures from what Gregory calls "touch memory" and
is able to understand objects through vision alone and not touching
them, though they are objects he has known from touch when blind (clock
on wall and written letters). This again touch memory. In his pictures
though he does not enter features which he "had not known previously by
touch".

This seems to answer differently than John Locke's answer to Molyneaux

However S.B had difficulty recognizing faces and facial expressions.
This is also the case for Michael May (blind and regained sight) that he
has difficulty with understanding faces and facial expressions. I was
thinking that perhaps the explanation to this is that facial
expressions, body language are something done "on the fly", there is
movement involved and this is something one can not experience with
touch. Transition of expression involves movement.

In lieu of this would it not seem fair to rephrase Molyneaux problem to:

"If a blind person gains sight will that person, soon after gaining
sight, understand an object from sight alone not having experienced it
by touch from before?"

and/or Could this be compared to an archaeologist who uncovers an object
and doesn't know what it is?

The idea being that touch is very important for sight

Is perhaps sight  the "servant" of touch? That sight discovers things
for us to touch? The original object of development is to touch and
verify?  Is sight the ability to "touch" at a distance? That sight
develops from touch? Sight developed to be able to touch farther away
then the lengths of our arms?

Thanks,
Lisa


http://www.richardgregory.org/

Other related posts: