Robert Acosta, President
Helping Hands for the Blind
(818) 998-0044
www.helpinghands4theblind.org
From: dan Thompson [mailto:dthompson5@xxxxxxxxx]
Sent: Monday, December 07, 2015 7:18 AM
To: dan Thompson
Subject: A survey of assistive technologies and applications for blind users on
mobile platforms: a review and foundation for research, Dan's tip for December
7 2015
Pondering question
Why do we recite at a play, and play at a recital?
Fact of the day:
The string attached to boxes of animal crackers was originally placed there so
that the container could be hung from the branches of a Christmas tree.
The shotput used by male athletes weighs 16 lbs., the same as the maximum
weight for a bowling ball.
Submitted by Bob Holt - Issaquah, WA
When business was slow in the early days of the Boeing Company, they had their
woodworkers make furniture.
The National Television Systems Committee is used by the U.S., Canada, and
Mexico. Why then is French the most popular secondary audio track on DVD using
the NTSC format? Mainly because approximately nine million Canadians claim
French as their primary language.
Submitted by Robert Lockwood - Quincy, California
Grover Cleveland is the only U.S. President to serve two non-consecutive terms.
He was the 22nd AND the 24th U.S. President.
A survey of assistive technologies and applications for blind users on mobile
platforms: a review and foundation for research
Ádám Csapó1,3 · György Wersényi1 · Hunor Nagy1 · Tony Stockman2
Received: 13 December 2014 / Accepted: 29 May 2015 / Published online: 18 June
2015
© The Author(s) 2015. This article is published with open access at
Springerlink.com
Abstract This paper summarizes recent developments in
audio and tactile feedback based assistive technologies targeting
the blind community. Current technology allows
applications to be efficiently distributed and run on mobile
and handheld devices, even in cases where computational
requirements are significant. As a result, electronic travel
aids, navigational assistance modules, text-to-speech applications,
as well as virtual audio displays which combine
audio with haptic channels are becoming integrated into standard
mobile devices. This trend, combined with the appearance
of increasingly user-friendly interfaces and modes of
interaction has opened a variety of new perspectives for the
rehabilitation and training of users with visual impairments.
The goal of this paper is to provide an overview of these
developments based on recent advances in basic research
and application development. Using this overview as a foundation,
an agenda is outlined for future research in mobile
interaction design with respect to users with special needs,
as well as ultimately in relation to sensor-bridging applications
in general.
Keywords Assistive technologies · Sonification ·
Mobile applications · Blind users
B Ádám Csapó
csapo.adam@xxxxxxxxx
1 Széchenyi István University, Gy ˝or, Hungary
2 Queen Mary University, London, UK
3 Institute for Computer Science and Control, Hungarian
Academy of Sciences, Budapest, Hungary
1 Introduction
A large number of visually impaired people use state-ofthe-art
technology to perform tasks in their everyday lives.
Such technologies consist of electronic devices equipped
with sensors and processors capable of making “intelligent”
decisions. Various feedback devices are then used to communicate
results effectively. One of the most important and
challenging tasks in developing such technologies is to create
a user interface that is appropriate for the sensorimotor capabilities
of blind users, both in terms of providing input and
interpreting output feedback. Today, the use of commercially
available mobile devices shows great promise in addressing
such challenges. Besides being equipped with increasing
computational capacity and sensor capabilities, these devices
generally also provide standardized possibilities for touchbased
input and perceptually rich auditory-tactile output. As
a result, the largest and most widespread mobile platforms
are rapidly evolving into de facto standards for the implementation
of assistive technologies.
The term “assistive technology” in general is used within
several fields where users require some form of assistance.
Although these fields can be diverse in terms of their scope
and goals, user safety is always a key issue. Therefore,
besides augmenting user capabilities, ensuring their safety
and well-being is also of prime importance. Designing navigational
aids for visually impaired people is an exemplary
case where design decisions must in no way detract from
users’ awareness of their surroundings through natural channels.In
this paper, our goal is to provide an overview of theoretical
and practical solutions to the challenges faced by
the visually impaired in various domains of everyday life.
Recent developments in mobile technology are highlighted,
with the goal of setting out an agenda for future research
123
276 J Multimodal User Interfaces (2015) 9:275–286
in CogInfoCom-supported assistive technologies. The paper
is structured as follows. In Sect. 2, <> a summary is provided
of basic auditory and haptic feedback techniques in general,
along with solutions developed in the past decades both
in academia and for the commercial market. In Sect. 3, <> an
overview is given of generic capabilities provided by stateof-the-art
mobile platforms which can be used to support
assistive solutions for the visually impaired. It is shown
that both the theoretical and practical requirements relating
to assistive applications can be addressed in a unified
way through the use of these platforms. A cross-section of
trending mobile applications for the visually impaired on the
Android and iOS platforms is provided in Sect. 4. <> Finally,
an agenda is set out for future research in this area, ranging
from basic exploration of perceptual and cognitive issues,
to the development of improved techniques for prototyping
and evaluation of sensor-bridging applications with visually
impaired users.
2 Overview of auditory and haptic feedback
methods
The research fields dealing with sensory interfaces between
users and devices often acknowledge a certain duality
between iconic and more abstract forms of communication.
In a sense, this distinction is intuitive, but its value in interface
design has also been experimentally justified with respect to
different sensory modalities.
We first consider the case of auditory interfaces, in which
the icon-message distinction is especially strong. Auditory
icons were defined in the context of everyday listening
as “caricatures of everyday sounds” [1–3 <> ]. This was the
first generalization of David Canfield-Smith’s original visual
icon concept [4 <> ] to modalities other than vision, through
a theory that separates ‘everyday listening’ from ‘musical
listening’. Briefly expressed, Gaver’s theory separates
cases where sounds are interpreted with respect to their
perceptual-musical qualities, as opposed to cases where they
are interpreted with respect to a physical context in which the
same sound is generally encountered. As an example of the
latter case, the sound of a door being opened and closed could
for instance be used as an icon for somebody entering or leaving
a virtual environment. Earcons were defined by Blattner,
Sumikawa and Greenberg as “non-verbal audio messages
used in the user–computer interface to provide information
to the user about some computer object, operation, or interaction”
[5 <> ]. Although this definition does not in itself specify
whether the representation is iconic or message-like, the
same paper offers a distinction between ‘representational’
and ‘abstract’ earcons, thus acknowledging that such a duality
exists. Today, the term ‘earcon’ is used exclusively in the
second sense, as a concept that is complementary to the iconic
nature of auditory icons. As a parallel example to the auditory
icon illustration described above, one could imagine a prespecified
but abstract pattern of tones to symbolize the event
of someone entering or leaving a virtual space. Whenever a
data-oriented perspective is preferred, as in transferring data
to audio, the term ‘sonification’ is used, which refers to the
“use of non-speech audio to convey information or perceptualize
data” [6 <> ] (for a more recent definition, the reader is
referred to [7 <> ]).
Since the original formulation of these concepts, several
newer kinds of auditory representations have emerged.
For an overview of novel representations—including representations
with speech-like and/or emotional characteristics
(such as spearcons, spindexes, auditory emoticons and spemoticons),
as well as representations used specifically for
navigation and alerting information (such as musicons and
morphocons)—the reader is referred to the overview provided
in [8 <> ].
In the haptic and tactile domains, a distinction that is somewhat
analogous to the auditory domain exists between iconic
and message-like communications, although not always
in a clearly identifiable sense. Thus, while MacLean and
Enriquez suggest that haptic icons are conceptually closer
to earcons than auditory icons, in that they are message-like
(“our approach shares more philosophically with [earcons],
but we also have a long-term aim of adding the intuitive
benefits of Gaver’s approach…”) [9 <> ], the same authors in a
different publication write that “haptic icons, or hapticons,
[are] brief programmed forces applied to a user through a
haptic interface, with the role of communicating a simple
idea in manner similar to visual or auditory icons” [10 <> ]. Similarly,
Brewster and Brown define tactons and tactile icons
as interchangeable terms, stating that both are “structured,
abstract messages that can be used to communicate messages
non-visually” [11 <> ]. Such a view is perhaps tenable as
long as no experimentally verifiable considerations suggest
that the two terms should be regarded as referring to different
concepts. Interestingly, although the terms ‘haptification’
and ‘tactification’ have been used in analogy to visualization
and sonification, very often these terms arise independently
of any data-oriented approach (i.e., it is the use of haptic
and tactile feedback—as opposed to no feedback—that is
referred to as haptification and tactification).
In both the auditory and haptic/tactile modalities, the
task of creating and deploying useful representations is a
multifaceted challenge which often requires a simultaneous
reliance on psychophysical experiments and trial-and-error
based techniques. While the former source of knowledge
is important in describing the theoretical limits of human
perceptual capabilities—in terms of e.g. just-noticeabledifferences
(cf. e.g. [12 <> –14 <> ]) and other factors—the latter
can be equally important as the specific characteristics of the
devices employed and the particular circumstances in which
123
J Multimodal User Interfaces (2015) 9:275–286 277
an application is used can rarely be controlled for in advance.
This is well demonstrated by the large number of auditory
and haptic/tactile solutions which have appeared in assistive
technologies in the past decades, and by the fact that, despite
huge differences among them, many have been used with a
significant degree of success. An overview of relevant solutions
is provided in the following section.
2.1 Systems in assistive engineering based on tactile
solutions
Historically speaking, solutions supporting vision using the
tactile modality appeared earlier than audio-based solutions,
therefore, a brief discussion of tactile-only solutions is provided
first.
Systems that substitute tactile stimuli for visual information
generally translate images from a camera into electrical
or vibrotactile stimuli, which can then be applied to various
parts of the body (including the fingers, the palm, the back or
the tongue of the user). Experiments have confirmed the viability
of this approach in supporting the recognition of basic
shapes [15 <> ,16 <> ], as well as reading [17 <> ,18 <> ] and localization
tasks [19 <> ].
Several ideas for such applications have achieved commercial
success. An early example of a device that supports
reading is the Optacon device, which operates by transcoding
printed letters onto an array of vibrotactile actuators in a
24 × 6 arrangement [17 <> ,20 <> ,21 <> ]. While the Optacon was relatively
expensive at a price of about 1500 GBP in the 1970s,
it allowed for reading speeds of 15–40 words per minute [22 <> ]
(others have reported an average of about 28 wpm [23 <> ], while
the variability of user success is illustrated by the fact that one
of the authors of the current paper knew and observed at least
two users with optacon reading speeds of over 80 wpm). A
camera extension to the system was made available to allow
for the reading of on-screen material.
An arguably more complex application area of tactile substitution
for vision is navigation. This can be important not
only for the visually impaired, but also for applications in
which users are subjected to significant cognitive load (as
evidenced by the many solutions for navigation feedback in
both on-the-ground and aerial navigation [24 <> –26 <> ]).
One of the first commercially available devices for navigation
was the Mowat sensor, from Wormald International
Sensory Aids, which is a hand-held device that uses ultrasonic
detection of obstacles and provides feedback in the
form of tactile vibrations that are inversely proportional to
distance. Another example is Videotact, created by ForeThought
Development LLC, which provides navigation cues
through 768 titanium electrodes placed on the abdomen [27 <> ].
Despite the success of such products, newer ones are still
being developed so that environments with increasing levels
of clutter can be supported at increasing levels of comfort.
The former goal is supported by the growing availability
of (mobile) processing power, while the latter is supported
by the growing availability of unencumbered wearable technologies.
A recent example of a solution which aims to make
use of these developments is a product of a company by the
name “Artificial Vision For the Blind”, which incorporates a
pair of glasses from which haptic feedback is transmitted to
the palm [28 <> ,29 <> ].
The effective transmission of distance information is a
key issue if depth information has to be communicated to
users. The values of distance to be represented are usually
proportional or inversely proportional to tactile and auditory
attributes, such as frequency or spacing between impulses.
Distances reproduced usually range from 1 to 15 m. Instead
of a continuous simulation of distance, discrete levels of distance
can be used by defining areas as being e.g. “near” or
“far”.
2.2 Systems in assistive engineering based on auditory
solutions
In parallel to tactile feedback solutions, auditory feedback
has also increasingly been used in assistive technologies oriented
towards the visually impaired. Interestingly, it has been
remarked that both the temporal and frequency-based resolution
of the auditory sensory system is higher than the
resolution of somatosensory receptors along the skin [30 <> ].
For several decades, however, this potential advantage of
audition over touch was difficult to take advantage of due
to limitations in processing power. For instance, given that
sound information presented to users is to be synchronized
with the frame rate at which new data is read, limitations
in visual processing power would have ultimately affected
the precision of feedback as well. Even today, frame rates
of about 2–6 frames per second (fps) are commonly used,
despite the fact that modern camera equipment is easily capable
of capturing 25 fps, and that human auditory capabilities
would be well suited to interpreting more information.
Two early auditory systems—both designed to help blind
users with navigation and obstacle detection—are SonicGuide
and the LaserCane [31 <> ]. SonicGuide uses a wearable
ultrasonic echolocation system (in the form of a pair of eyeglass
frames) to provide the user with cues on the azimuth
and distance of obstacles [32 <> ]. Information is provided to the
user by directly mapping the ultrasound echoes onto audible
sounds in both ears (one ear can be used if preferred, leaving
the other free for the perception of ambient sounds).
The LaserCane system works similarly, although its interface
involves a walking cane and it uses infrared instead
of ultrasound signals in order to detect obstacles that are
relatively close to the cane [33 <> ]. It projects beams in three
different directions in order to detect obstacles that are above
the cane (and thus possibly in front of the chest of the user),
123
278 J Multimodal User Interfaces (2015) 9:275–286
in front of the cane at a maximum distance of about 12 ft, and
in front of the cane in a downward direction (e.g., to detect
curbs and other discontinuities in terrain surface). Feedback
to the user is provided using tactile vibrations for the forwardoriented
beam only (as signals in this direction are expected
to be relatively more frequent), while obstacles from above
and from the terrain are represented by high-pitched and lowpitched
sounds, respectively.
A similar, early approach to obstacle detection is the Nottingham
Obstacle Detector that works using a ∼16 Hz ultrasonic
detection signal and is somewhat of a complement to
the Mowat sensor (it is also handheld and also supports obstacle
detection based on ultrasound, albeit through audio feedback).
Eight gradations of distance are assigned to a musical
scale. However, as the system is handheld, the position at
which it is held compared to the horizontal plane is important.
Blind users have been shown to have lower awareness of
limb positions, therefore this is a drawback of the system [34 <> ]
More recent approaches are exemplified by the Real-Time
Assistance Prototype (RTAP) [35 <> ], which is camera-based
and is more sophisticated in the kinds of information it conveys.
It is equipped with stereo cameras applied on a helmet,
a portable laptop with Windows OS and small stereo headphones.
Disadvantages are the limited panning area of ±32◦,
the lack of wireless connection, the laptop-sized central unit,
the use of headphones blocking the outside world, low resolution
in distance and “sources too close” perception during
binaural rendering, and the unability to detect objects at the
ground level. The latter can be solved by a wider viewfield or
using the equipment as complementary to the white stick that
is responsible for detection of steps, stones etc. The RTAP
uses 19 discrete levels for distance. It also provides the advantage
of explicitly representing the lack of any obstacle within
a certain distance, which can be very useful in reassuring
the user that the system is still in operation. Further, based
on the object classification capabilities of its vision system,
the RTAP can filter objects based on importance or proximity.
Tests conducted using the system have revealed several
important factors in the success of assistive solutions. One
such factor is the ability of users to remember auditory events
(for example, even as they disappear and reappear as the
user’s head moves). Another important factor is the amount
of training and the level of detail with which the associated
training protocols are designed and validated.
A recent system with a comparable level of sophistication
is the System for Wearable Audio Navigation (SWAN),
which was developed to serve as a safe pedestrian navigation
and orientation aid for persons with temporary or permanent
visual impairments [36 <> ,37 <> ]. SWAN consists of an audio-only
output and a tactile input via a dedicated handheld interface
device. Once the user’s location and head direction are determined,
SWAN guides the user along the required path using
a set of beacon sounds, while at the same time indicating
the location of features in the environment that may be of
interest to the user. The sounds used by SWAN include navigation
beacons (earcon-like sounds), object sounds (through
spatially localized auditory icons), surface transitions, and
location information and announcements (brief prerecorded
speech samples).
General-purpose systems for visual-to-audio substitution
(e.g. with the goal of allowing pattern recognition, movement
detection, spatial localization and mobility) have also
been developed. Two influential systems in this category are
the vOICe system developed by Meijer [38 <> ] and the Prosthesis
Substituting Vision with Audition (PSVA) developed
by Capelle and his colleagues [31 <> ]. The vOICe system translates
the vertical dimension of images into frequency and the
horizontal dimension of images into time, and has a resolution
of 64 pixels × 64 pixels (more recent implementations
use larger displays [30 <> ]). The PSVA system uses similar concepts,
although time is neglected and both dimensions of the
image are mapped to frequencies. Further, the PSVA system
uses a biologically more realistic, retinotopic model, in which
the central, foveal areas are represented in higher resolution
(thus, the model uses a periphery of 8 × 8 pixels and a foveal
region in the center of 8 × 8 pixels).
2.3 Systems in assistive engineering based on auditory
and tactile solutions
Solutions combining the auditory and tactile modalities have
thus far been relatively rare, although important results and
solutions have begun to appear in the past few years. Despite
some differences between them, in many cases the solutions
represent similar design choices.
HiFiVE is one example of a technically complex vision
support system that combines sound with touch and manipulation
[39 <> –41 <> ]. Visual features that are normally perceived
categorically are mapped onto speech-like (but non-verbal)
auditory phonetics (analogous to spemoticons, only not
emotional in character). All such sounds comprise three syllables
which correspond to different areas in the image: one
syllable for color, and two for layout. For example, “way-lairroar”
might correspond to “white-grey” and “left-to-right”.
Changes in texture are mapped onto fluctuations of volume,
while motion is represented through binaural panning.
Guided by haptics and tactile feedback, users are also enabled
to explore various areas on the image via finger or hand
motions. Through concurrent feedback, information is presented
on shapes, boundaries and textures.
The HiFiVE system has seen several extensions since it
was originally proposed, and these have in some cases led to
a relative rise to prominence of automated vision processing
approaches. This has enabled the creation of so-called audiotactile
objects, which represent higher-level combinations
123
J Multimodal User Interfaces (2015) 9:275–286 279
of low-level visual attributes. Such objects have auditory
representations, and can also be explored through ‘tracers’—
i.e., communicational entities which either systematically
present the properties of corresponding parts of the image
(‘area-tracers’), or convey the shapes of particular items
(‘shape-tracers’).
The See ColOr system is another recent approach to combining
auditory feedback with tactile interaction [42 <> ]. See
ColOr combines modules for local perception, global perception,
alerting and recognition. The local perception module
uses various auditory timbres to represent colors (through a
hue-saturation-level technique reminiscent of Barrass’s TBP
model [43 <> ]), and rhythmic patterns to represent distance as
measured on the azimuth plane. The global module allows
users to pinpoint one or more positions within the image
using their fingers, so as to receive comparative feedback
relevant to those areas alone. The alerting and recognition
modules, in turn, provide higher-level feedback on obstacles
which pose an imminent threat to the user, as well as
on “auditory objects” which are associated with real-world
objects.
At least two important observations can be made based on
these two approaches. The first observation is that whenever
audio is combined with finger or hand-based manipulations,
the tactile/haptic modality is handled as the less prominent of
the two modalities—either in the sense that it is (merely) used
to guide the scope of (the more important) auditory feedback,
or in the sense that it is used to provide small portions of the
complete information available, thereby supporting a more
explorative interaction from the user. The second observation
is that both of these multi-modal approaches distinguish
between low-level sensations and high-level perceptions—
with the latter receiving increasing support from automated
processing. This second point is made clear in the See ColOr
system, but it is also implicitly understood in HiFiVE.
2.4 Summary of key observations
Based on the overview presented in Sect. 2, <> it can be concluded
that feedback solutions range from iconic to abstract,
and from serial to parallel. Given the low-level physical
nature of the tasks considered (e.g. recognizing visual characters
and navigation), solutions most often apply a low-level
and direct mapping of physical changes onto auditory and/or
tactile signals. Such approaches can be seen as data-driven
sonification (or tactification, but in an analogous sense to
sonification) approaches. Occasionally, they are complemented
by higher-level representations, such as the earconbased
beacon sounds used in SWAN, or the categorical
verbalizations used in HiFiVE. From a usability perspective,
such approaches using mostly direct physical, and some
abstract metaphors seem to have proven most effective.
3 Generic capabilities of mobile computing
platforms
Recent trends have led to the appearance of generic mobile
computing technologies that can be leveraged to improve
users’ quality of life. In terms of assistive technologies, this
tendency offers an ideal working environment for developers
who are reluctant to develop their own, specialized hardware/software
configurations, or who are looking for faster
ways to disseminate their solutions.
The immense potential behind mobile communication
platforms for assistive technologies can be highlighted
through various perspectives, including general-purpose
computing, advanced sensory capabilities, and crowdsourcing/data
integration capabilities.
3.1 General-purpose computing
The fact that mobile computing platforms offer standard
APIs for general-purpose computing provides both application
developers and users with a level of flexibility that is
very conducive to developing and distributing novel solutions.
As a result of this standardized background (which
can already be seen as a kind of ubiquitous infrastructure),
there is no longer any significant need to develop one-of-akind
hardware/software configurations. Instead, developers
are encouraged to use the “same language” and thus collaboratively
improve earlier solutions. A good example of
this is illustrated by the fact that the latest prototypes of
the HiFiVE system are being developed on a commodity
tablet device. As a result of the growing ease with which
new apps can be developed, it is intuitively clear that a
growing number of users can be expected to act upon the
urge to improve existing solutions, should they have new
ideas. This general notion has been formulated in several
domains in terms of a transition from “continuous improvement”
to “collaborative innovation” [44 <> ]; or, in the case of
software engineering, from “software product lines” to “software
ecosystems” [45 <> ].
3.2 Advanced sensory capabilities
State-of-the-art devices are equipped with very advanced,
and yet generic sensory capabilities enabling tight interaction
with the environment. For instance, the Samsung Galaxy
S5 (which appeared in 2014) has a total of 10 built-in sensors.
This is a very general property that is shared by the
vast majority of state-of-the-art mobile devices. Typically
supported sensors include:
– GPS receivers, which are very useful in outdoor environments,
but also have the disadvantage of being inaccurate,
123
280 J Multimodal User Interfaces (2015) 9:275–286
slow, at times unreliable and of being unusable in indoor
environments
– Gyro sensors, which detect the rotation state of the mobile
device based on three axes (it is interesting to note that
the gyroscopes found on most modern devices are suf-
ficiently sensitive to measure acoustic signals of low
frequencies, and through signal processing and machine
learning, can be used as a microphone [46 <> ]).
– Accelerator sensors, which detect the movement state of
the device based on three axes
– Magnetic sensors (compass), which have to be calibrated,
and can be affected by strong electromagnetic fields.
The accuracy of such sensors generally depends on the specific
device, however, they are already highly relevant to
applications which integrate inputs from multiple sensors and
will only improve in accuracy in the coming years.
3.3 Crowdsourcing and data integration capabilities
A great deal of potential arises from the ability of mobile
communication infrastructures to aggregate semantically relevant
data that is curated in a semi-automated, but also
crowd-supported way. This capability allows for low-level
sensor data to be fused with user input and (semantic) background
information in order to achieve greater precision in
functionality.
A number of recent works highlight the utility of this
potential, for example in real-time emergency response [47 <> ,
48 <> ], opportunistic data dissemination [49 <> ], crowd-supported
sensing and processing [50 <> ], and many others. Crowdsourced
ICT based solutions are also increasingly applied in the assistive
technology arena, as evidenced by several recent works
on e.g. collaborative navigation for the visually impaired in
both the physical world and on the Web [51 <> –55 <> ]. Here, sensor
data from the mobile platform (e.g., providing location
information) are generally used to direct users to the information
that is most relevant, or to the human volunteer who
has the most knowledge with respect to the problem being
addressed.
4 State-of-the-art applications for mobile platforms
Both Android and iOS offer applications that support visually
impaired people in performing everyday tasks. Many of the
applications can also be used to good effect by sighted persons.
In this section, a brief overview is provided of various
solutions on both of these platforms.
Evidence of the take up of these applications can be seen
on email lists specific to blind and visually impaired users
[56 <> –58 <> ], exhibitions of assistive technology [59 <> ], and specialist
publications [60 <> ]. A further symptom of the growth
in mobile phone and tablet use by this population is the
fact that the Royal National Institute of the Blind in the UK
runs a monthly “Phone Watch” event at its headquarters in
London.
4.1 Applications for Android
In this subsection, various existing solutions for Android are
presented.
4.2 Text, speech and typing
Android includes a number of facilities for text-to-speech
based interaction as part of its Accessibility Service. In particular,
TalkBack, KickBack, and SoundBack are applications
designed to help blind and visually impaired users by allowing
them to hear and/or feel their selections on the GUI [61 <> ].
These applications are also capable of reading text out loud.
The Sound quality of TalkBack is relatively good compared
to other screen-readers for PCs, however, proper language
versions of SVOX must be installed in order to get the
required quality, and some of these are not free. On the other
hand, operating vibration feedback cannot be switched off
and the system sometimes reads superfluous information on
the screen. Users have reported that during text-messaging,
errors can occur if the contact is not already in the address
book. The software is only updated for the latest Android
versions. Although all of them are preinstalled on most con-
figurations, the freely available IDEAL Accessible App can
be used to incorporate further enhancements in their functionality.The
Classic Text to Speech Engine includes a combination
of over 40 male and female voices, and enables users to listen
to spoken renderings of e.g. text files, e-books and translated
texts [62 <> ]. The app also features voice support in key areas
like navigation. In contrast to SVOX, this application is free,
but with limited language support both for reading and the
human-device user interface.
BrailleBack works together with the TalkBack app to provide
a combined Braille and speech experience [63 <> ]. This
allows users to connect a number of supported refreshable
Braille displays to the device via Bluetooth. Screen content
is then presented on the Braille display and the user can navigate
and interact with the device using the keys on the display.
Ubiquitous Braille based typing is also supported through the
BrailleType application [64 <> ].
The Read for the Blind application is a communitypowered
app that allows users to create audio books for the
blind, by reading books or short articles from magazines,
newspapers or interesting websites [65 <> ].
The ScanLife Barcode and QR Reader [62 <> ] enables users
to read barcodes and QR codes. This can be useful not only
in supermarkets, but in other locations as well using even a
123
J Multimodal User Interfaces (2015) 9:275–286 281
relatively low resolution camera of 3.2 MP, however, memory
demands can be a problem.
The identification of colors is supported by application
such as the Color Picker or Seenesthesis [66 <> ,67 <> ]. More
generally, augmented/virtual magnifying glasses—such as
Magnify—also exist to facilitate reading for the weak-sighted
[62 <> ]. Magnify only works adequately if it is used with a camera
of high resolution.
4.3 Speech-based command interfaces
The Eyes-Free Shell is an alternative home screen or launcher
for the visually impaired as well as for users who are unable
to focus on the screen (e.g. while driving). The application
provides a way to interact with the touch screen to check
status information, launch applications, and direct dial or
message specific contacts [68 <> ]. While the Eyes-Free Shell
and the Talking Dialer screens are open, the physical controls
on the keyboard and navigation arrows are unresponsive and
different characters are assigned to the typing keys. Widgets
can be used, the volume can be adjusted and users can find
answers with Eyes-Free Voice Search.
Another application that is part of the Accessibility Service
is JustSpeak [69 <> ]. The app enables voice control of
the Android device, and can be used to activate on-screen
controls, launch installed applications, and trigger other commonly
used Android actions.
Some of the applications may need to use TalkBack as
well.
4.4 Navigation
Several Android apps can be used to facilitate navigation
for users with different capabilities in various situations (i.e.,
both indoor and outdoor navigation in structured and unstructured
settings).
Talking Location enables users to learn their approximate
position through WiFi or mobile data signals by shaking the
device [70 <> ]. Although often highly inaccurate, this can nevertheless
be the only alternative in indoor navigation, where
GPS signals are not available. The app allows users to send
SMS messages to friends with their location, allowing them
to obtain help when needed. This is similar to the idea behind
Guard My Angel, which sends SMS messages, and can also
send them automatically if the user does not provide a “heartbeat”
confirmation [71 <> ].
Several “walking straight” applications have been developed
to facilitate straight-line walking [72 <> ,73 <> ]. Such applications
use built-in sensors (i.e. mostly the magnetic sensor)
to help blind pedestrians.
Through the augmentation of mobile capabilities with data
services comes the possibility to make combined use of GPS
receivers, compasses and map data. WalkyTalky is one of
the many apps created by the Eyes-Free Project that helps
blind people in navigation by providing real-time vibration
feedback if they are not moving in the correct direction [62 <> ].
The accuracy based on the in-built GPS can be low, making
it difficult to issue warnings within 3–4 m of accuracy. This
can be eliminated by having a better GPS receiver connected
via Bluetooth.
Similarly, Intersection Explorer provides a spoken account
of the layout of streets and intersections as the user drags her
finger across a map [74 <> ].
A more comprehensive application is The vOICe for
Android [75 <> ]. The application maps live camera views to
soundscapes, providing the visually impaired with an augmented
reality based navigation support. The app includes a
talking color identifier, talking compass, talking face detector
and a talking GPS locator. It is also closely linked with
the Zxing barcode scanner and the Google Goggles apps by
allowing for them to be launched from within its own context.
The vOICe uses pitch for height and loudness for brightness
in one-second left to right scans of any view: a rising bright
line sounds as a rising tone, a bright spot as a beep, a bright
filled rectangle as a noise burst, a vertical grid as a rhythm
(cf. Section 2.2 and [38 <> ]).
4.5 Applications for iOS
In this subsection, various existing solutions for iOS are presented.4.6
Text, speech and typing
One of the most important apps is VoiceOver. It provides substantive
screen-reading capabilities for Apple’s native apps
but also for many third party apps developed for the iOS platform.
VoiceOver renders text on the screen and also employs
auditory feedback in response to user interactions. The user
can control speed, pitch and other parameters of the auditory
feedback by accessing the Settings menu. VoiceOver
supports Apple’s Safari web browser, providing element by
element navigation, as well as enabling navigation between
headings and other web page components. This sometimes
provides an added bonus for visually impaired users, as
the mobile versions of web sites are often simpler and less
cluttered than their non-mobile counterparts. Importantly
VoiceOver can easily be switched on and off by pressing the
home button three times. This is key if the device is being
used alternately by a visually impaired and a sighted user, as
the way in which interactions work is totally different when
VoiceOver is running.
Visually impaired users can control their device using
VoiceOver by using their fingers on the screen or by having
an additional keyboard attached. There are some third
123
282 J Multimodal User Interfaces (2015) 9:275–286
party applications which do not conform to Apple guidelines
on application design and so do not work with VoiceOver.
Voice Over can be used well in conjunction with the
oMoby app, which searches the Internet based on photos
taken with the iPhone camera and returns a list of search
results [76 <> ]. oMoby also allows for the use of images from
the iPhone photo library and supports some code scanning. It
has an unique image recognition capability and is free to use.
Voice Brief is another utility for reading emails, feeds,
weather, news etc. [77 <> ].
Dragon Dictation can help to translate voice into text [78 <> ].
Speaking and adding punctuation as needed verbally, Apple’s
Braille solution, BrailleTouch offers a split-keyboard design
in the form of a braille cell to allow a blind user to type [79 <> ].
The left side shows dots 1, 2 and 3 while the right side holds
dots 4, 5 and 6. The right hand is oriented over the iPhone’s
Home button with the volume buttons on the left edge.
In a way similar to various Android solutions, the Recognizer
app allows users to identify cans, packages and ID cards
by through camera-based barcode scans [79 <> ]. Like Money
Reader, this app incorporates object recognition functionality.
The app stores the image library locally on the phone and
does not require an Internet connection.
The LookTel Money Reader recognizes currency and
speaks the denomination, enabling people experiencing
visual impairments or blindness to quickly and easily identify
and count bills [80 <> ]. By pointing the camera of the iOS
device at a bill the application will tell the denomination in
real-time.
The iOS platform also offers color identification (for free)
called Color ID Free [81 <> ].
4.7 Speech-based command interfaces
The TapTapSee app is designed to help the blind and visually
impaired identify objects they encounter in their daily lives
[82 <> ]. By double tapping the screen to take a photo of anything,
at any angle, the user will hear the app speak the identification
back.
A different approach is possible through crowdsourced
solutions, i.e. by connecting to a human operator. VizWiz is
an application that records a question after taking a picture
of any object [83 <> ]. The query can be sent to the Web Worker,
IQ Engines or can be emailed or shared on Twitter. Web
worker is a human volunteer who will review and answer
the question. On the other hand, IQ Engines is an image
recognition platform.
A challenging aspect of daily life for visually impaired
users is that headphones block environmental sounds. “Awareness!
The Headphone App” allows one to listen to the
headphones while also hearing the surrounding sounds [84 <> ].
It uses the microphone to pick up ambient sounds while listening
to music or using other apps using headphones for
output. While this solution helps to mediate the problem of
headphones masking ambient sounds, its use means that the
user is hearing both ambient sounds and whatever is going
on in the app, potentially leading to overload of the auditory
channel or distraction during navigation tasks.
With an application called Light Detector, the user can
transform any natural or artificial light source he/she encounters
into sound [85 <> ]. By pointing the iPhone camera in any
direction the user will hear higher or lower pitched sounds
depending on the intensity of the light. Users can check
whether the lights are on, where the windows and doors are
closed, etc.
Video Motion Alert (VM Alert) is an advanced video
processing application for the iPhone capable of detecting
motion as seen through the iPhone camera [86 <> ]. VM Alert
can use either the rear or front facing camera. It can be con-
figured to sound a pleasant or alarming audible alert when it
detects motion and can optionally save images of the motion
conveniently to the iPhone camera roll.
4.8 Navigation
The iOS platform also provides applications for navigation
assistance. The Maps app provided by default with the iPhone
is accessible in the sense that its buttons and controls are
accessible using VoiceOver, and using these one can set
routes and obtain turn by turn written instructions.
Ariadne GPS also works with VoiceOver [87 <> ]. Talking
maps allow for the world to be explored by moving a finger
around the map. During exploration, the crossing of a
street is signaled by vibration. The app has a favorites feature
that can be used to announce stops on the bus or train,
or to read street names and numbers. It also enables users to
navigate large buildings by pre-programming e.g. classroom
locations. Rotating maps keep the user centered, with territory
behind the user on the bottom of the screen and what
is ahead on the top portion. Available in multiple languages,
Ariadne GPS works anywhere Google Maps are available.
Similarly to WalkyTalk, low resolution GPS receivers can
be a problem, however this can be solved through external
receivers connected to the device [70 <> ].
GPS Lookaround also uses VoiceOver to speak the name
of the street, city, cross-street and points of interest [79 <> ].
Users can shake the iPhone to create a vibration and swishing
sound indicating the iPhone will deliver spoken information
about a location.
BlindSquare also provides information to visually
impaired users about their surroundings [79 <> ]. The tool uses
GPS and a compass to identify location. Users can find out
details of local points of interest by category, define routes
to be walked, and have feedback provided while walking.
From a social networking perspective, BlindSquare is closely
linked to FourSquare: it collects information about the user’s
123
J Multimodal User Interfaces (2015) 9:275–286 283
environment from FourSquare, and allows users to check in
to FourSquare by shaking the iPhone.
4.9 Gaming and serious gaming
Although not directly related to assistive technologies, audioonly
solutions can help visually impaired users access
training and entertainment using so-called serious gaming
solutions [88 <> –90 <> ].
One of the first attempts was Shades of Doom, an audioonly
version of the maze game called Doom, which is not
available on mobile platforms [91 <> ]. The user had to navigate
through a maze, accompanied by steps, growling of a monster
and the goal was to find the way out without being killed. The
game represents interesting new ideas, but the quality of the
sound it uses, as well as its localization features have been
reported to be relatively poor.
A game called Vanished is another “horror game” that
relies entirely on sound to communicate with the player. It
has been released on iPhone, but there is also a version for
Android [92 <> ]. A similar approach is BlindSide, an audio-only
adventure game promising a fully-immersive 3D world [93 <> ].
Nevertheless, games incorporating 3D audio and directional
simulation may not always provide high quality experience,
depending on the applied playback system (speakers, headphone
quality etc.).
Papa Sangre is an audio-only guidance-based navigation
game for both Android and iOS [94 <> ]. The user can walk/run
by tapping left and right on the bottom half, and turn by sliding
the finger across the top half of the screen while listening
to 3D audio cues for directional guidance.
Grail to the Thief tries to establish a classic adventure
game like Zork or Day of the Tentacle, except that instead
of text-only descriptions it presents the entire game through
audio [95 <> ].
A Blind Legend is an adventure game that uses binaural
audio to help players find their bearings in a 3D environment,
and allows for the hero to be controlled through the phone’s
touchscreen using various multi-touch combinations in different
directions [96 <> ].
Audio archery brings archery to Android. Using only the
ears and reflexes the goal is to shoot at targets. The game
is entirely auditory. Users hear a target move from left to
right. The task consists in performing flicking motions on
the screen with one finger to pull back the bow, and releasing
it as the target is centered [97 <> ]. The game has been reported
to work quite well with playback systems with two speakers
at a relatively large distance which allows for perceptually
rich stereo imagery.
Agents is an audio-only adventure game in which players
control two field agents solely via simulated “voice calls”
on their mobile phone [98 <> ]. The task is to help them break
into a guarded complex, then to safely make their way out,
while helping them work together. The challenge lies is using
open-ended voice-only commands directed at an automatic
speech recognition module.
Deep sea (A Sensory Deprivation Video Game) and Aurifi
are audio-only games in which players have a mask that
obscures their vision and takes over their hearing, plunging
them into a world of blackness occupied only by the sound
of their own breathing or heartbeat [99 <> ,100 <> ]. Although these
games try to mimic “horrifying environments” using audio
only, at best a limited experience can be provided due to
the limited capabilities of the playback system. In general,
audio-only games (especially those using 3D audio, stereo
panning etc.) require high-quality headphones.
5 Towards a research agenda for mobile assistive
technology for visually impaired users
Given the level of innovation described in the above sections
in this paper, and the growth in take up of mobile devices,
both phones and tablets, by the visually impaired community
worldwide, the potential and prospects for research into
effective Interaction Design and User Experience for nonvisual
use of mobile applications is enormous. The growing
prevalence of small screen devices, allied to the continual
growth in the amount of information—including cognitive
content—that is of interest to users, means that this research
will undoubtedly have relevance to sensor-bridging applications
targeted at the mobile user population as a whole [101 <> ].
In the following subsections, we set out a number of areas
and issues for research which appear to be important to the
further development of the field.
5.1 Research into multimodal non-visual interaction
1. Mode selection. Where a choice exists, there is relatively
little to guide designers on the choice of mode in which
to present information. We know relatively little about
which mode, audio or tactile, is better suited to which
type of information.
2. Type selection: Little is also known about how information
is to be mapped to the selected mode, i.e. whether
through (direct) representation-sharing or (analogybased)
representation-bridging as described in [102 <> ,103 <> ].
3. Interaction in a multimodal context. Many studies have
been conducted on intersensory integration, sensory
cross-effects and sensory dominance effects in visually
oriented desktop and virtual applications [104 <> –107 <> ].
However, few investigations have taken place, in a nonvisual
and mobile computing oriented context, into the
consumption of information in a primary mode while
being presented with information in a secondary mode.
How does the presence of the other mode detract from
123
284 J Multimodal User Interfaces (2015) 9:275–286
cognitive resources for processing information delivered
in the primary mode? What information can safely be relegated
to yet still perceived effectively in the secondary
mode?
4. What happens when users themselves are given the ability
to allocate separate information streams to different
presentation modes?
5. What is the extent of individual differences in the ability
to process auditory and haptic information in unimodal
and multimodal contexts? Is it a viable goal to develop
interpretable tuning models allowing users to fine-tune
feedback channels in multimodal mobile interactions
[108 <> ,109 <> ]?
6. Context of use. How are data gained experimentally on
the issues above effected when examined in real, mobile
contexts of use? Can data curated from vast numbers
of crowdsourced experiments be used to substitute, or
complement laboratory experimentation?
5.2 Accessible development lifecycle
The following issues relate to the fact that the HCI literature
on user-centered prototype development and evaluation
is written assuming a visual context. Very little is available
about how to go about these activities with users with no or
little sight.
1. How might the tools used for quickly creating paperbased
visual prototypes be replaced by effective counterparts
for a non-visual context? What types of artifact are
effective in conducting prototyping sessions with users
with little or no vision? It should be a fundamental property
of such materials that they can be perceived and
easily altered by the users in order that an effective twoway
dialogue between developers and users takes place.
2. To what extent are the standard means of collecting evaluation
data from sighted users applicable to working with
visually impaired users? For example, how well does
speak-aloud protocol work in the presence of spoken
screen-reader output? What are the difficulties posed for
evaluation given the lack of a common vocabulary for
expressing the qualities of sounds and haptic interactions
by non-specialist users?
3. Are there techniques that are particularly effective in evaluating
applications for visually impaired users which give
realistic results while addressing health and safety? For
example, what is the most effective yet realistic way of
evaluating an early stage navigation app?
5.3 Summary
This paper briefly summarized recent developments on
mobile devices in assistive technologies targeting visually
impaired people. The focus was on the use of sound and vibration
(haptics) whenever applicable. It was demonstrated that
the most commonly used mobile platforms—i.e. Google’s
Android and Apple’s iOS—both offer a large variety of
assistive applications by using the built-in sensors of the
mobile devices, and combining this sensory information with
the capability of handling large datasets, as well as cloud
resources and crowdsourced contributions in real-time. A
recent H2020 project was launched to develop a navigational
device running on a mobile device, incorporating depthsensitive
cameras, spatial audio, haptics and development
of training methods.
Acknowledgments This project has received funding from the European
Union’s Horizon 2020 research and innovation programme under
grant Agreement No. 643636 “Sound of Vision”. This research was
realized in the frames of TÁMOP 4.2.4. A/2-11-1-2012-0001 “National
Excellence Program—Elaborating and operating an inland student and
researcher personal support system”. The project was subsidized by the
European Union and co-financed by the European Social Fund.
Open Access This article is distributed under the terms of the Creative
Commons Attribution 4.0 International License (http://creativecomm
ons.org/licenses/by/4.0/), which permits unrestricted use, distribution,
and reproduction in any medium, provided you give appropriate credit
to the original author(s) and the source, provide a link to the Creative
Commons license, and indicate if changes were made.
References
1. Gaver W (1986) Auditory icons: using sound in computer interfaces.
Human Comput Interact 2(2):167–177
2. Gaver W (1988) Everyday listening and auditory icons. PhD thesis,
University of California, San Diego
3. Gaver W (1989) The SonicFinder: an interface that uses auditory
icons. Human Comput Interact 4(1):67–94
4. Smith D (1975) Pygmalion: a computer program to model and
stimulate creative thought. PhD thesis, Stanford University, Dept.
of Computer Science
5. Blattner M, Sumikawa D, Greenberg R (1989) Earcons and icons:
their structure and common design principles. Human Comput
Interact 4(1):11–44
6. Kramer G (ed) (1994) Auditory display: sonification, audification,
and auditory interfaces. Santa Fe Institute Studies in the
Sciences of Complexity. Proceedings volume XVIII. AddisonWesley,
Reading
7. Hermann T, Hunt A, Neuhoff JG (2011) The sonification handbook.
Logos, Berlin
8. Csapo A, Wersenyi G (2014) Overview of auditory representations
in human–machine interfaces. ACM Comput Surveys 46(2)
9. Maclean K, Enriquez M (2003) Perceptual design of haptic icons.
In: In Proceedings of eurohaptics, pp 351–363
10. Enriquez M, MacLean K (2003) The hapticon editor: a tool in support
of haptic communication research. In: Proceedings of the 11th
symposium on haptic interfaces for virtual environment and teleoperator
systems (HAPTICS’03). IEEE Computer Society, Los
Angeles, pp 356–362
11. Brewster S, Brown L (2004) Tactons: structured tactile messages
for non-visual information display. In: Proceedings of the fifth
conference on Australasian user interface (AUIC’04), vol 28,
Dunedin, pp 15–23
123
J Multimodal User Interfaces (2015) 9:275–286 285
12. Picinali L, Feakes C, Mauro D A, Katz B F (2012) Spectral discrimination
thresholds comparing audio and haptics for complex
stimuli. In: Proceedings of international workshop on haptic and
audio interaction design (HAID 2012), pp 131–140
13. Picinali L, Feakes C, Mauro D, Katz BF (2012) Tone-2 tones discrimination
task comparing audio and haptics. In: Proceedings
of IEEE international symposium on haptic audio-visual environments
and games, Munich, pp 19–24
14. Picinali L, Katz BF (2010) Spectral discrimination thresholds
comparing audio and haptics. In: Proceedings of haptic and auditory
interaction design workshop, Copenhagen, pp 1–2
15. Kaczmarek KA, Haase SJ (2003) Pattern identification and perceived
stimulus quality as a function of stimulation current on a
fingertip-scanned electrotactile display. IEEE Trans Neural Syst
Rehabil Eng 11:9–16
16. Sampaio E, Maris S, Bach-y Rita P (2001) Brain plasticity:
‘visual’ acuity of blind persons via the tongue. Brain Res 908:204–
207
17. Bliss JC, Katcher MH, Rogers CH, Shepard RP (1970) Opticalto-tactile
image conversion for the blind. IEEE Trans Man Mach
Syst 11:58–65
18. Craig JC (1981) Tactile letter recognition: pattern duration and
modes of pattern generation. Percept Psychophys 30:540–546
19. Jansson G (1983) Tactile guidance of movement. Int J Neurosci
19:37–46
20. Levesque V, Pasquero J, Hayward V (2007) Braille display by
lateral skin deformation with the STReSS tactile transducer.
In: Proceedings of the second joint eurohaptics conference and
symposium on haptic interfaces for virtual environment and teleoperator
systems, Tsukuba, pp 115–120
21. Bach-y Rita P, Tyler ME, Kaczmarek KA (2003) Seeing with the
brain. Int J Hum Comput Interact 15(2):285–295
22. Craig JC (1977) Vibrotactile pattern perception: extraordinary
observers. Science 196(4288):450–452
23. Hislop D, Zuber BL, Trimble JL (1983) Characteristics of reading
rate and manual scanning patterns of blind optacon readers. Hum
Factors 25(3):379–389
24. Segond H, Weiss D, Sampaio E (2005) Human spatial navigation
via a visuo-tactile sensory substitution system. Perception
34:1231–1249
25. vanErp JBF, vanVeen AHC, Jansen C, Dobbins T (2005) Waypoint
navigation with a vibrotactile waist belt. Perception 2:
106–117
26. Jones LA, Lockyer B, Piateski E (2006) Tactile display and vibrotactile
recognition on the torso. Adv Robot 20:1359–1374
27. http://www.4thtdev.com. Accessed Mar 2015
28. http://unreasonableatsea.com/artificial-vision-for-the-blind/.
Accessed Mar 2015
29. http://www.dgcs.unam.mx/ProyectoUNAM/imagenes/080214.
pdf. Accessed Mar 2015
30. Kim JK, Zatorre RJ (2008) Generalized learning of visual-toauditory
substitution in sighted individuals. Brain Res 1242:263–
275
31. Capelle C, Trullemans C, Arno P, Veraart C (1998) A realtime
experimental prototype for enhancement of vision rehabilitation
using auditory substitution. IEEE Trans Biomed Eng
45(10):1279–1293
32. Kay L (1974) A sonar aid to enhance spatial perception of the
blind: engineering design and evaluation. IEEE Radio Electron
Eng 44(11):605–627
33. Murphy EF (1971) The VA—bionic laser can for the blind. In:
The National Research Council, (ed) Evaluation of sensory aids
for the visually handicapped. National Academy of Sciences, pp
73–82
34. Bissitt D, Heyes AD (1980) An application of bio-feedback in the
rehabilitation of the blind. Appl Ergon 11(1):31–33
35. Dunai L, Fajarnes GP, Praderas VS, Garcia BD, Lengua IL (2010)
Real-time assistance prototype—a new navigation aid for blind
people. In: Proceedings of IECON 2010—36th annual conference
on IEEE Industrial Electronics Society, pp 1173–1178
36. Walker BN, Lindsay J (2006) Navigation performance with a virtual
auditory display: effects of beacon sound, capture radius, and
practice. Hum Factors 48(2):265–278
37. Wilson J, Walker B N, Lindsay J, Cambias C, Dellaert F (2007)
SWAN: system for wearable audio navigation. In: Proceedings of
the 11th international symposium on wearable computers (ISWC
2007), USA
38. Meijer P (1992) An experimental system for auditory image representations.
IEEE Trans Biomed Eng 39(2):112–121
39. Dewhurst D (2007) An audiotactile vision-substitution system.
In: Proceedings of second international workshop on interactive
sonification, York, pp 1–4
40. Dewhurst D (2009) Accessing audiotactile images with HFVE
silooet. In: Proceedings of fourth international workshop on haptic
and audio interaction design. Springer, Berlin, pp 61–70
41. Dewhurst D (2010) Creating and accessing audio-tactile images
with “HFVE” vision substitution software. In: Proceedings of the
third interactive sonification workshop. KTH, Stockholm, pp 101–
104
42. Gomez Valencia JD (2014) A computer-vision based sensory substitution
device for the visually impaired (See ColOr), PhD thesis.
University of Geneva
43. Barrass S (1997) Auditory information design, PhD thesis. Australian
National University
44. Chapman RL, Corso M (2005) From continuous improvement
to collaborative innovation: the next challenge in supply chain
management. Prod Plan Control 16(4): 339–344
45. Bosch J (2009) From software product lines to software ecosystems.
In: Proceedings of the 13th international software product
line conference, pp 111–119
46. Michalevsky Y, Boneh D, Nakibly G (2014) Gyrophone: recognizing
speech from gyroscope signals. In: Proceedings of 23rd
USENIX security symposium, San Diego
47. Rogstadius J, Kostakos V, Laredo J, Vukovic M (2011) Towards
real-time emergency response using crowd supported analysis of
social media. In: Proceedings of CHI workshop on crowdsourcing
and human computation, systems, studies and platforms
48. Blum JR, Eichhorn A, Smith S, Sterle-Contala M, Cooperstock JR
(2014) Real-time emergency response: improved management of
real-time information during crisis situations. J Multimodal User
Interfaces 8(2):161–173
49. Zyba G, Voelker G M, Ioannidis S, Diot C (2011) Dissemination
in opportunistic mobile ad-hoc networks: the power of the crowd.
In: Proceedings of IEEE INFOCOM, pp 1179–1187
50. Ra M-R, Liu B, La Porta TF, Govindan R (2012) Medusa: a
programming framework for crowd-sensing applications. In: Proceedings
of the 10th international conference on mobile systems,
applications, and services, pp 337–350
51. Balata J, Franc J, Mikovec Z, Slavik P (2014) Collaborative
navigation of visually impaired. J Multimodal User Interfaces
8(2):175–185
52. Hara K, Le V, Froehlich J (2013) Combining crowdsourcing and
google street view to identify street-level accessibility problems.
In: Proceedings of the SIGCHI conference on human factors in
computing systems, pp 631–640
53. RiceMT, Aburizaiza AO, Jacobson RD, Shore BM, Paez FI (2012)
Supporting accessibility for blind and vision-impaired people with
a localized gazetteer and open source geotechnology. Trans GIS
16(2):177–190
54. Cardonha C, Gallo D, Avegliano P, Herrmann R, Koch F, Borger
S (2013) A crowdsourcing platform for the construction of
123
286 J Multimodal User Interfaces (2015) 9:275–286
accessibility maps. In: Proceedings of the 10th international crossdisciplinary
conference on web accessibility, p 26
55. Kremer KM (2013) Facilitating accessibility through crowdsourcing
(2013). http://www.karenkremer.com/kremercrow
dsourcingaccessibility.pdf. Accessed Mar 2015
56. http://apps4android.org/knowledgebase/screen_reader.htm.
Accessed Mar 2015
57. http://www.AppleVis.com. Accessed Mar 2015
58. https://groups.google.com/forum/#!forum/viphone. Accessed
Mar 2015
59. http://www.qac.ac.uk/exhibitions/sight-village-birmingham/1.
htm#.VQcQvM9ybcs. Accessed Mar 2015
60. Access IT magazine published monthly by the Royal National
Institute of the Blind UK. http://www.rnib.org.uk. Accessed Mar
2015
61. http://www.androlib.com/android.application.com-google-andro
id-marvin-kickback-FExn.aspx. Accessed Mar 2015
62. http://www.androidauthority.com/best-android-apps-visually-im
paired-blind-97471. Accessed Mar 2015
63. https://play.google.com/store/apps/details?id=com.googlecode.e
yesfree.brailleback. Accessed Mar 2015
64. http://www.ankitdaf.com/projects/BrailleType/. Accessed Mar
2015
65. https://andreashead.wikispaces.com/
Android+Apps+for+VI+and+Blind. Accessed Mar 2015
66. https://play.google.com/store/apps/details?id=com.beanslab.col
orblindhelper.helper. Accessed Mar 2015
67. https://play.google.com/store/apps/details?id=touch.seenesthesii
is. Accessed Mar 2015
68. http://accessibleandroid.blogspot.hu/2010/09/how-do-i-use-eye
s-free-shell.html. Accessed Mar 2015
69. http://eyes-free.blogspot.hu/. Accessed Mar 2015
70. http://www.androlib.com/android.application.
com-shake-locator-qzmAi.aspx. Accessed Mar 2015
71. http://www.prlog.org/11967532-guard-my-angel-mobile-app-a
dapted-to-visually-impaired-people.html. Accessed Mar 2015
72. Panëels SA, Varenne D, Blum JR, Cooperstock JR (2013) The
walking straight mobile application: helping the visually impaired
avid veering. In: Proceedings of ICAD13, Lódz, pp 25–32
73. https://play.google.com/store/apps/details?id=com.johnny.straig
htlinewalk.app. Accessed Mar 2015
74. http://www.androlib.com/android.application.com-google-andro
id-marvin-intersectionexplorer-qqxtC.aspx. Accessed Mar 2015
75. http://www.androlib.com/android.application.voice-voice-wiz.
aspx. Accessed Mar 2015
76. http://mashable.com/2010/03/04/omoby-visual-search-iphone/.
Accessed Mar 2015
77. https://itunes.apple.com/HU/app/id423322440?mt=8. Accessed
Mar 2015
78. https://itunes.apple.com/HU/app/id341446764?mt=8. Accessed
Mar 2015
79. http://www.eweek.com/mobile/slideshows/
10-iphone-apps-designed-to-assist-the-visually-impaired/#
sthash.0GDG5TYh.dpuf. Accessed Mar 2015
80. https://itunes.apple.com/HU/app/id417476558?mt=8. Accessed
Mar 2015
81. https://itunes.apple.com/HU/app/id402233600?mt=8. Accessed
Mar 2015
82. https://itunes.apple.com/us/app/taptapsee-blind-visually-impair
ed/id567635020?mt=8. Accessed Mar 2015
83. https://itunes.apple.com/HU/app/id439686043?mt=8. Accessed
Mar 2015
84. https://itunes.apple.com/HU/app/id389245456?mt=8. Accessed
Mar 2015
85. https://itunes.apple.com/HU/app/id420929143?mt=8. Accessed
Mar 2015
86. https://itunes.apple.com/HU/app/id387523411?mt=8. Accessed
Mar 2015
87. http://appadvice.com/applists/show/apps-for-the-visually-impai
red. Accessed Mar 2015
88. Balan O, Moldoveanu A, Moldoveanu F, Dascalu M-I (2014)
Audio games—a novel approach towards effective learning in
the case of visually-impaired people. In: Proceedings of seventh
international conference of education, research and innovation,
Seville
89. Lopez MJ, Pauletto S (2009) The design of an audio film for
the visually impaired. In: Proceedings of the 15th international
conference on auditory display (ICAD 09), Copenhagen, pp 210–
216
90. Balan O, Moldoveanu A, Moldoveanu F, Dascalu M-I (2014) Navigational
3D audio-based game- training towards rich auditory
spatial representation of the environment. In: Proceedings of the
18th international conference on system theory, control and computing,
Sinaia
91. http://www.gmagames.com/sod.html. Accessed Mar 2015
92. http://www.pixelheartstudios.com/vanished. Accessed Mar 2015
93. http://www.blindsidegame.com. Accessed Mar 2015
94. http://www.maclife.com/article/gallery/10_apps_blind_and_part
ially_sighted. Accessed Mar 2015
95. http://www.gizmag.com/grail-to-the-thief-blind-accessible-aud
io-adventure/31824/. Accessed Mar 2015
96. http://www.ulule.com/a-blind-legend/. Accessed Mar 2015
97. https://play.google.com/store/apps/details?id=net.l_works.audio
_archery. Accessed Mar 2015
98. http://www.gamecritics.com/brandon-bales/
sounds-great-a-peek-at-the-audio-only-agents#sthash.
oGSStwTg.dpuf. Accessed Mar 2015
99. http://wraughk.com/deepsea. Accessed Mar 2015
100. http://www.appfreakblog.com/blog/the-sound-only-game-aurifi
.html. Accessed Mar 2015
101. Baranyi P, Csapo A (2012) Definition and synergies of cognitive
infocommunications. Acta Polytech Hung 9(1):67–83
102. Csapo A, Baranyi P (2012) A unified terminology for the structure
and semantics of CogInfoCom channels. Acta Polytech Hung
9(1):85–105
103. Csapo A, Israel JH, Belaifa O (2013) Oversketching and associated
audio-based feedback channels for a virtual sketching
application. In: Proceedings of 4th IEEE international conference
on cognitive infocommunications, pp 509–414
104. Biocca F, Kim J, Choi Y (2001) Visual touch in virtual environments:
an exploratory study of presence, multimodal interfaces
and cross-modal sensory illusions. Presence Teleoper Virtual Environ
10(3):247–265
105. Biocca F, Inoue Y, Polinsky H, Lee A, Tang A (2002) Visual cues
and virtual touch: role of visual stimuli and intersensory integration
in cross-modal haptic illusions and the sense of presence. In:
Gouveia F (ed) Proceedings of presence, Porto
106. Hecht D, Reiner M (2009) Sensory dominance in combinations
of audio, visual and haptic stimuli. Exp Brain Res 193:307–314
107. Pavani F, Spence C, Driver J (2000) Visual capture of touch:
out-of-the-body experiences with rubber gloves. Psychol Sci
11(5):353–359
108. Csapo A, Baranyi P (2011) Perceptual interpolation and openended
exploration of auditory icons and earcons. In: 17th international
conference on auditory display, international community
for auditory display, Budapest
109. Csapo A, Baranyi P (2012c) The spiral discovery method: an interpretable
tuning model for CogInfoCom channels. J Adv Comput
Intell Intell Inform 16(2):358–367
123
“The Lord is a refuge for the oppressed, a stronghold in times of trouble.
Those who know your name trust in you, for you, Lord, have never forsaken those
who seek you.”
Psalm 9:9-10
To subscribe to Dan's tips or HotSpot with God Daily Devotional, send a blank
message to dthompson5@xxxxxxxxx and include "subscribe Dan's Tips" or Subscribe
HotSpot with God" in the subject line.