This is very interesting.
Sent: Wednesday, March 13, 2019 1:19 PM
To: Undisclosed-Recipient:;
Subject: Microsoft’s App For The Blind Called A Kid ‘Contemptuous.’ Here’s Why
That Matters.
Microsoft’s App For The Blind Called A Kid ‘Contemptuous.’ Here’s Why That
Matters.
Saqib Shaikh, one of Microsoft’s lead engineers behind Seeing AI.COURTESY OF
MICROSOFT figure
Microsoft's lead engineer on Seeing AI, Saqib Shaikh.
Saqib Shaikh, one of Microsoft’s lead engineers behind Seeing AI.COURTESY OF
MICROSOFT figure end
My neighbor Katina does not fit the description of an early adopter of
technology.
She is a 75-year-old sculptor and for much of her adulthood has been completely
blind. But technology sets the rhythm for her life. A handheld glucose
monitor reads a patch on her arm to manage her diabetes. Classical music plays
through her Google Home speaker. She navigates her iPhone, its screen always
blank, with a dizzying flurry of taps and swipes.
One day she sits down to show me a new app she is trying out, called Seeing AI.
Launched in July 2017 by Microsoft for people with impaired sight, it uses
the company’s vision-recognition software, a form of artificial intelligence
that it licenses to other businesses, to narrate the world through her phone’s
camera.
Eager to try it out myself, I use the app to take a photo of Katina. A robotic
male voice says, “Forty-five-year old woman with brown hair smiling.”
“I like this!” she says, clearly pleased at the age it has given her. My
8-year-old daughter is nearby. Try making a funny face, I say. When she does,
the app has this to report: “Six-year-old girl with brown hair looking
contemptuous.”
We can’t help but laugh. Later though, the app’s pronouncement gives me pause.
Its last judgment wasn’t exactly wrong, since my daughter had stuck her
tongue out. But this was a bold word for a computer to pick.
Within a few weeks I am speaking to Saqib Shaikh, one of Microsoft’s lead
engineers behind Seeing AI, who has also been blind since childhood. How had
his app picked a description like “contemptuous?” It turns out this came from
the data that Microsoft’s algorithms were trained on.
To build the system, researchers from Microsoft in Cambridge, U.K.,
had decided on a set of eight emotions
that spanned ethnicities and cultures, including sadness, anger, fear and
contempt, he explains. Microsoft
announced
that research in November 2015, when it released an emotion-recognition tool
that other software developers could use for building apps.
Happy or contemptuous? Chris Bishop, head of Microsoft Research Cambridge at a
conference in November 2015, demonstrates an early iteration of the software
that Seeing AI uses to recognize emotions. IMAGE VIA MICROSOFT figure
Chris Bishop, head of Microsoft Research Cambridge at a conference in November
2015
Happy or contemptuous? Chris Bishop, head of Microsoft Research Cambridge at a
conference in November 2015, demonstrates an early iteration of the software
that Seeing AI uses to recognize emotions. IMAGE VIA MICROSOFT figure end
“The scientists came up with a list of the things they thought were important,
like gender and age and hair color and emotion,” Shaikh says. Then a team
of humans classified thousands of images with those descriptions to teach the
software.
And it turns out, he adds, that same research underpins the more mainstream
vision-recognition software that Microsoft now sells to businesses under the
name Cognitive Services. A business intelligence firm called Prism Skylabs, for
instance,
has used it to help retailers
monitor video feeds to see when their delivery trucks have arrived.
Though Seeing AI has a small audience of users, its evolution points to how the
rest of us might use AI-powered technology like vision recognition in the
future. Already much of the tech that people use today, from the computer mouse
to text-to-speech software to predictive text, even the typewriter,
has its roots in disability research.
Audible arguably owes its existence to blind people, and Nuance, the voice
technology company once used to power Apple’s Siri, was born from disability
research, executives from that company told me. Necessity is the mother of
invention, and the deaf and visually impaired are technology’s unlikely
pioneers.
“We have evolved to process speech and language,” says Tony Allen, a former
lecturer at Nottingham Trent University who was working on speech-enabled
systems
and automated online-form filling for people with disabilities as early as
1998, predating Amazon’s Alexa by 16 years. “We need to adapt computers to us
rather than adapting us to computers.” Developing tech for people who are
blind, deaf or paralyzed can help push research further in that direction, he
adds.
Today’s souped-up hearing aids are already finding pockets of mainstream
adoption. Apple’s wireless AirPods have a feature for people who are hard of
hearing
called Live Listen, which lets them use an iPhone microphone to hear
conversations in a noisy room. People with perfectly good hearing have
exploited the feature for covert listening.
Starkey Hearing Technologies started in 1970 as a hearing-aid manufacturer in
Eden Prairie, Minnesota. Now it’s launching what it calls the world’s first
smart hearing aid, a sleek and tiny device that tracks steps, detects falls and
can translate more than 20 languages. Adding all those extra features “is
how hearing technologies will break into the mainstream,” says Starkey’s
technical director, Paul Lamb, who is a hearing aid user.
Katina uses Seeing AI to read handwritten cards and to count the change in her
purse. Robin Spinks, who is partially sighted, uses the app to read the
menu when he visits a restaurant. “I do it really discreetly so I don’t look
different from anyone else,” Spinks says, adding that the app reads directly
to an Apple AirPod in his ear. “It’s an incredibly normalizing effect.”
Another blind person has used the app to narrate subtitles from a foreign
language movie, while another points his phone at the TV during soccer matches
so it can read him the latest score, Shaikh says. Shaikh hopes to one day use
the app to recognize landmarks while he walks down the street, like shops
or new construction.
Shaikh, who is 37 and based in London, helped spearhead Seeing AI in 2014
during a gathering of Microsoft engineers.
Eventually they used Microsoft’s deep-learning software, a popular approach to
AI that uses layers of data to help predict an outcome. They added their
own training data to the system, taking thousands of photographs of $10 bills
so the app could recognize money and currencies.
A recently added feature allows users to swipe their fingers over a photo so
they can “touch” the edges of an object in the image like a table or a plant.
They can now sense the edges through tiny vibrations, known as haptic feedback,
and sounds.
But there are meatier ways that Shaikh could update his app.
I ask Katina if there are certain other details that she would like to know
about people she meets, narrated to her privately. “Cleanliness,” she answers
straightaway. “I find that extremely important. Whether a person looks groomed.
Well cared. They might have patched or torn clothes, but be clean.”
Spinks wishes for richer detail on the history of art, people or architecture
around him. He describes these visual clues as things that sighted people
take for granted. “The number one thing [blind] people want is parity with
fully sighted people,” says Spinks, who also works for the Royal National
Institute
of Blind people in the U.K.
A colleague of Spinks’ who is blind said he’d like an app that discreetly tells
him if someone is attractive or not. “That’s a genuine request,” says Spinks.
Would Microsoft imbue its AI with such an ability? What about someone’s race or
body shape?
“In general we believe that we should provide information that lets our
customers make judgments,” Shaikh says. “I don’t think it’s … you don’t want an
app to be making the judgments for you, because these are very personal
things.”
That’s a noble sentiment, but Seeing AI is already making personal judgements
about age and emotion. And while Shaikh says users can contact Microsoft
with corrections, many will simply trust what the algorithm tells them. Shaikh
doesn’t rule out the system identifying attractiveness or race down the
line.
If it does, that could bring blind people closer to the parity that Spinks
hopes for, but it could also throw up new complications around prejudice when
such technology goes mainstream. During our interview, Katina starts to walk
back her request for insight into tidiness when she recalls an encounter with
a man in her local park.
“We got into conversation, and it was perfectly nice and normal,” she says. As
they said their goodbyes, a nearby friend revealed that Katina wouldn’t
have spoken to the man if she could have seen what he looked like. “He was
filthy,” the friend said.
“But he was nice. Perfectly nice,” Katina remembers. “That taught me a lesson.”
