FreeLists / ossrp-control / [ossrp-control] The Longhorn Screen Reader Idea

["Will Pearson" <will-pearson@xxxxxxxxxxxxx>]

Hi,
As promised, here's more details on the Longhorn screen reader idea.

For those of you that are not ware of what a screen reader is, the basic idea 
is it reads any text that is displayed on a screen and turns it into speech 
output for use by someone who cannot see well enough to read the screen.

Why have we considered targeting Windows Longhorn?  As some of you will be 
aware Microsoft are bringing out a new version of their Windows operating 
system, codenamed Longhorn, next year.  This looks to be an exciting 
opportunity for access technology, but unfortunately also has it's drawbacks.  
Firstly, there's a lot of opportunities that are present in Longhorn that are 
not present in earlier versions of Windows, and I've listed a few below:
1. User Interface Automation (UIA)
UIA is the replacement for Microsoft Active Accessibility, or MSAA, that is 
found in current versions of Windows.  UIA, however, goes beyond MSAA and 
offers a few key benefits over MSAA.  Firstly, UIA has two purposes, not only 
does it provide accessibility information, but that accessibility information 
can be used in automated testing of software.  Hopefully, this dual-rolling of 
UIA will make it something that's common place in Windows applications, as 
automated testing is popular within the software industry, and therefore it 
should mean that more software will be accessible out of the box than with 
prior versions of Windows.
2. Longhorn will use vectors to draw most of it's graphics.  For me, this is an 
interesting point, and could mean diagrams become reasonably accessible.  At 
the moment, images and diagrams are just drawn using pixels, of certain 
colours, which are then arranged in a grid to form shapes.  With pixels, it's 
very hard to determine what shape something represents, and therefore give 
anything but unreliable and patchy access to graphics.  However, vectors change 
all this.  Vectors are just points in space, but importantly you can get access 
to the numbers that represent those points in space.  For example, if a line 
went from co-ordinate 0,0 to 10,10 you could determine where that line began, 
ended, and it's direction.  This tells you something about that line and it's 
role it plays in the overall image that you can then combine with details of 
other lines to form a mathematical model of how a shape is composed.  You can 
then compare this mathematical model to a database to put a name to that shape, 
thereby converting images to text, and that's what makes Longhorn's use of 
vectors exciting.
Unfortunately, some of the techniques used to create screen readers in earlier 
versions of Windows are either difficult to do or not viable options under 
Longhorn, making it difficult, although not impossible, to create a screen 
reader that will run on many different versions of Windows, something that will 
become easier after Longhorn has been released, as Microsoft plan to make 
Longhorn components, such as UIA, Indigo and Avalon, available for Windows XP 
and Server 2003.

So, what could be done to improve the basic screen reader experience?  There's 
been some ideas on this from various sources over the past few years, all of 
which are technically possible now.  Probably the biggest suggestion is to have 
output where you get to "see" more than one window at once, bringing screen 
reader users into line with how sighted people use computers.  The basic 
concept is to display the corners of the various windows, buttons, etc using 
either sound or tactile shapes.  Research exists demonstrating that it's 
possible to draw shapes using sound, just as it is to draw shapes using pen and 
paper, think of the sound as being equivalent to the ink of a pen.  So, all 
that would be needed to indicate four corners of a window would be four sets of 
two lines, both at 90º to one another, indicating the sides and corners of a 
window.  Displaying more than one window would allow a user to explore the 
spatial positioning of windows, and the spatial relationships between two or 
more windows, which is currently information lost to a screen reader user using 
any of the currently available packages.  However, this information is used a 
lot in conveying extra meaning about what action a button might perform, or the 
relevance of a specific window in the scope of the wider display.  Not only 
does it convey meaning, but spatial positioning and spatial relationships are 
necessary in performing drag and drop and point and click operations, something 
that it is possible for blind people to do using a spatialised display.  
Another benefit to be derived from this sort of display would be the user's 
ability to select what to read based on it's spatial location, which is 
something that makes sight so quick and easy.  It could be used to have skim 
reading that correctly mimics that found in sighted behaviour, it could be used 
to allow a user to quickly move to a specific window without having to tab 
through twenty other windows to get there, and more importantly, there wouldn't 
need to be a keyboard tab sequence, which should make those pieces of software 
that currently pose problems for keyboard users accessible.

Of course, this sort of display won't be practical for everyone, nor will it be 
practical all of the time, therefore a conventional focus based interface would 
also need to be created, but the benefits from a spatialised display look 
promising enough for it to be investigated.

That's just a few of the ideas that have been floated around regarding a 
potential screen reader.  As always, you have as much influence as we do, and 
questions, comments and suggestions are welcome.

Will