Researchers at the University of the Western Cape in South Africa are on the brink of a breakthrough in developing mobile applications that will revolutionise the way Deaf and hearing people communicate.
While the race is on around the world to develop innovative technology applications to enable Deaf and hearing-impaired people to communicate more effectively with hearing people, researchers at UWC are leading the pack – and are also pioneering technology that is tailored especially for users in the developing world.
Researchers at the Telkom/Cisco/Aria Technologies/THRIP Centre of Excellence for IP and Internet Computing, including scientists from Bridging Application and Network Gaps (BANG) and Integration of Signed and Verbal Communication: South African Sign Language Recognition, Animation and Translation (SASL) groups in the UWC computer science department, are working on several interconnected projects and programmes focusing on cellphone or mobile applications with the ability to translate spoken language into sign language and vice versa. The computer science department uses the capital D for Deaf to denote a cultural grouping, one that primarily uses South African Sign Language to communicate.
Estimates vary but it is believed that there are up to 1.5 million people with hearing disabilities in South Africa, a significant part of the population that is frequently overlooked as a potentially economically active workforce. Head of the Computer Science Department Associate Professor Isabella Venter says more than 45 postgraduate students are currently working on projects relating to technology applications. She highlights a video application called SignSupport for cellphones, in which pre-recorded videos for specific situations are loaded onto the phones, enabling Deaf people to interact with, for example, a pharmacist and to get medicine without the help of an interpreter or third person.
A key reason for the success of these projects, Professor Venter believes, is that they are inclusive – involving large industry partners and small NGOs as well as active engagement with ordinary South Africans with hearing impairments.
BANG leader and UWC Associate Professor in Computer Science William Tucker says that their research and collaboration with a local Deaf community has helped uncover the real needs of Deaf people. One of the big misconceptions about Deaf people is that they are able to read and write. But many Deaf people are in fact, functionally illiterate and unable to read, literate only in sign language. “Scientists have for years been building things for Deaf people that they don’t need. Often it was a case of computer geeks looking for problems they could think up cool solutions for, but which weren’t necessarily helpful to people in the real world,” he says.
Many Deaf people are also under-employed, poor, and unable to read lips or afford cochlear implants. “Many technology applications for Deaf people are based on text, but SignSupport is video-based to make it more accessible. So far, the Deaf people we have involved in the research think it’s great,” says Professor Tucker. SignSupport will be taken to the next level in 2014, when it is implemented at selected pharmacies following permission from the Department of Health.
In parallel with this, the SASL group led by James Connan and Mehrdad Ghaziasgar is working on a system to translate between South African Sign Language and English. Research has focused on recognising sign language by breaking gestures down into their various components, including facial expression recognition, hand shape estimation, hand tracking and various others. Work has also been done on rendering sign language using 3D avatars. This work will eventually be integrated into a fully-fledged translation system that will allow Deaf and hearing users to communicate using mobile phones.
The hearing user will use the camera on their phone to record a person using sign language and the phone will translate and play back the English speech. In the reverse, the Deaf user will use the microphone on the mobile phone to record what is being said in English and the sign language will be rendered by a 3D avatar on the display of the phone.
Microsoft research recently made headlines when they announced a similar system for Chinese Sign Language. The primary difference between their system and the local system is that the Microsoft system is based around their proprietary Kinect sensor hardware, while the local system is built using generic mobile phone technology.
Natural Sciences Faculty Dean Michael Davies-Coleman is very proud of the work done by the computer science department. “One of the core functions of UWC is to be an engaged university – engaged with society and committed to meaningful engagement with communities.” He says these flagship programmes show how scientists can engage directly with society and improve lives.
The mobile technology applications have been designed in such a way that they can be used by people in other countries and enable oral recordings as well, making it internationally competitive – but it is right here in the developing world where the technologies stand to have the greatest impact. “Cellphones have become ubiquitous, even in poor communities. So this has implications for especially rural areas, where people have amazing phones but where literacy levels are low. As one goes north on the continent, this will probably only become more pronounced,” says Professor Tucker.
He says that this is where the UWC has real potential to make a difference. While considerable work is being done all over the world regarding technology applications for Deaf people, there is very little for people with disabilities in developing regions. “We have become known world-wide for the work we are doing now,” acknowledges Professor Tucker.
Professor Venter says their research has already piqued the interest of American researchers who are keen to use the UWC-developed technology in their work.
In conclusion, she believes: “Alternative forms of communication as demonstrated by these projects can make a real and measurable difference for individuals and more broadly for economies and countries.”
For more information about particular research projects referred to in this article.