Applications

Application domains

We are particularly interested in the following application domains:

• Supporting sign language learners
• Improving accessibility of public services and information for deaf people
• Making the linguistic and cultural heritage of deaf communities visible and searchable
• Reducing communication barriers between deaf and hearing people

Supporting sign language learners

Most deaf children (95%) have hearing parents, and only a small percentage of these parents (in the Netherlands, around 10%) manage to learn sign language. As a consequence, during the first years of their lives, many deaf children receive little language input that is accessible to them, which in turn negatively affects linguistic and cognitive development as well as mental well-being throughout the lifespan. Therefore, it is very important to support parents of deaf children in learning sign language. Together with AURIS, an organisation which provides education for deaf children and support-programmes for their families, we are currently developing an app to support parents in learning NGT. In addition, for the platform www.gebarentaalvooronzekinderen.nl. we have made a compilation of scientific work that provides crucial information about language choices, cochlear implantation, relevant legislation, educational choices, and other major decisions that parents of deaf children have to make.

Another important group of sign language learners are sign language interpreters, both during their academic training and in their professional practice. We regularly provide professional development training sessions for sign language interpreters. An example can be found here.

Moreover, in collaboration with Prowise, a company providing digital education platforms, we are working on an online adaptive learning environment for children in primary school to learn the basics of NGT.

Improving accessibility of public services and information for sign language users

Evidently, most deaf people have no sensory access to spoken language and/or cannot or prefer not to produce speech. What is perhaps less evident is that written texts (e.g., captions, educational materials, healthcare information) are difficult to process for many deaf people as well. NGT is the preferred language for most deaf adults in the Netherlands, while Dutch is like a foreign language they are less comfortable with. 68% of deaf adults in the Netherlands are low literate in Dutch. This means that many deaf people have limited access to public information and services, including healthcare and education, with far-reaching consequences throughout the lifespan, including high rates of unemployment and health issues. For instance, in 2017 only 53.3% of deaf people in the US were employed, compared to 75.8% of hearing people. And in a Norwegian national well-being survey, 21% of deaf people (n ≈ 400) indicated that they felt hopeless, compared to 4% of hearing people (n ≈ 40000). It is important to address these issues.

In 2022, we did a pilot project with NS, the national railway company, to automatically translate railway travel announcements from Dutch to NGT using a signing avatar. In the future, we intend to substantially improve this first prototype system and to collaborate with other organizations as well to make public services and information accessible in NGT.

Making the linguistic and cultural heritage of deaf communities visible and searchable

The linguistic and cultural heritage of deaf communities is not systematically documented and preserved. We have coordinated the production of the documentary We were there... we are here, about the life stories of elderly signers, including their involvement in major events of the history of Europe. The interviews in the documentary are in Catalan Sign Language (LSC), French Sign Language (LSF), German Sign Language (DGS), Israeli Sign Language (ISL), Italian Sign Language (LIS), Sign Language of the Netherlands (NGT), Spanish Sign Language (LSE), and Turkish Sign Language (TİD).

To the extent that the linguistic and cultural heritage of deaf communities is documented on video, it is not easy for people who are interested in this heritage to find the video recordings that they are looking for and to search within these recordings, because search engines only work for videos with textual transcriptions. Manually transcribing sign language videos is incredibly time-consuming and not scalable. We are therefore developing tools that automatically transcribe sign language videos, in order to make them searchable.

Reducing communication barriers between deaf and hearing people

Reducing communication barriers is, technologically, by far the most challenging application domain, because communication involves two-way interaction with open-ended vocabulary, complex grammatical constructions, implicit meaning, and reference to the physical context in which the conversation takes place. We do not believe that machine translation can be applied in this domain in the foreseeable future. However, we still think that we can contribute in this area. One major problem is the huge shortage of sign language interpreters. This is partly a logistic problem. Often the need for an interpreter arises on short notice. But interpreters typically need to be booked weeks in advance, and need time to travel from one assignment to the next. We therefore want to develop a remote interpreter service that makes use of augmented reality technology. We will be working on this over the next couple of years within the TACIT project, in collaboration with Berengroep (the organisation responsible for interpreter services in the Netherlands), Hogeschool Utrecht (where sign language interpreters are trained), the Nederlands Gebarencentrum, and the Interactive Systems Group at CWI, which specializes in augmented reality technology.