Researchers of St. Petersburg Federal Research Center of the Russian Academy of Sciences (SPC RAS) have developed a software package able based on intelligent video analysis to recognize the sign language of people suffering hearing impairments and converting it into text. The application is planned to be used by medical institutions in order to improve the effectiveness of medical care for deaf patients. The development will contribute to improving the quality of life of people with disabilities; results of the study are published in the Scientific and Technical Journal of Information Technologies, Mechanics and Optics.

According to the World Health Organization, by 2021 approximately 466 million people in the world (over 5% of the total population of the globe, including 34 million children) are suffering the complete deafness or experiencing hearing problems. Every third person over 65 faces the problem of reduced hearing quality. Moreover, experts estimate that by 2050, almost 2 billion people are expected to suffer deafness or experience hearing problems.

People with hearing impairment may experience significant difficulties when visiting various government agencies, shopping malls or hospitals. Sometimes deaf citizens are provided with sign language interpreters, however, in practice their capabilities are insufficient. This is why, the development of intelligent systems for efficient automatic translation of machine sign language to facilitate communication between deaf people and the rest of society is becoming an urgent scientific direction.

“Representatives of a certain organization engaged in the rehabilitation of deaf people approached us regarding the problem of their visits to hospitals. For example, in order to explain the illness in detail, a deaf patient is forced to invite an interpreter, and the latter are in great demand and not always available. So, to solve this problem, we have developed software that recognizes sign language and converts it into text,” says Dmitry Ryumin, Candidate of Technical Sciences, Senior Researcher at the Laboratory of Speech and Multimodal Interfaces, SPC RAS.

To train the neural network, scientists have developed a unique database of gestures used by deaf people while describing areas related to medicine and health. It comprises 85 gestural utterances recorded by 12 speakers in five versions. At the same time, video recordings of all gestures are of different foreshortenings, what increases their information value. The total duration of all the videos was about seven hours.

The software works as follows: the video camera captures the patient’s gestures. Scientists have found out that the same gesture elements can vary significantly depending on the patient’s age and gender, so the system accounts for this factor to improve the interpretation accuracy. Then the program automatically converts the sign language into text in real time. The system accuracy exceeds 90%.

“We hope that in not too distant future our application will simplify the delivery of medical services both for doctors and people with disabilities. Currently we work on developing a user–friendly interface for the program,” explains Dmitry Ryumin.

The project is supported by the RSF Grant (No. 21-71-00141). The representatives of the Interregional Rehabilitation Center for People with Hearing Problems of the city of Pavlovsk have been engaged in this research.

The development of software products to improve the quality of life of deaf people is an important area in the work of the Laboratory of Speech and Multimodal Interfaces at SPC RAS. Earlier the scientists have developed a prototype of a special robotic trolley that allows the deaf to navigate supermarkets and shopping malls.