Speech and Multimodal Interfaces Laboratory

We received the RSF and the President grants, and the laboratory was awarded the status of a leading scientific school

We received grants from the Russian Science Foundation (RSF):

  • RSF No 21-71-00141 «Research and development of new methods and approaches to the task of automatic sign language recognition»

    Head: Ryumin Dmitry Alexandrovich
    Period: 2021-2023
    The purpose of the project is to develop information, mathematical and software that allows to improve the results available today in the field of automatic recognition of information transmitted by means of a gesture, including sign languages. The system prototype under development will support Russian sign language recognition. The results obtained during the project will be extremely important because it will contribute to improving the quality of life of people with disabilities, in particular, hearing impaired people.


  • RSF No 21-71-00132 «Development and research of End-to-end audio-visual speech recognition system using deep neural networks»

    Head: Ivanko Denis Viktorovich
    Period: 2021-2023
    The main goal of the project is to develop and research an end-to-end system for automatic recognition of audiovisual speech using deep neural networks. The objectives of this project are the development, theoretical and experimental research of mathematical, software and information-linguistic support for an integrated audio-visual speech recognition system. The results obtained in the course of this project will be used for further fundamental research and development in the field of speech technologies, dialogue systems, human-machine interaction and artificial intelligence.


  • RSF No 22-21-00843 «Automatic speech recognition for under-resourced languages of Russia (on the example of the Karelian language)»

    Head: Kipyatkova Irina Sergeevna
    Period: 2022-2023
    This project is aimed at development of a prototype system for automatic speech-to-text conversion for the Karelian language. The developing system can be used for machine translation from Karelian to Russian. In addition, automatic speech recognition systems can be used in automatic stenography systems for under-resourced and endangered languages, being a useful tool for preservation and research.


In addition, we received a Grant of the President and our laboratory was awarded the status of a leading scientific school:

  • Grant of the President: № MK-42.2022.4 «Investigation of the influence of the speaker's emotional state on the automatic audio-visual speech recognition»

    Head: Ivanko Denis Viktorovich
    Period: 2022-2023
    The main goal of the project is a comprehensive study of the influence of various emotional states of the speaker (such as fear, anger, sadness, happiness, etc.) on the accuracy of automatic speech recognition based on audio and video information. The research results will lead to an increase in the efficiency (accuracy and reliability) of modern automatic systems for recognizing emotional speech, based on the processing of audio and video information. It will also fill in the gaps in fundamental knowledge about the impact of different emotional states of a speaker on the accuracy of automatic speech recognition.


  • Scientific school: № NSh-17.2022.1.6 «Software for multimodal analysis of the behavior of participants in virtual communication»

    Head: Karpov Alexey Anatolyevich
    Period: 2022-2023
    The goal of the project is the development and experimental research of mathematical and software for multimodal analysis of the behavior of participants in virtual communication, expressed in the manifestations of emotions and the degree of involvement of participants in communication through audio and video information (facial expressions and gestures), using artificial intelligence methods.