Viterbi Faculty of Electrical Engineering, Technion
Few-Shot Learning Neural Network for Audio-Visual Speech Enhancement
Speech enhancement aims to improve speech quality and intelligibility when audio is recorded in noisy environments. Audio-visual speech enhancement models get a noisy audio input with the corresponding video frames, and produce an enhanced audio signal of the target speaker. A major drawback of existing audio-visual speech enhancement methods is speaker dependency, which entails sufficient training data of the target speaker. Speaker dependency prevents speech enhancement models from performing in real-time applications, where a large training set of the target speaker cannot be guaranteed.
In this talk, we address the problem of speaker dependency. We consider the realistic scenario when only a small number of training samples of the target speaker are available during model training. We show that this scenario resembles the task of few-shot learning in image classification. In order to overcome the problem of speaker dependency, we propose a fast adaptation speech enhancement (FASE) model. Our state-of-the-art visual speech enhancement neural network is inspired by meta-learning approaches, originally developed for the task of few-shot learning in image classification.
Our FASE model avoids speaker dependency and outperforms previous models in both quality and intelligibility measures when the number of training samples of the target speaker is small. The model also demonstrates an improvement in computational performance, which implies its potential applications in real-time and mobile systems.
*M.Sc. student under the supervision of Prof. Israel Cohen.
Maya Rapaport received her B.Sc. in electrical engineering from the Technion Israel Institute of Technology, in 2018. She has been with the Artificial Intelligence Products Group and the Computer Vision Group at Intel, before she moved to her current position as Computer Vision and Deep Learning Algorithm Engineer at Rafael. Among her volunteering activities are Paamonim organization branch manager, 8200Bio community organizer, chairperson of the EE students committee and coordinator of the SP&S seminars.
Online Zoom meeting link: https://technion.zoom.us/j/96390522948
Wed 24 Feb 2021
Start Time: 14:30
End Time: 15:30
Zoom meeting | Electrical Eng. Building