基于多模态生成对抗网络和三元组损失的说话人识别

为了挖掘说话人识别领域中人脸和语音的相关性，该文设计多模态生成对抗网络(GAN)，将人脸特征和语音特征映射到联系更加紧密的公共空间，随后利用3元组损失对两个模态的联系进一步约束，拉近相同个体跨模态样本的特征距离，拉远不同个体跨模态样本的特征距离。最后通过计算公共空间特征的跨模态余弦距离判断人脸和语音是否匹配，并使用Softmax识别说话人身份。实验结果表明，该方法能有效地提升说话人识别准确率。

Speaker Recognition Based on Multimodal GenerativeAdversarial Nets with Triplet-loss

In order to explore the correlation between face and audio in the field of speaker recognition, a novel multimodal Generative Adversarial Network (GAN) is designed to map face features and audio features to a more closely connected common space. Then the Triplet-loss is used to constrain further the relationship between the two modals, with which the intra-class distance of the two modals is narrowed, and the inter-class distance of the two modals is extended. Finally, the cosine distance of the common space features of the two modals is calculated to judge whether the face and the voice are matched, and Softmax is used to recognize the speaker identity. Experimental results show that this method can effectively improve the accuracy of speaker recognition.