Speech enhancement has recently achieved great success with various
deep learning methods. However, most conventional speech enhancement
systems are trained with supervised methods that impose two
significant challenges. First, a majority of training datasets for speech
enhancement systems are synthetic. When mixing clean speech and
noisy corpora to create the synthetic datasets, domain mismatches
occur between synthetic and real-world recordings of noisy speech
or audio. Second, there is a trade-off between increasing speech
enhancement performance and degrading speech recognition (ASR)
performance. Thus, we propose an unsupervised loss function to
tackle those two problems. Our function is developed by extending
the MixIT loss function with speech recognition embedding and
disentanglement loss. Our results show that the proposed function
effectively improves the speech enhancement performance compared
to a baseline trained in a supervised way on the noisy VoxCeleb
dataset. While fully unsupervised training is unable to exceed the
corresponding baseline, with joint super- and unsupervised training,
the system is able to achieve similar speech quality and better ASR
performance than the best supervised baseline.