Why Do Masked Neural Language Models Still Need Semantic Knowledge in Question Answering?

Abstract

Pre-trained language models have widely been used to solve various natural language processing tasks. Especially, masked neural language models, which are composed of huge neural networks that are trained to restore the masked tokens, have shown outstanding performance in many tasks including text classification and question answering. However, it is challenging to identify what knowledge are trained inside due to the ‘black box’ nature of deep neural networks with numerous and intermingled parameters. There have been recent studies that try to approximate how much knowledge is learned in masked neural language models. However, a recent study reveals that the models do not precisely understand semantic knowledge while they show superhuman performance. In this work, we empirically verify that questions that require semantic knowledge are still challenging for masked neural language models to solve in question answering. Therefore, we suggest a possible solution that injects semantic knowledge from external repositories into masked neural language models.