Part of the book series:Lecture Notes in Computer Science ((LNCS,volume 13734))
Included in the following conference series:
Abstract
Commonsense question answering has always been a challenging task due to the wide-domain coverage and the implicity of commonsense knowledge. Few works are tackling the answer generation of commonsense questions, which is more difficult than multiple-choice. This motivates us to delve into the answer generation ability of pre-trained language models (PLMs). Other than utilizing knowledge bases to extract commonsense-related knowledge to answer commonsense questions, we exploit the latent knowledge within PLMs to solve this task. In this work, we reformulate this generative task into a masked token prediction task and experiment with masked language models (MLMs) and generative language models (GLMs). Experimental results on the ProtoQA dataset demonstrate the effectiveness of our proposed method. Our work finds that both MLMs and GLMs are good at masked token prediction and that PLMs have acquired commonsense knowledge through large-corpus pre-training.
This is a preview of subscription content,log in via an institution to check access.
Access this chapter
Subscribe and save
- Get 10 units per month
- Download Article/Chapter or eBook
- 1 Unit = 1 Article or 1 Chapter
- Cancel anytime
Buy Now
- Chapter
- JPY 3498
- Price includes VAT (Japan)
- eBook
- JPY 6291
- Price includes VAT (Japan)
- Softcover Book
- JPY 7864
- Price includes VAT (Japan)
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Boratko, M., Li, X., O’Gorman, T., Das, R., Le, D., McCallum, A.: ProtoQA: A question answering dataset for prototypical common-sense reasoning. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1122–1136. Association for Computational Linguistics, Online, November 2020.https://aclanthology.org/2020.emnlp-main.85
Chang, H.S., McCallum, A.: Softmax bottleneck makes language models unable to represent multi-mode word distributions. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8048–8073. Association for Computational Linguistics, Dublin, Ireland, May 2022.https://aclanthology.org/2022.acl-long.554
Davison, J., Feldman, J., Rush, A.: Commonsense knowledge mining from pretrained models. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 1173–1178. Association for Computational Linguistics, Hong Kong, China, November 2019.https://aclanthology.org/D19-1109
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186 (2019)
Kuhn, H.W.: The hungarian method for the assignment problem. Naval Res. Logist. Quart.2(1–2), 83–97 (1955)
Li, X.L., Liang, P.: Prefix-tuning: optimizing continuous prompts for generation. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4582–4597. Association for Computational Linguistics, Online, August 2021.https://aclanthology.org/2021.acl-long.353
Ma, K., Ilievski, F., Francis, J., Ozaki, S., Nyberg, E., Oltramari, A.: Exploring strategies for generalizable commonsense reasoning with pre-trained models. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5474–5483. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, November 2021.https://aclanthology.org/2021.emnlp-main.445
Munkres, J.: Algorithms for the assignment and transportation problems. J. Soc. Ind. Appl. Math.5(1), 32–38 (1957)
Petroni, F., et al.: Language models as knowledge bases? In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2463–2473. Association for Computational Linguistics, Hong Kong, China (Nov 2019),https://aclanthology.org/D19-1250
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.: Language models are unsupervised multitask learners. OpenAI Blog1(8), 9 (2019)
Raffel, C., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res.21(140), 1–67 (2020).http://jmlr.org/papers/v21/20-074.html
Sap, M., Rashkin, H., Chen, D., Le Bras, R., Choi, Y.: Social IQa: commonsense reasoning about social interactions. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pp. 4463–4473. Association for Computational Linguistics, Hong Kong, China, November 2019.https://aclanthology.org/D19-1454
Shin, T., Razeghi, Y., Logan IV, R.L., Wallace, E., Singh, S.: AutoPrompt: eliciting knowledge from language models with automatically generated prompts. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4222–4235. Association for Computational Linguistics, Online, November 2020.https://aclanthology.org/2020.emnlp-main.346
Talmor, A., Herzig, J., Lourie, N., Berant, J.: Commonsenseqa: a question answering challenge targeting commonsense knowledge. arXiv preprintarXiv:1811.00937 (2018)
Acknowledgement
This work was partially supported by the National Natural Science Foundation of China (62006062, 62176076), Shenzhen Key Technology Project JSGG20210802154400001.
Author information
Authors and Affiliations
Harbin Institute of Technology (Shenzhen), Shenzhen, China
Xuan Luo, Yihui Li & Ruifeng Xu
- Xuan Luo
You can also search for this author inPubMed Google Scholar
- Yihui Li
You can also search for this author inPubMed Google Scholar
- Ruifeng Xu
You can also search for this author inPubMed Google Scholar
Corresponding author
Correspondence toRuifeng Xu.
Editor information
Editors and Affiliations
Tsinghua University, Shenzhen, Tsinghua, China
Yujiu Yang
University of Science and Technology Beijing, Beijing, China
Xiaohui Wang
Kingdee International Software Group Co.,Ltd, Shenzhen, China
Liang-Jie Zhang
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Luo, X., Li, Y., Xu, R. (2022). Solving a Cloze Test for Generative Commonsense Question Answering. In: Yang, Y., Wang, X., Zhang, LJ. (eds) Cognitive Computing – ICCC 2022. ICCC 2022. Lecture Notes in Computer Science, vol 13734. Springer, Cham. https://doi.org/10.1007/978-3-031-23585-6_2
Download citation
Published:
Publisher Name:Springer, Cham
Print ISBN:978-3-031-23584-9
Online ISBN:978-3-031-23585-6
eBook Packages:Computer ScienceComputer Science (R0)
Share this paper
Anyone you share the following link with will be able to read this content:
Sorry, a shareable link is not currently available for this article.
Provided by the Springer Nature SharedIt content-sharing initiative