Lifelong learning aims to accumulate knowledge and alleviate catastrophic forgetting when learning tasks sequentially. However, existing lifelong language learning methods only focus on the supervised learning setting. Unlabeled data, which can be easily accessed in real-world scenarios, are underexplored. In this paper, we explore a novel setting, semi-supervised lifelong language learning (SSLL), where a model learns sequentially arriving language tasks with both labeled and unlabeled data. We propose an unlabeled data enhanced lifelong learner to explore SSLL. Specially, we dedicate task-specific modules to alleviate catastrophic forgetting and design two modules to exploit unlabeled data: (1) a virtual supervision enhanced task solver is constructed on a teacher-student framework to mine the underlying knowledge from unlabeled data; and (2) a backward augmented learner is built to encourage knowledge transfer from newly arrived unlabeled data to previous tasks. Experimental results on various language tasks demonstrate our model’s effectiveness and superiority over competitive baselines under the new setting SSLL.
Yingxiu Zhao, Yinhe Zheng, Bowen Yu, Zhiliang Tian, Dongkyu Lee, Jian Sun, Yongbin Li, and Nevin L. Zhang. 2022.Semi-Supervised Lifelong Language Learning. InFindings of the Association for Computational Linguistics: EMNLP 2022, pages 3937–3951, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
@inproceedings{zhao-etal-2022-semi, title = "Semi-Supervised Lifelong Language Learning", author = "Zhao, Yingxiu and Zheng, Yinhe and Yu, Bowen and Tian, Zhiliang and Lee, Dongkyu and Sun, Jian and Li, Yongbin and Zhang, Nevin L.", editor = "Goldberg, Yoav and Kozareva, Zornitsa and Zhang, Yue", booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2022", month = dec, year = "2022", address = "Abu Dhabi, United Arab Emirates", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2022.findings-emnlp.290/", doi = "10.18653/v1/2022.findings-emnlp.290", pages = "3937--3951", abstract = "Lifelong learning aims to accumulate knowledge and alleviate catastrophic forgetting when learning tasks sequentially. However, existing lifelong language learning methods only focus on the supervised learning setting. Unlabeled data, which can be easily accessed in real-world scenarios, are underexplored. In this paper, we explore a novel setting, semi-supervised lifelong language learning (SSLL), where a model learns sequentially arriving language tasks with both labeled and unlabeled data. We propose an unlabeled data enhanced lifelong learner to explore SSLL. Specially, we dedicate task-specific modules to alleviate catastrophic forgetting and design two modules to exploit unlabeled data: (1) a virtual supervision enhanced task solver is constructed on a teacher-student framework to mine the underlying knowledge from unlabeled data; and (2) a backward augmented learner is built to encourage knowledge transfer from newly arrived unlabeled data to previous tasks. Experimental results on various language tasks demonstrate our model`s effectiveness and superiority over competitive baselines under the new setting SSLL."}
%0 Conference Proceedings%T Semi-Supervised Lifelong Language Learning%A Zhao, Yingxiu%A Zheng, Yinhe%A Yu, Bowen%A Tian, Zhiliang%A Lee, Dongkyu%A Sun, Jian%A Li, Yongbin%A Zhang, Nevin L.%Y Goldberg, Yoav%Y Kozareva, Zornitsa%Y Zhang, Yue%S Findings of the Association for Computational Linguistics: EMNLP 2022%D 2022%8 December%I Association for Computational Linguistics%C Abu Dhabi, United Arab Emirates%F zhao-etal-2022-semi%X Lifelong learning aims to accumulate knowledge and alleviate catastrophic forgetting when learning tasks sequentially. However, existing lifelong language learning methods only focus on the supervised learning setting. Unlabeled data, which can be easily accessed in real-world scenarios, are underexplored. In this paper, we explore a novel setting, semi-supervised lifelong language learning (SSLL), where a model learns sequentially arriving language tasks with both labeled and unlabeled data. We propose an unlabeled data enhanced lifelong learner to explore SSLL. Specially, we dedicate task-specific modules to alleviate catastrophic forgetting and design two modules to exploit unlabeled data: (1) a virtual supervision enhanced task solver is constructed on a teacher-student framework to mine the underlying knowledge from unlabeled data; and (2) a backward augmented learner is built to encourage knowledge transfer from newly arrived unlabeled data to previous tasks. Experimental results on various language tasks demonstrate our model‘s effectiveness and superiority over competitive baselines under the new setting SSLL.%R 10.18653/v1/2022.findings-emnlp.290%U https://aclanthology.org/2022.findings-emnlp.290/%U https://doi.org/10.18653/v1/2022.findings-emnlp.290%P 3937-3951
Yingxiu Zhao, Yinhe Zheng, Bowen Yu, Zhiliang Tian, Dongkyu Lee, Jian Sun, Yongbin Li, and Nevin L. Zhang. 2022.Semi-Supervised Lifelong Language Learning. InFindings of the Association for Computational Linguistics: EMNLP 2022, pages 3937–3951, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.