Repository files navigation

Multiple Different Natural Language Processing Tasks in a Single Deep Model. This, in my opinion, is indeed a very goodpaper. It demonstrates how a neural model can be trained from low-level to higher level in a fashion such that lower layerscorrespond to word-level tasks and the higher layers correspond to tasks which are performed at sentence level.The authors also show how to retain the information at lower layers while training the higher layers by successive regularization.It is also clearly shown that transfer learning is possible where different datasets are exploited simultaneously after jointlypre-trained for word embeddings. Catastrophic inference is a very crucial thing to deal with in this mode.It is basically the inference in other layer's learned parameters while training a particular layer. As an example, you want to retain information about POS while training for, say, chunking later!

• Conll2000 (http://www.cnts.ua.ac.be/conll2000/chunking/)
• SICK data (http://clic.cimec.unitn.it/composes/sick.html)

• POS Tagging (word-level)
• Chunking (word-level)
• Semantic Relatedness (sentence-level)
• Textual Entailment (sentence-level)

data.py - Preprocesses data for the model

run.py - Runs the main model.

task_desc= {'pos':'this has increased the risk','chunk':'this has increased the risk','relatedness': ['two dogs are wrestling and hugging','there is no dog wrestling and hugging'],'entailment': ['Two dogs are wrestling and hugging','There is no dog wrestling and hugging']}

The original paper contains one more task which is dependency parsing. Currently, that is not incorporated in the model due tonon-availability of good public data. Also need to add successive regularization.

A Joint Many-Task Model: Growing a Neural Network for Multiple NLP TasksKazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, Richard Socher

https://arxiv.org/abs/1611.01587