Repository files navigation

This is the code for the ACL 2023 paper "Linear Classifier: An Often-Forgotten Baseline for Text Classification". The repository is used to reproduce the experimental results in our paper. If you find our work useful, please consider citing the following paper:

@InProceedings{YCL22a,author ={Yu-Chen Lin and Si-An Chen and Jie-Jyun Liu and Chih-Jen Lin},title ={Linear Classifier: An Often-Forgotten Baseline for Text Classification},booktitle ={Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL)},year ={2023},url ={https://www.csie.ntu.edu.tw/~cjlin/papers/text_classification_baseline/text_classification_baseline.pdf},note ={Short paper}}

Please feel free to contactYu-Chen Lin if you have any questions about the code/paper.

It is optional but highly recommended to create a virtual environment. For example, you can first refer to thelink for the installation guidances of Miniconda and then create a virtual environment as follows.

conda create -n baseline python=3.8conda activate baseline

We recommend usingPython 3.8 because that is the version we used in our experiments. Please also install the following packages.

pip install -r requirements.txtpip install -r requirements_parameter_search.txt

If you have a different version of CUDA, follow the installation instructions for PyTorch LTS on theirwebsite.

You can simply run the following script to generate all the needed data.

bash generate_data.sh

There are three formats of data. For all the formats, we use the same sets as the ones used inLexGLUE. We process the sets toLibMultiLabel format and then modify them to be runnable on LibMultiLabel. We briefly explain what we do as follows.

• linear: We combine training and validation subsets as the new training set, so only training and test sets are available.
• nn: Training, validation, and test sets are available.
• hier (hierarchical): We need to employ the hierarchical setting of BERT to reproduce the results in Chalkidis et al. (2022). Specifically, we add a special symbol to the data and modify the code in LibMultiLabel for conducting the experiments of the hierarchical BERT. Training, validation, and test sets are available.

In Table 2, we present our investigation on two types of approaches: Linear SVM and BERT.

In Table 9, we present additional results from BERT. Note that theOurs setting (BERT) in Table 2 is the same as thetuned setting (BERT in LibMultiLabel) in Table 9.

We show how to conduct the experiments of each method as follows.

bash run_experiments.sh [DATA] [METHOD]

First, you need to determine which data and method you want to try. Then, you should refer to the following lookup table for the value of[DATA] and[METHOD] arguments.

For example, if you want to usethresholding techniques on the setUNFAIR-ToS, you should run the following command.

bash run_experiments.sh unfair_tos thresholding

If you aim to deal with the data setECtHR (B) with theBERT_default setting, you should place the arguments like the following command.

bash run_experiments.sh ecthr_b bert_default

Additional information is shown as follows.

• For theBERT_tuned setting, we have already tuned the parameters for you. The script only runs the experiments using the tuned parameters. The running time will be different from Table 10 because, in Table 10, the time for the parameter search is also included. However, if you want to tune the parameters by yourself, we also provide a script and a search space configuration to do that. Please check the following command.

# Conduct the hyper-parameter searchbash search_params.sh [DATA]# Replace the given tuned configuration with the searched parametersmv runs/[DATA]_bert_tune_XXX/trial_best_params/params.yml config/[DATA]/bert_tuned.yml# Run the BERT_tuned settingbash run_experiments.sh [DATA] bert_tuned

• To conduct theBERT_reproduced method on the data setsECtHR (A),ECtHR (B), andSCOTUS, you need a GPU that includes more than 16GB of GPU memory.

For comparison purposes, we followed Chalkidis et al. (2022) to deal with unlabeled datasets during the evaluation process,though this setting is not a standard practice in multi-label classification, nor is it supported byLibMultiLabel.

For our experimental results, linear methods were run on the CPUIntel Xeon E5-2690, while for BERT we used the GPUNvidia V100. You may notice some minor differences in results between your running of our scripts and our paper results, especially on the BERT results. If you want to fully reproduce our results, you should carefully follow the items below.

• Make sure you install our suggested package version.
• Use the same device as ours.
• Because our BERT results are based on the average results from five runs of different seeds (1,2,3,4,5), you should modifyrun_experiments.sh and follow us to do five runs.