Repository files navigation

The directory is aFlask web application set-up for hosting onPivotal servers. The description of files and folders can be found below:

1. app.py --The file used to start the Flask server.
2. requirements.txt --Containing all Python dependencies of the project.
3. Procfile -- Needed to setup Pivotal.
4. templates --Folder containing HTML/CSS files.
5. Models --Folder containing the saved model.

1. Open theTerminal.
2. Clone the repository by enteringhttps://github.com/devil-cyber/Reddit-Flair-Detection
3. Ensure thatPython3 andpip is installed on the system.
4. Create avirtualenv by executing the following command:virtualenv -p python3 env.
5. Activate theenv virtual environment by executing the follwing command:source env/bin/activate.
6. Enter the cloned repository directory and executepip install -r requirements.txt.
7. Enterpython shell andimport nltk. Executenltk.download('stopwords') and exit the shell.
8. Now, execute the following command:python manage.py runserver and it will point to thelocalhost with the port.
9. Hit theIP Address on a web browser and use the application.

The following dependencies can be found in requirements.txt:

1. praw
2. scikit-learn
3. nltk
4. Flask
5. bs4
6. pandas
7. numpy

Going through various literatures available for text processing and suitable machine learning algorithms for text classification, I based my approach using[2] which described various machine learning models like Naive-Bayes, Linear SVM and Logistic Regression for text classification with code snippets. Along with this, I tried other models like Random Forest Algorithm. I have obtained test accuracies on various scenarios which can be found in the next section.

The approach taken for the task is as follows:

1. Collect 1800 India subreddit data for each of the 15 flairs usingpraw module[1].
2. The data includestitle, comments, body, url, author, score, id, time-created andnumber of comments.
3. Forcomments, only top level comments are considered in dataset and no sub-comments are present.
4. Thetitle, comments andbody are cleaned by removing bad symbols and stopwords usingnltk.
5. Five types of features are considered for the the given task:

a) Titleb) Commentsc) Urlsd) Bodye) Combining Title, Comments, Body and Urls as one feature.

6. The dataset is split into70% train and30% test data usingtrain-test-split ofscikit-learn.
7. The dataset is then converted into aVector andTF-IDF form.
8. Then, the following ML algorithms (usingscikit-learn libraries) are applied on the dataset:

a) Naive-Bayesb) Linear Support Vector Machinec) Logistic Regressiond) Random Forest

9. Training and Testing on the dataset showed theLinear Support Vector Machine showed the best testing accuracy of77.97% when trained on the combination ofTitle + Comments + Body + Url feature.
10. The best model is saved and is used for prediction of the flair from the URL of the post.

The features independently showed a test accuracy near to82% with theURL feature giving the worst accuracies during the training.