Repository files navigation

This was made as a part of Machine Learning course at IIIT,Delhi.Link to blog

We are using theGTZAN dataset which contains a total of 1000 audio files in .wav format divided into 10 genres. Each genre has 100 songs of 30-sec duration. Along with the audio files, 2 CSV files containing features of the audio files. The files contain mean and variance calculated over multiple features that can be obtained from an audio file. The other file has the same composition, but the songs were divided into 3-second audio files.

We convert every audio file to signals with a sampling rate to analyze its characteristics.Every waveform has its features in ttwo forms:

• Time domain- nothing much information about music quality can be extracted and explored apart from visual distinction in the waveforms
• Frequency domain which we get after fourier transform of two types: Spectral features and Rhythm features

MFCC and Rhythm feature plots provide a matrix based information for the unique features. Both the features have been mapped with the duration of the music file.

After extraction of features, all columns were not null. So extra values were not added.Why is it important to preprocess the data?

• The variables will be transformed to the same scale.
• So that all continuous variables contribute equally
• Such that no biased results
• PCA very sensitive to variances of initial variablesIf large variance range difference between features , the one with larger range will dominate
• The boxplots of each feature shows some features have very large differences in their variances.
• PCA with both normalisation(minMaxScaler) and standardisation(StandardScaler) is done and difference noted.

Feature extraction -> correlation matrix -> PCA

• With 30 secs sample
• With 3 secs sample
• Less outliers/ variance for some classes found in principal components:

• pca.explained_variance_ratio_=[0.20054986 0.13542712]Shows pc1 holds 20% percent of the data, pc2 holds 13% of the data
• Big clusters of metal , rock, pop ,reggae, classical can be seen.
• Jazz ,country, are separable to one extent.
• Hip-hop,disco,blues are very dispersed and can’t be seen
• Majority are easily separable classes
• Decided to proceed to modelling phase by using 3 sec sampled feature set with standardization as it aggregated the genres into more linearly separable clusters than normalisation

This model is a predictive analysis algorithm based on the concept of probability. GridSearchCV was used to pass all combinations of hyperparameters one by one into the model and the best parameters were selected.

Without Hyperparameter tuning:

Using Hyperparameter tuning:

Took SGD as baseline model and performed hyperparameter tuning for a better performance.Though difference werent that great even after HP tuning.

Without Hyperparameter tuning:

With Hyperparameter tuning:

We used a Simple Naive Bayes classifier, one vs Rest Naive Bayes as baseline models.Then used Hyperparameter testing to get better performance.

Without Hyperparameter tuning:

With Hyperparameter tuning:

This model almost outperformed compared to Gaussian NB models.As we can see , after HP tuning , correlation between the features has decreased, some had even 0 correlation.

Without Hyperparameter tuning:

Using hyperparameter tuning :

Best params:{'metric': 'manhattan', 'n_neighbors': 1, 'weights': 'uniform'}

• Took DT as baseline model which didnt give great results, with accuracy around 65%.

• Used ADA boosting which reduced the performance(rock,pop,disco)

• Then gradient boosting which increased the accuracy exponentially.

• CatBoost was having high AUC for all genres unlike gradient which had low accuracy for some genres

• Cat boost outperformed ensemble methods. Gradient boost was close enough with 82% accuracy, rest all were in between 50-60%

• As shown here RF was having around 80% accuracy but XGB boosting reduced the accuracy to 75%

• Cross Gradient Boosting on Random Forest reduced the accuracy , it even reduced precision ,recall to large extent.

• Correlation matrix shows there is very less correlation among variables

• Best performed model among all DT and RF models with every genre was classified with atleast 85+% accuracy
• Genres like classical,hiphop had even 100% accuracy
• XGBoost improves upon the basic Gradient Boosting Method framework through systems optimization and algorithmic enhancements.
• Evaluations

This model is an Artificial Neural Network involving multiple layers and each layer has a considerable number of activation neurons. The primary training included random values of hyperparameters except the activation function . This initiation reflected overfitting in the data for different activation functions :

From the following graph, we choose softplus to be the best activation function, considering softmax to be fixed for output
Upon looking the graph, we can conclude a very high variance in testing and training accuracy and so we know that our model is overfitting. In fact the testing loss starts to increase which indicates a high cross entropy loss. This will be dealt later. For now we see that softplus, relu and sigmoid, all 3 have performed similar on training and testing set thus we will go with softplus since it provides a little less variance than others.

• Learning rate
  activation = softmax
  no. of hidden layers = 3; neurons in each = [512,256,64]
  activation of output layer is fixed to be softmax epochs = 100
  

From the above graphs we see that 0.01 definitely results in over convergence and bounces as reflective from the accuracy graph. 0.001 has a very high variance and loss increases margianally with low acuracy so it isn't appropriate as well.

The best choice for alpha is either 0.0001 or 0.00001.
0.00001 has a relatively low variance and loss converges quickly with epochs but accuracy on training and testing set is pretty low.
0.0001 has a better performance but variance is very high

• no.of hidden layers
  activation = softmax
  learning rate = 0.0001
  activation of output layer is fixed to be softmax epochs = 100
  

In conclusion, increasing or decreasing the number of layers have no effect on variance. This is because we have too many neurons per layer. So we take 3 layers and reduce the number of neurons.

• Number of neurons
  activation = softmax
  learning rate = 0.0001
  number of layers = 3
  activation of output layer is fixed to be softmax epochs = 100
  drop out probability = 0.3
  alpha = 0.001
  

Now for the same neuron set, we apply regularization and neuron dropout to find any change in the variance for high number of neurons with reducing the number of neurons

• regularization and decomposition
  

So in conclusion, if we have high number of neurons per layer, then applying regularization techniques will increase the accuracy and decrease the variance overall. If we do not apply any regularization techniques then we can have moderate number of neurons to have a decent accuracy on training and testing set with low accuracy.

From all our analysis and extra experimentation we conclude our model with following metrics:

• activation : softmax
• learning rate : 0.0001
• number of hidden layers = 3
• number of neurons in each layer = [512,256,128]
• epochs = 100
• regularization and dropout true

Precision on the model : 0.5774000692307671
Recall on the model : 0.583
F1score on the model : 0.5801865223684216
Accuracy on the model : 0.6130000042915345

Even after hyperparameter tuning, the best accuracy is just above 60%. The reason is simply because of overfitting and underperformance due to inability to pick up each feature. This creates amazing accuracy on the training set but always misses out on the testing set.

This model outperformed every other model and gave the best accuracy. Manual hyperparameter tuning was done. Linear, polynomial and RBF kernel were compared using confusion matrix.

• SVMs performed the best among all classifiers with 94% accuracy
• Gaussian outperformed polynomial kernel in almost all iterations
• XGB classifiers were the best among all ensembling methods with 90% accuracy.
• Since genre classes were balanced , the tradeoff between precision and recall was less observed.
• Among all KNN,DT and ensemble classifiers , precision was more than recall
• While in case of LR,SGD,NB,MLP,SVM recall was observed more than precision.