Repository files navigation

This repository contains the architectures, pretrained models, logs, etc pertaining to the SimpNet Paper (Towards Principled Design of Deep Convolutional Networks: Introducing SimpNet) :https://arxiv.org/abs/1802.06205

Major winning Convolutional Neural Networks (CNNs), such as VGGNet, ResNet, DenseNet, etc, include tens to hundreds of millions of parameters, which impose considerable computation and memory overheads. This limits their practical usage in training and optimizing for real-world applications. On the contrary, light-weight architectures, such as SqueezeNet, are being proposed to address this issue. However, they mainly suffer from low accuracy, as they have compromised between the processing power and efficiency. These inefficiencies mostly stem from following an ad-hoc designing procedure. In this work, we discuss and propose several crucial design principles for an efficient architecture design and elaborate intuitions concerning different aspects of the design procedure. Furthermore, we introduce a new layer calledSAF-pooling to improve the generalization power of the network while keeping it simple by choosing best features. Based on such principles, we propose a simple architecture calledSimpNet. We empirically show thatSimpNet provides a good trade-off between the computation/memory efficiency and the accuracy solely based on these primitive but crucial principles. SimpNet outperforms the deeper and more complex architectures such as VGGNet, ResNet, WideResidualNet \etc, on several well-known benchmarks, while having 2 to 25 times fewer number of parameters and operations. We obtain state-of-the-art results (in terms of a balance between the accuracy and the number of involved parameters) on standard datasets, such as CIFAR10, CIFAR100, MNIST and SVHN.

The main contributions of this work are as follows:

1. Introducing several crucial principles for designing deep convolutional architectures, which are backed up by extensive experiments and discussions in comparison with the literature.

2. Based on such principles, It puts under the test the validity of some of the previously considered best practices. such as Strided Convolutions vs MaxPooling, Overlapped Pooling vs Nonoverlapped Pooling, etc. Furthermore, it tries to provide intuitive understanding of each point as to why one should be used instead of the other.

3. A new architecture called SimpNet is proposed to verify the mentioned principles. Based on such design principles, the architecture is allowed to become superior to its predecessor (SimpleNet), while still retaining the same number of parameters and maintaining simplicity in design, while outperforming deeper and more complex architectures (from 2 to 25X), such as Wide Residual Networks, ResNet, FMax, etc., on a series of highly compatative benchmark datasets (e.g., CIFAR10/100, SVHN and MNIST).

If you find SimpleNet useful in your research, please consider citing:

@article{hasanpour2018towards,  title={Towards Principled Design of Deep Convolutional Networks: Introducing SimpNet},  author={Hasanpour, Seyyed Hossein and Rouhani, Mohammad and Fayyaz, Mohsen and Sabokrou, Mohammad and Adeli, Ehsan},  journal={arXiv preprint arXiv:1802.06205},  year={2018}}

(†): Unfinished tests. the results are not finalized and training continues. These models are simply tested without any hyperparameter tuning, only to show how they perform compare to the DeseNet and WRNs. As the prelimnery results show, they outperform both architectures. The full details will be provided after the tests are finished.

• The slim version achieves 1.95% error rate.

• The slim version achives 99.73% accuracy.

As indicated in the paper, CIFAR10/100 use zero-padding and horizontal filipping.The script used for preprocessing CIFAR10/100 can be accessed fromhere

Here is a quick overview of the tests conducted for every principle.
For the complete dicussion and further explanations concerning these experimentsplease read the paper.

Demonstrating how gradually expanding the network helps obtaining better performance. Increasing the depth up to a certain point improves the accuracy (up to 10 layers) and then after that it starts to degrade, indicating PLD issue taking place.

Shallow vs Deep: showing how a gradual increase can yield better performance with fewer number of parameters.

Accuracy for different combinations of kernel sizes and number of network parameters, which demonstrates how correlation preservation can directly affect the overall accuracy.

Different kernel sizes applied on different parts of a network affect the overall performance, the kernel sizes that preserve the correlation the most yield the best accuracy. Also, the correlation is more important in early layers than it is for the later ones.

SqueezeNet test on CIFAR10 vs SimpNet (slim version).

Correlation Preservation: SqueezeNet vs SimpNet on CIFAR10. Byoptimizedwe mean, we added Batch-Normalization to all layers and used the sameoptimization policy we used to train SimpNet.

The effect of using pooling at different layers. Applying pooling earlyin the network adversely affects the performance.

Effect of using strided convolution ((†)) Max-pooling ((*)).Max-pooling outperforms the strided convolution regardless of specificarchitecture. First three rows are tested on CIFAR100 and two last onCIFAR10.

Table [tab:max_perf] demonstrates the performanceand elapsed time when different kernels are used. (3 × 3) has thebest performance among theothers.

Maximum performance utilization using Caffe, cuDNNv6, networks have 1.6Mparameters and the same depth.

Balanced distribution scheme is demonstrated by using two variants ofSimpNet architecture with 10 and 13 layers, each showing how thedifference in allocation results in varying performance and ultimatelyimprovements for the one with balanced distribution ofunits.

The importance of experiment isolation using the same architecture onceusing (3 × 3) and then using (5 × 5)kernels.

Wrong interpretation of results when experiments are not compared inequal conditions (Experimentalisolation).

UsingSAF-pooling operation improves architecture performance. Testsare run on CIFAR10.