Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up

🔪 Elimination based Lightweight Neural Net with Pretrained Weights

License

NotificationsYou must be signed in to change notification settings

snoop2head/ElimNet

Repository files navigation

Wandb Log

ElimNet: Eliminating Layers in a Neural Network Pretrained with Large Dataset for Downstream Task

📂 Please refer to README.pdf for further information.

  • Removed top layers from pretrained EfficientNetB0 and ResNet18 to construct lightweight CNN model with less than 1M #params.
  • Assessed onTrash Annotations in Context(TACO) Dataset sampled for 6 classes with 20,851 images.
  • Compared performance with lightweight models generated with Optuna's Neural Architecture Search(NAS) constituted with same convolutional blocks.
  • It is speculated that such elimination method will work on neural networks with residual connections, according to the paper ofVeit et al(2016) where it shows residual networks behave like ensembles of networks. Refer to Figure 5 attached below, where error when deleting layers linearly increase rather than exponentially increasing.

image-20230330172447858

Quickstart

Installation

# clone the repositorygit clone https://github.com/snoop2head/elimnet# fetch image dataset and unzip!wget -cq https://aistages-prod-server-public.s3.amazonaws.com/app/Competitions/000081/data/data.zip!unzip ./data.zip -d ./

Train

# finetune on the dataset with pretrained modelpython train.py --model ./model/efficientnet/efficientnet_b0.yaml# finetune on the dataset with ElimNetpython train.py --model ./model/efficientnet/efficientnet_b0_elim_3.yaml

Inference

# inference with the lastest ran modelpython inference.py --model_dir ./exp/latest/

Performance

Performance is compared with (1) original pretrained model and (2) Optuna NAS constructed models with no pretrained weights.

  • Indicates that top convolutional layers eliminated pretrained CNN models outperforms empty Optuna NAS models generated with same convolutional blocks.
  • Suggests that eliminating top convolutional layers creates lightweight model that shows similar(or better) classifcation performance with original pretrained model.
  • Reduces parameters to 7%(or less) of its original parameters while maintaining(or improving) its performance. Saves inference time by 20% or more by eliminating top convolutional layters.

ELimNet vs Pretrained Models (Train)

[100 epochs]# of Parameters# of LayersTrainValidationTest F1
Pretrained EfficientNet B04.0M352Loss: 0.43
Acc: 81.23%
F1: 0.84
Loss: 0.469
Acc: 82.17%
F1: 0.76
0.7493
EfficientNet B0 Elim 20.9M245Loss:0.652
Acc: 87.22%
F1: 0.84
Loss: 0.622
Acc: 87.22%
F1: 0.77
0.7603
EfficientNet B0 Elim 30.30M181Loss: 0.602
Acc: 78.17%
F1: 0.74
Loss: 0.661
Acc: 77.41%
F1: 0.74
0.7349
Resnet1811.17M69Loss: 0.578
Acc: 78.90%
F1: 0.76
Loss: 0.700
Acc: 76.17%
F1: 0.719
-
Resnet18 Elim 20.68M37Loss: 0.447
Acc: 83.73%
F1: 0.71
Loss: 0.712
Acc: 75.42%
F1: 0.71
-

ELimNet vs Pretrained Models (Inference)

# of Parameters# of LayersCPU times (sec)CUDA time (sec)Test Inference Time (sec)
Pretrained EfficientNet B04.0M3523.9s4.0s105.7s
EfficientNet B0 Elim 20.9M2454.1s13.0s83.4s
EfficientNet B0 Elim 30.30M1813.0s9.0s73.5s
Resnet1811.17M69---
Resnet18 Elim 20.68M37---

ELimNet vs Empty Optuna NAS Models (Train)

[100 epochs]# of Parameters# of LayersTrainValidTest F1
Empty MobileNet V34.2M227Loss 0.925
Acc: 65.18%
F1: 0.58
Loss 0.993
Acc: 62.83%
F1: 0.56
-
Empty EfficientNet B01.3M352Loss 0.867
Acc: 67.28%
F1: 0.61
Loss 0.898
Acc: 66.80%
F1: 0.61
0.6337
Empty DWConv & InvertedResidualv3 NAS0.08M66-Loss: 0.766
Acc: 71.71%
F1: 0.68
0.6740
Empty MBConv NAS0.33M141Loss: 0.786
Acc: 70.72%
F1: 0.66
Loss: 0.866
Acc: 68.09%
F1: 0.62
0.6245
Resnet18 Elim 20.68M37Loss: 0.447
Acc: 83.73%
F1: 0.71
Loss: 0.712
Acc: 75.42%
F1: 0.71
-
EfficientNet B0 Elim 30.30M181Loss: 0.602
Acc: 78.17%
F1: 0.74
Loss: 0.661
Acc: 77.41%
F1: 0.74
0.7603

ELimNet vs Empty Optuna NAS Models (Inference)

# of Parameters# of LayersCPU times (sec)CUDA time (sec)Test Inference Time (sec)
Empty MobileNet V34.2M227413-
Empty EfficientNet B01.3M3523.7803.78268.4s
Empty DWConv &
InvertedResidualv3 NAS
0.08M6613.561.1s
Empty MBConv NAS0.33M1412.147.20167.1s
Resnet18 Elim 20.68M37---
EfficientNet B0 Elim 30.30M1813.0s9s73.5s

Background & WiP

Background

Work in Progress

  • Will test the performance of replacing convolutional blocks with pretrained weights with a single convolutional layer without pretrained weights.
  • Will add ResNet18's inference time data and compare with Optuna's NAS constructed lightweight model.
  • Will test on pretrained MobileNetV3, MnasNet on torchvision with elimination based lightweight model architecture search.
  • Will be applied on other small datasets such as Fashion MNIST dataset and Plant Village dataset.

Others

  • "Empty" stands for model with no pretrained weights.
  • "EfficientNet B0 Elim 2" means 2 convolutional blocks have been eliminated from pretrained EfficientNet B0. Number next to "Elim" annotates how many convolutional blocks have been removed.
  • Table's performance illustrates best performance out of 100 epochs of finetuning on TACO Dataset.

Authors

About

🔪 Elimination based Lightweight Neural Net with Pretrained Weights

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages


[8]ページ先頭

©2009-2025 Movatter.jp