Repository files navigation

Example result of a sketch simplification. Image copyrighted by Eisaku Kubonouchi (@EISAKUSAKU) and only non-commercial research usage is allowed.

This code provides pre-trained models used in the research papers:

   "Learning to Simplify: Fully Convolutional Networks for Rough Sketch Cleanup"   Edgar Simo-Serra*, Satoshi Iizuka*, Kazuma Sasaki, Hiroshi Ishikawa (* equal contribution)   ACM Transactions on Graphics (SIGGRAPH), 2016

and

   "Mastering Sketching: Adversarial Augmentation for Structured Prediction"   Edgar Simo-Serra*, Satoshi Iizuka*, Hiroshi Ishikawa (* equal contribution)   ACM Transactions on Graphics (TOG), 2018

See ourproject page for more detailed information.

• PyTorch (version 0.4.1)torchvision
• pillow

All packages should be part of a standard PyTorch install. For information on how to install PyTorch please refer to thetorch website.

Before the first usage, the models have to be downloaded with:

bash download_models.sh

Next test the models with:

python simplify.py

You should see a file calledout.png created with the output of the model.

Application options can be seen with:

python simplify.py --help

Using the same interface it is possible to perform pencil drawing generation. In this case, the input should be a clean line drawing and not a rough sketch, and the line drawings can be generated by:

python simplify.py --img test_line.png --out out_rough.png --model model_pencil2.t7

This will generate a rough version oftest_line.png asout_rough.png. By changing the model it is possible to change the type of rough sketch being generated.

• model_mse.t7: Model trained using only MSE loss (SIGGRAPH 2016 model).
• model_gan.t7: Model trained with MSE and GAN loss using both supervised and unsupervised training data (TOG 2018 model).
• model_pencil1.t7: Model for pencil drawing generation based on artist 1 (dirty and faded pencil lines).
• model_pencil2.t7: Model for pencil drawing generation based on artist 2 (clearer overlaid pencil lines).

For replicability we include code to replicate the figures in the paper. After downloading the models you can run it with:

./figs.sh

This will convert the input images infigs/ and save the output inout/. We note that there are small differences with the results in the paper due to hardware differences and small differences in the torch/pytorch implementations. Furthermore, results are shown without the post-processing mentioned in the notes at the bottom of this document.

Please note that we do not have the copyright for all these images and in general only non-commercial research usage is permitted. In particular,fig16_eisaku.png,fig06_eisaku_robo.png,fig06_eisaku_joshi.png, andfig01_eisaku.png are copyright by Eisaku Kubonoichi (@EISAKUSAKU) and only non-commercial research usage is allowed.The imagesfig14_pepper.png andfig06_pepper.png are licensed by David Revoy (www.davidrevoy.com) under CC-by 4.0.

Please see thetraining readme.

• Models are in Torch7 format and loaded using the PyTorch legacy code.
• This was developed and tested on various machines from late 2015 to end of 2016.
• Provided models are under a non-commercial creative commons license.
• Post-processing is not performed. You can perform it manually withconvert out.png bmp:- | mkbitmap - -t 0.3 -o - | potrace --svg --group -t 15 -o - > out.svg.

If you use these models please cite:

@Article{SimoSerraSIGGRAPH2016,   author    = {Edgar Simo-Serra and Satoshi Iizuka and Kazuma Sasaki and Hiroshi Ishikawa},   title     = {{Learning to Simplify: Fully Convolutional Networks for Rough Sketch Cleanup}},   journal   = "ACM Transactions on Graphics (SIGGRAPH)",   year      = 2016,   volume    = 35,   number    = 4,}

and

@Article{SimoSerraTOG2018,   author    = {Edgar Simo-Serra and Satoshi Iizuka and Hiroshi Ishikawa},   title     = {{Mastering Sketching: Adversarial Augmentation for Structured Prediction}},   journal   = "ACM Transactions on Graphics (TOG)",   year      = 2018,   volume    = 37,   number    = 1,}

This work was partially supported by JST CREST Grant Number JPMJCR14D1 and JST ACT-I Grant Numbers JPMJPR16UD and JPMJPR16U3.

This sketch simplification code is freely available for free non-commercialuse, and may be redistributed under these conditions. Please, see thelicensefor further details.