Repository files navigation

Expert eyeglasses recommendation system with Generative Adversarial Networks written in Python, 2020.

Live Demo:Glasses:eyes:Expert

• Description
• Installation
• Reminder about API-services
• Additional features
  • Download pretrained GAN models
  • Download images for A/B testing
• How to use ExpertGlasses
  • In code
  • GUI
• Performance benchmarks
• Sources

The task of recommending eyeframes based on people's appearance is quite hard and there are no similar projects available at the moment. This system takes into account over 20 facial attributes, maps them into eyeglasses features with the help of expert module (which consists with over 40 branches). Expert module apply written beforehand rules to get necessary mappings. High interpretability of such approach guaranties user's understanding and loyality towards the system.

To get necessary attributes from the photo this system uses bunch of machine learning algorithms and existing free services, i.eBetaFaceAPI andFace++. Face++ allows to locate bounding boxes of faces in images, while BetaFace uses various classifiers to get most of the secondary-importance features. So to use this repository, you should have access to the internet.

To detect face shape probabilities, iris color, forehead size, jawtype type, skintone the system uses own pretrained convolutional neural networks (CNNs). These models run on your local machine on CPU.

To get all necessary eyeframes attributes, the large dataset (>8k records) of eyeframes was parsed and processed. Because in real life there are not so many eyeframe models in the local shop available, the generation of unique eyewear by given features was implemented. The system use a conditional GAN followed by Super Resolution GAN to create non-existing high-definition images of the eyeframes.

Requirements:

• Python 3.6 or greater
• requests
• numpy
• pandas
• opencv_python_headless
• scikit_image
• tensorflow
• Keras_Applications
• PySimpleGUI

For the full list of requirements consider to check requirements.txt

To install repository, use the following commands:

Windows:

$ git clone https://github.com/Defasium/expertglasses$ cd expertglasses/$ pip3 install -r requirements.txt

Linux:

$ git clone https://github.com/Defasium/expertglasses$ cd expertglasses/$ sudo pip3 install -r requirements.txt

Linux inside virtualenv:

$ git clone https://github.com/Defasium/expertglasses$ cd expertglasses/$ pip3 install -r requirements.txt

In current version without Face++ api key you can't use Expert Recommender System!

While BetaFace api has the same free api key for everyone with some limitations, allowing you to upload up to 500 images from your IP per day. Face++ utilizes a different strategy, requiring you to create an account on their site. In this account you can generate your own free api key and api secret in the Apps -> API key section:

After that consider changing values of API_KEY and API_SECRET in thefaceplusplus.py module:

...API_KEY="api_key"API_SECRET="api_secret"RETURN_ATTRIBUTES="gender,beauty,age,smiling,headpose,ethnicity,skinstatus"RETURN_LANDMARK=1...

In the github repository there are no currently some functionality, e.g. pretrained GAN models and best selling eyeglasses. If you want to include this features in the system, you need to download necessary files. Links are listed bellow.

Download 2 files,cgan.h5 andsrgan.h5, and place them inutils/ directory.

Download files, create a directory namedabtest/ and extractman/ andwoman/ directories inside of it.

At the current state there are only 2 ways of using expert eyeglasses recommender system: in your python code and via GUI.

To use this system, simply import class in your python code:

fromexpert_backendimportExpertEyeglassesRecommender

After that create an instance of this class with a specified path to the image:

ins=ExpertEyeglassesRecommender('test.jpg')# by passing a `lang` parameter you can specify language, which will be used at explanation stepins=ExpertEyeglassesRecommender('test.jpg',lang='en')

Initialization of class may take quite a long time (from 30 second up to 2 minutes).

After initialization recomendations will be completed and to get the top 6 best images of eyeglasses use:

ins.plot_recommendations()

By changing strategy value you can obtain different results:

# in standart strategy feature with the biggest value impacts the results the mostins.plot_recommendations(strategy='standart')# in factorized strategy (default value) features are divided into three main groups:# shape of eyewear, its rim and other features, thus making results more variousins.plot_recommendations(strategy='factorized')# in factorized plus strategy every feature greatly influence the final result# thus providing many varios nonsimilar eyeframesins.plot_recommendations(strategy='factorized_plus')# in color only strategy the resulting images will have the same color as in color vectorins.plot_recommendations(strategy='color_only')# shape only strategy's results are similar to standart's one but doesn't take into account colorins.plot_recommendations(strategy='shape_only')

If you havedownloaded abtest directory, you can also try recommending random most popular eyeglasses (so the system don't use any information about your appearance):

# will output 6 randomly chosen eyeglassesins.plot_recommendations(strategy='most_popular')

To get explanation of the system try:

print(ins.description)

To work with new image use:

ins.update_image('test2.jpg')ins.expert_module()

You can alse use url as an argument:

ins.update_image('https://github.com/Defasium/expertglasses/blob/master/assets/gui.png?raw=true')ins.expert_module()

To generate unique image with GANs (implying that you have downloaded pretrained models, lookDownload pretrained GAN models section) use:

image=ins.generate_unique(show=True)

Will generate eyeglasses according to features, plot it with matplotlib if show is True and returns an numpy.ndarray

Graphical user interface is powered by PySimpleGui framework, which allows fast and simple prototyping for cross-platform systems.

To launch GUI, use the following command from terminal:

$ python gui.py

In the first popup window you should load an image to process, i.e. the fast start, module would automatically calculate all the necessary stuff in background. After that all machine learning models will be loaded in RAM (loading usually takes from 30 seconds up to 2 minutes). When progressbar will be filled, the main interface will appear.

In this section you can find information about current performance of the system, speed benchmarks and so on.

Speed benchmarks were tested on 2 different machines:

• Windows 10 with 4 cores 2.3 GHz Intel® Core™ i5-4200U and 4 GB of RAM (low spec)
• Linux Ubuntu 16.04 LTS with 4 cores 3.4 GHz Intel® Core™ i7-4770 and 20 GB of RAM (high spec)

The results are listed below:

To measure quality of recommendations A/B testing was made with ~50 participants. After that normalized discounted cumulative gain at top 5 items was calculated. The results are shown below:

As you can see, expert recommender system outperforms the results of randomly chosen best selling eyeframes.

For classifiers this system uses modified version of ShuffleNetV2, implemented by@opconty.

For upscaling generated images this system uses modified version of@krasserm Super Resolution GAN implementation.