Repository files navigation

Transformer-based dialog models work better and we recommend using them instead of RNN-based CakeChat. See, for examplehttps://github.com/microsoft/DialoGPT

CakeChat is a backend for chatbots that are able to express emotions via conversations.

CakeChat is built onKeras andTensorflow.

The code is flexible and allows to condition model's responses by an arbitrary categorical variable.For example, you can train your own persona-based neural conversational model^[1]or create an emotional chatting machine^[2].

• python 3.5.2
• tensorflow 1.12.2
• keras 2.2.4

1. Network architecture and features
2. Quick start
3. Setup for training and testing
   1. Docker
      1. CPU-only setup
      2. GPU-enabled setup
   2. Manual setup
4. Getting the pre-trained model
5. Training data
6. Training the model
   1. Fine-tuning the pre-trained model on your data
   2. Training the model from scratch
   3. Distributed train
   4. Validation metrics calculation
   5. Testing the trained model
7. Running CakeChat server
   1. Local HTTP-server
      1. HTTP-server API description
   2. Gunicorn HTTP-server
   3. Telegram bot
8. Repository overview
   1. Important tools
   2. Important configuration settings
9. Example use cases
10. References
11. Credits & Support
12. License

Model:

• Hierarchical Recurrent Encoder-Decoder (HRED) architecture for handling deep dialog context^[3].
• Multilayer RNN with GRU cells. The first layer of the utterance-level encoder is always bidirectional.By default, CuDNNGRU implementation is used for ~25% acceleration during inference.
• Thought vector is fed into decoder on each decoding step.
• Decoder can be conditioned on any categorical label, for example, emotion label or persona id.

Word embedding layer:

• May be initialized using w2v model trained on your corpus.
• Embedding layer may be either fixed or fine-tuned along with other weights of the network.

Decoding

• 4 different response generation algorithms: "sampling", "beamsearch", "sampling-reranking" and "beamsearch-reranking".Reranking of the generated candidates is performed according to the log-likelihood or MMI-criteria^[4].Seeconfiguration settings description for details.

Metrics:

• Perplexity
• n-gram distinct metrics adjusted to the samples size^[4].
• Lexical similarity between samples of the model and some fixed dataset.Lexical similarity is a cosine distance between TF-IDF vector of responses generated by the model and tokensin the dataset.
• Ranking metrics: mean average precision and mean recall@k.^[5]

In case you are familiar withDocker here is the easiest way to run a pre-trained CakeChatmodel as a server. You may need to run the following commands withsudo.

CPU version:

docker pull lukalabs/cakechat:latest && \docker run --name cakechat-server -p 127.0.0.1:8080:8080 -it lukalabs/cakechat:latest bash -c "python bin/cakechat_server.py"

GPU version:

docker pull lukalabs/cakechat-gpu:latest && \nvidia-docker run --name cakechat-gpu-server -p 127.0.0.1:8080:8080 -it lukalabs/cakechat-gpu:latest bash -c "CUDA_VISIBLE_DEVICES=0 python bin/cakechat_server.py"

That's it! Now test your CakeChat server by running the following command on your host machine:

python tools/test_api.py -f localhost -p 8080 -c "hi!" -c "hi, how are you?" -c "good!" -e "joy"

The response dict may look like this:

{'response': "I'm fine!"}

Docker is the easiest way to set up the environment and install all the dependencies for training and testing.

Note:We strongly recommend using GPU-enabled environment for training CakeChat model.Inference can be made both on GPUs and CPUs.

1. InstallDocker.

2. Pull a CPU-only docker image from dockerhub:

docker pull lukalabs/cakechat:latest

3. Run a docker container in the CPU-only environment:

docker run --name <YOUR_CONTAINER_NAME> -it lukalabs/cakechat:latest

1. Installnvidia-docker for the GPU support.

2. Pull GPU-enabled docker image from dockerhub:

docker pull lukalabs/cakechat-gpu:latest

3. Run a docker container in the GPU-enabled environment:

nvidia-docker run --name <YOUR_CONTAINER_NAME> -it cakechat-gpu:latest

That's it! Now you can train your model and chat with it. See the corresponding section below for further instructions.

If you don't want to deal with docker, you can install all the requirements manually:

pip install -r requirements.txt -r requirements-local.txt

NB:

We recommend installing the requirements inside avirtualenv to preventmessing with your system packages.

You can download our pre-trained model weights by runningpython tools/fetch.py.

The params of the pre-trained model are the following:

• context size3 (<speaker_1_utterance>, <speaker_2_utterance>, <speaker_1_utterance>)
• each encoded utterance containsup to 30 tokens
• the decoded utterance containsup to 32 tokens
• both encoder and decoder have2 GRU layers with768 hidden units each
• first layer of the encoder is bidirectional

The model was trained on a preprocessed Twitter corpus with ~50 million dialogs (11Gb of text data).To clean up the corpus, we removed

• URLs, retweets and citations;
• mentions and hashtags that are not preceded by regular words or punctuation marks;
• messages that contain more than 30 tokens.

We used our emotions classifier to label each utterance with one of the following 5 emotions:"neutral", "joy", "anger", "sadness", "fear", and used these labels during training.To mark-up your own corpus with emotions you can use, for example,DeepMoji tool.

Unfortunately, due to Twitter's privacy policy, we are not allowed to provide our dataset.You can train a dialog model on any text conversational dataset available to you, a great overview of existingconversational datasets can be found here:https://breakend.github.io/DialogDatasets/

The training data should be a txt file, where each line is a valid json object, representing a list of dialog utterances.Refer to ourdummy train dataset to see the necessaryfile structure. Replace this dummy corpus with your data before training.

There are two options:

1. training from scratch
2. fine-tuning the provided trained model

The first approach is less restrictive: you can use any training data you want and set any config params of the model.However, you should be aware that you'll need enough train data (~50Mb at least), one or more GPUs and enoughpatience (days) to get good model's responses.

The second approach is limited by the choice of config params of the pre-trained model – seecakechat/config.py forthe complete list. If the default params are suitable for your task, fine-tuning should be a good option.

1. Fetch the pre-trained model from Amazon S3 by runningpython tools/fetch.py.

2. Put your training text corpus todata/corpora_processed/train_processed_dialogs.txt. Make sure that yourdataset is large enough, otherwise your model risks to overfit the data and the results will be poor.

3. Runpython tools/train.py.

   1. The script will look for the pre-trained model weights inresults/nn_models, the full path is inferred from theset of config params.
   2. If you want to initialize the model weights from a custom file, you can specify the path to the file via-iargument, for example,python tools/train.py -i results/nn_models/my_saved_weights/model.current.
   3. Don't forget to setCUDA_VISIBLE_DEVICES=<GPU_ID> environment variable (with <GPU_ID> as in output ofnvidia-smi command) if you want to use GPU. For example,CUDA_VISIBLE_DEVICES=0 python tools/train.py will run thetrain process on the 0-th GPU.
   4. Use parameter-s to train the model on a subset of the first N samples of your training data to speed uppreprocessing for debugging. For example, runpython tools/train.py -s 1000 to train on the first 1000 samples.

Weights of the trained model are saved toresults/nn_models/.

1. Put your training text corpus todata/corpora_processed/train_processed_dialogs.txt.

2. Set up training parameters incakechat/config.py.Seeconfiguration settings description for more details.

3. Consider runningPYTHONHASHSEED=42 python tools/prepare_index_files.py to build the index files with tokens andconditions from the training corpus. Make sure to setPYTHONHASHSEED environment variable, otherwise you may getdifferent index files for different launches of the script.Warning: this script overwrites the original tokens index filesdata/tokens_index/t_idx_processed_dialogs.json anddata/conditions_index/c_idx_processed_dialogs.json.You should only run this script in case your corpus is large enough to contain all the words that you want your modelto understand. Otherwise, consider fine-tuning the pre-trained model as described above. If you messed up with indexfiles and want to get the default versions, delete your copies and runpython tools/fetch.py anew.

4. Consider runningpython tools/train_w2v.py to build w2v embedding from the training corpus.Warning: this script overwrites the original w2v weights that are stored indata/w2v_models.You should only run this script in case your corpus is large enough to contain all the words that you want your modelto understand. Otherwise, consider fine-tuning the pre-trained model as described above. If you messed up with w2vfiles and want to get the default version, delete your file copy and runpython tools/fetch.py anew.

5. Runpython tools/train.py.

   1. Don't forget to setCUDA_VISIBLE_DEVICES=<GPU_ID> environment variable (with <GPU_ID>as in output ofnvidia-smi command) if you want to use GPU. For exampleCUDA_VISIBLE_DEVICES=0 python tools/train.pywill run the train process on the 0-th GPU.
   2. Use parameter-s to train the model on a subset of the first N samples of your training data to speed uppreprocessing for debugging. For example, runpython tools/train.py -s 1000 to train on the first 1000 samples.

6. You can also setIS_DEV=1 to enable the "development mode". It uses a reduced number of model parameters(decreased hidden layer dimensions, input and output sizes of token sequences, etc.) and performs verbose logging.Refer to the bottom lines ofcakechat/config.py for the complete list of dev params.

Weights of the trained model are saved toresults/nn_models/.

GPU-enabled docker container supports distributed train on multiple GPUs usinghorovod.

For example, runpython tools/distributed_train.py -g 0 1 to start training on 0 and 1 GPUs.

During training the following datasets are used for validations metrics calculation:

• data/corpora_processed/val_processed_dialogs.txt(dummy example, replace with your data) – for thecontext-sensitive dataset
• data/quality/context_free_validation_set.txt – for the context-freevalidation dataset
• data/quality/context_free_questions.txt – is used for generatingresponses for logging and computing distinct-metrics
• data/quality/context_free_test_set.txt – is used for computing metrics ofthe trained model, e.g. ranking metrics

The metrics are stored tocakechat/results/tensorboard and can be visualized usingTensorboard.If you run a docker container from the provided CPU or GPU-enabled docker image, tensorboard server should startautomatically and serve onhttp://localhost:6006. Open this link in your browser to see the training graphs.

If you installed the requirements manually, start tensorboard server first by running the following command from yourcakechat root directory:

mkdir -p results/tensorboard && tensorboard --logdir=results/tensorboard 2>results/tensorboard/err.log &

After that proceed tohttp://localhost:6006.

You can run the following tools to evaluate your trained model ontest data(dummy example, replace with your data):

• tools/quality/ranking_quality.py –computes ranking metrics of a dialog model
• tools/quality/prediction_distinctness.py –computes distinct-metrics of a dialog model
• tools/quality/condition_quality.py –computes metrics on different subsets of data according to the condition value
• tools/generate_predictions.py –evaluates the model. Generates predictions of a dialog model on the set of given dialog contexts and then computesmetrics. Note that you should have a reverse-model in thedata/nn_models directory if you want to use "*-reranking"prediction modes
• tools/generate_predictions_for_condition.py –generates predictions for a given condition value

Run a server that processes HTTP-requests with given input messages and returns response messages from the model:

python bin/cakechat_server.py

SpecifyCUDA_VISIBLE_DEVICES=<GPU_ID> environment variable to run the server on a certain GPU.

Don't forget to runpython tools/fetch.py prior to starting the server if you want to use our pre-trained model.

To make sure everything works fine, test the model on the following conversation

– Hi, Eddie, what's up?
– Not much, what about you?
– Fine, thanks. Are you going to the movies tomorrow?

by running the command:

python tools/test_api.py -f 127.0.0.1 -p 8080 \    -c "Hi, Eddie, what's up?" \    -c "Not much, what about you?" \    -c "Fine, thanks. Are you going to the movies tomorrow?"

You should get a meaningful answer, for example:

{'response': "Of course!"}

JSON parameters are:

POST /cakechat_api/v1/actions/get_responsedata: { 'context': ['Hello', 'Hi!', 'How are you?'], 'emotion': 'joy'}

200 OK{  'response': 'I\'m fine!'}

We recommend usingGunicorn for serving the API of your model at production scale.

1. Install gunicorn:pip install gunicorn

2. Run a server that processes HTTP-queries with input messages and returns response messages of the model:

cd bin && gunicorn cakechat_server:app -w 1 -b 127.0.0.1:8080 --timeout 2000

You can run your CakeChat model as a Telegram bot:

1. Create a telegram bot to get bot's token.
2. Runpython tools/telegram_bot.py --token <YOUR_BOT_TOKEN> and chat with it on Telegram.

• cakechat/dialog_model/ – contains computational graph, training procedure and other model utilities
• cakechat/dialog_model/inference/ – algorithms for response generation
• cakechat/dialog_model/quality/ – code for metrics calculation and logging
• cakechat/utils/ – utilities for text processing, w2v training, etc.
• cakechat/api/ – functions to run http server: API configuration, error handling
• tools/ – scripts for training, testing and evaluating your model

• bin/cakechat_server.py –Runs an HTTP-server that returns response messages of the model given dialog contexts and an emotion.Seerun section for details.
• tools/train.py –Trains the model on your data. You can use the--reverse option to train a reverse-model used in "*-reranking" responsegeneration algorithms for more accurate predictions.
• tools/prepare_index_files.py –Prepares index for the most commonly used tokens and conditions. Use this script before training the model from scratchon your own data.
• tools/quality/ranking_quality.py –Computes ranking metrics of a dialog model.
• tools/quality/prediction_distinctness.py –Computes distinct-metrics of a dialog model.
• tools/quality/condition_quality.py –Computes metrics on different subsets of data according to the condition value.
• tools/generate_predictions.py –Evaluates the model. Generates predictions of a dialog model on the set of given dialog contexts and then computesmetrics. Note that you should have a reverse-model in theresults/nn_models directory if you want to use "*-reranking"prediction modes.
• tools/generate_predictions_for_condition.py –Generates predictions for a given condition value.
• tools/test_api.py –Example code to send requests to a running HTTP-server.
• tools/fetch.py –Downloads the pre-trained model and index files associated with it.
• tools/telegram_bot.py –Runs Telegram bot on top of trained model.

All the configuration parameters for the network architecture, training, predicting and logging steps are defined incakechat/config.py. Some inference parameters used in an HTTP-server are defined incakechat/api/config.py.

• Network architecture and size

  • HIDDEN_LAYER_DIMENSION is the main parameter that defines the number of hidden units in recurrent layers.
  • WORD_EMBEDDING_DIMENSION andCONDITION_EMBEDDING_DIMENSION define the number of hidden units that eachtoken/condition are mapped into.
  • Number of units of the output layer of the decoder is defined by the number of tokens in the dictionary in thetokens_index directory.

• Decoding algorithm:

  • PREDICTION_MODE_FOR_TESTS defines how the responses of the model are generated. The options are the following:
    • sampling – response is sampled from output distribution token-by-token.For every token the temperature transform is performed prior to sampling.You can control the temperature value by tuningDEFAULT_TEMPERATURE parameter.
    • sampling-reranking – multiple candidate-responses are generated using sampling procedure described above.After that the candidates are ranked according to their MMI-score^[4]You can tune this mode by pickingSAMPLES_NUM_FOR_RERANKING andMMI_REVERSE_MODEL_SCORE_WEIGHT parameters.
    • beamsearch – candidates are sampled usingbeam search algorithm.The candidates are ordered according to their log-likelihood score computed by the beam search procedure.
    • beamsearch-reranking – same as above, but the candidates are re-ordered after the generation in the sameway as in sampling-reranking mode.

  Note that there are other parameters that affect the response generation process.SeeREPETITION_PENALIZE_COEFFICIENT,NON_PENALIZABLE_TOKENS,MAX_PREDICTIONS_LENGTH.

By providing additional condition labels within dataset entries, you can build the following models:

• A Persona-Based Neural Conversation Model — a model that allows to condition responses on a persona ID to makethem lexically similar to the given persona's linguistic style.
• Emotional Chatting Machine-like model — a model that allows conditioning responses on different emotions to provideemotional styles (anger, sadness, joy, etc).
• Topic Aware Neural Response Generation-like model — a model that allows to condition responses on a certaintopic to keep the topic-aware conversation.

To make use of these extra conditions, please refer to the sectionTraining the model.Just set the "condition" field in thetraining set to one of thefollowing:persona ID,emotion ortopic label, update the index files and start the training.

• ^[1]A Persona-Based Neural Conversation Model
• ^[2]Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memory
• ^[3]A Hierarchical Recurrent Encoder-Decoder For Generative Context-Aware Query Suggestion
• ^[4]A Diversity-Promoting Objective Function for Neural Conversation Models
• ^[5]Quantitative Evaluation of User Simulation Techniques for Spoken Dialogue Systems
• ^[6]Topic Aware Neural Response Generation

CakeChat is developed and maintained by theReplika team:

Nicolas Ivanov,Michael Khalman,Nikita Smetanin,Artem Rodichev andDenis Fedorenko.

Demo byOleg Akbarov,Alexander Kuznetsov andVladimir Chernosvitov.

All issues and feature requests can be tracked here –GitHub Issues.

© 2019 Luka, Inc. Licensed under the Apache License, Version 2.0. See LICENSE file for more details.