Repository files navigation

GPT-2 Discord Bot and Steps to Train Something Like You

I finetuned my gpt2 models locally on a RTX 2080 Super. Overclocked to around 1550MHz of memory allocated. With this using the355M (Medium sz) Model, I was able to reach 10k iterations within 4 hours. This will vary but you can even finetune enough for a workable bot within 5k iterations using a CPU.

For a chatbot like this I recommend you try to reach at least 8500-10000 iterations. If you reach an unchanging or NaN difference in learning rate stop the model, it may have trained to much and you will encounter really weird bugs. For this you should save every 500-1000 iterations so you have snapshots to revert to.

My discord bot runs with8560 iterations using the355M on a dataset with1375352 tokens (about 500k individual chat messages from discord). The messages are trained from a single discord channel which included myself and 4 friends. This resulted in a very specific personality set the bot picked up where it would make jokes that only we would pick up and call out each other even when others spoke to it. This isn't desireable unless you want to just have it in one server.

To combat this you should grab a lot of conversations that you take part in with many different people. Then in your final dataset make sure you have a lot of back and forth conversation. General and off-topic chats are good for this.

• Links. You probably want to parse these from your dataset or your bot will start to send randomly generated links that look real. Sometimes it can send a real link and this can be funny but it's a rare occasion.
• Bot messages and server messages. Messages from bots can sounds like messages from bots and you don't want your life like AI to sound like a bot do you? Parse these out along with discord'sJoined the Server. messages.
• Language. If you want your bot to be nice, don't put toxic stuff in the dataset. AI doesn't have feelings and GPT-2 especially doesn't care about your views or how moral you are. If you are toxic in discord your bot is going to be toxic.

USE A VIRTUALENV!!!

First clonegpt-2-simple's source code locally.

git clone https://github.com/minimaxir/gpt-2-simplecd gpt-2-simple

Setup a virtualenv inside of the repo

virtualenv .env# Wait for it to setup.env\Scripts\activate # On Windowssource .env\bin\activate # On Unixpip install -r requirements.txt # IF USING A GPU TO FINETUNE/GENERATEpip install -r requirements-gpu.txt

Download the models you need:

gpt-2-simple download MODELNAME

Models are:124M,355M,774M,1.5B

**You will not be able to run anything more than the355M model on a gaming graphics card. Don't bother wasting the time to download the higher memory models unless you have a Titan or Quadro or something. If you are using Colab the774M can work sometimes. **

You should use a discord chat exporter likethis and export it to txt. The format for my dataset was as follows:

name1:conversational message heremaybe another one herename2:conversational reply herename1:replyname2:reply replyname3:blah blah blah

Obviously it had real contextual conversation going on but this is the format it was in.

If you are running the finetuning on a CPU I recommend using the smaller124M model. If you have a beefy GPU like an RTX card or a high level GTX/RX card you can probably try the355M model.

Below is code to finetune a model basically:

importgpt_2_simpleasgpt2fromdatetimeimportdatetimefile_name="dataset.txt"# File name of datasetsess=gpt2.start_tf_sess()gpt2.finetune(sess,dataset=file_name,model_name='355M',# Model you have already downloadedsteps=-1,# -1 will do unlimited. Enter number of iterations otherwiserestore_from='latest',# Also allows 'fresh' which will overwrite old trainingrun_name='discord',# The name to pull or create a checkpoint underprint_every=50,# Print iterations every X numebrsample_every=150,# Generate a text sample ever X number of iter.save_every=500,# Save a snapshot every X number of iter.learning_rate=0.0001,# Lower to 0.00001 if you are not getting massive changes in resultsbatch_size=1# Keep at 1 or 2, will use up more memory if you raise this)

You can run this everytime and it will train your model and pick up from where it left off, or start a new one if you have a newrun_name.

Finetune your model until it reaches around 8k-10k iterations.

Create a discord bot on the discord site.

First you need to go to theApplications Section of the developer panel. Inside you need to create a new app.

Add a name and go to the new application's settings. You need to create a Bot User:

Now copy the token for the bot.

Below you will find the code to run a basic answer bot using the GPT-2 model you trained with the steps above:

importdiscord,datetimeimportgpt_2_simpleasgpt2importasyncio,random,stringsess=gpt2.start_tf_sess()gpt2.load_gpt2(sess,run_name='discord')# The name of your checkpointYOURNAME="yourdiscordname"client=discord.Client()@client.eventasyncdefon_ready():print('We have logged in as {0.user}'.format(client))activity=discord.Activity(name="How to Become Human",type=discord.ActivityType.watching)awaitclient.change_presence(status=discord.Status.online,activity=activity)@client.eventasyncdefon_message(message):ifclient.user.mentioninmessage.content.replace('<@!','<@'):ifmessage.author==client.user:returnelse:ifclient.is_ready:uses_con=Falseasyncwithmessage.channel.typing():if"makeconvo"inmessage.content:print("Gen Convo")uses_con=Trueresults=gpt2.generate(sess,run_name='discordlarge',temperature=0.9,nsamples=1,batch_size=1,prefix=message.author.name+":\n"+message.content+"\n\n",length=350,include_prefix=True,return_as_list=True)awaitmessage.channel.send("```\n"+str('='*20).join(results)+"\n```")else:print("Generating")final=''prefix=""last_author=""old=awaitmessage.channel.history(limit=9).flatten()old.reverse()formsginold:iflast_author==msg.author.name:iflen(msg.mentions)>0:formentioninmsg.mentions:msg.content.replace("<@!"+str(mention.id)+">","@"+mention.name)prefix=prefix+msg.content+"\n"else:iflen(msg.mentions)>0:formentioninmsg.mentions:msg.content.replace("<@!"+str(mention.id)+">","@"+mention.name)last_author=msg.author.nameprefix=prefix+"\n\n"+msg.author.name+":\n"+msg.content+"\n"whileTrue:results=gpt2.generate(sess,run_name='discordlarge',temperature=0.9,nsamples=3,batch_size=3,prefix=prefix+"\n\n"+YOURNAME+":\n",length=250,return_as_list=True,include_prefix=False,truncate="\n\n")res_split=random.choice(results).split('\n')ok= []forrinres_split:ifnotr.endswith(":")andlen(r)>2and"http"notinr:ok.append(r)iflen(ok)>0:breakfori,msginenumerate(ok):ifi== (len(ok)-1):awaitasyncio.sleep(random.randint(0,1))awaitmessage.channel.send(msg)else:asyncwithmessage.channel.typing():awaitmessage.channel.send(msg)awaitasyncio.sleep(random.randint(1,3))else:returnclient.run('YOUR DISCORD TOKEN')

When you run this it will respond to you or anybody who pings it using the last 9 messages as context for the bot to continue the conversation!