About the Dataset

allserieslines = {seriesname :                    {episode number :                        {character : alllines                        }                    }}

We choose a single series, DS9 i.e. Deep Space 9 for further investigation

Answering some questions from the given Dataset using Natural Language Proccessing and Machine Learning techniques

Question 1. Who has the most lines? Who are the top 5?

Hence, we see that the top 5 characters with the most lines are :

Question 2. Who is the most verbose?

Hence, SISKO is the most verbose character

Question 3. What are the main topics each main character talks about? Define how you determined main character.

If we notice the bar plots above, we notice that there is a sharp fall in number of lines/words for characters beyond'DAX'

Now, we useLatent Dirichlet Allocation (LDA) for Topic Modelling of the dialogues of the main characters in the Series in an unsupervised setup.

About LDA

Note: We are assuming that there are at most 10 most frequent topics that the series is built around.

On training a multicore LDA model on our corpus with Bag-of-Words embedded text data, we obtained the 10 following topics in the Script:

Finally, we test the dialogues of each main character to find out theTOP 2 topics they talk about throughout the series.

Conclusions

Based on the obtained results we see that the Main Characters talk about the following in the series:

Question 4. How do the topics change throughout the series?

Conclusions

Based on the results as depicted on theiPython Notebook, here are the top 2 topics each of the main characters talks about with changing seasons in the series:

Season 1

Season 2

Season 3

Season 4

Season 5

Season 6

Season 7

Question 5. Do one more thing with the data that is interesting to you!

Let us investigate the simialrities between theMain Characters in the series based on their dialogues and how these similaritieschange over seasons. A close observation of the results would reveal the influence the characters have on each over time in the Star Trek universe, which in my opinion, is a very interesting insight.

We have used theSentence BERT (Pre-trained Transformer Network - RoBERTa Large) model to encode sentences into 1024-dimensional embedded vectors.

Plotting Season-wise Similarity Matrices between the Main Characters

Season 1

Season 2

Season 3

Season 4

Season 5

Season 6

Season 7

Conclusions

As we can observe from the heatmap of the similarity matrices for the main characters, the similarities are lower in the first season, signifying that the characters are new and unbiased and not influenced by the others. But with increasing seasons, we notice a gradual shift towardshigher similarities between the characters, with Season 4 and Season 5 showing significantly higher average similarities. This indicates that the characters themselves begins to influence the scriptwriting as the show progresses.

This is an interesting observation.

Lastly, the last season, i.e.Season 7 shows an abnormally high degree of similarity between the characters, except DAX. This could be due to the following reasons:

Question 6. If you had more time, what would you do with this data?

If I had more time, I would try to model the interactions between some of the important characters in the show by finding out the top supporting characters each of them share scenes with, most frequently. Building aconnected graph of these scene associations with the characters being vertices and the strengths of the edges being proportional to the screen time shared between any 2 of them, would allow deeper understanding of the substructures of the different scenes, settings and moods of the movie in conjunction with the topics covered by each character.

Understanding these structures in the Character Interaction Graph would be like knowing the fundamental premises of the series without actually watching it !