The Presidential Debates Seen Through a Prism of Art and Natural Language Processing

In this work, computational linguistics is performed on transcripts of recent debates involving Hillary Clinton and Donald Trump.

Politicians have a tendency to repeat core concepts. I determined which phrases are used most frequently, focusing on three, five and seven word phrases and applying natural language processing algorithms.

I parsed over half a million words: 218,265 from Donald Trump and 308,319 from Hillary Clinton using an open source Python Library, Natural Language Toolkit (NLTK) (1).

I then take the results and infuse them onto vector graphics and animation tools to represent the data in an artistic form. Press “run” above to interact with the art.

If you're interested in the code and larger data sets, see below.

Ngrams Presidential Debates: 3, 5 and 7 word phrases.

The dataset

The University of California Santa Barbara provides the main primary debates. In order to get the corpus of each candidate we extract the text of each candidate.

Python has a great Natural Language Processor called Natural Language Toolkit (NLTK). It was developed by Steven Bird and Edward Loper in the Department of Computer and Information Science at the University of Pennsylvania. (1)

              import nltk
              # from nltk import word_tokenize
              from nltk.tokenize import TweetTokenizer
              from nltk.util import ngrams
              from collections import Counter
              import string

              #open and read file, make all text lowercase
              text = open("debates/hill/all.txt", "r").read().lower() 
              # remove all punctionation from text.
              text = "".join([ch for ch in text if ch not in string.punctuation])


              # that TweetTokenizer does not split the contraction into two parts: didn't, 'did', "n't" tockenize the text
              tknzr = TweetTokenizer()
              token = tknzr.tokenize(text)


              # counter
              trigrams = ngrams(token,3)
              fivegrams = ngrams(token,5)
              sevengrams = ngrams(token,7)

              #output three, five and seven word phrases
              print Counter(trigrams)
              print Counter(fivegrams)
              print Counter(sevengrams)