Pride & Prejudice & Cucumber

Pride and Prejudice is the only book I have read so many times that I have lost count. I find it fun to use as a text for analysis in Mathematica, especially since doing that combines two of my favorite things.

This activity is a compilation of three smaller explorations inspired by discussions in class.

TextStructure

While reading EIWL, I saw an example using TextStructure (Chapter 35: Natural Language Understanding). This breaks down a sentence into grammatical components.

TextStructure["You can do so much with the Wolfram Language."]

TextStructure["You can do so much with the Wolfram Language.","ConstituentGraphs"]

Let’s try it with some longer sentences. We will randomly select sentences from Pride and Prejudice.

First, download the text of the novel using ExampleData. Without a semicolon ; the entire text appears as output. Include the semicolon to hide all of that. We will call this text pp.

pp=ExampleData[{"Text","PrideAndPrejudice"}];

Evaluating pp will show the text. Note that it is all contained within one string (grouped with "  "). For quotes typed within the text \" is used. (Delete the output cell after evaluating the following and seeing how it is constructed.)

InputForm[pp]

We can use TextSentences to break a string into a list of sentences. Each sentence is now its own string.  Name this list. This takes a moment to evaluate.

ppsentences=TextSentences[pp];

Let’s see what some of this looks like. The following takes parts 500 through 510 of the list. Once evaluated, click somewhere within the output and press “Enter.” This will turn the output into input and you can see the list of strings a little more clearly. Can you count the 11 sentences?

ppsentences[[500;;510]]

Next assign a variable name to one sentence randomly selected from the text.

ppsent=RandomChoice[ppsentences]

Look at the text structure of that sentence. (Scroll to the right if you have a very long sentence.)

TextStructure[ppsent]

Reevaluate the previous two inputs to try new sentences.

Sentiment Analysis

What is the current sentence selected?

ppsent

The Classify function can access some built-in classifiers that have already been trained. One of these is “Sentiment” that classifies a string as Positive, Negative, or Neutral. Sometimes it returns Indeterminate.

Classify["Sentiment",ppsent]

Put the random sentence selection together with the classifier. Do not use a semicolon after the RandomChoice so that you are able to see the sentence. Evaluate this several times.  What do you think? Do you agree with the sentiment classifications? What might be elements of a sentence that yield positive or negative results?

ppsent=RandomChoice[ppsentences]
Classify["Sentiment",ppsent]

Mostly I find that I agree with the sentiment result, though not always. How definitive is the result produced by Classify? When a classifier function chooses a category, it actually generates probabilities for each category then outputs the one with the highest probability. Add the option “Probabilities” to Classify.

ppsent=RandomChoice[ppsentences]
Classify["Sentiment",ppsent,"Probabilities"]

The result is an Association \[LeftAssociation] \[RightAssociation]. Sorting more easily displays the highest probability each time we evaluate it.

ppsent=RandomChoice[ppsentences]
ReverseSort[Classify["Sentiment",ppsent,"Probabilities"]]

How many sentences are in the entire text?

Length[ppsentences]

Is it possible to classify them all at once? Yes! The Classify function in this case is listable. Instead of asking for the sentiment of each sentence individually, we can have it classify the entire list of sentences at once. (Evaluate this without and then with a semicolon.)

sentlist=Classify["Sentiment",ppsentences];

Since that is difficult to look at, let’s simply count the number of times each answer appears.

Tally[sentlist]

We can visualize this information as a PieChart. However, we need to reorganize the data a bit first.

data=Transpose[Tally[sentlist]]

PieChart values will be the second part of data (data[[2]]) and the labels are the first part of data (data[[1]]).

PieChart[data[[2]],ChartLabels->data[[1]]]

Click on the largest sector of the PieChart.

Putting this all Together

Bring it together to make a program that downloads text and generates a PieChart of the sentiment analysis.

sents=TextSentences[ExampleData[{"Text","PrideAndPrejudice"}]];
data=Transpose[Tally[Classify["Sentiment",sents]]];
PieChart[data[[2]],ChartLabels->data[[1]]]

How does this look with different text? Within ExampleData, replace “PrideAndPrejudice” with “USConstitution.”

sents=TextSentences[ExampleData[{"Text","USConstitution"}]];
data=Transpose[Tally[Classify["Sentiment",sents]]];
PieChart[data[[2]],ChartLabels->data[[1]]]

What other texts are easily accessible within Mathematica? ExampleData is one available resource.

ExampleData["Text"]

sents=TextSentences[ExampleData[{"Text","AliceInWonderland"}]];
data=Transpose[Tally[Classify["Sentiment",sents]]];
PieChart[data[[2]],ChartLabels->data[[1]]]

WikipediaData is another resource.

sents=TextSentences[WikipediaData["computer"]];
data=Transpose[Tally[Classify["Sentiment",sents]]];
PieChart[data[[2]],ChartLabels->data[[1]]]

To read that article, evaluate WikipediaData[“computer”] or view the article directly (linked here):
https://en.wikipedia.org/wiki/Computer

WikipediaData["computer"]

Try another article.

The Wolfram Data Repository has additional texts easily downloaded, though you may need to be logged in using your Wolfram ID. The Wolfram Data Repository has much more than just text if you are looking for computable data.

ResourceObject["The Scarlet Letter"]

To extract the text, use ResourceData as below. View the text, but then delete that output since it is so long.

ResourceData[ResourceObject["The Scarlet Letter"],"Plaintext"]

Did you notice that it begins with a rather long introduction? Perhaps we do not want that included in our sentiment analysis. To get only the text of the novel, we can find the beginning. First, name the whole text as a variable.

scarlet=ResourceData[ResourceObject["The Scarlet Letter"],"Plaintext"];

StringPosition tells the where in the text the given string begins and ends. These are positions by character.

StringPosition[scarlet,"I. THE PRISON DOOR"]

StringTake will take a specified section of a string. The double semicolon ;; represents the span of elements through the start position and end position.

StringTake[scarlet,86288;;87000]

Without an end position, we get the rest of the text without knowing the value of the final position. Since this is what we want, give it a variable name for easier use. (Delete the output or reevaluate with a semicolon.)

scarlet2=StringTake[scarlet,86288;;]

sents=TextSentences[scarlet2];
data=Transpose[Tally[Classify["Sentiment",sents]]];
PieChart[data[[2]],ChartLabels->data[[1]]]

Activity or Project Idea

To see what else is available in the Wolfram Data Repository go to https://datarepository.wolframcloud.com and “Browse by Type” or for text directly visit https://datarepository.wolframcloud.com/type/Text.

I noticed within that list “State of the Union Addresses: Complete text of State of the Union addresses from 1790 to 2018.” For a dataset like this, there should be a systematic way to extract speeches by year or by decade or by century! It could be interesting to compare the sentiment analysis for different time periods or different presidents or different political parties. Are there trends? Or do presidents prefer to pontificate positive proclamations?

Network Graphing A.I.

An activity in class with some images led us to the following. Rather than using images, for now it will be simpler to use color swatches.

colors=RandomColor[50]

We can organize the colors into a network by nearest neighbors.

NearestNeighborGraph[colors]

However, that does not tell us much without knowing what color each vertex represents.

NearestNeighborGraph[colors,VertexLabels->Automatic]

The second argument of NearestNeighborGraph indicates levels of neighbors to include. Try different values.

NearestNeighborGraph[colors,4,VertexLabels->Automatic]

Do you notice any grouping? CommunityGraphPlot finds groups and illustrates those distinctions more clearly.

CommunityGraphPlot[NearestNeighborGraph[colors,4,VertexLabels->Automatic]]

Try changing the level of neighbors. Try using a new set of random colors or more colors (not too many).

CommunityGraphPlot[NearestNeighborGraph[RandomColor[100],4,VertexLabels->Automatic]]

Communities of Words

Would this same type of network graph work with text? I used Pride and Prejudice again to play with this.

pp=ExampleData[{"Text","PrideAndPrejudice"}];

Rather than breaking up the text into a list of sentences, we will break it up into a list of words.

ppwords=TextWords[pp]

Now select 30 random words from the text.

ppsample=RandomSample[ppwords,30]

Use the same code as for the Community Graph Plot of colors above, but change change RandomColor[...] to the name of your word list.

CommunityGraphPlot[NearestNeighborGraph[ppsample,4,VertexLabels->Automatic]]

How does this compare to a FeatureSpacePlot?
Try changing the level of neighbors above and comparing to the FeatureSpacePlot.

FeatureSpacePlot[ppsample]

When I did this in class, the following list of words came up and we put them in a CommunityGraphPlot.

ppsample0={"public","though","cucumber","inconvenience","Elizabeth","again","though","than","She","and","will","bring","a","And","she","your","of","has","Hunsford","readily","not","the","you","him","in","room","married","Collins","Elizabeth","was"};
CommunityGraphPlot[NearestNeighborGraph[ppsample0,4,VertexLabels->Automatic]]

More than one student remarked on the word “cucumber” since it seemed incongruous with the other words and the types of words expected by those familiar with the novel.

Cucumber

The cucumber comments lead me to wonder where in the book that word appears. I know the story rather well and there are only a few scenes describing food. It seems unlikely that cucumber would be a common word within Pride and Prejudice. I have an idea where it might be, but let’s try to approach finding it computationally.

StringCount will count the number of times a particular string appears within another string.

StringCount["This is a string.","is"]

Why is “is” in there twice? StringPosition will tell us where “is” is.

StringPosition["This is a string.","is"]

We have two positions given. Using spaces we can isolate “is” as its own word.

StringPosition["This is a string."," is "]

So, how many times does “cucumber” appear in the entire Pride and Prejudice string?

StringCount[pp,"cucumber"]

Where does it appear?

StringPosition[pp,"cucumber"]

That is not particularly useful with such a long text. How long is the entire text string?

StringLength[pp]

Well, that gives us some idea that “cucumber” is roughly in the middle of the story. (How close to the middle is it? Can you figure that out?)

Let’s try to identify the sentence containing the word. Remember that we broke up the text into sentences earlier.

ppsentences=TextSentences[pp]

If we pull out one sentence, we can test it for containing the word. Rather than using StringPosition to find it, we can ask whether a smaller string is contained in a larger string with StringContainsQ. See the examples below.

StringContainsQ["This is a string.","is"]

StringContainsQ["This is a string.","that"]

StringContainsQ["This is a string.","this",IgnoreCase->True]

Evaluate the following a few times.

ppsent=RandomChoice[ppsentences]
StringContainsQ[ppsent,"Elizabeth"]

Doing this to find cucumber would be very time consuming. However, we will check all of the sentences at once!

Set it up so that if the string contains what we want, the sentence is the output. If it does not have the desired string, it returns nothing. Evaluate the following a few times to see how it works.

ppsent=RandomChoice[ppsentences]
If[StringContainsQ[ppsent,"Elizabeth"],ppsent,Nothing]

Recall that Map will apply a function to every element of a list.

Map[f,{a,b,c,d}]

We will Map the “If” across the entire list of sentences. (Note that we use a pure function here. Each sentence within the list ppsentences goes into the slot # position, just where ppsent was above. Remember that an ampersand & is needed to mark the end of a pure function.) Our answer will be a list of sentences containing the given string. (Change the word to “prejudice.” There are too many sentences with “Elizabeth.”)

Map[If[StringContainsQ[#,"prejudice"],#,Nothing]&,ppsentences]

Now, let’s find that one instance of “cucumber.”

Map[If[StringContainsQ[#,"cucumber"],#,Nothing]&,ppsentences]

We now have the one sentence in the book that contains the word “cucumber”! However, is that enough to confirm where it is in the book? By “where” I mean, what is happening in the book at that point?

I believe it agrees with where I think it is in the story, but I’d like to see a little more around it. The following code finds the position of that sentence and then outputs the sentence along with its neighbor sentences.

pos=Position[ppsentences,Map[If[StringContainsQ[#,"cucumber"],#,Nothing]&,ppsentences][[1]]][[1,1]];
ppsentences[[pos-1;;pos+1]]

Exactly the scene I expected!

Other Single Words

Is “cucumber” special because it appears only once in the entire book? Or are there other singletons?

ppwords=TextWords[pp]

We used Tally above to count the sentiments. What if we tally all of the words individually.

Tally[ppwords]

We can organize this better with Sort.

Sort[Tally[ppwords]]

What did that do? More useful would be to sort by the number of times a word appears. Note above that Tally produces a list of lists. We want to sort the tally results by the second element (the last element) of each pair.

SortBy[Tally[ppwords],Last]

There are so many words, that this list is unmanageable to look through. Let’s extract any word that has a tally of 1. We can use Cases and look for elements of the tally list that match the pattern {_, 1} where the underscore _ indicates anything in that position.

singles=Cases[Tally[ppwords],{_,1}]

How many singletons are there?

Length[singles]

Take a close look at the singles list. Simply look at the words that appear in the portion of the list above since clearly we are not seeing all of them. Some words are capitalized like “Single” and I know that word appears more than once since it is in the famous first line of the book.

First[ppsentences]

Tally apparently pays attention to capitals and lower-case letters. Also, in the tally list I notice some hyphenated words. Do we really want to count those as unique words? I don’t; at least not for the purposes of this exercise.

I am going to change the book. First we can make the entire book lower case.

ToLowerCase[pp]

We can change all hyphens to spaces. (Either use a semicolon or delete the output after you look for any hyphens. There won’t be any hyphens, so don’t spend too much time looking.)

StringReplace[ToLowerCase[pp],"-"->" "]

Now, let’s break this edited version into a list of words.

ppedit=TextWords[StringReplace[ToLowerCase[pp],"-"->" "]]

And we will do the Tally and Cases again.

singles=Cases[Tally[ppedit],{_,1}]

Length[singles]

Well, we eliminated about 300 words. Let’s look at the singletons now. It will be simpler to rename the list to be just the words since we don’t need the tallies anymore.

singles=singles[[All,1]]

Can you find “cucumber” in all of that? In what order are the words listed? What is the longest word in the list?

SortBy[singles,StringLength]

Reverse@SortBy[singles,StringLength]

Apply the same code we used to find the “cucumber” sentence to find sentences containing other interesting singleton words.

Map[If[StringContainsQ[#,"cucumber"],#,Nothing]&,ppsentences]

Map[If[StringContainsQ[#,"uniting"],#,Nothing]&,ppsentences]

Last[ppsentences]

Map[If[StringContainsQ[#,"calico"],#,Nothing]&,ppsentences]

Map[If[StringContainsQ[#,"elope"],#,Nothing]&,ppsentences]

What happened here? The word “elope” is in the singleton list, yet we just found about a dozen sentences.

Map[If[StringContainsQ[#," elope "],#,Nothing]&,ppsentences]

Let’s try one more. This word stood out when I glanced through the list since I was surprised it was used only once.

Map[If[StringContainsQ[#,"superiority"],#,Nothing]&,ppsentences]

Let’s see a little more around it.

pos=Position[ppsentences,Map[If[StringContainsQ[#,"superiority"],#,Nothing]&,ppsentences][[1]]][[1,1]];
ppsentences[[pos-1;;pos+1]]

Classify["Sentiment",%]

Update: One-Liner Competition 2018

In October 2018, I attended the annual Wolfram Technology Conference in Champaign, Illinois. Each year as part of the conference, attendees are invited to participate in the annual One-Liner Competition by submitting “the most stupendous, entertaining, amazing, surprising, astounding effects you can achieve in 128 characters or fewer of tweetable Wolfram Language code.”

I used the sentiment analysis of Pride and Prejudice as inspiration and was able to condense the process into exactly 128 characters. I actually had it down to 126 characters, so I decided to make it 3D!

Though it didn’t win any awards, I’m proud to share below what I submitted as “Pride & Prejudice & Sentiment.”

{l,d}=Transpose@Tally@Classify["Sentiment",TextSentences@ExampleData[{"Text","PrideAndPrejudice"}]];PieChart3D[d,ChartLabels->l]

To see more from the 2018 One-Liner Competition, read the article by Chris Carlson linked below.
http://blog.wolfram.com/2018/10/25/the-winners-of-the-2018-one-liner-competition

Additional Activity Ideas

Apply some of the analysis in this activity to other texts.

What is the probability that “cucumber” would appear when randomly selecting 30 words from the text? What is the probability that any singleton would be in such a list? What is the probability that more than one singleton would appear in such a list?

Try the “State of the Union” project described above.

Find the average number of words per sentence in the book. What about another book? Try several books with the same author. Could you identify an author (from a group of authors) simply by this value?

What is the largest (smallest) number of characters used in a sentence? What sentence is it? What is the average? How many sentences have exactly the average number of characters? (Mean? Median? Mode?)

What is the percentage of unique words in the text? Compare different authors with this value. Do “easier” books have fewer unique words? Do “easier” books have fewer long words (words greater than some threshold number of characters)? Answer similar questions for sentence length.

Credits

EIWL: An Elementary Introduction to the Wolfram Language, Second Edition by Stephen Wolfram, 2018

Pride and Prejudice by Jane Austen, 1813

The Scarlet Letter by Nathaniel Hawthorne, 1850

Abby Brown
www.abbymath.com
CT@TP: Computational Thinking at Torrey Pines High School
Spring 2018 (Updated November 2018)