Authors & Average Sentence Lengths

Christine Tsu

Can we identify the author of a book from its average sentence length?

Given a book, can we use computational methods to identify its author from a specific group of authors? In the following activity, we will first explore some of the Wolfram Language’s basic text analysis methods, then apply them to create an author guessing game using Dynamic!

Basic Functions & Background

The Wolfram Language has multiple different sources of text, one of which is ExampleData. We can download and use one of the texts in ExampleData for some introductory text analysis.

ExampleData["Text"]

Let’s first try the Declaration of Independence. We can initialize it as a variable so that we don’t have download it every time we run a piece of code.

declaration=ExampleData[{"Text","DeclarationOfIndependence"}]

First, we will take a look at how many words are in the Declaration of Independence using the function WordCount.

WordCount[declaration]

Digging a bit deeper, we can look at how many times each word appears in the text. There are lots of different words in the text, so the output for WordCounts is likely going to be very long and unwieldy. To make it a bit more manageable, we’re going to use Part (the double square brackets) to take just the first twenty items, which will be the top twenty most common words.

WordCounts[declaration][[1;;20]]

As might be expected, the words that appear the most are basic articles, prepositions, and pronouns. If we want to check what part of speech a word is, we can use the function PartOfSpeech.

PartOfSpeech["and"]

Stop words are common words that hold little meaning, like “the,” “and,” and “a.” We can use the function DeleteStopwords to get rid of them, then apply WordCounts again, which will likely give us more interesting results.

WordCounts[DeleteStopwords[declaration]][[1;;20]]

Now we can see more unique words, including people’s names (Thomas, John, William, George, etc.) and nouns like “Laws” and “Colonies.”

We can also create a word cloud for the Declaration of Independence. Words that appear more frequently will be larger, while words that appear less frequently will be smaller.

WordCloud[declaration]

TextSentences creates a list of the sentences in a given text. Its output will be very long, so we can add a semicolon to suppress it.

sentences=TextSentences[declaration];

As an example, we can get the 7th sentence in the Declaration of Independence.

sentences[[7]]

TextStructure allows us to see the structure of the sentence in a visual diagram format.

TextStructure[sentences[[7]]]

We can also find the sentiment of this sentence using the Wolfram Language’s built-in classifier function, which will tell us if the sentence is Positive, Neutral, Negative, or Indeterminate.

Classify["Sentiment",sentences[[7]]]

What if we want to try more sentences? We can use RandomChoice to get a random sentence and classify its sentiment.

randomSent=RandomChoice[sentences]
Classify["Sentiment",randomSent]

It seems like a lot of these sentences are neutral, which makes sense because of the formal diction and older language of the Declaration of Independence. We can see exactly how many sentences are Positive, Neutral, and Negative using Counts, Map, and a pure function. As a reminder, pure functions use a slot (#) and are tagged with an ampersand (&). The shortcut for Map is /@, so basically, each element in the list of sentences below will be inserted at the slot in turn.

Counts[Classify["Sentiment",#]&/@sentences]

As might be expected, the overwhelming majority of sentences are Neutral.

Getting the Text of Books from Entities

We can get text from entities, as well. As a brief review, entities are objects that store a lot of information and data, and they can quickly be accessed through natural language input. Press Ctrl + = to enter free-form input.

LinguisticAssistant

Let’s type in Lewis Carroll, the author of Alice in Wonderland. Press Enter to create the entity. The gray text at the right side of the entity box tells you the type of entity that the free-form input was interpreted as. If it is correct, click the check mark. If not, three dots will appear, and you can select an alternate interpretation from there. For our example, Lewis Carroll the person immediately comes up, so we can just click the check mark.

(Entity["Lewis Carroll", Entity["Person","LewisCarroll::5kn6j"], "Entity[\"Person\", \"LewisCarroll::5kn6j\"]", "person"])

Now that we have our entity, we can see what properties it has!

(Entity["Lewis Carroll", Entity["Person","LewisCarroll::5kn6j"], "Entity[\"Person\", \"LewisCarroll::5kn6j\"]", "person"])["Properties"]

We can ask for all kinds of information, from Lewis Carroll’s astrological sign to notable facts. Keep in mind, though, that some of these properties may have missing data.

Let’s say we’d like to see Lewis Carroll’s notable books. We can do this in two ways; both will give us the same output.

(Entity["Lewis Carroll", Entity["Person","LewisCarroll::5kn6j"], "Entity[\"Person\", \"LewisCarroll::5kn6j\"]", "person"])[(EntityProperty["notable books", EntityProperty["Person","NotableBooks"], "EntityProperty[\"Person\", \"NotableBooks\"]"])]

(Entity["Lewis Carroll", Entity["Person","LewisCarroll::5kn6j"], "Entity[\"Person\", \"LewisCarroll::5kn6j\"]", "person"])["NotableBooks"]

Now we can get the text of one of his notable books. The property is called “Plaintext” because it’s regular text, not code.

EntityValue[(Entity["Through the Looking Glass", Entity["Book","ThroughTheLookingGlass"], "Entity[\"Book\", \"ThroughTheLookingGlass\"]", "book"]),"Plaintext"]

Getting the Text of Books by Specific Authors

As mentioned above, some entities have properties with missing data. Not every book’s text is available through the entities. For our activity, we want to have the text for as many books as possible. This will give us more accurate average sentence lengths, and we will also have more texts for our author guessing game. We will use Arthur Conan Doyle, Jane Austen, and Charles Dickens, as they all have many notable books listed in their entities. Their writing is usually quite different in style, content, and genre, so hopefully this will also help us distinguish between them.

As an example, we can quickly see Arthur Conan Doyle’s notable books.

EntityValue[(Entity["Arthur Conan Doyle", Entity["Person","ArthurConanDoyle::f98ww"], "Entity[\"Person\", \"ArthurConanDoyle::f98ww\"]", "person"]),"NotableBooks"]

Using all of these books may become a bit unwieldy, so let’s just focus on the first five for now. We can get the plaintext for those books as follows.

EntityValue[#,"Plaintext"]&/@{(Entity["The Adventures of Sherlock Holmes", Entity["Book","TheAdventuresOfSherlockHolmes"], "Entity[\"Book\", \"TheAdventuresOfSherlockHolmes\"]", "book"]),(Entity["The Complete Sherlock Holmes", Entity["Book","TheCompleteSherlockHolmes"], "Entity[\"Book\", \"TheCompleteSherlockHolmes\"]", "book"]),(Entity["The Hound of the Baskervilles", Entity["Book","TheHoundOfTheBaskervilles"], "Entity[\"Book\", \"TheHoundOfTheBaskervilles\"]", "book"]),(Entity["Vampire Stories", Entity["Book","VampireStoriesSirArthurConanDoyle"], "Entity[\"Book\", \"VampireStoriesSirArthurConanDoyle\"]", "book"]),(Entity["The Lost World", Entity["Book","TheLostWorld1912"], "Entity[\"Book\", \"TheLostWorld1912\"]", "book"])};

Review of Association

It might be helpful to keep track of the titles of the works, as well, so let’s use an association to link each book’s title to its plaintext. Associations are like lists, but every item in the list consists of a key linked to a value. We saw an association in the output of WordCounts and Counts above, but let’s look at another example.

association=Counts[{a,b,c,d,e,a,b}]

Here, the function Counts used an association to track how many times each letter appeared in the list. We can ask for just the keys or just the values, and we can also ask for the value that corresponds to a certain key.

Keys[association]

Values[association]

association[a]

Creating an Association of Titles and Plaintexts

To create our association, we first need to add a piece of code that gets the book’s title (its “name”) before the plaintext. Then, we can link the two values using an arrow (type “-” followed by “>”). Finally, we apply Association to the entire thing.

textAssociation=Association[(EntityValue[#,"Name"]->EntityValue[#,"Plaintext"])&/@{(Entity["The Adventures of Sherlock Holmes", Entity["Book","TheAdventuresOfSherlockHolmes"], "Entity[\"Book\", \"TheAdventuresOfSherlockHolmes\"]", "book"]),(Entity["The Complete Sherlock Holmes", Entity["Book","TheCompleteSherlockHolmes"], "Entity[\"Book\", \"TheCompleteSherlockHolmes\"]", "book"]),(Entity["The Hound of the Baskervilles", Entity["Book","TheHoundOfTheBaskervilles"], "Entity[\"Book\", \"TheHoundOfTheBaskervilles\"]", "book"]),(Entity["Vampire Stories", Entity["Book","VampireStoriesSirArthurConanDoyle"], "Entity[\"Book\", \"VampireStoriesSirArthurConanDoyle\"]", "book"]),(Entity["The Lost World", Entity["Book","TheLostWorld1912"], "Entity[\"Book\", \"TheLostWorld1912\"]", "book"])}];

That output is a bit hard to read, so we can use Part to take a look at each item in the list.

textAssociation[[1]]

When we go through the list, we can see that items 2 and 4 just say “Missing[NotAvailable].” To clean the data, we can simply use DeleteMissing to remove these items.

textAssociation[[2]]

textAssociation[[4]]

doyle=DeleteMissing[textAssociation];

Let’s check the length of the association to make sure the items were deleted.

Length[doyle]

Creating a Function for Getting Plaintext

Now we have the text of three books for Arthur Conan Doyle. We need to apply this same process for Jane Austen and Charles Dickens, so we can create a function for the sake of efficiency. This will also allow us to easily add more authors if we decide to later. Here is a simple example of the format for a user-defined function:

calculation[x_]:=x^2+2

The function name is “calculation,”and its argument is x. To signify this, we add a pair of square brackets after “calculation” and put x with an underscore inside. Once we add a colon and equals sign, the argument x is labeled with a green color.

calculation[2]

For our function, we can just condense all of the above steps into a single piece of code. As the argument, we’ll take the author name as a string. Instead of using Ctrl + = to get the author entity, we can use the function Interpreter, which will take a string and try to interpret it in the specified form (in this case, as a person). Although we’d likely get more accurate results with a larger sample size of books, that would take a long time to run, so we’ll only be taking the first 5 notable books for each author.

EntityValue[Interpreter["Person"]["jane austen"],"NotableBooks"]

getPlaintext[author_]:=DeleteMissing[Association[EntityValue[#,"Name"]->EntityValue[#,"Plaintext"]&/@EntityValue[Interpreter["Person"][author],"NotableBooks"][[1;;5]]]]

austen=getPlaintext["jane austen"];

If we want to see which Jane Austen books ended up in our association, we can simply use Keys.

Keys[austen]

Let’s do the same for Charles Dickens.

dickens=getPlaintext["charles dickens"];
Keys[dickens]

Calculating Average Sentence Lengths

Using these texts, let’s calculate average sentence lengths for each author. The average sentence length of a text is just the number of words divided by the number of sentences. We can use WordCount to get the number of words. Then, TextSentences can be used to break the text into sentences, and we can apply Length to the list of sentences to find how many sentences there are.

WordCount[doyle[[1]]]/Length[TextSentences[doyle[[1]]]]

Our answer is automatically outputted as a fraction, so we probably want to change this to a decimal value.

N[WordCount[doyle[[1]]]/Length[TextSentences[doyle[[1]]]]]

Let’s use Map with a pure function to apply this to the entire list of Arthur Conan Doyle texts.

N[WordCount[#]/Length[TextSentences[#]]]&/@doyle

The average sentence lengths appear to be similar. Although our sample size is very small, this could mean that Doyle’s writing is pretty consistent overall — which would be very helpful for distinguishing it from others!

Let’s now calculate an overall average sentence length for Doyle so that we can compare it to the other authors. To do this, we’ll first need to combine his texts using StringRiffle, which concatenates all elements of a given list with a separator between each one. For our separator, let’s use a period and a space. This will ensure that TextSentences can split apart the different texts after they’ve been joined into one string.

StringRiffle[{"a","b","c"},". "]

doyleString=StringRiffle[Values[doyle],". "];
doyleSL=N[WordCount[doyleString]/Length[TextSentences[doyleString]]]

Creating a Function to Calculate Average Sentence Length

Like before, creating a function will make this process more efficient for other authors.

avgSentLength[text_]:=N[WordCount[text]/Length[TextSentences[text]]]

We can now apply this function to Austen and Dickens. This may take a little while to run.

austenSL=avgSentLength[StringRiffle[Values[austen],". "]]

dickensSL=avgSentLength[StringRiffle[Values[dickens],". "]]

Let’s display this data in a bar chart so that we can visualize it more easily.

BarChart[{doyleSL,austenSL,dickensSL},ChartLabels->{"Doyle","Austen","Dickens"},ChartStyle->"Rainbow"]

We’ve only run the functions on a small sample of the data, so our results are likely not very accurate. However, it looks like the three authors have pretty distinct average sentence lengths. This could just be a personal stylistic choice, but an interesting consideration is that these authors all wrote their books in different time periods — Austen’s novels are from the late 18th century, Dickens’s are from the mid-19th century, and Doyle’s are from the late 19th century to early 20th century. With this in mind, differing average sentence lengths could also be an indication of literary trends/conventions of the time — though probably a limited one, at best.

Testing a New Book

Let’s try calculating the average sentence length for another book: A Study in Scarlet by Doyle.

scarletSL=avgSentLength[(Entity["A Study in Scarlet", Entity["Book","AStudyInScarlet1887"], "Entity[\"Book\", \"AStudyInScarlet1887\"]", book])["Plaintext"]]

Creating a number line could help us see which author this value is closest to. First, let’s create a list of our average sentence lengths for each of the authors. We will wrap each name in a tooltip so that when we hover over it on the number line, it will display the author’s name. Tooltip’s first argument is the expression that will be labeled, and the second argument is the label.

Tooltip[1,"x"]

tooltipSLs={Tooltip[doyleSL,"Doyle"],Tooltip[austenSL,"Austen"],Tooltip[dickensSL,"Dickens"]}

Now we can put this on a NumberLinePlot.

NumberLinePlot[tooltipSLs]

Finally, let’s add the new value for A Study in Scarlet.

NumberLinePlot[{tooltipSLs,Tooltip[scarletSL,"A Study in Scarlet"]}]

We can clearly see that A Study in Scarlet is closest to the other Doyle books, so we would easily be able to identify him as its author.

Author Guessing Game

Let’s now test if, given a random book by Doyle, Austen, or Dickens, we are able to identify the author just by looking at the average sentence length. We will use some randomness to create a guessing game interface.

Calculating Avg. Sentence Lengths for a New Sample of Books

We can first compile a new sample of 3-4 plaintexts from each author. We could definitely try more books, but this would take longer to run. For the most part, we’ll try to take different books than we previously used. However, because Austen has fewer books available, we will have to use many of the same books for her.

Join[EntityValue[(Entity["Arthur Conan Doyle", Entity["Person","ArthurConanDoyle::f98ww"], "Entity[\"Person\", \"ArthurConanDoyle::f98ww\"]", "person"]),"NotableBooks"][[14;;16]],EntityValue[(Entity["Jane Austen", Entity["Person","JaneAusten::n2229"], "Entity[\"Person\", \"JaneAusten::n2229\"]", "person"]),"NotableBooks"],EntityValue[(Entity["Charles Dickens", Entity["Person","CharlesDickens::h6573"], "Entity[\"Person\", \"CharlesDickens::h6573\"]", "person"]),"NotableBooks"][[6;;10]]]

Applying logic similar to before, we will make a list of rules, this time linking the author to the plaintext.

allBooks=DeleteMissing[(#["Author"]->#["Plaintext"])&/@Join[EntityValue[(Entity["Arthur Conan Doyle", Entity["Person","ArthurConanDoyle::f98ww"], "Entity[\"Person\", \"ArthurConanDoyle::f98ww\"]", "person"]),"NotableBooks"][[14;;16]],EntityValue[(Entity["Jane Austen", Entity["Person","JaneAusten::n2229"], "Entity[\"Person\", \"JaneAusten::n2229\"]", "person"]),"NotableBooks"][[1;;5]],EntityValue[(Entity["Charles Dickens", Entity["Person","CharlesDickens::h6573"], "Entity[\"Person\", \"CharlesDickens::h6573\"]", "person"]),"NotableBooks"][[6;;10]]]];

We can now calculate the average sentence length of each of the books. This may take a couple of minutes to run.

allSL=avgSentLength[#]&/@Values[allBooks]

To organize this information, we want to create a list of pairs in which the first element is the author name and the second element is the average sentence length. This will help us easily pull out the information we want for the guessing game.

Transpose[{{a,b,c},{d,e,f}}]

Using Transpose, we can combine the list of allBooks keys (author names) and the list of average sentence lengths to create our pairs.

authorsSLs=Transpose[{Keys[allBooks],allSL}]

Now, we can pick a random text using RandomChoice, and it will give us the author name and the text’s average sentence length.

randomText=RandomChoice[authorsSLs]

To get the author’s name, we’ll need to dig into two levels of lists:

randomText[[1,1]]

We can display the author’s name with a string using Row.

Row[{"Author: ",randomText[[1,1]]}]

We can also get the average sentence length by asking for the second part of the list.

randomText[[2]]

Buttons

Buttons perform a certain action when they are clicked. The first argument is the label, which will be displayed on the button, and the second argument specifies the action.

Button["Click Me",Print[5]]

Dynamic

We can make a button that, when clicked, will reveal the author’s name. We’ll create a variable called showAuthor, initialized as False. When showAuthor is True, the author name will be displayed.

We’ll need to use Dynamic for this because when showAuthor is changed to True, we want the If statement below to automatically execute again.

showAuthor=False;
Button["Click to show the author!",showAuthor=True]
Dynamic[If[showAuthor,Row[{"Author: ",randomText[[1,1]]}],""]]

To organize our code a bit, let’s define our variables locally using DynamicModule.  We’ll also place the button and author line in a column.

DynamicModule[
{showAuthor=False},
Column[{
Button["Click to show the author!",showAuthor=True],
Dynamic[If[showAuthor,Row[{"Author: ",randomText[[1,1]]}],""]]
}]
]

Now let’s add the average sentence length above this.

DynamicModule[
{showAuthor=False},
Column[{
StringJoin["Avg. Sentence Length: ",ToString[randomText[[2]]]],
Button["Click to show the author!",showAuthor=True],
Dynamic[If[showAuthor,Row[{"Author: ",randomText[[1,1]]}],""]]
}]
]

This is the basic interface for the random text we chose. However, we want to be able to randomly pick and guess more texts, so we will make a button for that, inserting it in a row with our other button.

DynamicModule[
{showAuthor=False},
Column[{
StringJoin["Avg. Sentence Length: ",ToString[randomText[[2]]]],
Row[{
Button["Click to show the author!",showAuthor=True],
Button["Generate a new text",randomText=RandomChoice[authorsSLs]]
}],
Dynamic[If[showAuthor,Row[{"Author: ",randomText[[1,1]]}],""]]
}]
]

We need to fix a couple of things in our code above: first, the average sentence length should be dynamic because it needs to update when the random text changes. Second, we need to set showAuthor to False when a new text is generated so that the author name stays hidden.

DynamicModule[
{showAuthor=False},
Column[{
Dynamic[StringJoin["Avg. Sentence Length: ",ToString[randomText[[2]]]]],
Row[{
Button["Click to show the author!",showAuthor=True],
Button["Generate a new text",{randomText=RandomChoice[authorsSLs],showAuthor=False}]
}],
Dynamic[If[showAuthor,Row[{"Author: ",randomText[[1,1]]}],""]]
}]
]

Let’s also localize the randomText variable so that everything we need is wrapped in DynamicModule.

DynamicModule[
{showAuthor=False,randomText=RandomChoice[authorsSLs]},
Column[{
Dynamic[StringJoin["Avg. Sentence Length: ",ToString[randomText[[2]]]]],
Row[{
Button["Click to show the author!",showAuthor=True],
Button["Generate a new text",{randomText=RandomChoice[authorsSLs],showAuthor=False}]
}],
Dynamic[If[showAuthor,Row[{"Author: ",randomText[[1,1]]}],""]]
}]
]

We can now organize the interface so that it appears more clean. We will use alignment and spacing options for Column and add a frame around the entire thing.

DynamicModule[
{showAuthor=False,randomText=RandomChoice[authorsSLs]},
Framed[
Column[{
Dynamic[StringJoin["Avg. Sentence Length: ",ToString[randomText[[2]]]]],
Row[{
Button["Click to show the author!",showAuthor=True],
Button["Generate a new text",{randomText=RandomChoice[authorsSLs],showAuthor=False}]
}],
Dynamic[If[showAuthor,Row[{"Author: ",randomText[[1,1]]}]," "]]
},Alignment->Center,Spacings->2],
FrameMargins->5,ImageSize->{350,250},Alignment-> Center
]]

Adding the Number Line

The final step is to add our number line so that we can easily compare the new average sentence length with the known ones! Let’s put it after the average sentence length, making sure to add Dynamic so that it automatically updates, as well.

DynamicModule[
{showAuthor=False,randomText=RandomChoice[authorsSLs]},
Framed[
Column[{
Dynamic[StringJoin["Avg. Sentence Length: ",ToString[randomText[[2]]]]],
Dynamic[NumberLinePlot[{tooltipSLs,Tooltip[randomText[[2]],"Mystery Text"]},ImageSize->300]],
Row[{
Button["Click to show the author!",showAuthor=True],
Button["Generate a new text",{randomText=RandomChoice[authorsSLs],showAuthor=False}]
}],
Dynamic[If[showAuthor,Row[{"Author: ",randomText[[1,1]]}]," "]]
},Alignment->Center,Spacings->2],
FrameMargins->5,ImageSize->{350,250},Alignment-> Center
]]

There are definitely a couple of outliers here, which is to be expected because our data samples were so small. However, overall, it’s interesting to see how we can use average sentence lengths to identify authors.

Extension: Shortest & Longest Sentences

We’ve looked at average sentence lengths, but what if we want to see the shortest and longest sentences in a text? As an example, let’s take Jane Austen’s first plaintext.

austen[[1]];

Because quotation marks do not add any meaning and may interfere with our calculation of sentence lengths, let’s delete them from the string. We can use the function StringDelete for this.

StringDelete["abcd efgh","f"]

In order to delete quotation marks, we need to type “\””.

StringDelete[austen[[1]],"\""];

Let’s now split the text into sentences.

austenSents=TextSentences[StringDelete[austen[[1]],"\""]];

Cleaning the List of Sentences

austenSents[[1;;3]]

As we can see above, this list includes some elements that are not sentences (e.g. “(1814)”). We can’t completely eliminate this issue, but to reduce the error, we can use StringContainsQ.

StringContainsQ["(1814)",Alphabet[]]

We can now apply this to every element in our list using Table. If the line of code above returns false, that means the element in the list doesn’t have any letters and therefore isn’t a valid sentence, so we will get rid of it.

cleanedSents=Table[If[StringContainsQ[sent,Alphabet[]]==True,sent,Nothing],{sent,austenSents}];

Sorting the List of Sentences

We can sort these sentences by length, which will allow us to easily get the shortest and longest ones. For this, we will use SortBy. As its first argument, SortBy takes the list to be sorted. As its second argument, it takes the function that it will use to sort the list.

SortBy[{"abc","a","ab"},StringLength]

We can apply this to our list of sentences. Let’s take a look at the first 100 sentences in the sorted list, which should be the 100 shortest ones.

SortBy[cleanedSents,StringLength][[1;;100]]

We can condense this list using DeleteDuplicates.

sortedSents=DeleteDuplicates[SortBy[cleanedSents,StringLength]];

Getting the Shortest & Longest Sentences

Finally, we can get the first item in the list, which should be the shortest sentence!

sortedSents[[1]]

To get the longest sentence (the last item in the list), we can ask for Part -1.

sortedSents[[-1]]

To clean this up a bit, we can use StringReplace to change the extra line breaks into spaces.

StringReplace[sortedSents[[-1]],"\n"->" "]

WordCount[StringReplace[sortedSents[[-1]],"\n"->" "]]

There’s clearly a large difference between the shortest and longest sentences! The shortest sentence is just a 1-word exclamation, while the longest sentence (180 words) is very complex and includes a lot of clauses.

References

This activity idea and the inspiration/model for a few “Basic Functions & Background” examples are from Ms. Brown’s CT@TP activity “Pride & Prejudice & Cucumber”: http://www.abbymath.com/CT/
Stop words definition: https://towardsdatascience.com/text-pre-processing-stop-words-removal-using-different-libraries-f20bac19929a

Christine Tsu - Spring 2022
Torrey Pines High School
Advanced Topics in Math II & Computational Thinking Club CT@TP