Trends in Wikipedia "Request for Adminship" Elections - Part 1

This is the first post I'll make about a nice Wikipedia data set I just got my hands on. This post will focus solely on the characteristics of voting and election averages over time.

An election win means that an editor becomes a Wikipedia page administrator, giving them more editorial power. In order for an election to commence, a candidate must be nominated. The nodes in the data set, for each election, are labeled as 'nominator', 'candidate', or 'voter.' The three kinds of votes a voter can cast are 1, 0, and -1, indicating, yea, nea, and neutral, respectively.

There are 2,794 elections, 2,391 candidates, 7,194 nodes, and 110,087 edges. 44.6 % of elections resulted in a win, and 55.4 % resulted in a loss. Interesting to note that, in every election which resulted in a loss for the candidate, the nominator was labeled 'UNKNOWN.' I don't know if that label exists before the election started, and therefore no one knew who the nominator was, or if the label was changed from a unique identifier to UNKNOWN after the election outcome, presumably to mitigate the shame of the nominator at having nominated a losing candidate.

The first plot shows the average vote score a candidate receives during an election. The first three subplots are broken down by the first vote the candidate received. For example, the subplot in the upper left shows the moving average of votes, given that the candidate's first vote was positive (+1). The upper middle subplot shows average vote for candidates whose first vote was negative. The upper right subplot shows candidates whose first vote was neutral. The bottom subplot is the aggregate. In each subplot, a line represents a single election. The y coordinate corresponds to the average value of all votes up to and including that value. Tap or zoom in on the images to see more clearly.

We can see that those whose initial vote was positive (blue) tend to vote positive thereafter, while those initial vote is negative (red) or neutral (green) tend to average out near zero. If the first vote is positive, there is a 61 % chance of winning the election. If the first vote is negative, there is only a 39 % chance. If the first vote is neutral, the chances of winning and loosing are 0.8 % and 5 %, respectively.

Now let's talk about average scores, since that's what the plot shows. Since the votes can only take the values +1, 0, -1, the average must be between +1 and -1. If the first vote is positive, the average vote at the end of the election is 0.6; if the first vote is negative, the ending average is -0.65; if the first vote is neutral, the average vote is -0.12. Interestingly, if the first vote is positive, the chance of losing the election is about 30 %; however, if the first vote is negative, the chance of winning the election is only 0.8 %. The key takeaway here is to get a first vote that is positive if you want to win.

It is also worth mentioning that the average length (number of votes) in a winning election is 57, while the average length in a losing election is 27. This difference is significant. It may be that people will not continue to vote if they perceive an imminent loss for the candidate.

Now that we've seen data on elections, let's look at candidates over all elections. In the plot below, each line represents a candidate across all elections for which they ran.

Looks fairly similar to the previous one. That is because most people only run once. In fact, the average number of election in which a candidate participates is just the number of elections divided by the number of candidates: 2,794 / 2,391 = 1.168. However, if a candidate wins their first election, their average campaigns jumps to 3.23.

Now let's look at the average voter.

Those who voted positive tend to vote more, as indicated by the length of the line (how far to the right the line stretches). Since the proportion of voters whose first vote is positive account for 76.8 % of all voters, the line in the bottom subplot (grey) tend to terminate with a positive average. Note the x-axis is log scaled. If a voter cast a positive first vote, their average vote is 0.79; if their first vote is neutral, their average is 0.21; and if their first vote is negative, their average vote is -0.43. This one sample, the first vote, says a lot.

Finally, let's look at a moving average of election outcomes per candidate.

Here we see that just under half of candidates won their first election. Those who won tend to run fewer times, as well, with a max of 3 campaigns. While those who lost their first election ran more times, with a max of 5 campaigns. Note the x axis is linear and goes from 0 to 4, giving 4-0+1=5 distinct values. Main takeaway here is that once people are elected, they are content and don't tend to run more. However, those who loose have everything to gain, and tend to run more in order to win. In this plot, as in the previous ones, the thickness of the line conveys the proportion of people who follow that path. Notice the Thickness of the red line between x = 0 and 1, where the y value increases from -1 to 0. This means that they lost their first election (average of -1 is -1), and they won their second (average of -1 and +1 is 0), at which point they stopped running. Ditto for the red line between x = 1 and 2, and between 2 and 3. Again, the bottom subplot is an aggregate of the top 2 subplots.

Upcoming posts will also include average election size (how many people are voting), who is nominating candidates, visualizations, time series, and more.

Anagrams using MapReduce and mrjob

This code discovers anagrams. The input is a text file dictionary with one word per line. The output is a text file.  

#! /usr/bin/env python

from mrjob.job import MRJob

class Anagram(MRJob):
 def mapper(self, _, word):
  letters = list(word)
  letters.sort()
  if word != '':
   yield letters, word

 def reducer(self, _, words):
  anagrams = [w for w in words]
  if len(anagrams) >= 2:
   yield len(anagrams), anagrams

if __name__ == '__main__':
 Anagram.run()

Call it like this:

Sample output:

The output file consists of two columns: the first column is the number of word in the set, and the second column is the set itself. 

Counting Letter Frequencies with MapReduce

This code shows how to use the mrjob library in python to count the occurrence of letters in a document.

#! /usr/bin/env python

from mrjob.job import MRJob

class LetterCount(MRJob):
 def mapper(self, key, value):
  for word in value.split():
   for letter in list(word):
    yield letter.lower(), 1
 def reducer(self, key, value):
  yield key, sum(value)

if __name__ == '__main__':
 LetterCount.run()

The input and output files can be passed in like so:

We pass dict.txt--just a text file with about 118,000 words, one per line--as the input and we specify lettercounts.txt as the output. The '<' and '>' symbols are called 'redirect' operators for stdin and stdout, respectively. Below is some sample output. Observe that 'e' is the most frequent letter, a fact that makes code breaking slightly easier. 

Movie and Genre Similarity using Link Analysis

Here we again measure movie genre similarity, but instead of simply counting genre co-occurrences, as in the previous post, we use link analysis. That is, we construct a bipartite graph, where the first part represents movies and the second part consists of genres, and . See the plot below for a sample from the imdb dataset. 

Each movie is connected to one or more genres. The average number of genres per movie is about 2.7. The data above represents a small portion of the total dataset. 
For this analysis below, we partition the dataset by decade, as before in the previous post. The set of movies and genres are denoted A and B, respectively Define s_A(X, Y) to be the similarity between movies, and s_B(x, y) to be the similarity between genres. We use the following equations to calculate the similarity between two movies X,Y ∈ A:

Similarly, we use the following equations to calculate the similarity between two genres x,y ∈ B:

Note that if X = Y, then s_A(X, Y) = 1. Also, if x = y, then s_B(x, y) = 1. That is, a movie X is 100% similar to itself; likewise, a genre x is perfectly similar to itself. 
C₁ and C₂ are decay constants in the range (0, 1). O(X) is the number of edges of movie X, and O_i(X) is the genre, x ∈ B, of the ith edge of X. The terms that use Y can be treated the same. I(x) is the number of edges of genre x, and I_i(x) is the movie, X ∈ A, of the ith edge of x. The terms that use y can be treated the same. 
We initialize all s_A(X, Y) and s_B(x, y) to 0 where X ≠ Y and x ≠ y; otherwise, s_A(X, Y) and s_B(x, y) are set to 1. 
All combinations of elements of A must be calculated, as must all combinations of elements of B. There are $ \binom{|A|}{2}} $  pairs of X,Y ∈ A, and \$ |B| \choose 2 \$ pairs of x,y ∈ B. The two equations above, for s_A(X, Y) and s_B(x, y), must be calculated for each pair, resulting in $ \binom{|A|}{2} + \binom{|B|}{2} $ calculations. These calculations must be repeated until the values of s_A and s_B converge. This may take several iterations. Here the criteria is that the total change over all pairs is less than 0.1.
After  s_A and s_B have been calculated
One advantage of the method above over the co-occurrence method is that we calculate movie similarity as well as genre similarity. This allows us to give movie recommendations. If we know a user likes movie X, then all we need to do is find movies similar to X and recommend them. For example, the results of the above algorithm gives that the four most similar movies to The Shawshank Redemption are:
What Becomes of the Broken Hearted?

Hidden Agenda

South Central

Some Mother's Son

This seems right. These movies definitely share some characteristics of The Shawshank Redemption. 

Another example should suffice. Here are the 4 most similar movies to The Matrix:

Dragon Inn

American Ninja V

Operation Delta Force 3: Clear Target

The Defender

A third example should further suffice things up. X = Pulp Fiction. Recommendations =

Safe House

The Rich Man's Wife

Darr

Incognito

Note: these similar movies are calculated by the decade; that is, if the query is a movie from the 1990's, the recommendations will only be movies from the 1990's. Doing all movies together takes too much time with the algorithm above.

Below is a side-by-side comparison of the results of this post and the results in Co-occurrences. Again, each column is for a decade, and the higher genres are the ones closest (most similar to) the Comedy genre.

So now we have genre-genre similarity and movie-movie similarity. In order to determine genre-movie similarity, we will employ personalized PageRank, a topic for another post.

Movie Genre Similarity using Genre Co-occurrences

This page shows a few ways of visualizing movie genre similarity. The similarity scores were obtained by iterating through an IMDB dataset of 10,000 movies from the 1950's to the 2010's, and counting the number of times each genre appeared with the other genres. For example, if a movie's genres were Action, Adventure, and Comedy, the co-occurrence counts would go up for the following pairs: (Action, Adventure), (Action, Comedy), and (Adventure, Comedy). The resulting co-occurrence count matrix is shown below. 
The color map is as follows: 1.0 is perfect similarity, and 0 is total non-similarity. 

The next three visualizations focus on individual genres. For the Comedy genre plot below, the more similar a genre, the higher the label. The plot below is not to scale. The genres are simply ordered most-to-least similar, top-to-bottom. The data is also time sliced by decade, as indicated at the bottom of the plot. Observe the changes in rank over time. 

This plot is to scale, although that scale is not shown. 

The plot below shows essentially the same thing as above, but using line plots, showing scale, and with lower similarity genres filtered out. 

There are 24 genres in total, and not enough room for them here. Feel free to request a particular genre, and I'll try to post it. 

For those interested in more detailed view, below is a plot using 1-year time slices (as opposed to the 10-year slices used above), for the Sci-Fi genre. 

The plot below, for the Mystery genre, is for 2-year time slices. Notice the rise in similarity to the Thriller genre, and the jagged wave shape of the Horror genre. Apparently, intense feelings of fear and shock, combined with puzzlement, are popular approximately once per generation. 

This analysis was made with python, specifically numpy, pandas, matplotlib. 

•