A post-election analysis of the Twitter Vote Report

“Our sweepers were empowered to ‘dismiss’ certain reports.”*
 – Twitter Vote Report’s “Lead Sweeper”

The Twitter Vote Report was promoted as a “non-partisan, all-volunteer network” in which voters with cell phones and accounts on the micro-blogging site Twitter could text message or post voting experiences. An analysis of the Twitter Vote Report shows, however, that the volunteer, social media project aimed at publicly identifying voting problems in the 2008 election was subject to indeterminate amounts of human judgment and technological error.

In running data on Nevada’s publicly posted reports and in email correspondence with Twitter Vote Report volunteers, I found that the Twitter Vote Report had so many potential loopholes that the report’s results should be interpreted only as a glimpse at the sharing of views of a relatively small number of voters — just over 11,000 out of more than 110 million. [NOTE: Read the complete data analysis here: Twitter Vote Report Data Analysis.]

The Twitter Vote Report provided a novel opportunity for people to share their voting experiences in real-time. At the same time, the report’s processes for collecting and posting user experiences indicates too many subjective interpretations of what to post and what not to post in order to be considered reliable examples of the average voter experience. 

Other problems were that:

Vote reports did not follow recommended use of hashtags (hashtags are words preceded with a number sign and are searchable in Twitter)

Vote reports were potentially dominated by a minority of users (one user in Nevada had 38% of the posted reports for that state)

Vote reports were subjectively approved to be publicly posted on the Vote Report site 

Vote reports were subjectively “dismissed” from public posting using inconsistently enforced criteria (see below)

Vote reports were duplicated

Vote reports were posted that appeared irrelevant as to intent of the Vote Report (one example from Nevada: “@DwayneH dude I love egg salad sandwiches, but liquor store is scary. downtown scarier, even. best of luck. #votereport”).

Based on a data analysis of Nevada’s 72 posted reports (retrieved November 7, 2008), combined with posts that ended up only being viewed on individual Twitter accounts and not on the Twitter Vote Report Web site, it is clear the Twitter Vote Report did not post an undetermined number of vote reports because of user error, volunteer judgment and because of technological factors.

In an email to me, the Vote Report’s Allison Fine says,

“All of the tweets that had the vote report hashtag made in into the datastream that was ran down the middle of the site — or could be seen on Twitter directly. If the message didn’t have a location (which automatically came through the cell phone apps on iphone and android but had to be included in the message if straight through Twitter) then it couldn’t be posted on the visuals. One thing that became clear as a lesson learned for us was that we were successful in getting the word out (all by word of mouth and free media, btw, there was no money spent on this effort at all) about the votereport hashtag, but users didn’t pay much attention to the other hashtags of zip code, machines or registration if they were having problems which makes visualizing tweets challenging.”

To test the system, I posted my vote report on election day, a link to my audio log of my own voter experience (in which I had a lot of fun). I simply posted my tweet with the #votereport hashtag. My friend Ryan, who was blogging for Reno-area newspapers during the election, also noticed a number of tweets were not being displayed on the Vote Report site. We wanted to know why. What criteria were being used to prevent posts from being publicly viewed?

So I asked a Vote Report volunteer. He said:

“Our sweepers were empowered to ‘dismiss’ certain reports. Nothing was deleted from the archive of reports, but anything dismissed was kept out of the live stream (emphasis added). The guideline for accepting a report was that it needed to have location and at least one other valid tag (#machine, #registration, #wait, etc.). When possible, the sweepers appended a valid tag if it was implied by the content of the report.”*

In analyzing what was publicly posted, it appears a good deal of empowerment was going on. Results of what was actually posted in Nevada indicate volunteers did not follow their own guidelines, or significant technological errors occurred which led to the appearance of widespread “dismissal” of vote reports. (In the world of quantitative research, anytime data is “dismissed,” ignored or thrown out, a red flag should be raised and results should be considered suspect.)

Had the Twitter Vote Report followed clearly specified criteria for posting reports, in addition to employing a system for rectifying possible volunteer bias, it is likely many of these problems would not have occurred.

The irony is that in the process of trying to identify so-called voter suppression, the 2008 election’s results ended up more accurate and reliable than the Twitter Vote Report.

Nevertheless, the report is considered a success by supporters. Allison Fine says that the Twitter Vote Report,

“was an amazing ride that went from an idea posted on a blog to a real-life site in less than a month powered entirely by volunteers. It is the continuation of ways that open-source efforts using cell phones can contribute to large-scale events like natural disasters and elections — and it’s just the beginning of what we’ll see develop over the next few years.”

Let us hope that better protocols are developed and followed before that happens.

[NOTE: Read the complete data analysis here: Twitter Vote Report Data Analysis.]

———————————————————–

*Here is the email from the Twitter Vote Report’s “lead sweeper”:

“1. Our sweepers were empowered to “dismiss” certain reports. Nothing was deleted from the archive of reports, but anything dismissed was kept out of the live stream.  The guideline for accepting a report was that it needed to have location and at least one other valid tag (#machine, #registration, #wait, etc.).  When possible, the sweepers appended a valid tag if it was implied by the content of the report.

“Ultimately, it was up to individual judgment, and errors were possible as well.  Also, there were a number of other technological factors that might have led to a report not being posted.  

“To explain the reports that you saw that seemed not to meet any criterion for relevance:

• you may have seen reports to which sweepers had appended appropriate tags.  we did not display those tags alongside the reports, because we wanted to publish the reports as the user submitted them
• some reports never got swept.  all reports went live automatically, and were pulled later if they were dismissed

“2. I’m fairly certain that all the data is available and queryable if you know where to look and how to query.  I’ll follow up woth the tech team to facilitate your  request.”