How we are tracking the US Presidential Campaign in Real Time on Twitter
A large number of users share and discuss their interests or what’s happening in the world, on Twitter. And they do this almost every single day of their lives. These tweets open the door to a variety of insights — so much so that marketers can use the aggregated Twitter data to spot trends, analyze sentiment, and connect with their target audience in a far more granular way.
At Media iQ, we want to deliver robust insights. So, for the 58th US Presidential campaign, our marketing analysts at Media iQ felt it would be great to see if Americans were tweeting #Democrats more or less than #Republicans, or if the social media sentiment soared high for any particular candidate.
We built a real-time social activation political dashboard that enables “live” updates of how social reactions for each party and election candidate changed as the election campaigning continues.
The MiQ team worked hard for days to provide a solution that enables real time social media insights and tracks real-time variation in social sentiment. For instance, the image above depicts the geographies that favour ‘Democratic’ or ‘Republican’ in the number of tweets. It shows the social media sentiment for both the parties, top keywords etc.
Media iQ’s system analyzes over 100,000 tweets per day, and the entire dashboard has been built to help clients inform their campaigning with valuable insights.
Real-time Social Media Tracking and Insights Using GNIP
With Twitter’s integration with GNIP, Media iQ chose to partner with them to create more sophisticated data sets and build better data enrichments.
So how does it work? Saksham Srivastava, Senior Software Engineer at Media iQ talks about it here.
Firstly, the rule server is set up in our system. When the analysts type in a rule
(an example is listed below),
(profile_country_code:us) (brand name OR @brand name ) tag:brand name, and tags the brand name, it identifies the type of data that is being received.
The rules are tied into the rule server and these rules are ingested into the GNIP server every night. Based on these rules, GNIP receives the full firehose of data and uses PowerTrack as a way to filter down to get complete coverage of the data needed. To handle real time data we use Rabbit Message Queue as the message broker. It gives the application a common platform to ingest and consume tweets, and tweets are stored here until received by our database. Rabbit MQ comes with an easy-to-use management UI that allows us to monitor and control every aspect of the route.
Once the tweets are in the message queue we apply an algorithm that picks up 150 tweets at a time for sentiment analysis in real time, which could be either negative, positive or neutral sentiment corresponding to the twitter text. The tweets are sent to the sentiment analyzer in bulk and the responses are received in bulk as well.
We then add geolocation mapping to tweets from our extensive geo data received in partnership with data providers like Digital Element. This is important to understand the true location from where every single tweet is coming. With this, we are able to classify tweets accurately based on the country.
We run a regular ‘feed service’ that picks up the tweets accumulated over time and then populates our RedShift database. The tables on RedShift is populated with data from every single tweet, and the tables have columns like time zone, mentions, hashtags, language, date, sentiment, keywords among others.
Post this, using our in-house BI (Business Intelligence) tool we can run social sync streams and generate extensive insights from Twitter data for our clients or campaigns. Currently, the average rate of data ingestion is about 100k tweets per day.
Our Social Sync solution uses this social data, and allows advertisers to deliver impactful messages during significant ‘Social Moments’. With the help of our proprietary analytical engine- AiQ it ingests real-time feed of social data, analyses the sentiment trends and layers this with user-level ‘micro’ data in order to deliver a synced ad to social commenters.
Therefore, by leveraging Twitter data, we are quickly able to compare the number of the tweets mentioning either #Democrats or #Republicans, the geographies that are in favour of each party, and even the social media sentiment for each candidate.
This gives users insights in real time on who is leading the race for the Presidential campaign, and which candidate has a better clout than the rest.