Category: MIB Mood Table

Men In Blazers PL Mood Table – How It Works

Men In Blazers PL Mood Table – How It Works

Odds are, if you’re reading this, you represent the part of the Venn diagram where my Microsoft data platform buddies and Premier League soccer fans intersect or you clicked the link in today’s Men In Blazers Raven because you were curious how the Premier League mood table works. Rest assured, if you can think of different ways to do this and tweaks to make, I’d love to hear about them in the comments (unless you’re an Arsenal supporter – mostly kidding)! I’m passionate about data solutions and am always trying to stay abreast of rapidly changing platforms and technologies. Feel free to reach out to correct me, congratulate me, or just to connect with me professionally!

Before we go any farther, I owe my buddy Bradley Ball (b|t) a tip of the cap and several cold beverages for the original blog that provided the inspiration behind the Mood Table. If you’re interested in a deeper dive than this high-level overview, click through to my deep dive post here or his original post here.

First off, the foundation of the mood table begins with creating an Azure Logic App (basically a workflow container) for each of the twenty Premier League clubs. Those logic apps contain a Twitter connector that, when active (more on that later), searches Twitter for tweets containing each club’s official Twitter handle. Those tweets are collected by the Twitter connector and fed into the Sentiment Analysis function of the Cognitive Services API found within Azure. The Sentiment Analysis function adds a sentiment score to the tweet (0 is completely negative and 1 is completely positive) and passes it on to the final couple of steps in the process.

Once the sentiment score of the tweet is calculated, the tweet and its corresponding data (text, score, originating account, date created, etc.) is fed into a loop container that separates each club handle in the tweet and connects the tweet to every club tagged within the tweet. Finally, all of this data is stored in an Azure SQL Database (basically a database in the cloud) for persistent storage, querying, and analysis.

It is important to point out that each club’s logic app runs for a 10-minute window around the end of each match. It captures roughly the last 2-3 minutes of match action as well as the 7-8 minutes immediately following the full time whistle. As Rog told me, the intention is to capture the sentiment of tweets akin to when you turn to your mates (or buddies, in ‘Merican) at the pub and offer your instant analysis on your club’s success, failure, or utter mediocrity in the match.

Before I get into a brief description of the query I use to provide this critical information to Rog and Davo, I do want to mention that there was originally a real-time component of the mood table as well. I had written a Power BI report that showed a live table throughout the matches as tweets and their data streamed into the Power BI streaming dataset. After chatting with Rog, though, that was deemed far too optimal for MIB (not to mention quite pricey for my personal Azure account) and we settled on the current sentiment information surrounding the final whistle. It may make an appearance at some point, though!

Once all this Azure coolness has done the heavy lifting, the actual query that compiles the results is fairly straightforward. It takes the average of the sentiment scores tagged to a specific club and then ranks those clubs (and their corresponding average scores) from the highest average to the lowest average. The only data it excludes is tweets that contain only an image (as the sentiment analysis cannot score images – yet). Sadly, your hilarious soccer GIFs are not yet understood by the mood table.

In a nutshell, that’s it. I look forward to comments, questions, and connections. As I mentioned, I have a much more detailed technical deep dive into that that you can read here. This has been a blast to work on and we have some potentially interesting plans for it moving forward. Finally, there is no truth to the rumor that Rog is paying the Russians to create bots to elevate Everton to the top spot. That is just the natural optimism and enthusiasm of Evertonians shining through! To the football…

Men In Blazers Premier League Mood Table – Deep Dive

Men In Blazers Premier League Mood Table – Deep Dive

Welcome nerdy GFOPs! If you’ve made it this far, you’re likely prepared for the nerdiness (and lengthy post) that you are about to dive into. While our assumption was that some GFOPs would be curious, at a high level, about how the Premier League mood table works, my hope was that some GFOPs who work in analytics, data science, or anywhere in the data platform realm would be curious enough about this to dig into what I’ve done, discuss it, and hopefully build upon it to do cool things with the mood table or in our professional work. With that greeting, let me get to a couple acknowledgements and then we’ll get nerdy.

There are two blogs that provided much of the foundation for the mood table. First, the inspiration for the mood table came from a blog written by a friend/former boss of mine that now works at Microsoft, Brad Ball (b|t). Please go check out his blog post to better understand the foundation of the mood table. The second blog that proved to be an immense help can be found here. It was written by Toon Vanhoutte. I don’t know Toon, but I owe him a beer or three for a wonderful blog post laying out, in necessary detail, how to schedule the execution of the Azure Logic Apps. Without being able to schedule the enabling and disabling of these logic apps, the mood table would likely be too expensive to run regularly. So, cheers to you, Brad and Toon! Let’s get into the guts of the mood table.

The foundation of the mood table is an Azure Logic App created for each of the 20 clubs. Because of character limits on some of the object names, I was forced to use creative abbreviations for some of the clubs. Apologies for the name below, Arsenal supporters, but I am a Spurs supporter after all! This screenshot shows what is inside each logic app – I’ll go into more detail about each component below the screenshot.

logic_app_1

The first component (labeled “When a new tweet is posted”) is the Twitter connector itself. It signs into Twitter and searches for tweets containing the club’s official Twitter handle. It passes those tweets into the second component (labeled “Detect Sentiment”). That is where the sentiment analysis actually takes place. That component analyzes the text (not images – yet) of the tweet and scores it from 0 (most negative) to 1 (most positive). The sentiment score is added to that dataset and passed to the third component – the for each loop container.

The loop container (labeled “For each 2”) parses each tweet and tags it with all clubs mentioned within the tweet. It then inserts that data into an Azure SQL Database (via the “Insert row” component in red) for storage, querying, and analysis.

When I spun up the mood table for the first time, this is all it was. There was no scheduling component and it was just streaming tweets into my Azure SQL DB during all the matches. That proved to be prohibitively expensive, so let me walk you through some gotchas before we get into the scheduling components of the mood table.

Gotchas (i.e. how to not bankrupt yourself in Azure)

Gotcha 1: Twitter Connector Does Not Die (unless you kill it)

As I first started using the Twitter connector (screenshot below) I noticed that it asked a relatively simple question: “How often do you want to check for items?” Based on my research and discussion with others, our understanding was that the connector would wake up on that defined interval, check for tweets meeting the search criteria, and then shut down again until the next interval.

logic_app_2

Unfortunately, that understanding was incorrect. The first full weekend I ran the mood table it cost several hundred dollars as the connector never stopped running because there were always tweets for it to find. Because all of this is billed at a fixed cost per operation, the cost increased exponentially throughout the weekend, especially for matches involving large clubs with supporters throughout the world. My wife was relatively unimpressed by my mistake, so I wanted to mention it here to save you from a similar fate should you begin experimenting with this.

Gotcha 2: Scheduling Logic Apps Always Works (Eventually)

After a discussion with Azure Support to clarify exactly how the connector works, I dug into trying to figure out how to schedule these logic apps so they would enable and disable on command. That would allow me to do two things: 1) capture the time intervals Rog and Davo wanted during or after the match and 2) keep the cost manageable so I would not have to sell a limb or one of my children to continue financing the mood table.

That led me to Azure Job Collections and Azure Scheduler and Toon’s wonderful blog on how to wire this up. I will not rehash his blog here, but please go read it if you want a step-by-step walkthrough of what I did. One note I will add to Toon’s blog is that, wherever he references <workflow name>, that is referring to the name of the logic app itself. It took some trial and error for me to discover that so I want to save my readers that time and effort.

My job collection contains an enable and disable job for each club’s logic app (for a total of 40 jobs). They run on set schedules (that I manually update) to capture the 2-3 minutes before the final whistle and the 7-8 minutes after the final whistle. Rog wants those instant post-match sentiments, the gut reaction you have to your club’s match as the final whistle blows.

logic_app_3

This all seemed a brilliant idea until the next set of matches rolled around and I realized that some of the disable jobs failed their initial attempt – leading to yet more money accidentally donated to Microsoft. The Azure team’s holiday party likely got a bit better as a result of my mistakes!

Luckily, you can apply retry logic to these jobs (see below). I strongly recommend this as it will save you many disapproving looks from your significant other for making it rain in the Azure bill once again.

logic_app_4.PNG

Gotcha 3: Having A Laptop At The Pub Is Dorky (use the Azure iOS app)

The first weekend or two I ran the mood table stuff I was monitoring it via a laptop while watching matches at the pub. I’m told, from reliable sources, that that made me look like a dork. It also meant that I could not reliably monitor the jobs enabling and disabling on schedule unless I was somewhere with 1) room for a laptop and 2) decent Wi-Fi.

To use an obviously hypothetical example relayed to me by a friend of mine (wink wink, nudge nudge), let’s say a big match ended in the middle of a Sunday church service and “my friend” wanted to make sure the jobs kicked off on schedule. Taking out a laptop during the service would be deemed inappropriate by my friend’s family, his priest, and likely incurs a risk of him being struck by lightning were the Good Lord to catch a glimpse of such activity. How could my friend solve this issue?

He used the Azure Portal iOS app found here (it’s also available for Android). That allowed “my friend” to ensure that the club’s logic apps enabled and disabled on schedule and that the tweets were properly captured. I’m also delighted to report that “my friend” was not struck by lightning while checking this during church.

Things Deemed Too Optimal For The Mood Table

Originally the mood table had a live component with it: a Power BI report that accepted the tweet information via a streaming dataset and showed a live table throughout the matches with up-to-the-second rankings. While it was fascinating, Rog wanted to focus on the end-of-match sentiments for the static table.

The ability to analyze streaming sentiment data in this way, however, may make an appearance for individual Premier League matches or, possibly, a major soccer tournament taking place this summer (no, not the Alternative Facts World Cup).

Conclusion

If you’ve read all the way to the end, I salute you! I’ve enjoyed working with the guys on the Premier League mood table and I’ve enjoyed putting together this deep dive post for those of you that are interested. As I mentioned, I look forward to questions or corrections (throw them in the comments) or connections via Twitter or LinkedIn. Cheers (and come on you Spurs)!

 

 

 

 

 

Coming 12/8/17 – Premier League Mood Table How-to

Coming 12/8/17 – Premier League Mood Table How-to

Greetings GFOPs (and others who have clicked on this post)! There were a few mentions on social media that I’d be publishing two blogs here this afternoon. The first was going to be a high-level, layman’s explanation for how the Premier League Mood Table I did for Men In Blazers (b|t) works and the second was going to be a deep dive into the nuts and bolts of how I built it, the mistakes I made, the ways I fixed those mistakes, and just how flipping cool Azure Logic Apps and Social Sentiment Analysis are. That second blog will form the foundation for a SQL Saturday/conference talk I intend to submit and present throughout 2018 (feel free to contact me if this interests you and your user group/conference/organization).

These blogs were originally timed with the 12/1 release of MIB’s latest Raven (their newsletter for readers who don’t listen to their podcast – register here if you’re not a subscriber) containing an interview with me, thus the handful of tweets and other social posts mentioning today as the day to take a look at my site.

Unfortunately, in typically suboptimal MIB fashion, the 12/1 Raven has been delayed until 12/8, thus Mauricio Pochettino (and I) make the sad face above. My blogs will be released in conjunction with that Raven, so we’ll see you back here next Friday. If you’re interested in soccer and/or data and/or social media analysis potentially relevant to your company, it will be worth a read. Have a great weekend and see you back here Friday, December 8!