Welcome nerdy GFOPs! If you’ve made it this far, you’re likely prepared for the nerdiness (and lengthy post) that you are about to dive into. While our assumption was that some GFOPs would be curious, at a high level, about how the Premier League mood table works, my hope was that some GFOPs who work in analytics, data science, or anywhere in the data platform realm would be curious enough about this to dig into what I’ve done, discuss it, and hopefully build upon it to do cool things with the mood table or in our professional work. With that greeting, let me get to a couple acknowledgements and then we’ll get nerdy.
There are two blogs that provided much of the foundation for the mood table. First, the inspiration for the mood table came from a blog written by a friend/former boss of mine that now works at Microsoft, Brad Ball (b|t). Please go check out his blog post to better understand the foundation of the mood table. The second blog that proved to be an immense help can be found here. It was written by Toon Vanhoutte. I don’t know Toon, but I owe him a beer or three for a wonderful blog post laying out, in necessary detail, how to schedule the execution of the Azure Logic Apps. Without being able to schedule the enabling and disabling of these logic apps, the mood table would likely be too expensive to run regularly. So, cheers to you, Brad and Toon! Let’s get into the guts of the mood table.
The foundation of the mood table is an Azure Logic App created for each of the 20 clubs. Because of character limits on some of the object names, I was forced to use creative abbreviations for some of the clubs. Apologies for the name below, Arsenal supporters, but I am a Spurs supporter after all! This screenshot shows what is inside each logic app – I’ll go into more detail about each component below the screenshot.
The first component (labeled “When a new tweet is posted”) is the Twitter connector itself. It signs into Twitter and searches for tweets containing the club’s official Twitter handle. It passes those tweets into the second component (labeled “Detect Sentiment”). That is where the sentiment analysis actually takes place. That component analyzes the text (not images – yet) of the tweet and scores it from 0 (most negative) to 1 (most positive). The sentiment score is added to that dataset and passed to the third component – the for each loop container.
The loop container (labeled “For each 2”) parses each tweet and tags it with all clubs mentioned within the tweet. It then inserts that data into an Azure SQL Database (via the “Insert row” component in red) for storage, querying, and analysis.
When I spun up the mood table for the first time, this is all it was. There was no scheduling component and it was just streaming tweets into my Azure SQL DB during all the matches. That proved to be prohibitively expensive, so let me walk you through some gotchas before we get into the scheduling components of the mood table.
Gotchas (i.e. how to not bankrupt yourself in Azure)
Gotcha 1: Twitter Connector Does Not Die (unless you kill it)
As I first started using the Twitter connector (screenshot below) I noticed that it asked a relatively simple question: “How often do you want to check for items?” Based on my research and discussion with others, our understanding was that the connector would wake up on that defined interval, check for tweets meeting the search criteria, and then shut down again until the next interval.
Unfortunately, that understanding was incorrect. The first full weekend I ran the mood table it cost several hundred dollars as the connector never stopped running because there were always tweets for it to find. Because all of this is billed at a fixed cost per operation, the cost increased exponentially throughout the weekend, especially for matches involving large clubs with supporters throughout the world. My wife was relatively unimpressed by my mistake, so I wanted to mention it here to save you from a similar fate should you begin experimenting with this.
Gotcha 2: Scheduling Logic Apps Always Works (Eventually)
After a discussion with Azure Support to clarify exactly how the connector works, I dug into trying to figure out how to schedule these logic apps so they would enable and disable on command. That would allow me to do two things: 1) capture the time intervals Rog and Davo wanted during or after the match and 2) keep the cost manageable so I would not have to sell a limb or one of my children to continue financing the mood table.
That led me to Azure Job Collections and Azure Scheduler and Toon’s wonderful blog on how to wire this up. I will not rehash his blog here, but please go read it if you want a step-by-step walkthrough of what I did. One note I will add to Toon’s blog is that, wherever he references <workflow name>, that is referring to the name of the logic app itself. It took some trial and error for me to discover that so I want to save my readers that time and effort.
My job collection contains an enable and disable job for each club’s logic app (for a total of 40 jobs). They run on set schedules (that I manually update) to capture the 2-3 minutes before the final whistle and the 7-8 minutes after the final whistle. Rog wants those instant post-match sentiments, the gut reaction you have to your club’s match as the final whistle blows.
This all seemed a brilliant idea until the next set of matches rolled around and I realized that some of the disable jobs failed their initial attempt – leading to yet more money accidentally donated to Microsoft. The Azure team’s holiday party likely got a bit better as a result of my mistakes!
Luckily, you can apply retry logic to these jobs (see below). I strongly recommend this as it will save you many disapproving looks from your significant other for making it rain in the Azure bill once again.
Gotcha 3: Having A Laptop At The Pub Is Dorky (use the Azure iOS app)
The first weekend or two I ran the mood table stuff I was monitoring it via a laptop while watching matches at the pub. I’m told, from reliable sources, that that made me look like a dork. It also meant that I could not reliably monitor the jobs enabling and disabling on schedule unless I was somewhere with 1) room for a laptop and 2) decent Wi-Fi.
To use an obviously hypothetical example relayed to me by a friend of mine (wink wink, nudge nudge), let’s say a big match ended in the middle of a Sunday church service and “my friend” wanted to make sure the jobs kicked off on schedule. Taking out a laptop during the service would be deemed inappropriate by my friend’s family, his priest, and likely incurs a risk of him being struck by lightning were the Good Lord to catch a glimpse of such activity. How could my friend solve this issue?
He used the Azure Portal iOS app found here (it’s also available for Android). That allowed “my friend” to ensure that the club’s logic apps enabled and disabled on schedule and that the tweets were properly captured. I’m also delighted to report that “my friend” was not struck by lightning while checking this during church.
Things Deemed Too Optimal For The Mood Table
Originally the mood table had a live component with it: a Power BI report that accepted the tweet information via a streaming dataset and showed a live table throughout the matches with up-to-the-second rankings. While it was fascinating, Rog wanted to focus on the end-of-match sentiments for the static table.
The ability to analyze streaming sentiment data in this way, however, may make an appearance for individual Premier League matches or, possibly, a major soccer tournament taking place this summer (no, not the Alternative Facts World Cup).
Conclusion
If you’ve read all the way to the end, I salute you! I’ve enjoyed working with the guys on the Premier League mood table and I’ve enjoyed putting together this deep dive post for those of you that are interested. As I mentioned, I look forward to questions or corrections (throw them in the comments) or connections via Twitter or LinkedIn. Cheers (and come on you Spurs)!
Like this:
Like Loading...
