How to Measure Super Bowl Success
The Super Bowl, despite being a decent game this year, is a pretty fascinating marketing event. Our culture, which will generally go to great lengths to avoid commercials, will, on this one day of the year, pay as much attention to the advertising as they do to the entertainment programming that it’s interrupting.
Television advertising has generally been sustained by its flawed metrics. Viewership counts are wildly inflated by companies like Neilsen and exhaustive data about how people generally ignore commercials and are rarely able to retain even the simplest message is universally discarded.
This football game may be the only time all year that reach numbers are actually accurate since viewers are actually paying attention and are more likely to go to the bathroom on 3rd and 2. Under that assumption, it’s hard to put any ad ahead of any other on the basis of reach. So how do you measure success?
The USA Today, which represents a fairly decent cross section of the country, launched their Super Bowl Ad Meter to try to address the problem. The methodology was similar to many focus groups where about 300 volunteer viewers were given a dial to express when they saw something they liked, disliked or were neutral about. The results were somewhat discouraging to many advertising people, as the group generally responded more favorably to the low concept work from brands like Bud Light, Doritos and Pepsi Max, the last of which employed the old can-in-the-balls punchline. Whether or not these 300 people were in line with the demographic that will run out and buy a new Chrysler or Web domain is debatable but it’s just about as good as you can do in measuring at least the entertainment value of the spots.
Since we’re pretty much at the peak of inflated expectations in the evolution of social media, it was no surprise that Radian6 and Mullen got together to come up with a social media measurement of success, the Brand Bowl. It was an interesting idea except for the fact that it is based largely on an automated sentiment analysis technology that has some major shortcomings. The methodology takes a large number of keywords related to the spots and computes positive tweets plus neutral tweets minus negative tweets to score each ad. This works great if every tweet was like “I loved the VW ad…Darth Vader rocks!” or “God I hate Bud Light…Coors 4ever!” but that’s not how social media works. Tweets like “What’s up with Chrysler? Do I need to love Detroit to buy one of their gas guzzlers now?” would be recorded as neutral, which is scored the same as positive, when it’s not. There’s no super computer that has been able to detect sarcasm and Twitter is rife with it. Back when I had interns, I tested the sentiment engines of multiple monitoring services against the common sense of college graduates and found that sentiment engines were always off my a minimum of 20%, which isn’t statistically acceptable.
So, which do you trust, the minute-by-minute feelings of a random group of 300 people or a large volume of Tweets sent through a potentially flawed sentiment engine? Neither is perfect but which is more flawed?
Let’s look at the somewhat controversial Groupon spots by CP+B.
The USA Today focus group didn’t really like this campaign. In fact, in came in at #41, well into the lower half of all the spots surveyed.
In Radian6/Mullen’s Brand Bowl, the spot is currently resting at #3, up from #7 when I looked last night. There have been over 4,000 tweets as of writing this, yet it is the only spot in the top 10 with a negative sentiment score. People are talking about this spot…but what are they saying?
You don’t need a algorithm to see that people were left with a bad taste in their mouth from these spots. Everyone from Read Write Web to Paid Content to the commenters on Groupon’s blog are chiming in and the vast majority have concluded that it is an insensitive, ham-fisted approach to commercial comedy (sadly, I just found out the Christopher Guest was involved with directing the spots). Groupon, which didn’t suffer from much of an awareness problem before the ads, is now a central brand in the discussion of how brands exploit a crisis without sensitivity. I’m sure Kenneth Cole is very happy they’ve joined the club.
So, this brings us back to the classic word-of-mouth marketing argument: is it a success if people are talking about it? CP+B’s mission statement is to create advertising that people talk about but the don’t really specify whether or not they need to be positive about it. This is somewhat of a no brainer for an agency because controversy gives your work more exposure, as opposed to most immediately forgettable advertising. It helps agencies but does it actually help the brands?
I’m hoping, possibly in vain, that this represents a bit of a tipping point in the overly simplistic idea that marketing can be successful if you can only make people talk about it. 10-15 years ago I could see why this myth persisted. You had anecdotal evidence that people were talking and there was enough neutral editorial coverage to make a case that overall sentiment wasn’t trending negative.
We don’t live in those times anymore.
The reaction to the Groupon spots must have been foreseen since the Kenneth Cole flub was such a close parallel. The only difference is that the Kenneth Cole tweet may have been a momentary lack of judgment as opposed to being carefully orchestrated in a multi-million dollar campaign. The intelligent people at Groupon and their agency knew this would offend people and still made conscious decisions, like leaving out the charity angle in the television creative, to allow the reaction to play out. They didn’t care if people had a negative reaction, as long as they were talking about it.
Creatively, this is lazy and it undermines a brand that was on the verge of becoming a household name. Groupon is often dismissed as a coupon site but there is a much bigger story there. People are connecting with small businesses in new ways and are being incentivized to get out and explore local experiences of all kinds. I’ve heard dozens of these stories in the past year, from people trying out kayaking because of a deal or some amazing massage that helped someone discover a new independently owned spa in their neighborhood. There is so much that is central to the stakeholder value in Groupon that is undermined by a cheap publicity ploy. Sure, you’ll hear all about how many more people visited Groupon.com this week than previously but squandering an opportunity to create long term brand value on one of the highest impact marketing platforms of the year is really a shame at this stage of their development.
Chrysler’s ad had a similar fate to Groupon in the Ad Meter and Brand Bowl. In fact, it ranked just below Groupon in the Ad Meter and just above them in the Brand Bowl. They key difference was that Chrysler had a +14 sentiment score versus Groupon’s -4. While the sentiment scoring can be dismissed to a degree, the message you see being pulled through in social media can’t. Wieden + Kennedy’s spot challenged what people thought about Detroit and American car manufacturers and actually made a persuasive argument for buying a Chrysler. It was suddenly a brand that stood for something and the discussion you’ll find around it pertains to that very important aspect of the creative.
There are a lot of ways to measure success in advertising and changes in media habits and the amount of data we can collect may be resulting in a constantly changing set of standards for this success. Focus groups will always be compelling due to the level of feedback you can get but scaling them to represent a larger demographic and peer influence is always going to be an obstacle. Likewise, monitoring sentiment is social media will always be a challenge and may never be fully automated but it’s a step in the right direction. With measurement standards for creative completely absent, you’re left with instinct and your best judgment but also a giant media toolbox. The challenge isn’t in finding the shortcuts, it’s in having the foresight to avoid them.
I should probably preface this post by saying that I’ve never been impressed with Facebook as a technology or a form of media. When you deconstruct Facebook, there are very few Facebook features that aren’t executed better somewhere else. Photo sharing is weak at best compared to Flickr and others. The new location-based services are a shadow of Gowalla and Foursquare. Even status updates as a microblogging platform are lagging behind Friendfeed, a company Facebook acquired a long time ago. In fact, aside from the default news feed algorithm, I can’t point to one thing that Facebook does better than anyone else. Yet nothing really compares to it’s user base and a social network is ultimately defined by its user base so Facebook wins. For now.
Jeremiah 
Social media presents an interesting challenge to agency creative departments. In many ways, the rise of a largely consumer generated media entity is a potential pitfall for ad creatives. By definition, it seems that they are almost completely unnecessary in a consumer generated world yet they’re still part of it.![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_b.png?x-id=13c9e8ba-27fa-4acc-b18c-37fad4d47de8)
Brands have struggled with how to integrate with emerging media channels for a long long time now. Social networks are no longer “new” to most people under 40-years-old. Brands have been creating ![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_b.png?x-id=d9bd936d-0a26-49bd-85bd-2a896d512abb)
Jamie Monberg just finished a panel on taking interactive beyond the screen and into real interactions with stakeholders. One great example used was McDonalds, a company that spends millions in marketing but has done very little to improve the drive-in experience. Another example used on the power of interactive design was how Apple overcame Xerox through interface design and understanding that their success depended on personal interactions (although I’m ot sure if Apple is a great example due to their limited communications strategy).
Sarah Lacy stoked the fires of a debate that a lot people, including myself, aren’t looking forward to having in a recent ![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_b.png?x-id=0eab978e-feb1-4385-ac0d-06e6fd8d2dc9)


Apparently Twitter can cause quite a 