by Freddy Tran Nager, Founder of Atomic Tango + Guy Who Questions The Numbers; photo via Wikimedia Commons…
Imagine that you own a basketball team, and you send your talent scout out to find a new player. He comes back and presents two options:
- Player A: averages 9 points a game, including 1 three-point basket per game
- Player B: averages 24 points a game, including an average of 2 three-point baskets per game
Both play the same number of minutes per game, and both want the same amount of money. To your surprise, your scout recommends hiring Player A. “Why?!” you ask incredulously. “Because,” the scout replies, “Player A gets 33% of his points from three-point shots, and Player B gets only 25% of his points from three-point shots. That means Player A gets 32% more points from three-pointers than Player B.”
As you stare in stunned disbelief, the scout takes it as a signal to continue…
“Now you know that three-point shots are worth 50% more than two-point shots, so if we multiply the 50% by 32%, that makes Player A 1600% more valuable than Player B.”
Of course, that makes your choice easy: you fire the scout.
Sounds silly, doesn’t it? But that’s how some people are measuring social media “effectiveness.”
I came across this recently in an article that claimed, “Top B2B Firms Gaining 230% More Leads via Social Media Than Peers.”
“Best-in-Class B2B companies generate on average 17% of their leads from social media channels, roughly 230% more marketing-generated leads than other companies (5%).”
That’s some mad mathematical mayhem there – what financial-analyst Ron Shevlin would call “quantipulation.” Where did that figure of 230% come from? I tried to do the math: 17 is 240% more than 5, so somewhere along the way, we lost 10% and…
Hold on there: this whole case smells like something beached itself and died. I find myself suddenly remembering Darrell Huff‘s classic book, How to Lie with Statistics, which demonstrated how you can say anything you want with numbers — particularly if you use those ever deceptive percentages.
So I contact Shevlin, who’s a professional marketing analyst in the financial services industry. He cuts to the chase:
“NEVER EVER calculate the percentage difference between two percentages.”
Thank you, Sensei, for dispensing with that nonsense.
But I wasn’t done with this article yet.
The “So What” Factor
As I tell my students, the most important question to ask when analyzing social media in business is, “So what?” It’s the perfect response to any of these statements:
- “I’ve got one quadrillion Twitter followers!”
- “My video went viral!”
- “Our free burrito offer got a bunch of likes!”
- “According to Klout, I’m the world’s most influential person in three categories!”
Ask “So what?” — and the answer better have a $ sign attached.
So let’s look at our 230% overachievers, shall we? And let’s be good sports and grant them that number. So here it goes: so what if you got 230% more leads?
- What are the absolute numbers? 17% sounds like more than 5%, but you don’t need to be a financial analyst to understand that 17% of 100 is a lot less impressive than 5% of 1,000,000.
- What was the ultimate value of those leads? Did they all convert into sales, or were they junk? And if they did convert, at what dollar amount?
- What was each of those firms selling? Some products and services are a lot easier to sell through social media than others. (Say, an MP3 download vs. nuclear accident clean-ups.)
- Are the top firms getting more leads because they’re good at social media, or because they’re top firms?
- If a higher percentage of a company’s leads comes from social media, couldn’t that mean that they suck at traditional media? Or, conversely, perhaps the 5-percenters are so good at traditional media, that they don’t need to bother with social. (There’s a company like that you may have heard of: Apple.)
- How much did the 17-percenters spend in time, money, and opportunity costs to land those leads? What did the 5-percenters spend? Of course, EVERYONE knows by now that social media isn’t free and can be even more expensive than traditional media, right?
- If the firms had invested all that time and money into traditional media – from which 83-95% all leads in this case come from – what would the results have been?
- And, wait, if non-social media is generating 83-95% of all leads for these firms, why is this even a story? I know where I’d invest my time and money, don’t you?
Now don’t get me wrong: social media has actually generated leads for my own B2B agency, almost entirely through LinkedIn. So yes, it can be productive and lucrative. For some… at some times… under some situations… for some prospective clients.
The key is discerning what that “some” is, and we can’t discern squat from distorted stats. They’re no way to pick a basketball player, and certainly no way to run a business.
I’m 230% sure about that.
Spot on. If more SM experts asked “so what”….well, there would be fewer SM experts.
But there are two more reasons why the statement in question — “Best-in-Class B2B companies generate on average 17% of their leads from social media channels, roughly 230% more marketing-generated leads than other companies (5%)” — is so stupid.
1. OK, so best-in-class company A gets 17% of its leads from SM, while loser company B gets just 5%. Where do the REST of the leads come from?
It’s entirely feasible that company A gets 59% of its leads from email, while company B gets just 33%. Wouldn’t that be an indication that email is the superior lead-generating channel?
2. (Dr. Tango: This is the point I really hope you hammer home w/ your students.) You can’t judge marketing effectiveness by simply looking at the channel. You have to look at other factors, an important one being the message, or the creative component. Explanation: It’s also entirely possible that the reason best-in-class companies generate a higher percentage of leads from SM than other companies is that the CONTENT of their SM campaigns is superior. One message might resonate with consumers better than another. In that case, it should be more effective in WHATEVER channel it’s disseminated in. The only way to understand the impact of the channel is to TEST. Unfortunately, this seems to be a foreign concept to many social media experts.
Great points. I should have also noted that the offending article is essentially a tautology: “Companies that do more social media marketing get more of their leads from social media, while companies that don’t do as much social media marketing get fewer leads from social media.” And for just $399, they’ll sell you all the insights from their study.
Great post. There is nothing wrong with using percentages as benchmarks as long as you put them in context with the real world, but most of the time, be it in the news or politics or marketing pitches, percentages are used to hide the real quality of the big picture.
The extremely inflated numbers that various companies are boasting about now tend to actually have a negative affect rather than a positive one. Although companies that often post these high percentages often think it brings them credibility, in all honesty, it does nothing more then damage their “brand” among professionals for posting dishonest numbers. It’s mind-numbing how they can believe that most people do not see right through them! After all, if you cannot trust what a company says about themselves, can you honestly feel secure in conducting business with them?
[…] merito alla follia dei numeri, trovo geniale l’analisi su social media e le statistiche di Freddy Nager, un mio ex professore, che si arrovella nel cercare di capire che utilità può […]