The WSJ Shows No Love for Aberdeen
What followed was essentially a critical examination of the Aberdeen business model. To summarize, vendors pay what the story suggests is around $30,000 on average, to be featured as a sponsor of a single survey (there were 212 reports published last year each "typically with four or five sponsors"). According to the columnist, Lee Gomes, "Most of the half-dozen Aberdeen sponsors I talked with described the attraction of sponsoring a report as a chance to rise above the noise of the marketplace by being associated with something customers consider 'research'."
But how unbiased this research is, the article suggests, is open to interpretation. "The reports seem to invariably discover that 'best in class' companies use, or are thinking of using, or somehow embody, whatever technology the report happens to be discussing," the column notes. Now to be fair to Aberdeen, I do believe the firm has published at least a few critical reports in the past year in the Spend Management space (Vance Checkett's expose on supplier enablement -- and how most companies come up short -- is a perfect example).
But in general, there certainly does seem to be a formula to the Aberdeen model -- and one where the reports could almost write themselves without any critical thinking or subject matter expertise wrapped around them. Perhaps that's why I've heard that Aberdeen is talking to numerous people I've spoken with about "outsourcing" the research and analysis for report writing to them. Then, the firm will essentially slap its logo onto a report and sell sponsorships. So much for quality control -- if it ever existed -- I suppose ...
But my real issue with Aberdeen has nothing to do with the formulaic research findings or the lack of expert analysis in what they present. Rather, my beef comes down to two main issues. First, I don't believe the firm should pretend to be an analyst firm when it is in fact something entirely different -- a lead generation source for vendors (and sometimes, a good one, from what I hear). Second, I seriously question the quality of the data that goes into the benchmarking research. Unlike firms like the Hackett Group which specifically pre-qualify all potential responders to survey research, Aberdeen lacks a data quality control process that would stand up to outside scrutiny (e.g., verifying the employment, companies, titles and responsibilities of those responding to the web-based requests).
At the end of the article, Gomes asks Aberdeen's President Steven Gold why the old Aberdeen business model came up short. His response is that Aberdeen "had become complacent about actual research. We were following a non-scalable and non-value-added model". Hmmm ... I would agree that such research minds as Tim Minahan, who single handily built and preserved Aberdeen's reputation in the dark years following the B2B bust, are not scalable.
After all, talent is hard to find. It's much easier -- as Aberdeen is in the process of doing -- to simply outsource the analysis of questionable survey data to some unnamed third party in its entirety. But to argue that the old Aberdeen model was non-value added is a joke. For years when I was at FreeMarkets, I hired Aberdeen analysts to speak at events, conduct expert research and give us unbiased, hard-hitting advice. These types of activities, ironically, are precisely what the new Aberdeen is not focusing on, despite having a number of smart, capable minds in our sector that could provide such advice and counsel to their vendor clients if they made it a focus. I suppose "non-value added" is in the eyes of the beholder.
In any event, I'd be curious to get your thoughts. Was the Journal article too harsh?
- Jason Busch
Look, I completely understand why vendors should sponsor research. In fact, I think its acceptable as long as the vendors don't affect the impartiality of the results. However, when vendors 1) pay for the research, 2) structure the research work in a manner than benefits their products, and 3) openly promote their solutions in the research it's just too much. The fact that a WSJ article exposes this in the press really shouldn't be a surprise to anyone who has read Aberdeen's reports.
It's a darn shame that more of spend management leaders don't jointly fund research. For example, I'd sure like to see a longitudinal study of spend management analysis tools or "procurement transformations". However, this gets to the core of the problem. Spend management teams simply are not buying research. Particularly for indirect spend. At $1000 to $3000 a pop, who can afford to purchase reports that lack expert insight, as you allude to?
Jason, you published a blog entry recently where you touched on Ariba and Denali's market research models. I read Ariba's research and it's downright fascinating, even if it is not actionable enough to run a sourcing project from. Ariba knows it can't charge for this stuff, but a free publication demonstrating industry knowledge really builds their brand. Denali would love to make money off its deep expertise and we'll see if they do, though I doubt many firms will buy some of the narrowly focused articles (or at least in sufficient quantities to make them profitable products).
I sure wouldn't pay $$$ for Aberdeen reports, though the old ones were okay.
Maybe you can comment on how you maintain impartiality in producing Spend Matters while taking vendor sponsorships?
How the heck did you gete to be an supposed executive if you write like an imbecile? Not very "proffesional".
First, “supply chain executive”, my guess is that you’re in the minority regarding your opinion of this blog among practitioners who do read it (which certainly is only a small subset of the potential market at this point, though it’s growing every month). I spend less than a third of my posts discussing technology – the majority of my commentary is about economic, trade and political topics that impact procurement and supply chain areas, which you’d realize if you read this blog with any regularity. And the feedback I’ve gotten from practitioners, for the most part, is that Spend Matters is a breath of pragmatic and useful fresh air. But obviously you’re entitled to your opinion (even though you sound like to me someone from Aberdeen, to be totally candid, considering you’re not disclosing who you are). Why don’t you use your real name and email address? If you are truly an executive in the sector I’d welcome your candid feedback on a call or in person. Please get in touch.
Second, regarding the question regarding balancing sponsorship with impartial coverage, the answer is that opinion is always inherently subjective, yet I strive to not let sponsorship or consulting dollars influence my views (long-time Spend Matters readers might want to chime in here as well as they will attest that I am equally objective regarding coverage of sponsors and non sponsors alike). I am happy -- and usually a few times a week -- take calls and have meetings / briefings with vendors, consultants, and outsourcing providers whom I have no -- and have no intention of having -- commercial relationships with. The point of these discussions is information exchange. That’s how I stay current and get outside opinions. But perhaps the most important tenant I have is full disclosure of all relationships. That you can find here: http://www.spendmatters.com/Disclosures.cfm I have long encouraged analysts, trade publications and other bloggers to make similar disclosures as well.
One last point ... I forgot to answer one point from “Supply Chain Executive” … and that’s: why don’t I attempt to monetize Spend Matters by having practitioner sponsor or pay for research. The answer is simple: I believe the world is headed towards a free content model where you can only make money based on advertising (unless of course, it is highly, highly strategic content that you can charge a fortune for). This was a business decision that I made when I started the blog and continue to evaluate on a frequent basis.
I hope that this answers the questions.
What's funny is that nobody from Aberdeen ever chimes in on this blog defending themselves and their "honor" (ok , stop laughing). Why? Because they know it to be true and the analysts are using Aberdeen (to build their personal brands) as much as Aberdeen uses them. To quote Bob Seeger, they were "getting their share".
But, there shouldn't be any surprise to all this. As Groucho Marx quipped, "I wouldn't want to belong to a club that had me as a member". I work with CPOs quite extensively (as Dr. Jekyl vs. my foolish Mr. Hyde or rather 'Hide' persona), and I can tell you that they are not reading blogs and certainly do not base their decisions on Aberdeen reports.
However, the reports ARE useful for education and for including 3rd party data (for business case support) on the cheap when nothing else is available. Reminds me of the Lynard Skynard song "Don't ask me no questions, and I won't tell you no lies".
So, to Jason's point, let's not "see the corruption inherent in the system" (quiz: name the movie) onto itself, but just make sure that everyone realizes what it really is - and that was in fact the WSJ point as well.
I do almost feel a little bad for Harte Hanks, but, hey, they've only themselves to blame.
So CPOs are not reading blogs? I know a number who do, but as a general rule, you're probably right. Still, I speak to practitioners everyday who read this and other blogs (so things are changing, that's for sure). Though I reckon the CPO question will change dramatically in the coming years as younger staff are promoted through the ranks to the top.
This subject is a bit too close to home for anyone to speak to with immutable disinterest...somewhere in the Churchill continuum between "Any man under 30 who isn't a liberal has no heart, any man over 30 who isn't a conservative has no brains" lies the answer to our personal stances on the blog/analyst question.
Yet, we must pick our side and fight for it.
In their defense…
• Aberdeen is the only analyst that cranks out hard data. Gartner, Forrester, and bloggers crank out opinions. Both solution buyers and vendors can leverage this data to create ROI assumptions, support their business cases, and offer up third-party backing to their numbers.
• Aberdeen has never asked me to direct the topic of their next report.
• Aberdeen is a fantastic lead gen source, and my customers – supply management practitioners – place a lot of stock in what Aberdeen puts out there.
Now, to pile on…
• Aberdeen’s numbers are suspect at best (but they are the only thing out there). If you’ve ever really looked into the “research methodology” at the end of their reports, you’ll see that only a small percentage of the respondents are decision makers, most are from outside of the US, and well over half are from smaller (<$1B) companies.
• Aberdeen routinely asks vendors to offer their opinion on and add to their questionnaires before they go out. A crafty vendor can carefully word a question to help ensure a favorable result. No, Aberdeen doesn’t have to take the vendor’s input, but the interaction itself brings up questions about impartiality.
• Aberdeen routinely asks vendors to offer their opinion on and add to their draft reports before they are published. Again, vendors have the ability to smooth talk their way out of a negative assessment of market direction, or reword a “finding” that could potentially help them and/or hurt a competitor.
Bottom line – as a vendor (which is why I didn't give my name), Aberdeen presents an opportunity to drive leads, get my company some ink, and provide me with ammo to win more deals. Regardless of their shortcomings, as long as people believe them, I’d be remiss not to capitalize on the opportunities that they offer. I'm just playing within the system. If buyers ever realize that analysts are a scam, then I'll have to "advertise" elsewhere.
Where Pinky and the Brain (Re)Define Best In Class
http://blog.sourcinginnovation.com/2008/01/19/wher...
Where the Brain gives Pinky a Lesson in Statistics
http://blog.sourcinginnovation.com/2008/01/26/wher...
As to "Because it's there"''s points in Aberdeen's defense.
* What about Hackett Group? I seem to recall they publish a Book of Numbers every year of a quality that far surpasses anything Aberdeen cranks out, or has cranked out, in quite a while. What do you think all-knowing SpendFool. Is my memory working?
* But I bet they didn't stop you from suggesting a report they should do. (You haven't tried you say? Try it, try it, and they may I say.)
* I have to ask just how you qualify a lead. Sure you get 600 to 1000 names with contact information from an average report, but how many of these are leads? You'll have a lot of academics looking for data to crunch, a lot of media people looking for the next big thing, a lot of current customers trying to get something for their 15K to 30K+ subscription, and a lot of random individuals who just stumble upon the report and think it might be what they're looking for.
But when you ask - how many of these "leads" are buyers in sourcing / procurement organizations? How many of these "leads" are in organizations planning to buy a solution in the next year? .. and how many of these "leads" are actually looking for a solution of the type I'm offering? How many of these "leads" actually have any impact on the decision whatsoever? How many of these "leads" will be willing to honestly consider and evaluate my solution? And, most importantly, How many of these "leads" do I at least have equal odds of converting into a "customer" when the vendor selection process begins, then, IF even 1% of the "leads" are left after that last question, you're doing pheneomal. Most of the organization's I've worked with are lucky when they get "1" sale as a result of an Aberdeen report sponsorship and that sale covers the cost of the report sponsorship in the first place. And what do a few measely impression do for brand building? Not much!
As to "Because it's there"''s pile-on points:
* Very suspect. Read the posts.
* I'm betting you're the crafty type of vendor, aren't you? Otherwise, I would bet that you wouldn't be too happy with the lead generation you got when you just gave them money and hoped for the best, as too many companies have found out lately.
* I have to thank you for making this last point and repeat it. "Aberdeen routinely asks vendors to offer their opinion on and add to their draft reports before they are published. Again, vendors have the ability to smooth talk their way out of a negative assessment of market direction, or reword a “finding” that could potentially help them and/or hurt a competitor." Statistics are supposed to be unbiased. That's about as biased as you get!
As for your final point, I think you'd be surprised at how many buyers who now realize that most analysts are just, to use your words, a scam. I think you meant to say "as long as the cheque cutting executives believe in them", because I know many an intelligent buyer, and even manager, like the kind who read this blog and Sourcing Innovation, who believe that you can't count on getting your money's worth with Aberdeen anymore. These buyers and managers just haven't worked their way up the ranks yet in the bigger companies. However, now that almost every spark of talent has fled Aberdeen, I don't think it will be much longer before the masses start to realize the truth.
Now, I'll have to admit this is a bit selfish, but I would suggest you start looking at other advertising options now so that you're ready when the time comes where you cease getting a positive ROI on your Aberdeen sponsorships (which could be sooner than you think), and in new media (blog) sponsorships in particular, such as those available here on Spend Matters and over on Sourcing Innovation.
Information on Spend Matter Sponsorship slots, when they are available, can be found in this post:
http://www.spendmatters.com/index.cfm/2007/10/22/M...
And information on Sourcing Innovation Sponsorship slots, which are available now, can be found in these posts:
http://blog.sourcinginnovation.com/2007/11/11/the-...
http://blog.sourcinginnovation.com/2007/11/11/wher...
http://blog.sourcinginnovation.com/2007/11/11/why-...
Haven't had a chance to read the Aberdeen PO report yet... but will definitely comment on it when i do.
I recall the dark days Aberdeen just did a load of sponsored white papers for individual vendors ("Pay for Play"), so this move towards "Pay for Praise" is a different proposition (and I have seen other research firms do these as "multiclient studies" on many occasions). However, I would add that hyping up an industry to help a group of sponsors sell product/service is much more dangerous than simply making a single supplier look good. Everyone knows a white paper sponsored by a single vendor is blatant marketing and gives it little mindshare, but when the research is disguised as an "industry report" and only focuses on hyping up a space, then the research firm is making some potentially dangerous statements. They have to be balanced and focus on the challenges and issues of an industry / technology, and not just the upside. For example, we all know procurement outsourcing is an immensely challenging, complex - and not fully developed - business proposition, where firms need to conduct a great deal of self-analysis, so I sincerely hope Aberdeen makes this point when I review their report.
PF
My time at Aberdeen was before the Harte Hanks acquisition, so things may have changed with the influx of HH's money and lead lists. But while I was there, one of the many outright deceitful acts I saw at Aberdeen was the manner in which they generated leads for each report.
Frequently, when downloads were not up to snuff on a report, names were pilfered from downloads of other reports and added to the lists as if these individuals had downloaded the content of their own volition, effectively providing hundreds of irrelevant or uninterested names to companies that had paid for qualified leads.
This is unfair to both the companies who are footing the bill and those individuals who downloaded reports who were contacted by sales reps trying to sell them technologies in which they had no interest.
You have to also question the quality of Aberdeen's research itself. While I was there, we received maybe 70-100 responses to surveys that were sent out via e-mail to anybody who had shown interest in the topic, report: downloaders, low-level titles, etc. Heck, I still receive Aberdeen surveys to a test account I used while there...these surveys are for all sorts of technologies for which I have no responsibility. What's to stop me from filling out these surveys and skewing Aberdeen's research?