A Free Look at a Gartner's Sourcing Magic Quadrant
In the old days, kids and adults alike would pay pennies at the circus or traveling freak shows to see oddities that defy nature. Dogs with extra limbs. Siamese twins. Blind fire-eaters. The list goes on. Part of the appeal of these shows remained the thrill of spending a few cents to see something that was off-limits to everyone else. But in today's world, there's no need to pony up your hard-earned shillings to peer behind the closed door. That's right. You, too, can see Gartner's recent Sourcing MQ for free, by visiting and registering on Emptoris' website (click here to get to the actual report). Its contents will certainly amaze you -- and maybe even create shock.
In fact, I bet the reason that Emptoris decided to license it and make it available was not just to show off their ranking, but rather to build their lead list with curious onlookers who might not otherwise register to read a whitepaper or attend a webinar. In my view, this makes sense. Companies should decide for themselves whether the quadrant gets billing in one of the three circus-rings or should be relegated to freak-show status. At the time of my initial review of the quadrant, I wrote that "having looked at it, I must say that the relative rankings of smaller vendors -- in proportion to those in the Leader's Quadrant -- takes away from its overall credibility ... [For example] I simply can't understand why there are two vendors who I rarely see in deals significantly out rank the Leader's ability to execute (not just by a thread, but by a significant margin)." Now you can read it for yourself and form your own opinion, courtesy of Emptoris.
- Jason Busch
In fact, I bet the reason that Emptoris decided to license it and make it available was not just to show off their ranking, but rather to build their lead list with curious onlookers who might not otherwise register to read a whitepaper or attend a webinar. In my view, this makes sense. Companies should decide for themselves whether the quadrant gets billing in one of the three circus-rings or should be relegated to freak-show status. At the time of my initial review of the quadrant, I wrote that "having looked at it, I must say that the relative rankings of smaller vendors -- in proportion to those in the Leader's Quadrant -- takes away from its overall credibility ... [For example] I simply can't understand why there are two vendors who I rarely see in deals significantly out rank the Leader's ability to execute (not just by a thread, but by a significant margin)." Now you can read it for yourself and form your own opinion, courtesy of Emptoris.
- Jason Busch
One of the big mistakes is merely reading a Magic Quadrant research note as you suggest in this post. Because a published MQ research note is static and has so little context, only reading the research provides an incomplete understanding of what the IT manager needs to know to effectively use this commentary. If a tech buyer does not have client access to Gartner analysts and the ability to talk to the analysts, then our recommendation is not to use a Magic Quadrant at all in forming vendor short lists or other purchasing purposes.
For more tips on how research consumers should approach the MQ please see:
”IT managers, it’s never, ever only about the upper right dot when it comes to Forrester Waves or Gartner Magic Quadrants"
http://www.sagecircle.wordpress.com/2008/01/16/it-...
You seem to be stopping short, leaving your devotees like me champing at the bit to know what you think the motivation behind the errors and omissions is.
As to why it was flawed specifically, that's a topic that I could go into in more detail -- besides what I've said already -- but I'd rather leave it to those who read it to form an opinion for themselves (which thanks to Emptoris, they now can).
Personally I thought the rest of the report had value, i.e. the strength and weakness areas as well as general sourcing trends. In particular, I am glad to see that Gartner recognizes the bifurcation into strategic and tactical component of sourcing solutions.
A download - and read ! - is definitely recommended. Kudos to Emptoris - and to Jason for pointing out this freebie.
re: "So why is Gartner's research flawed, Jason?"
There are a variety of reasons why Gartner's - and other firms' - research has shortcomings. Here are a few:
1. Analysts are not given the time or resources to do deep research. See
http://sagecircle.wordpress.com/2008/07/09/why-ana...
2. Analysts rely too much on the vendors for information -- Many vendors do not do a good job educating the analysts on a continuous basis. This isn't a case of "blaming the victim," but pointing out that good AR is a part of good marketing and selling. Vendors that don't invest in top notch AR are often disappointed in placement in analyst reports, which translates into lost sales.
3. Vendors do not use best practices for dealing with the Magic Quadrant. A subset of #2. See the series on AR MQ best practices:
http://sagecircle.wordpress.com/category/magic-qua...
4. Gartner analysts rely too much on informal conversations with their clients -- While end-user client inquiry can provide valuable insights, that data set is deeply flawed and skewed. For example: only Gartner clients are included, it is self selecting as a client has to initiate the call, the actual number of inquiries on any particular vendor is quite small. Etc.
5. Lack of transparency on the exact methodology, criteria, source of data and weights
I could go on. Some of these issues are something that Gartner could fix. Others are items that vendors need to address. Bottom line: everybody could be doing a better job.
Regarding Point #4 above, Debbie Wilson also spoke with customer references provided by the vendors. They were not screened for whether or not they were Gartner clients. Also, she reached out to initiate the interviews, and therefore they were not self-selecting customers.
I will go back to a fundamental point regarding analyst research. Should a company make a technology decision based solely on one analyst report? No. However, do analyst vendor rankings provide useful information for this company? Of course. A company considering a major technology investment should use reports like this as part of a toolkit in their selection process. Points to consider include the following: what has the trend been among vendors over time within the Magic Quadrant? Is this the first time a vendor has appeared, or has the vendor shown consistent presence and improvement over a period of years? What criteria are considered that make a vendor a leader - is it based on low price delivery models, superior service execution, unique commodity expertise, or innovative and heavy weight functionality? It is important to understand that there can be many reasons for garnishing the leader status, but rarely do two vendors become a leader for the same reasons. Finally, how has this vendor performed in other analysts evaluations (e.g., The Forrester Wave, AMR surveys). If multiple analysts reach similar conclusions, then companies can feel more secure in the conclusions drawn.
Kevin Potts
Vice President of Marketing
www.emptoris.com
Good to see the points from Carter & Kevin.
On balance I enjoyed reading the report. My main issue is its name: By calling itself "Magic Quadrant" it implies that there is one magic quadrant and that there is something, well, magical about being in that quadrant. But as Kevin points out you'd hope that readers of the report would be wise enough to gather multiple data points to make their decisions.