Debbie Wilson: Baking up a New Magic Quadrant … Too Little Time in the Oven?

Over on Gartner Blogs, Debbie Wilson shared with her readers after Thanksgiving that "In between baking pies, eating turkey, watching hockey (go Monarchs!), playing tennis, and spending time with my family [that] first pass ratings are underway for the updated Magic Quadrant for Strategic Sourcing suites." While I'm a huge fan of multitasking in the kitchen and in life, I've discovered that one of the major challenges of doing too many things at once is simply running out of space in the oven, not to mention hours in the day. But when it comes to how analyst firms often involve vendors -- not to mention their references -- in competitive comparisons like a Quadrant, there's another angle to this as well. To wit: It's not just your time as the reviewer that's involved; it’s others' as well. Which is why this particular quadrant analysis, whatever the results, will go down in my history as the most tainted and flawed piece of comparative research ever done in the space.

Now, perhaps when Debbie is forced to defend it to angry vendors, she won’t go to the extremes one former analyst I know did when he decided to wear army fatigues to a meeting with the CEO of a vendor, post-ranking, to defend his work. But the process and timeline itself in this case should be enough to give us all pause. In a post from October, just after Debbie sent out her information request to vendors, I wrote that Gartner was requiring participants to “fill out a complete survey response in approximately a 10-day period” when in fact “these surveys can take 50+ hours to sufficiently complete depending on the level of detail they ask for.” At the time, I opined that “because of the short time frame in this case, there's no question vendors will be stretched thin filling out this information (not to mention the fact that they're also currently filling out similar requests from competing firms who are also updating their research in this and related areas, but allowing more time).”

I also suggested that, “Given this, I suspect the results will end up flawed, representing a fire drill based around who had the most time to fill in a response and who is lucky enough to have enough customers not currently going through a renewal process who will serve as rapid- fire references.” In this particular case, I now know Gartner deviated from past practices by driving vendor references to an online web- based survey environment to fill out information versus taking time on the phone. Is this an improvement on the past given the short timeframe? Perhaps. But I also suspect such a digital response will fail to yield the color and anecdotes that are so critical to such analyses. Maybe at the end of the day everyone would have been better off delaying the analysis instead of doing things based around Gartner’s perceived need to crank out this piece of research as quickly as possible.

After all, unlike a pie, you can’t simply put the thing back in the oven if it’s not done. Or run out to the bakery and buy a new one if your home-cooked dessert ends up inedible.

- Jason Busch