How Much Can Demonstrations Teach You About a Solution? (Part 1)
Over on Debbie Wilson's blog, there's a good entry that raises the question of the role of product demonstrations in evaluating technology solutions. Even though Debbie tackles it from the role of analyst, I think there are some lessons and broader discussion points to come out of it. For one, consider the role of references in decisions as much as a demonstration and RFP response. However, the best references are not necessarily the ones provided by a vendor -- they often come through your own network. If a vendor provides them, make sure you ask as many open-ended questions as possible to gauge the level of enthusiasm in the response. "Damning with faint praise" should damn a vendor off your shortlist. I often request references from vendors but rely on industry (or channel) connections when possible to do my real homework.
But back to the topic at hand -- the value of demonstrations. All too often, unless the analyst or prospect orders up the demonstration himself or herself, requesting their own demo scripts and scenarios, vendors will attempt to pull the shades over the shortcomings in their solutions (or at least provide basic whitewash to areas where functionally, they can't quite compare to others). In other words, be prescriptive about what you want to see. In a follow-up to this post, I'll provide some specific ideas about how you can drive vendors to provide demonstrations that will most closely mirror your own environment versus what they want to show.
Overall, how should demonstrations fit into your evaluation of a product? In short, as Debbie notes, "If you aren't really sure what a solution does, a demo is a great way to get, at a glance, the highlights of functionality. Often, a picture is worth a thousand words." Moreover, "A demo can also give you a rough idea of how aesthetically pleasing a solution is to use." This last point is essential. Business users must prioritize usability alongside absolute capability in the case of solutions that will touch more than a handful of users. For example, when it comes to spend analysis (not broader spend visibility), I'd personally want the most powerful solution out there, even if it looked like a relic of the client/server era and required an IQ of 120+ to work with. But for eProcurement, sourcing, contract management and other areas -- not to mention broader spend visibility and related reporting -- ease of use matters, potentially uber alles. And if you don't like what you see during a demonstration, I'd scratch the potential vendor off your shortlist as quickly as possible.
Tweet
TweetBacks
Analysts don't really know what to ask. They are generally not experts at anything, certainly not software or technology (unless they make an effort to involve informed third parties, and I've yet to see that happen). Their stock in trade is "inside information" consisting of the whispers of Joe Marketing in ABC Corporation multiplied by {however many large vendors can pony up their fees}. This is why hiring analysts is generally a bad idea. Once their whisper network is severed, their value is nil.
The whole thing reminds me of the first volume of Asimov's Foundation Trilogy, which describes a degenerate future society in which the idea of "analysis" has devolved into weighing the opinions of others rather than doing any original thinking.
With regard to customer references, it's well understood that you can't ask customers what they really want: they don't know, and they can't tell you. They'll know it when they see it (like the iPad or the iPhone), but if you believe what they tell you a priori, you'd best have a bankruptcy attorney standing by. You can't believe reviews from customers who own a solution, either. Half of them have no clue how it works or what it can do, slept through training, and can't be believed one way or the other. That's as true of Oracle ERP as it is of some point solution.
I like this perspective better: http://blog.sourcinginnovation.com/2010/07/08/anal...
Michael (who you link to) impresses me very much with his technical skills, his reputations for managing demoes, etc. But the approach, in isolation, that he recommends is what led far too many people down the wrong ERP, eProcurement, etc. path originally. Technologists should be key drivers of evaluations, but there are many subtleties that transcend binary responses or the ability to script a good demo. Miss those and it could cost you millions. Tradex, i2 and many others used to put on good demoes or POCs (and could ace the demo), but when it came to implemented solutions, they pulled a fast one over far too many companies. Unless you are to be the reference and are willing to embrace that risk/reward, you better darn well learn about the experience of others from the horses mouth.
Wait and see about the business scenario approach I'll posit in Part II.
The core problem is that they are feature/function oriented - "Here's a list of f/f that we'd like - please give us a Y/N + comment response as to your product's capabilities." This sort of buying may work very well when you're purchasing, say, a bridge. Your engineers received the same training as those at the construction company(s) from whom you're purchasing the solution, and what you're buying is the construction company's ability to most efficiently put the engineered solution into place.
Jason, were you buying a procurement solution, I have no doubts that you know what you're looking for, and that you understand the space (problems, solutions, efficiencies, etc.) as well as anyone at the solution provider. What you'd be looking for is my ability to deliver an engineered solution. THAT IS NOT NORMALLY THE CASE!
Far better is a broad-scoped RFP saying something along the lines of "We'd like to create substantial efficiencies in our procurement process. Here's our current situation: (Situation described). How would you propose to change our business process, and what value would we derive from doing so? Additionally, we'd like to talk to customers who have been in the same situation and have successfully made the transition."
You do need to look for your own references, but as much as anything, you're looking for lessons learned - things to avoid.
Nice article, BTW! Always enjoy your stuff.
In terms of demos, stock demos obviously only go so far, and smarter firms will use some type of conference room pilot methodology to take actual process flows and build it into the vanilla software and make sure it does what it's supposed to. It won't fix all the problems, but sure does highlight a lot of stuff that demo's surely don't.