Most selection procedures require the bidders to identify reference accounts. The wording in many RFPs is similar:
"Proposals must include a list of references including names and telephone numbers for whom the bidder has performed similar work. These references might be contacted during the proposal evaluation phase to determine their satisfaction with the work carried out.
"Provide 3 references, for work performed within the last two years, which you judge to be of similar scope and complexity."
Invariably, someone from the selection team makes the calls.
If this is your job, what do you say? What questions do you ask? What is the purpose of the exercise? Clearly, you want to verify that the supplier did in fact do a very good job at that company or agency.
But of course we all know that suppliers will never submit the name of a bad reference. The existence of a few solid references only demonstrates that the bidder has the potential for excellent work. Obviously, a bidder will only submit the names of companies that they know will provide them with wonderful references.
There is another, more aggressive tactic. Phone the reference accounts, not only to learn about the bidder, but also to learn the names of other purchasers. Once this has been accomplished, call all of the purchasers who weren’t listed as references.
The purpose of these calls to "non reference" accounts is to learn about the supplier from a broad range of customers. These customers will relate both good and bad stories about the bidder and its products or services.
The purpose in contacting "non reference" accounts is not simply to identify problem situations (which may have been caused by the supplier or its products, the purchaser, third parties, etc.). The purpose is to learn what the supplier did when difficulties were encountered. What did the supplier do when its project leader left? What happened when the key user became ill? What did the supplier do when you required more product on short notice? Did the supplier act in the customer’s best interest? Did the supplier simply disavow responsibility? Did the supplier and purchaser solve the problem by working as partners?
If these conversations cause you to be concerned, talk to the supplier. Listen to the supplier’s side before accepting the information.
Listed below is part of one organization’s reference questionnaire used in a telephone survey of current users (of a software supplier’s financial modeling system):
- Was an evaluation of various vendors carried out?
- If yes, who was considered and why was this vendor/product chosen?
- How long have you had this program?
- What training was provided? Rate it 1 to 10.
- Did you find many bugs or errors in the system?
- How responsive was the vendor to your queries and problems? Did the vendor in fact take an active role in solving any problems?
- How would you rate the vendor’s support on a scale of 1 to 10?
- How have you used support?
- What do you consider to be the major limitations of the system?
- Are you considering an alternative at this time?
If you decide to incorporate this approach to reference checking into your evaluation procedure, we suggest that you modify your RFP so that the bidders are aware of this new process. Here’s how one RFP informed the bidders that the reference checking would include a broad range of customers, not just the names they provided:
"Our evaluation will be made primarily via checks with bidder-provided references and other industry sources and users known to the evaluation team."
