eBook | Bidding Opportunities | Industry Specific | Market Intelligence | Buyer Contacts Proposal Writing | Buyer Products | Vendor Database | Professional Services Referrals 1. Establish Compliance | 2. Score the Proposals | 3. Impose Upset Levels | 4. Evaluate the Cost 5. Develop a Short-list | 6. Interview Suppliers | 7. Best and Final Offers | 8. Reference Checking The Eight Building Blocks of the Evaluation Process 2. Score the Proposals In most evaluation processes, scoring is performed as the second task, immediately following the determination of compliance. Most proposals require that the financial information be provided separately. This is to ensure that the technical evaluators have no knowledge of the pricing proposal. This avoids the debate over whether knowledge of the pricing proposal influenced an evaluator's assessment of technical factors. Upon receipt of the proposals, the financial section is removed and given to the analyst for evaluation. This person receives the information, establishes the costs to be used in the evaluation, and determines the score (if required). Costing is discussed in Section 7.5. Copies of the technical/management proposal are distributed to the Evaluation Team as required. For small proposals, each member of the Evaluation Team may read the entire proposal (except for cost) and perform the evaluation. On larger or more complex proposals, specific sections are usually assigned to individuals. For example, the communications expert on the Evaluation Team and the Project Manager might be the only people who evaluated the proposed network. The Evaluators then meet to review each other's evaluations, to resolve differences, and to ensure that they share the same understanding of each proposal. This process results in scores for each evaluation criterion. The purpose of the meeting, and the team effort for that matter, is to discuss, to understand and to resolve differences - not simply to average the scores. If two evaluators, both experts in the same area, score the same proposal as a '2' and a '6' respectively, there is some fundamental difference in each evaluator's interpretation of the proposal. It is not good enough to give the proposal a '4', the average of the two scores. Fairness dictates that the two evaluators discuss the issue, identify the differences in interpretation, and agree upon a score. (However, in some jurisdictions, evaluators do their scoring independent of one another and scores are then averaged.) Three components are required to score a proposal: a detailed set of evaluation criteria; an assignment of weights to reflect the importance of each factor; and a method for establishing a score. Each of these components is discussed in the remainder of this section. Evaluation CriteriaEvaluation Criteria Evaluation criteria are as different as people. Some are very specific and easy to assess. Others are vague and highly subjective; and, some would argue, arbitrary. There are three major families of criteria: Technical, Management and Cost. Technical criteria usually include the following: understanding of the problem, soundness of the approach and solution, ability to satisfy the stated requirements, service and support capabilities, analysis of risks, and testing methodology. Management criteria usually include the following: project plans, management approach, qualifications of key people, project timetable, and corporate experience. Cost is often evaluated in terms of the following criteria: total life cycle costs, cost controls, and consistency with technical and management plans. The quote which follows is one of the best which I have found that deals with the effectiveness of evaluation criteria: For evaluation criteria to be effective, they should ideally have the following characteristics:
Relate to the requirements definition: address all key elements Discriminating: separate best, average and weaker proposals Non-discriminatory: are fair and reasonable Realistic: given the contract nature and/or value Measurable: use measurable standards Economical to use: do not consume an unreasonable amount of time or resources An Example of Detailed Selection Criteria Here is how an RFP might set out the evaluation criteria. This particular example is from an RFP for an integrated law office system. Vendor proposals will be evaluated based upon the following criteria. Criteria are listed in order of importance: (a) the ability of the vendor to propose a complete solution to the documented needs; (b) the degree to which the proposed solution fully integrates the data processing, office automation, and word processing capabilities of the system, providing transparent access to all system functions via a single user terminal or work station; (c) the 4-year present value of the total cost of the proposed solution, including initial front-end costs, recurring annual costs and incremental costs for upgrades and expansion; (d) the financial stability and degree of established business presence of the vendor in the New York legal community; (e) the success of the vendor in installing similar systems of comparable size and complexity; (f) the degree to which the proposed software, after modifications which the vendor is willing to perform, will meet the documented application needs; (g) the amount of modification required to be performed on the application software; (h) the hardware reliability, and the quality and availability of maintenance and support; and (i) the quality of user training, implementation assistance, documentation and the application software support. Proposals are evaluated by scoring them in each of the selection categories. The raw scores, usually out of '10', are multiplied by the weight and totaled. The supplier with the highest score becomes the first choice. For example, the first selection criterion deals with providing a complete solution to the customer's requirements. This selection criteria had a weight of 20%. The Buyer had determined before issuing the RFP that there were 10 important elements to an integrated law office system. To determine the score for this section, a supplier would receive between 0 and 2 points for each application or feature which they could provide. They would receive a 0 if this feature was absent from their proposal. They would receive a 1 if the feature was there but at a rudimentary or unsophisticated level. Suppliers proposing a sophisticated version of a feature, such as an advanced word processing package, with a large, well-established user base would receive a score of 2. There were 10 required features. which are listed below: accounting word processing work product retrieval internal databases external legal and non-legal databases client matter management electronic mail/office functions Support hardware installation and support network design, installation and support user training and system documentation Evaluation Criteria are those factors which are used in the assessment of the proposals' merits. Selecting or constructing effective criteria is sometimes difficult. Here's some advice from Nova Scotia's guidelines: At a minimum evaluation criteria should be designed to assess the proponent's proposal in terms of:
Where local content or support requirements are other than purely operational in nature they must be both clearly identified and consistent with Nova Scotia's obligations under its trade agreements. New Criteria Once the evaluation criteria are published in the RFP, it is difficult to deal with factors that have not been included. How do you handle the proposal from a supplier that suggests an unexpected approach, a new piece of technology, or provides information which changes the way you think about the project? How do you build this unknown into the RFP? You certainly cannot include a factor labeled "unexpected information". If you did, you would be accused of being arbitrary. One approach which seems to be gaining in popularity is to include "Management of Risk" as a factor. This permits the Evaluators to take into account new, unexpected information. For all new information influences the Evaluators' assessment of the risks of the project. Other criteria such as "understanding of our requirements" can be used by the Evaluators to reward suppliers for innovation and unexpected approaches. The remainder of this section contains examples of evaluation criteria as published in RFPs over the last few years. Each example is preceded by a brief description and a score indicating my assessment of its merits. Each of the guidelines which follows has been rated using a numerical score of 1 through 5, with five being the highest possible and zero the lowest. In general terms, the more words spent on defining and explaining the scoring system, the more the supplier understands your requirements.
Scoring Guidelines - Example 1 One: Inadequate, fails to meet perceived needs. This 28-word descriptions does not provide adequate information:
Three: Acceptable, meets minimum requirements. This statement of evaluation criteria provides some direction but lacks important details in several areas.
Scoring Guidelines - Example 3 Four: Above average, exceeds minimum in some areas.
The Commissioner of DAS will establish an Evaluation Committee to evaluate the Proposals. The evaluation Committee will review the Proposals for format to ensure conformance with the requirements of this RFP. Failure to meet these requirements might result in rejection of your organization's Proposal. The Evaluation Committee can waive minor irregularities if, in its judgment, to do so would be in the best interests of Connecticut. Evaluations will be based on the Proposals, and additional information requested by Connecticut, applying the following criteria as to each Proposal: (a) Proposer's understanding of the project, its purpose and scope, and proposer's plan for performing the IT services, as evidenced by the proposed solution (b) Proposer's ability to perform the scope of the IT services, as reflected by its experience in performing such services and by the qualifications and abilities of the key individuals proposed as proposer's team (c) Proposer's demonstrated ability to make available the key personnel and facilities to perform the IT services at the time of contracting and to keep them on the project thereafter (d) Proposer's specific record of past performance of similar IT services (e) Price competitiveness of proposed solution and cost savings demonstrated (f) Proposer's ability to provide IT services form off-site facilities in Connecticut and to foster job retention and job creation in Connecticut (g) Demonstration of commitment to affirmative action by full compliance with regulations of the Connecticut Commission on Human Rights and Opportunities (h) Previous experience and customer references in government-sector IT services (i) Expertise in managing complex integrated systems and services and implementing and maintaining evolving leading-edge technologies (j) Expertise in business process reengineering, for purposes of developing new system architectures and developing plans for changes in computing environments (k) Expertise in consolidating mainframe environments and in migrating systems (in whole or in part) from mainframe environments to distributed-computing environments. (l) Financial strength and depth necessary to sustain a long-term relationship and long-term growth as Connecticut's IT services requirements change (m) Readiness to assume full accountability to Connecticut, its Agencies and its citizens for performance including commitments to perform IT services at levels that meet acceptable performance criteria, and commitments to an open-book approach and financial-reporting requirements (n) Proposer's demonstrated ability to protect highly sensitive and confidential information of its customers (o) A focus on delivering value-added services Weights Weights are used to assign a relative importance to each evaluation criterion or each group of criteria. They are usually determined by the Project Manager in consultation with the users and the Steering Committee prior to release of the RFP. Weights reflect the degree of importance of various criteria. Consider the following example:
The proposals constructed by vendors would be very different if (b) Technical response, or (a) Price response were worth 80%. The significance of weights is discussed in section 7.3.1. Scoring Systems Scoring systems are used to establish a numerical value indicating how well the proposal satisfies each selection criterion. Numerical scoring systems are easier to understand and explain than those based on only descriptive phrases. Evaluators sometimes have a difficult time deciding on the specific score of a factor. Suppose you were evaluating the plan for a project, or the project manager's experience. When is it worth 5 out of 10? Why isn't it worth a score of 6? Or 4? The more general question is how do we eliminate personal bias and take some of the arbitrariness out of scoring? Can we ensure that each of the evaluators are using the same scheme? There are two major types of scoring systems. The first is generic; it's the system used for the last 100 years in public schools to grade students' compositions. Ten out of 10 is excellent. Five is acceptable. Most evaluations use some variation of this method. The second type of system is based on the specific characteristics of the procurement and requires a lot more work to be done prior to issuing the RFP. In this system, positive and negative indicators are developed for each factor. For example, in evaluating the strength of the project team, a positive factor would be the proposed manager's experience with a similar system. A negative factor would be the use of a part-time project manager. After reviewing the proposals and these indicators, the evaluators would assign a score, usually using the zero-to-ten scale. The use of these indicators is a powerful tool in performing an objective evaluation, and one that is easy to justify should it come under public scrutiny. If these indicators are not formally developed and written down when the RFP is being constructed, they always emerge when evaluators are discussing their score for a specific proposal. These indicators simply reflect the concerns that an informed person would have related to each evaluation criterion. For example: Is the project manager experienced? Is the project manager full-time? The remainder of this section contains examples of scoring systems. A scoring system published in an RFP should help the supplier understand the process and create a better proposal. Scoring systems are intended to assist the evaluators to identify the merits or the deficiencies in a proposal in an unbiased, objective way. They should be easily understood and simple to use. Each example has been rated using the scoring system described below. In general, the greater the number of words contained in the description, the higher the score.
Example #1: One: Inadequate, fails to meet perceived needs.
7= Good - most requirements met, it is good enough 4= Satisfactory - some requirements met, not sufficient 1= Unsatisfactory - requirements essentially not met Example #2: Two: Fair, partially unresponsive
2 = Below Average - Response is substandard to that which is average or expected as the norm. 5 = Average - Baseline score for each item with adjustments based on evaluator's review and evaluation of the response. 8 = Above Average - Response is better than that which is average or expected as the norm. 10 = Exceptional - Response is clearly superior to that which is average or expected as the norm. Other Scoring Techniques There are many other techniques in common usage. In some RFPs, each proposal automatically receives a 5 out of 10 for a given factor. This amount is then adjusted up or down based on a consensus of the evaluators' opinions. This technique is less precise and more arbitrary than some of the preceding examples. Some organizations do not use a scoring system, but rather assign points for specific values of each factor. The maximum points represent the importance of the factor. For example, prior experience on similar projects could be worth a total of 8 points. The proposal would receive 2 points for each previous project up to the maximum. Not all organizations use scoring systems based on points. Some use words, others have a color-coding scheme. Here's an example of an adjectival scoring system. One of its good features is that the evaluation incorporates the "probability of success".
Exceptional - The submission exceeds expectations, excellent probability of success and in achieving all objectives. Very innovative. Good - Very good probability of success. Achieves all objectives in reasonable fashion. Acceptable - Has reasonable probability of success. Some objectives may not be met. Poor - Falls short of expectations and has a low probability of success.Unacceptable - Submission fails to meet requirements and the approach has no probability of success. |
Home
| About | Articles
| Products
| Services
| Seminars
| Site Map
| Contact |
