Step-by-Step Guide on How to Evaluate Submissions
When managing a solicitation that will be awarded based on the highest score(s), evaluating submissions in a fair, open and transparent manner can be daunting – the Province not only needs to be fair, open and transparent, but also needs to be able to prove that it was. Therefore, it’s essential that solicitation decisions are supported by the process and documentation.
For more information on evaluations, refer to the Preparing RFPs: A Ministry Guide to the Request for Proposals Process or the Ministry Writers' Guide: Developing and Managing Short-form Requests for Proposals. In addition, videos specific to the evaluation process for the case study are available to assist.
Step 1: Develop and Finalize the Evaluation Handbook
Step 2: Record Submission Arrival Dates and Times
Step 3: Check Mandatory Requirements
Step 4: Distribute Submissions
Step 5: Individual Evaluations
Step 6: Consensus Evaluations
Every solicitation process requires documentation on how submissions were evaluated (see the Core Policy and Procedures Manual (CPPM) section 6.3.3.e.1); the document with the evaluation criteria that is developed for each solicitation process is generally known as the “evaluation handbook”. Ideally, the initial draft of the evaluation handbook is developed before the solicitation is released, as errors in the solicitation document will be more obvious when the detailed evaluation criteria are being developed. The evaluation handbook should be finalized before opening any submissions; in order to avoid delays in starting the evaluation process, finalize the evaluation handbook before closing.
The evaluation handbook should be based on the mandatory and weighted criteria disclosed in the solicitation document. Remember the Golden Rule: Say what you’re going to do, and do what you said you would. Do not deviate from the process and criteria described in the solicitation, as these deviations can form the basis for allegations of unfair treatment.
However, this doesn’t mean that the solicitation document should contain everything required to score 100% of the available points. Disclosing approximately 70% of the evaluation criteria in the solicitation will help to balance the need to inform vendors what is being sought and to award highest scores only to high quality submissions.
There is no one format that works best for all solicitations, but consider the following when developing this document:
- Match exactly the wording used for mandatory requirements in the solicitation document;
- Closely match the detailed weighted criteria being scored to the relevant information provided in the solicitation document;
- Additional weighted criteria that were not disclosed in the solicitation document may be included, but such criteria should be limited, and should apply only to those elements that an experienced vendor would likely address and that the ministry would find valuable;
- Use the same order of information as found in the solicitation for ease of reference between the solicitation document, the evaluation handbook, and submissions;
- As much as possible, include complete and objective information on what is awarded points, in order to assist evaluators to be consistent with each submission;
- Provide a space for evaluators to document their scores;
- Provide room for evaluators to note any comments they may wish to make to support their scores; and
- Provide sufficient details so that a completed evaluation handbook can be used to explain to a proponent / respondent what was needed to score 100% of the available points.
Evaluation handbooks will also need a breakdown of available points. If the solicitation document provided a detailed breakdown, the evaluation handbook needs to match it. If the solicitation document provided a high-level overview of points, the evaluation handbook needs to be consistent but usually further breaks down those points.
Once the evaluation handbook is finalized and the submissions are opened, no further changes can be made (with the exception of correcting errors in mathematical formulas, if required) to avoid the perception that the evaluation criteria have been changed to favour or disfavour a particular proponent / respondent.
If an error is found in the evaluation handbook after opening submissions that is not related to a mathematical formula, determine a fair manner to apply the applicable criteria to all submissions. The evaluation team will collectively decide on the approach to take, which may include (but is not restricted to) a reasonable interpretation of the criteria, or awarding the same score to all submissions. Be sure to document the approach chosen and the reason for adopting this approach.
The date and time that each submission arrives at the closing location should be recorded in order to provide proof of whether the submission arrived on time. If allowing eBids through the BC Bid website, the system will record this date and time. If allowing emailed submissions, follow the information found in Accepting Emailed Submissions to determine the receipt date and time. If allowing hard copy submissions (i.e. hand delivery or courier), date and time stamp each submission as it arrives.
It can be helpful to assign someone with the task of recording the names of the proponents / respondents. For example, a record can be maintained of the names of each proponent / respondent as their submissions are received. This can assist with ensuring that no submission is missed in the evaluation process, and to respond to proponents / respondents asking whether or not their submission has been received. However, the person who records these names should not be on the evaluation team and the submissions should not be opened until after the closing date and time has passed and the evaluation handbook has been finalized.
After the evaluation handbook is finalized and the solicitation closes, open each submission received and check the mandatory requirements. Designate who will be responsible to check mandatory requirements, which may include one or more evaluators and/or someone else. Do not skip any submissions; if the solicitation process included options for delivery (e.g. hard copy and eBids through BC Bid), check for all options.
If the solicitation included a mandatory requirement to be on time, the date-and-time stamp on the outside of the submission package (for hard copy submissions) or the electronically recorded time (for either eBids or emailed submissions) will indicate if it has been met or not. If the submission is late, it should not be opened. For other mandatory requirements, open the submission to determine whether or not they have been met. Not all submission will clearly mark where mandatory requirements have been addressed, and therefore a search throughout the submission may be required.
The ministry can ask proponents / respondents for clarification on their submission under limited circumstances, which may be helpful to confirm whether or not a mandatory has been met. Refer to Clarifications after the solicitation has closed in Contract Finalization for more information.
If a submission has missed one or more mandatory requirements, use the evaluation handbook to document exactly what was missed. Do not forward any submissions that have missed a mandatory to the evaluation team. These submissions cannot receive any further consideration and should be set aside. Ministries may decide to either inform the proponents / respondents that missed a mandatory requirement immediately of the situation, or may wait until announcing results of who was successful. See Announcing Results for more information.
If there is any doubt as to whether or not a submission has met a mandatory requirement, consult with the ministry’s Procurement Specialist, Legal Services Branch or Procurement Services Branch. Consider that sometimes substantial compliance may apply. Where there is a question of whether a substantial (or material) compliance standard will apply, or whether such a standard has been met, consult with a Procurement Specialist or Legal Services Branch.
If the solicitation included a mandatory requirement that the submission be received before the closing date and time (as most do), stringently apply this date and time to all submissions. For example, if the solicitation states that submissions must be received before 2:00 pm on a certain date, any submission that arrives at 2:00 pm on that date or later is late. The Province should not accept a late submission – even if it is one minute late. Instead, record when the submission was received in that vendor’s evaluation handbook, and otherwise do not consider the submission. If the late submission arrived as an eBid through BC Bid, the system will record the fact that it was late and will inform the vendor. Refer to Accepting Emailed Submissions for information on how to manage a late emailed submission, if emails were permitted. For submissions that arrive in hard copy, take a photocopy of the envelope or package with the date stamp visible, and return the package (unopened if possible) to the vendor.
Refer to the video on non-compliant proposals for more information.
Distribute copies of all submissions that have met the mandatory requirements to the evaluators. This can be done electronically through email, and/or by mailing hard copies. Keep one copy of all submissions for the procurement file.
Once all proponents / respondents are known, a list of these organizations should be sent to all evaluators. Evaluators should all confirm that they have no conflicts of interest with any proponent / respondent. Refer to Evaluation Teams for more information on conflicts of interest.
Evaluation team members should read and evaluate each submission on their own, using the finalized evaluation handbook to document what they find (one handbook for each submission). This documentation should include scores awarded, rationales for those scores, and page numbers where the pertinent information was found in the submissions.
All evaluation documentation, including individual notes, is subject to the Freedom of Information and Protection of Privacy Act. For this reason, keep all comments professional and factual, with supporting facts for any opinions expressed.
Evaluators should not discuss the evaluation of submissions during their individual assessments, in order to avoid unduly influencing each other. However, someone who is not an evaluator should be designated as the contact for evaluators to bring forward any questions on how to manage unusual or unexpected situations. This contact person should have a good understanding of the particular solicitation and procurement generally, in order to be able to provide advice to individual evaluators or the team overall, as required.
During individual evaluations, the evaluators should evaluate all submissions completely, even if they feel that the minimum score, if any apply, will not be met or that the submission could not possibly be successful. This approach has the following benefits:
- The individual evaluator may be scoring submissions more strictly than the rest of the team. If in the consensus meeting the submission actually meets the minimum scores or scores high enough to be successful, this evaluator will require evaluation notes on the entire submission to ensure the fairness of the process.
- Even if the team collectively determines that the minimum score is not met, individual notes on the entire submission can be used to provide informal feedback during a debrief.
- Documented evaluation information is required for all submissions (see Core Policy and Procedures Manual (CPPM), section 6.3.3.e.1); dismissing a submission as not being successful without this documentation may create issues if the solicitation is subject to a Freedom of Information request, an audit, or a vendor complaint.
- All proponents / respondents are entitled to a debrief (see CPPM 6.3.3.c.1), and therefore ministries require documentation that will allow them to provide clear, factual and detailed information on the strengths and weaknesses of their own submission.
What to Evaluate
Only the information contained in the submissions can be considered for the evaluation process. Similar to an interview process, evaluators cannot give points for something they know a proponent / respondent has or does but is not included in their submission. In addition, a proponent / respondent should be given credit for everything stated in their submission even if the evaluators are not convinced that it is true. Such issues can be resolved through reference checks (see Using References for more information) if it’s related to experience, or through the contract finalization process if it relates to anything else. This process ensures that the evaluators are treating those proponents /respondents they know the same as those they do not know.
When all evaluators have completed their individual evaluations, they should bring their completed evaluation handbooks to the consensus meeting. Using these notes, the team should discuss each section of submissions, and agree on the scores and comments to be captured. These consensus discussions will determine the final results of the evaluation, and therefore should be well documented.
Evaluators should not change their scores or comments on their individual notes during the consensus process. Individual notes should be retained in the solicitation file, to demonstrate the decision-making process. It is expected that individual scores and notes will differ from the final decisions made during the consensus process; this demonstrates a true consensus, rather than consensus being heavily influenced by only one evaluator.
If the team has trouble reaching consensus on a particular point, average each evaluator’s scores after discussion, in order to come to agreement on a final score. However, be sure that the comments support the score that is ultimately determined.
If a minimum score applies and is not met in the consensus process, end the consensus evaluation of that submission as the proponent / respondent cannot be successful without meeting all minimum scores. Many evaluation teams start with the section(s) that have minimum scores to save time.
Some solicitations state that all proponents / respondents who meet the mandatory requirements and minimum scores will be successful. For these solicitations, a summary of the individual scores can be developed and, for those submissions where all team members individually agree that the minimum score(s) has been met or not met, the scores averaged. The consensus process will then focus only on those submissions where some evaluators determined that the minimum score(s) was met and others did not. However, this approach only works if the individual evaluation notes document clear rationales for the scores awarded.
Once the consensus evaluation process is complete, all evaluators should indicate their agreement with the final results. This can be done via email or by signing a hard copy print-out of each final consensus evaluation handbook. The full consensus documentation – including individual and consensus notes, as well as agreements with final results – should be included in the procurement file. Refer to Records Management and the Freedom of Information and Protection of Privacy Act for more information.
Identical or Unclear Results
Although this rarely happens, it is possible for two or more submissions to receive the exact same highest score in a solicitation process. If this occurs for a solicitation resulting in a single successful proponent / respondent, the evaluation team should confirm the scores awarded to all, to ensure no errors in math were made. Note that a small difference – even less than one point – still identifies a highest scoring proponent / respondent, and the award should be made accordingly.
Once the scores are confirmed to be exactly the same, the successful proponent / respondent could be determined using a random selection process. In this case, invite all proponents / respondents with the same highest score to attend a meeting where the successful submission will be chosen. Attendance could be via video conference if required. Explain what happened, and how the successful submission will be selected. Ensure that each proponent / respondent clearly understands and agrees with the process, and award to whichever one is successful using the method chosen.