Understanding evaluation scoring

Last updated September 27, 2023
Written by Princess Luzadas

Felix provides several options for organisations to build evaluation templates. Templates can include provision for free text responses, N/A responses for evaluators, the ability for evaluators to skip questions, and a variety of numerical scoring scales.


When drilling into evaluation scores, the following principles are important to understand.


As a reminder, templates are made up of one or more criteria, each criterion having a % weighting. Criteria can include one or more questions.


All questions are weighted equally within a criteria set

Felix does not currently allow each question within a criteria to be individually weighted. If a criterion has three scorable questions, they are each worth 33%. If there are four scorable questions, they are each worth 25%, etc.


For Yes/No questions, a Yes is always a 100% score

This is important when your users are setting up templates and the way questions are worded to get the right response from evaluators.


Free Text questions are not scorable

This might seem obvious, however, important to highlight when a criteria set only includes free text questions.


Evaluation criteria will be 0% weighting if all questions are free text

As all free text questions are non-scorable, a criteria setup like this will be 0% weighting and effectively act purely as a way to capture qualitative feedback in the same location as quantitative feedback in an evaluation template.


Skipped questions or N/A responses are effectively removed from the scoring for a criteria

If a question was set as Optional or a N/A response option was offered on a scorable question, then Felix does not penalise or reward the vendor for this question. It is removed from the calculation for that criteria and the remaining scored questions are rebalanced.


Example: A criteria has four questions. Three are scored and one is left Unscored or with No Response.


The default behaviour is that all questions within a criteria are equally weighted.


Criteria 1
Question Question WeightingScore
125%1/5
225%1/3
325%10/10
425%Unscored or No Response


Rebalanced, so that we do not reward the vendor for the last question with 100% nor penalise them with 0%.


Criteria 1 (rebalanced)
QuestionQuestion WeightingScore
133%1/5
233%1/3
333%10/10
4
Unscored or No Response


With a resultant score for this criterion of:


(0.33*0.2)+(0.33*0.33)+(0.33*1) = 50%


Criteria with all N/A responses are effectively removed from the scoring

There are possible cases where a criterion has all scorable questions answered with N/A (Unscored). In these cases, the criteria are effectively excluded and Felix does not penalise nor reward the vendor for this fact. It is removed from the calculation template and the remaining scored criteria are rebalanced.


Criteria with all skipped questions are given a score of 0

Unlike N/A responses from evaluators, where they feel they are not in a position to score this topic or where all questions in criteria are skipped (where they are not mandatory), then the criteria are given a score of 0 for that evaluation.


Combining scores across evaluations for the same template follows similar rules to the treatment of questions inside the criteria

Similar to how N/A (Unscored) and skipped (No Response) questions are treated inside criteria for a single evaluation when combining scores from multiple evaluations on the same template, rebalancing of weightings occurs.


Example: Two evaluations are completed by two users, using the same template. The template has three criteria. 


User 1
CriteriaWeightingScore
150%60%
210%70%
330%Unscored
TOTAL
=(0.6*0.5+0.7*0.1) / (0.5+0.1) = 61.7%


For User 1, they responded with all N/A responses in Criteria 3. Criteria 3 is therefore Unscored and does not penalise nor reward the vendor. The evaluation score is rebalanced across the scored criteria.


User 2
CriteriaWeightingScore
150%60%
210%No Response
330%40%
TOTAL
= (0.6*0.5+0*0.1+0.4*0.3) = 42%


For User 2, they skipped all responses in Criteria 2. Criteria 2 is therefore marked as No Response and receives a score of 0.


Felix then combines these two evaluations' totals (of the same template) to create an aggregate score, with a simple average:


(61.7%+42%) / 2 evaluations = 51.9%

Was this article helpful?