Felix provides several options for organisations to build evaluation templates. Templates can include provision for free text responses, N/A responses for evaluators, the ability for evaluators to skip questions, and a variety of numerical scoring scales.
When drilling into evaluation scores, the following principles are important to understand.
As a reminder, templates are made up of one or more criteria, each criterion having a % weighting. Criteria can include one or more questions.
Felix does not currently allow each question within a criteria to be individually weighted. If a criterion has three (3) scorable questions, they are each worth 33%. If there are four (4) scorable questions, they are each worth 25%, etc.
This is important when your users are setting up templates and the way questions are worded to get the right response from evaluators.
This might seem obvious, however, it's important to highlight when a criteria set only includes free text questions.
As all free text questions are non-scorable, a criteria setup like this will be 0% weighting and effectively act purely as a way to capture qualitative feedback in the same location as quantitative feedback in an evaluation template.
If a question was set as Optional or a N/A response option was offered on a scorable question, then Felix does not penalise or reward the vendor for this question. It is removed from the calculation for that criteria and the remaining scored questions are re-balanced.
Example: A criteria has four questions. Three are scored and one is left Unscored or with No Response.
The default behaviour is that all questions within a criteria are equally weighted.
| Criteria 1 | ||
| Question | Question Weighting | Score |
| 1 | 25% | 1/5 |
| 2 | 25% | 1/3 |
| 3 | 25% | 10/10 |
| 4 | 25% | Unscored or No Response |
Re-balanced, so that we do not reward the vendor for the last question with 100% nor penalise them with 0%.
| Criteria 1 (re-balanced) | ||
| Question | Question Weighting | Score |
| 1 | 33% | 1/5 |
| 2 | 33% | 1/3 |
| 3 | 33% | 10/10 |
| 4 | Unscored or No Response | |
With a resultant score for this criterion of:
(0.33*0.2)+(0.33*0.33)+(0.33*1) = 50%
There are possible cases where a criterion has all scorable questions answered with N/A (Unscored). In these cases, the criteria are effectively excluded and Felix does not penalise nor reward the vendor for this fact. It is removed from the calculation template and the remaining scored criteria are rebalanced.
Unlike N/A responses from evaluators, where they feel they are not in a position to score this topic or where all questions in criteria are skipped (where they are not mandatory), then the criteria are given a score of 0 for that evaluation.
In order to respect the intended weightings of criteria and any N/A responses that can distort this effect, Felix first averages criteria scores across completed evaluations, then sums these averages for a total.
Example: Two evaluations are completed by two users, using the same template. The template has three criteria.
| User 1 | ||
| Criteria | Weighting | Score |
| 1 | 50% | 60% |
| 2 | 20% | 70% |
| 3 | 30% | Unscored |
| TOTAL | =(0.6*0.5+0.7*0.2) / (0.5+0.2) = 61.7% | |
For User 1, they responded with all N/A responses in Criteria 3. Criteria 3 is therefore Unscored and does not penalise nor reward the vendor. The evaluation score is re-balanced across the scored criteria.
| User 2 | ||
| Criteria | Weighting | Score |
| 1 | 50% | 60% |
| 2 | 20% | No Response |
| 3 | 30% | 40% |
| TOTAL | = (0.6*0.5+0*0.2+0.4*0.3) = 42% | |
For User 2, they skipped all responses in Criteria 2. Criteria 2 is therefore marked as No Response and receives a score of 0.
Felix then combines these two evaluations as follows:
Criteria 1: ( 60% + 60% ) / 2 evaluations scored = 60% of 50% weighting = 30%
Criteria 2: ( 70% + 0% ) / 2 evaluations scored = 35% of 10% weighting = 3.5%
Criteria 3: ( 40% ) / 1 evaluation scored = 40% of 30% weighting = 12%
For a total score for the vendor across the two (2) evaluations of 45.5%
Note: Felix used to simply average the two evaluation scores, which in the example above would be 51.9% (which is (61.7 + 42)/ 2). This method was changed to better respect criteria in the case of N/A scores.
