Once a candidate has completed your Codility test, you'll gain access to their test report to review their score. Reading these reports correctly is really important to ensure you're identifying which candidates you want to move forward with first.
A candidate test report is made up of a number of data points, divided per task. To see a live example, click here.
The first thing you'll see is the candidate's overall test score, made up of the combined total points achieved on each task. You can stack rank candidates by this overall score, and then dig deeper to see how they reached this total, task by task. Every task is scored for correctness, and where possible, also performance.
"Performance" corresponds to how their solution performs with large data-sets in terms of running time complexity. To produce scalable code, a candidate requires good algorithmic knowledge and developed programming skills. In other words, for a candidate’s solution to be scalable, it must be suitably efficient and practical when applied to large data-sets.
Symbols of the form "O(…):", presented in the evaluation report under the label “Detected time complexity”, describe the scalability of the solution. This notation shows the precise relationship between the input data size and how quickly the solution is able to compute the answer. For example, if the data consists of N elements, then the expression "O(N2)" (read: "N squared") means that there is a quadratic relationship between the number of elements and the running time of the solution. In other words, increasing the input size by 3× will roughly increase the execution time by 9×, increasing the input size by 5× will roughly increase the execution time by 25×, and so on. O-notation is a popular way of describing the behavior of algorithms (see Wikipedia), and Codility employs it to describe the scalability of the solution program.
"Correctness" is the program’s ability to produce correct results for inputs of moderate sizes, including various corner cases. For a programmer to produce code that runs correctly requires care, attention to detail and a good understanding of both the problem and the programming language. Where performance cannot be assessed, correctness is the only contributor towards the total score. If a candidate passes all the test cases, he or she gets the maximum score. If the answer is wrong the details of what was expected and what they actual output was is provided.
There's a lot more to reviewing a candidate's test report, so if you'd like a tour of the insights features, or have a specific question about a candidate, feel free to contact us via chat, or email email@example.com and we'll be happy to help.