AI-Therapy creates online fully automated programs using the latest evidence-based treatments, such as cognitive behavioural therapy. The social anxiety program has been tested with a randomized controlled trial. To find out more visit:
A receiver operating characteristic (aka the "ROC curve") displays the performance of a classification system over a range of threshold values. It plots the false positive rate on the x-axis and the true positive rate on the y-axis. The curve is based on signal detection theory, and is used widely in a number of fields. In fact, psychological applications among the earliest.
Imagine you are conducting a "human lie detector" experiment. You have a number of videos of people who have been accused of a crime, but are claiming to be innocent. Some of the suspects are telling the truth, whlie others are lying (assume that you, the experimenter, know the truth for each video).
The experiment is conducted as follows: you show all of the videos to a subject, and for each video he or she gives a value in the range 0-100: 0 means "I think this person is definitely telling the truth" and 100 means "I think this person is definitely lying".
How can we evaluate the subject's ability to detect lies? This involves classifying each video based on the participant's judgement. In order to label the videos we need to select a threshold value. Any video with a confidence number above this threshold will be classified as "lie", and the others will be classified as "truth". For example, let's consider a threshold value of 50. In this case, all videos that the subject gave a score of 50 or higher will be classified as "lie". Some of these classifications will be correct (a true positive) and some will be incorrect (a false positive). Each threshold value leads to a true positive rate and false positive rate.
Assume our participant gives the following confidence estimates for the videos of people telling lies:
54, 34, 87, 76, 57, 23, 88, 97, 76, 75, 56, 73, 89, 92, 36, 55, 91, 52, 48, 72
and the following estimates for the videos without lies:
45, 34, 67, 45, 65, 12, 5, 87, 34, 39, 28, 64, 6, 48, 22, 28, 11, 43, 50, 55
At a threshold value of 50, the true positives are the green samples and the false positives are the red samples. This leads to a true positive rate of 80% (16/20) and a false positive rate of 30% (6/20).
An ROC curve shows the true positive rate and false negative rate for each threshold value. This gives an overall indication of a subject's lie detection skill: if the curve is close to the upper left hand corner of the plot, this is high performance since there is a low false positive rate and a high true positive rate.