As a chartered clinical psychologist within the British Psychology Society, I get sent a magazine called “The Psychologist” every month. In the August 2013 issue there was an article called “Why are effect sizes still neglected?” by Peter Morris and Catherine Fritz. The gist of the article is that when psychologist evaluate treatments using hypothesis testing, the results are often misleading. In particular, if you have large enough sample size you will almost always get a statistically significant result, even for minor and inconsequential effects. Consequently, it is important to always report an effect size when publishing results in an academic journal.
A common way to establish the effect size is to compare the symptoms for patients before and after a treatment. In the following video I give a tutorial on how to use data stored in an Excel spreadsheet with the AI-Therapy effect size calculator (it’s best to view full screen at 720p HD):
Fjola Helgadottir, PhD, MClinPsych, is a clinical psychologist, a senior research clinician at the University of Oxford, and is a co-creator of AI-Therapy.com, an online CBT treatment program for overcoming social anxiety
Last month I attended the 7th World Congress for CBT in Lima, Peru. Conferences are a great way to get up to speed on the latest developments in a field, and this conference was no exception. Overall, the presentations made me very optimistic about the future of online therapy. There is a lot of exciting and encouraging research being conducted.
As part of a symposium on internet-based treatment, I presented some of the latest results from AI-Therapy’s Overcome Social Anxiety program:
Dr Fjola Helgadottir presenting AI-Therapy overcoming social anxiety at the CBT World Congress 2013
Investigation into real world treatment data
My presentation was somewhat unusual for an academic conference in that it was based on real world data. Typically, talks are based on carefully controlled trials. There is an important reason for this – one goal of a trial is to make the results reproducible by other researchers. This is a key aspect of scientific research. However, there is an important question that is often ignored: will the results translate into the real world? The real world is chaotic, users are not screened, users are not monitored, there is less control over the equipment used, etc. In the past it has been found that treatments that work well in a laboratory environment cease to have the same impact when they are released to the general population. One goal of my talk was to present data from a commercially available treatment program, and contrast this with the latest results from academic systems.
Visitors to the AI-Therapy website
Before continuing, I should mention that all AI-Therapy users are anonymous, and their results are kept strictly confidential. The only data I presented are aggregated, showing average scores across groups of users.
As can be seen in the slide above, we have had almost 20,000 unique visitors to the website since our launch about a year ago. The top 5 countries for visitors are:
USA
UK
Iceland
Australia
Canada
These results are roughly what I would expect. The US is our largest market, but a significant margin. The reason Iceland has made the top 3 is due to some media coverage we have received there.
Effective social anxiety treatment
In order to assess the efficacy of the Overcome Social Anxiety program, I determined its pre-post effect size. When using the program, users fill out a series of questionnaires before starting, and the same questionnaires after completion. The effect size is a standardized measures of the reduction in symptoms over this period (see this page for information about effect sizes, and effect size calculators).
The effect size for the first 19 people who completed all sections of the program was 1.7. An effect size of 0.2 is considered small, an effect size of 0.5 is considered medium, and effect size of 0.8 is considered large. Therefore, an effect size of 1.7 is very large. (It is important to note that this value has been calculated based on people who completed the whole program, and does not include people who started the program, but did not reach the end. We intend to write up a more detailed analysis, and release it as a white paper on this site. Please watch this space.) The primary conclusion is that online treatment programs for social anxiety can be an effective treatment strategy for real world patients.
I am already looking forward to the 8th CBT World Congress, which will be held in Melbourne Australia in 2016. I look forward to seeing the advances that will be made in the online therapy field over the next three years!
Fjola Helgadottir, PhD, MClinPsych, is a clinical psychologist, a senior research clinician at the University of Oxford, and is a co-creator of AI-Therapy.com, an online CBT treatment program for overcoming social anxiety
You may have noticed we have been a little slow on the blog updates lately. The reason is that we have been very busy working on our new product, and are pleased to announce:
This is a little different than our core area (online self-help), so let me explain why we’ve created it. There are lots of statistics programs out there. However, some of them are hard for non-statisticians to use, some are expensive, and most require you to install software on your computer. We wanted something accessible, easy to use, and not loaded with options we don’t need. Given our extensive background in statistics and software, we decided to create our own solution.
These are our target audiences:
Academia: The online calculators provide academic researchers with tools to help them distinguish between effective and non-effective therapies. Naturally our focus is psychology, but the tools themselves will be equally useful to researchers in any field.
Clinicians: Too often clinicians never give stats a second thought after they finish their degree. However, chances are that you are surrounded by interesting data in your clinic. We encourage you to run some therapy data through the program, as you may be surprised by what you find!
Students: Learning stats isn’t easy. What we’ve created is something between an online tutorial and an interactive calculator. One way to learn the concepts is by interacting with the tools and visualizations. Try to get a feel for how changes to the input impact the results.
Everyone else: Statistics is much more important in our day to day lives than most people give it credit for. In particular, it helps us make sense of the world around us. Take the plunge, and see what it’s all about!
Please get in touch and let us know if you have any feedback or feature requests. Also, help us spread the word if you know of anyone who might find this useful.
