ACAT

You are here: Home > Reformulation > Winter 2018 [Issue 51] > Capturing Service Users’ Views of Their Experiences of Cognitive Analytic Therapy (CAT): A Pilot Evaluation of a Questionnaire

Capturing Service Users’ Views of Their Experiences of Cognitive Analytic Therapy (CAT): A Pilot Evaluation of a Questionnaire

Phyllis Annesley and Alex Barrow, 2018. Capturing Service Users’ Views of Their Experiences of Cognitive Analytic Therapy (CAT): A Pilot Evaluation of a Questionnaire. Reformulation, Winter, pp.14-21.

Download PDF

Summary

The pilot and revision of a questionnaire on service users’ views of receiving Cognitive Analytic Therapy (CAT) are described. The revised questionnaire has been named the Annesley and Barrow CAT Questionnaire (AB-CAT Questionnaire). Recommendations for further development of the AB-CAT Questionnaire are made.

Introduction

Cognitive Analytic Therapy (CAT) as an integrative and relational psychotherapy can be an effective intervention across a wide range of psychosocial difficulties (Ryle et al., 2014). Core components of CAT are its delivery in pre-determined time limits (usually 16 or 24 sessions); its focus on the establishment of a collaborative and mutually respecting working relationship with service users; and its use of six central CAT tools.

These are The Psychotherapy File, Reformulation Letter, CAT Diagram, Rating Sheets, Patient/Client Goodbye Letter and Therapist Goodbye Letter (Annesley and Sheldon, 2012).

The Reformulation Letter is a descriptive reformulation of problems to understand past and present difficulties and the impact of the therapy relationship (Ryle et al., 2014). The evidence base for CAT is accumulating but needs further development and support (Ryle et al., 2014).

Service users’ experiences of therapies received

Please note that within this paper the term “service user” is employed reflecting the fact that a wide range of people are accessing CAT, both within hospital and secure settings and community settings.

The emergence of practice-based evidence (Barkham et al., 2010) has meant that there is a growing expectation that psychological therapy services evidence the effectiveness of what they do (Department of Health, 2010). Furthermore, the “Payment by Results Initiative” means that commissioners pay healthcare providers for each patient seen or treated and outcomes achieved (Department of Health, 2013). Thus far, outcome evidence for CAT has focussed on the use of psychometrically robust measures assessing changes in symptomatology (Calvert and Kellett, 2014; Marriott and Kellett, 2009) rather than also considering service users’ experiences of therapy. This pattern of ignoring the service user’s experience of therapy (neglecting to obtain client feedback on therapy) is also seen in other psychotherapies such as Cognitive Behavioural Therapy (CBT) as illustrated in McHugh et al.’s (2014) study which evaluated brief CBT in primary care. Consequently, within published literature there is a dearth of measures to ascertain service users’ experiences of CAT, and specifically the different aspects of CAT, including the key CAT tools and the focus on the therapeutic relationship. CAT’s focus on the therapeutic relationship is important as meta-analyses of studies examining the linkage between alliance and outcomes in both adult and youth psychotherapy (Martin et al., 2000; Shirk and Karver, 2003; Karver et al., 2006) have confirmed that the quality of alliance is more predictive of positive outcome than the type of intervention.

Brown et al. (2014) similarly noted that the vast majority of existing psychotherapy measures focus on access to care or quantifying the number of visits rather than on the content of care or outcomes of care. Asking about service users’ experiences of specific aspects of therapies is important as Department of Health (2001) observed that asking about specific elements of care discloses problems that would not emerge from more general feedback questionnaires.

Understanding the service user’s experience of CAT including its specific components is also crucial for a number of other reasons. Firstly, it can assist in ensuring that clinicians consistently deliver therapies that service users consider to have been beneficial. This is important as evidence suggests that the service user’s assessment is a better predictor of the outcome of psychotherapy than the therapist’s (Castonguay et al., 2006). Secondly, it can inform the development of CAT and understanding important findings such as its popularity with service users (Ryle et al., 2014) as evidenced in the low attrition rate in CAT compared to other therapies. Thirdly, it can provide important information on specific aspects of CAT, helping clinicians to understand the specific roles that each aspect plays. For example, the ways in which CAT letters and diagrams assist service users in making changes and overall in their progress. Consequently, the authors aimed to conduct a pilot study including the development and evaluation of a questionnaire to ascertain service users’ experiences of receiving CAT.

Method

Aims

The authors aimed to design and evaluate a user-friendly and accessible questionnaire on service users’ experiences of CAT that could be used with a diverse range of client groups and within different clinical settings including secure and community settings.

Ethics

The study was approved by Nottinghamshire Healthcare NHS Foundation Trust’s Research Management and Governance Department. Participants could complete the questionnaire anonymously and they could indicate if they did not wish for their specific feedback to be published.

Evaluation Design

Three documents were developed in line with the study aims (contact authors for pilot elements of the study):

Document 1:

The first document was an Information sheet for participants entitled “Pilot Study of an Evaluation of Cognitive Analytic Therapy (CAT)”. This was designed to introduce participants to the study.

Document 2:

The second document was a questionnaire to ascertain service users’ experiences of CAT received. This was named “Questionnaire about your experiences of Cognitive Analytic Therapy”. This questionnaire was designed to capture feedback on all key aspects of CAT. It was designed by reviewing the literature (e.g. Khan and Beail, 2016) and other questionnaires on service users’ views of therapy including The Satisfaction with Therapy and Therapist Scale – Revised STTS-R (Oei and Shuttleword, 1999); the Experience of Service Questionnaire (Commission for Health Improvement, 2002); and The Client Satisfaction Questionnaire CSQ-8 (Larsen et al, 1979). The authors also drew on their extensive experience of CAT in designing this specific CAT questionnaire. Questions asked about CAT therapy received (number of sessions and the service where the therapy took place), participants’ overall experience of having CAT, their overall satisfaction with their CAT, and their satisfaction with the length of the therapy. Six further questions asked about participants’ experiences of the working therapeutic relationship. Following these questions participants were asked to rate how helpful and understandable each CAT tool was, and the helpfulness of their therapy in eight key areas related to the core aims of CAT (examples include “Understanding yourself more”, “Getting on with other people better”). All of the aforementioned questions from their overall satisfaction with their CAT incorporated a six point Likert scale (Extremely, Very much, Quite a lot, Somewhat, A little and Not at all). In addition, there were three other questions. The first asked “How likely would you be to recommend CAT to a friend or someone you are close to with similar difficulties?” The second asked “Was there anything about CAT you didn’t expect or think people starting CAT therapy should know?” The final question gave participants an opportunity to add further comments about their CAT therapy.

Document 3:

The third document was a feedback form to ascertain service users’ feedback on their experiences of completing the questionnaire. This was named “Pilot Study of an Evaluation of Cognitive Analytic Therapy (CAT): Feedback on the Questionnaire”. The feedback form contained seven questions covering clarity of the reasons for asking people to complete the questionnaire, views about the length of the questionnaire, time taken to complete it, clarity of the questionnaire layout, whether the questions were easy to understand, if there were was anything else the authors should ask people in terms of CAT, participants’ willingness in future to complete the questionnaire (if they had further CAT), and a space for further feedback about completing the questionnaire or suggestions for improving it.

Participants

Ten participants took part in the pilot study. Participants were from the authors’ workplaces within the National Women’s Service at Rampton Hospital (n=6) and Nottingham’s low secure community forensic service (n=2) plus participants (n=2) from community adult mental health services in other parts of the country. People who had completed CAT within the workplaces of the authors were invited to take part in the pilot study. At the time of the study eight current patients within the National Women’s Service had received CAT; the uptake by six participants therefore meant a seventy-five percent response rate. Because many patients who had received CAT were no longer within these services the authors liaised with the Association for Cognitive Analytic Therapy (ACAT) and tried to recruit additional participants subject to interested clinicians obtaining their own consent for the study from their relevant Trust. Despite a lot of interest initially from therapists whom the authors approached, only two completed forms were returned from service users external to the study centre. Therapists may have been put off inviting their service users to take part by the authors asking the therapists to obtain their own consent.

Results

Participants completed the questionnaire and the feedback form fully with very little missing information.

Participants said they were clear about the authors’ reasons for asking them to complete the questionnaire (eighty percent of participants answered this question). Most participants (80%) felt the questionnaire was the right length, while the remainder (20%) felt it was too long. Participants took an average of 7.77 minutes (SD = 2.64) to complete the questionnaire with times taken varying from 5-10 minutes.

Participants stated that the layout of the questionnaire was clear with 90% answering this question. One participant (10%) said that question 4f (“How much did you talk about patterns from your life generally also being present between you and your therapist?”) was “difficult to understand”. Another participant suggested adding “more colour”.

In response to the question “Is there anything else we should ask people about in terms of CAT?” (80% response rate) most participants (6/8, 75%) wrote “no” or “n/a”. One participant suggested seeking information when people have experienced other therapies about which therapy they found more useful. Another remarked that they would have liked a longer therapy.

Ninety percent of participants responded to the question “If you had CAT in the future how happy would you be to complete the questionnaire about your experiences of CAT?”.

Among these, the majority (8/9, 89%) indicated they would be willing to do so. Replies included “happy”, “very” and “I recommend CAT to everyone”. The ninth participant said she finds questionnaires “tedious”.

Ninety percent of participants responded to the question “Do you have any other feedback about completing the questionnaire or any suggestions as to how we could improve it?” Six out of these nine participants (67%) had no additional feedback.

Among the other three, one participant acknowledged getting “a bit confused as I do not remember filing out a questionnaire at the beginning” (referring to the Psychotherapy File, one of the CAT tools). A second participant suggested taking into consideration the long-term effects of CAT as exemplified here:

“To take into consideration the ethics and long after effects that could be immersed into. It may be brilliant initially but what could be the long-term effects”.

Finally, a third participant wrote:

“I think the questionnaire is very helpful, very well set out and easy to understand. I don’t think any improvements need to be added”.

Discussion

Initial feedback suggests that the questionnaire was clear, easy to understand and patients indicated they would be willing to complete it after therapy.

Following the pilot study changes were made to the questionnaire in line with participants’ feedback. An additional colour of blue was added to the questionnaire to assist participants’ understanding and to make it more attractive visually and some questions were amended. Question 3 which asked “How satisfied were you with the length of your CAT therapy?” was augmented with a part b asking “What, if anything would you like to have changed about the length of your CAT therapy?” Question 4f was amended to make it clearer by including an example. It was changed from “How much did you talk about patterns from your life generally also being present between you and your therapist?” to “How much did you talk about patterns from your life generally also being present between you and your therapist? For example, if you are someone who likes to please others, you may have also wanted to please your therapist and you and your therapist may have spoken about this”. The introduction was also changed to make the questionnaire amenable for use beyond the initial pilot. One further amendment was made splitting question 9 into two parts as part of the process of having this paper reviewed for publication.

The pilot study was limited by the small number of participants and the fact that most participants were detained patients in secure settings with very severe mental health problems including mental illnesses and personality disorders. However, CAT has been developed as a treatment for people with more severe problems including personality disorders and as such it is very positive that the questionnaire was accessible to this hard to reach group. With the advantage of hindsight an initial multi-site NHS pilot with ethical approval would probably have worked better as the authors observed that some clinicians appeared to be put off taking part by having to seek their own consent.

As noted in summary at the outset of this paper the revised questionnaire (Appendix 1) incorporating changes suggested within this study has been named the Annesley and Barrow CAT questionnaire (AB-CAT Questionnaire). Feedback from the pilot study suggests that the AB-CAT Questionnaire could potentially be very beneficial in assisting individual therapists in receiving feedback on their clinical practice and also more widely in the development of the evidence base for CAT and informing the future development of CAT so that it remains focussed on delivering positive outcomes for service users.

In terms of the future, the AB-CAT Questionnaire needs to be trialled with a larger number of participants who reflect diversity including all aspects of the protected characteristics as outlined in the Equality Act 2010 (Government Equalities Office, 2013). The sample also needs to include people who have accessed CAT within a range of different clinical settings. Furthermore, the reliability and validity of the AB-CAT Questionnaire need to be established. Finally, since CAT is being used with people with intellectual disability (ID) the authors strongly advocate the development of an ID specific version of the questionnaire.

Conclusion

The study has successfully piloted and further developed a highly accessible and user-friendly questionnaire (AB-CAT Questionnaire) that shows much promise in ascertaining service users’ views of their experiences of CAT received. The authors are currently liaising with Nottingham University to make progress with establishing the AB-CAT Questionnaire’s reliability and validity. Alongside this, the authors would greatly value receiving readers and ACAT’s views about the potential for the AB-CAT Questionnaire to become part of routine evaluation in CAT.

Authors

Dr Phyllis Annesley, Consultant Clinical Psychologist, Nottinghamshire Healthcare NHS Foundation Trust, National Women’s Service and National High Secure Learning Disability Service, Rampton Hospital, Retford, Nottinghamshire, DN22 0PD.

E mail: phyllis.annesley@nottshc.nhs.uk

Dr Alex Barrow, Clinical Psychologist, Nottinghamshire Healthcare NHS Foundation Trust, The Wells Road Centre, The Wells Road, Nottingham, NG3 3AA.

Email: alex.barrow@nottshc.nhs.uk

Acknowledgements

The authors would like to thank the study participants for their hugely valued contributions to the study. They would also like to thank their managers and colleagues for generously supporting the study.

References

Annesley P., and Sheldon, K., (2012), “Cognitive analytic therapy within the perimeter fence: an exploration of issues clinicians encounter in using CAT within a high secure hospital”, The British Journal of Forensic Practice, 14(2), 124-137.

Barkham, M., Hardy, G.E. and Mellor-Clark, J. (2010), Developing and delivering practice-based evidence: A guide for the psychological therapies, Wiley, Chichester.

Brown J., Hudson Scholle, S. and Azur M., (2014), Strategies for Measuring The Quality of Psychotherapy: A White Paper to Inform Measure Development and Implementation, Mathematica Policy Research, Washington DC.

Calvert, R. and Kellett, S. (2014), “Cognitive Analytic Therapy: A review of the outcome evidence base for treatment”, Psychology and Psychotherapy: Theory, Research and Practice, 87, 253-277.

Castonguay L. G., Constantino M. J., Grosse Holtforth M. (2006), “The working alliance: where are we and where should we go?”, Psychotherapy, 43, 271–279

Commission for Health Improvement (2002), The Experience of Service Questionnaire Handbook, Commission for Health Improvement, London.

Department of Health (2001), Intermediate care, Department of Health, London.

Department of Health (2010), Equity and excellence: Liberating the NHS, Retrieved 19 August 2016 from: http://www.dh.gov.uk/en/Publicationsandstatistics/Publications/

PublicationsPolicyAndGuidance/DH_123759

Department of Health (2013), A Simple Guide to Payment by Results, Department of Health, London.

Government Equalities Office (2013), Equality Act 2010: Specific Duties To Support The Equality Duty What Do I Need To Know? A Quick Start Guide for Public Sector Organisations, Government Equalities Office, London.

Karver M. S., Handelsman J. B., Fields S., Bickman L. (2006), “Meta-analysis of therapeutic relationship variables in youth and family therapy: the evidence for different relationship variables in the child and adolescent treatment outcome literature”, Clinical Psychology Review, 26, 50–65

Khan, M. M. and Beail, N. (2016), “Service user satisfaction with individual psychotherapy for people with intellectual disabilities”, Advances in Mental Health and Intellectual Disabilities, 7(5), 277-283.

Larsen, D.L., Attkisson, C.C., Hargreaves, W.A., and Nguyen, T.D. (1979), “Assessment of client/patient satisfaction: Development of a general scale”, Evaluation and Program Planning, 2, 197-207.

Marriott, M. and Kellett, S. (2009), “Evaluating a cognitive analytic therapy service; practice-based outcomes and comparisons with person-centred and cognitive-behavioural therapies”, Psychology and Psychotherapy: Theory, Research and Practice, 82, 57–72.

Martin D. J., Garske J. P., Davis M. K. (2000), “Relation of the therapeutic alliance with outcome and other variables: a meta-analytic review”, Journal of Consulting Clinical Psychology, 68 (3), 438–450.

McHugh, P. Gordon, M. and Byrne, M. (2014), “Evaluating brief cognitive behavioural therapy with primary care”, Mental Health Review Journal, 19(3), 196-206.

Oei, T. P. S. and Shuttleword, G. J. (1999), “Development of a satisfaction with therapy and therapist scale”, Australian and New Zealand Journal of Psychiatry, 33 (5), 748-573.

Ryle, A., Kellett, S., Hepple, J. and Calvert, R. (2014), “Cognitive analytic therapy at 30”, Advances in Psychiatric Treatment, 20, 258-268.

Shirk S. R., Karver M. (2003), “Prediction of treatment outcome from relationship variables in child and adolescent therapy: a meta-analytic review”, Journal of Consulting and Clinical Psychology, 71 (3), 452–464.

Erratum

“Clinical Outcomes Of Cognitive Analytic Therapy Delivered By Trainees”

by Darongkamas, J., Newell, A. Hewitt-Moran, T. & Jordan, S

published in the Winter 2017 Issue of Reformulation was first published in Mental Health Today July 2015, pp. 24-27.

www.pavpub.com

Petition to NHS England - The Case for Funding Training in the NHS 2021 Alert!
ACAT's online payment system has been updated - click for more information

Full Reference

Search the Bibliography

Type in your search terms. If you want to search for results that match ALL of your keywords you can list them with commas between them; e.g., "borderline,adolescent", which will bring back results that have BOTH keywords mentioned in the title or author data.

The CAT Articles Review
Knight, A., 2009. The CAT Articles Review. Reformulation, Summer, p.32.

A Randomised Controlled Trial of the Efficacy of CAT for the Treatment of Personality Disorder
Clarke, S., 2001. A Randomised Controlled Trial of the Efficacy of CAT for the Treatment of Personality Disorder. Reformulation, ACAT News Autumn, p.x.

A Brief Survey of Perceptions of Cognitive Analytic Therapy Within Local Mental Health Systems
Turley, A., Faulkner, J., Tunbridge, V., Regan, C. and Knight, E., 2009. A Brief Survey of Perceptions of Cognitive Analytic Therapy Within Local Mental Health Systems. Reformulation, Winter, p.26.

Research Into the Use of CAT Rating Sheets
Coombes, J., Taylor, K. and Tristram, E., 2009. Research Into the Use of CAT Rating Sheets. Reformulation, Winter, pp.28-29.

The 4P’s model: A Cognitive Analytic Therapy (CAT) derived tool to assist individuals and staff groups in their everyday clinical practice with people with complex presentations
Phyllis Annesley and Lindsay Jones, 2016. The 4P’s model: A Cognitive Analytic Therapy (CAT) derived tool to assist individuals and staff groups in their everyday clinical practice with people with complex presentations. Reformulation, Summer, pp.40-43.

Help

This site has recently been updated to be Mobile Friendly. We are working through the pages to check everything is working properly. If you spot a problem please email support@acat.me.uk and we'll look into it. Thank you.

Search

Capturing Service Users’ Views of Their Experiences of Cognitive Analytic Therapy (CAT): A Pilot Evaluation of a Questionnaire

Full Reference

Search the Bibliography

Related Articles

Other Articles in the Same Issue

Help