FIT BUT scientists are developing an app for psychotherapists. Using deep learning to analyze therapy sessions

To offer psychotherapists systematic feedback on individual therapies and to improve the overall quality of psychotherapeutic care in the Czech Republic - this should be achieved by the new DeePsy application enabling automatic speech processing. Researchers from the BUT Speech@FIT group and their colleagues from Masaryk University are currently working on its development. The application should be completed in June next year.

In their practice, psychotherapists lack feedback that would allow them to continuously evaluate their work. "Psychotherapy is a demanding activity in which therapists process a considerable amount of information. They analyse some information consciously, but much more is processed unconsciously and intuitively. They can thus easily miss, for example, subtle signals of a client's discomfort or even deterioration. Clients are usually solving their own problems rather than evaluating the professionalism of the therapist's performance. In addition, some research has revealed a declining level or stagnation of psychotherapeutic qualities over time," explains Pavel Matějka from the BUT Speech@FIT group.

Manual transcription of individual sessions and their subsequent analysis are too time-consuming. That's why experts from Masaryk University turned to researchers from FIT BUT, who specialise in automatic processing and mining of information from speech. The test version of the DeePsy application, which works on the principle of deep learning, offers psychotherapists automatic transcription of sessions and analysis of their content.

Graphs comparing client and therapist speech show who spoke more during the session and what the average number of words per minute was. Keyword analysis can also reveal what emotions were prevalent in the speech or what proportion of verbs were phrased in past, present or future tense. The app will also assess the frequency of the most used words.

"Research studies show that when the language of the client and therapist differs significantly, either in content or form, it can indicate problems in the therapeutic relationship. DeePsy will alert the therapist to such a mismatch. How this information is handled, however, is up to the therapist. We only provide information to the therapist," adds Matějka.

To extract information from speech, the FIT researchers use automatic speech recognition, natural language processing and machine learning technologies. They trained the neural network algorithm on several thousand hours of audio recordings - from interviews to phone calls to spoken monologues. Yet right from the start, they ran into a problem: "We found that in therapy sessions, speech is very different from regular speech. Clients are usually emotionally distraught, so they repeat words much more often - perhaps three to five times before moving on. It took us a lot more time to come up with a meaningful transcript of the interviews from the start," Matějka adds.

The DeePsy app also includes a client questionnaire system that, together with the audio recordings, allows for systematic feedback to work with clients. "We will also work on evaluating therapist interventions. The algorithm should be able to recognise whether the therapist frequently asks questions, interprets, provides information or makes recommendations," says Matějka.

The web application, which is being developed as part of a Czech Technology Agency project, is currently being tested by researchers together with therapists at the Psychosomatic Clinic and the Therapeutic Port. It should be ready in June next year. "We hope that it will provide psychotherapists with user-friendly and useful feedback that will enable them to improve the quality of psychotherapeutic care in the Czech Republic," Matějka concludes.