Warning on AI and Data in mental health: ‘Patients are drowning’

Innovators in AI and Data need to act now to prevent harm to patients using digital tools to support their mental health, said a speaker at Digital Health’s AI and Data event on Monday.  

Umar Nizamani, CEO, International, at NiceDay, emphasised that AI will inevitably become an essential tool in mental health care: “I am very confident AI will not replace therapists – but therapists using AI will replace therapists not using AI.”

But he also stressed that technology companies working in the mental health field need to address the risks in the technology. Nizamani said there had been suicides caused by AI, citing the case of a person in Belgium who died by suicide after downloading an anxiety app.

The individual was anxious about climate change. The app suggested “if you did not exist” it would help the planet, said Nizamani.

He called on the industry to come together to ensure that mental health systems using AI and data are “explainable’, “transparent”, and “accountable”.  Ignoring the risks could be risky for the industry, as well as patients, he suggested: “The self-driving car doesn’t go to jail, the driver does.”

Nizamani was speaking during a panel discussion on utilising AI and data to provide effective mental health services.

After the session, he told Digital Health News that legislators were moving too slowly to regulate the industry, which was driven by innovation and the financial rewards for innovation. “Everyone is pouring money in, because there is money to be made.”

He added: “The patients are drowning. They don’t have any support.”

Companies needed to “build responsibly” he said. And he suspected that many cases of harm to patients were currently going unreported because the link between an app and a death was hard to prove. “It’s going to be like the tobacco industry,” he said.

Panellists also spoke of the positive difference AI and data is making to mental health services. Arden Tomison, CEO and founder of Thalamos, revealed how his company has digitised Section 136 in London, a highly restrictive pathway of care which involves complicated NHS and police workflows.

The change has not only relieved the “huge admin burden” of the paper system but provided accurate information which could be used to improve the care of highly vulnerable patients.

Dr Stefano Goria, co-founder and CTO at thymia, gave an example of “frontier AI”: “mental health biomarkers” which are “driving towards precision medicine” in mental health. Goria said thymia’s biomarkers (e.g. how someone sounds, or how they appear in a video) could help clinicians be aware of symptoms and diagnose conditions that are often missed.

Earlier, the president of The Royal College of Radiologists Dr Katherine Halliday said that “we need to proceed with caution” as AI in healthcare develops.