Patients trust AI health advice but expect doctors to take responsibility

People often trust chatbots with intimate problems and accept their advice, yet still expect human doctors to provide care and take responsibility, ERR technology commentator Kristjan Port writes.

European Union rules that took effect at the end of summer require people to be clearly told when they are interacting with a chatbot, unless that is already obvious. Port argues that such a warning does not necessarily stop people from attributing human qualities to a machine.

The Guardian has described people who, while preparing wills, also worry about what will happen to their chatbots. Data from the United States and the United Kingdom show that 27% to 31% of adults use chatbots as emotional companions for personal matters. Many see them as trusted friends or therapists.

People in the UK regularly turn to chatbots for emotional support, discuss ways to deal with their problems and seek help making decisions. Even a prominent warning on screen may not stop someone in distress from treating a machine as if it were alive and confiding intimate concerns in it. Two-thirds of respondents, for example, trusted a chatbot’s health advice.

Doctors use chatbots, too. OpenEvidence, a platform that provides health information using artificial intelligence, says about two-thirds of US doctors, or roughly 650,000, use its medical advice chatbot, along with 1.2 million doctors outside the United States. The company says uptake has grown sharply in the past few years.

OpenEvidence is considered comparable to asking a colleague for an expert opinion. The service can also prepare hospital discharge summaries and provide a study platform for medical exams. The company says 60% of searches on its platform are related to clinical decisions.

In many cases, patients do not know whether a decision came from a doctor’s own assessment, a conversation between the doctor and an AI system, or advice generated by AI alone. Doctors must agree with the advice and take responsibility for passing it on. But when patients expect decisions to reflect a doctor’s own expertise, it can be unclear whether the doctor would have reached the same conclusion independently.

That raises a further question, Port writes: how much must a doctor be able to explain a decision in order to be held responsible for it? By adopting AI-generated medical advice, doctors may also be validating the technology’s clinical competence.