Recording a patient’s clinical history is an essential part of a visit to the doctor. But note-taking is time-consuming and can affect the capacity of a clinician to build a rapport with their patient.
Doctors traditionally took notes by hand to assist recall or to share information with colleagues. In the informatics era, this has evolved into an Electronic Health Record (EHR) containing a summary of clinical conversations produced either during or after each medical appointment.
David Fraile Navarro, a medical doctor and postdoctoral research fellow at the Australian Institute of Health Innovation at Macquarie University, knows first-hand the importance of recording a patient consultation as well as the burden it places on doctors.
“As I was a GP, one of the most cumbersome tasks has always been dealing with the EHR and adding or extracting information there,” he says.
Patients, meanwhile, may worry their doctor is not paying enough attention to them, which may be interpreted as a lack of empathy, or physically examine them properly because they are sitting at a computer typing notes.
“Some studies have measured that half of the time of a consultation for some doctors is spent on documentation,” Dr Navarro says.
When clinicians looked blindly at these notes, it turns out the quality of the human and the AI-generated one by ChatGPT was similar.
Up to one-in-three medical practices in Australia already have some form of AI tool embedded in their record-keeping software despite concerns about the technology’s accuracy, reliability and privacy.
“More than anything we need to be sure they are robust and accurate, and proper data management is in place,” Dr Navarro says.
In Expert evaluation of large language models for clinical dialogue summarization, published in Nature Scientific Reports, a team of researchers led by Dr Navarro subjected AI to “a battery of tests” to determine whether large language models (LLMs) such as those powering ChatGPT could be trusted to record a patient’s medical history.
“The most basic one compares notes generated by AI with a human generated one word-by-word and seeing how similar they are,” he says. “Then more advanced testing explored the quality of the note, its length, coherency, consistency and so on.”
Dr Navarro’s team also asked clinicians to judge the clinical quality of notes generated by AI and doctors in a blind test.
“When clinicians looked blindly at these notes, it turns out the quality of the human and the AI-generated one by ChatGPT were similar,” he says.
Free to focus
Dr Navarro says AI has the potential to improve the patient experience during a medical consultation by allowing a clinician to focus on them rather than taking notes.
AI may also help doctors produce documents such as clinical letters, referrals and discharge summaries.
“Improved treatment is further away, but as documentation improves it may indirectly help patient management,” he says.
However, Dr Navarro says more testing is needed before AI can be trusted with recording clinical histories.
“We need to find out if the errors AI makes are similar or different than doctors’ errors,” he says.
Concerns about the use of AI range from maintaining the privacy of patients and sensitive health information to accuracy and reliability, which may be life-threatening, for example, if names of medications are recorded incorrectly.
Dr Navarro says the use of AI also raises the problem of automation bias - the tendency of humans to place too much trust in machines - as well as insufficient training or poor implementation of AI tools.
“We may need to work on setting up testing standards and work with regulators to ensure these tools are safe to use,” he says. “The burden of having a proper clinical note still lies on the clinician, so they need to be aware that they still are responsible for ensuring whatever goes on record is correct.”
David Fraile Navarro is a postdoctoral research fellow in the Australian Institute of Health Innovation at Macquarie University.
