ChatGPT Not Yet Ready for Clinical Practice

ChatGPT is the fastest growing app in the world, surpassing 1 million users five days after launch, according to DemandSage. The website has been visited over 1 billion times in the last month. It can write poetry, essays, summaries, and more. When it comes to technology, we want it to help more than harm, but sometimes new technologies are rapidly adopted, and it isn’t until later that we learn the ramifications. We see this with social media platforms that we now know are addictive and associated with anxiety, depression, and even physical ailments.

Psychiatrists are already saying that ChatGPT has the potential to be a valuable tool in psychiatric practice and that students should use it to enhance clinical acumen and increase their skill at integrating technology into patient care. While I agree that ChatGPT is a revolutionary tool with remarkable potential, there are several important concerns to address before utilizing ChatGPT for psychiatric practice.

First, let’s consider the sources and quality of information that ChatGPT uses to generate its answers. The large language models underlying ChatGPT are pre-trained with text—16% of the text comes from books and news articles, and 84% comes from webpages. The webpage data include high-quality text such as Wikipedia but also low-quality text like spam mail and Reddit, according to a paper by Wayne Xin Zhao and colleagues published in May in arXiv, a Cornell University publication. Knowing the source of information provides insight into not only the accuracy of the information but also what biases may exist. Bias can originate from data sets and become magnified through the machine learning development pipeline, leading to bias-related harms. For example, false news and racist viewpoints are often shared on Reddit. Another way to think of ChatGPT is as a user interface that packages internet information that existed before 2021. While these sources can be accurate at times, there are more reliable and high-quality resources for students to access. Taking excerpts from a reliable text and asking ChatGPT to summarize it may be a way around this limitation.

Second, let’s consider privacy violations. The OpenAI privacy policy states that it collects all conversation history and that data collected trains the algorithm, which users may opt out of. Personal identifiers, commercial information, network activity information, geolocation data, and account login credentials may be disclosed to “affiliates, vendors and service providers, law enforcement, and parties involved in Transactions.” Given this privacy policy, it is ill advised to use ChatGPT to assist with patient documentation or the handling of patient information until safeguards for health care data are in place.

Finally, it is difficult to say who is accountable for what and to whom under which circumstances. In computerized systems, “bugs” or accuracy ranges can divert accountability and chalk up an error to just falling within the margin of error. The problem with this is that sometimes the margin of error is a result of human-made decisions along the machine learning pipeline that could be better optimized, blurring accountability, according to a paper by A. Feder Cooper et al. published in arXiv in May. A lack of transparency in chosen models, data sources, and training methods further obscures the picture. Additionally, there is room for influencing decision-making when an algorithm decides which information to present or exclude.

Ultimately, while this technology has amazing potential in psychiatry, these mainstream, large language models are not built with adequate sources, privacy protections, or robust enough models to prevent unforeseen harm. Regulation may help increase transparency, accuracy, accountability, and privacy. Striking a balance between innovation and regulation is needed if we want to enjoy this technology and limit the harm. When that day comes, I will sing its praises!

An opposing viewpoint to this article is posted here. ■

Darlene King, M.D., is an assistant professor in the Department of Psychiatry at UT Southwestern Medical Center, deputy medical information officer at Parkland Health, and the chair of APA’s Committee on Mental Health Information Technology. She graduated from the University of Texas at Austin with a degree in mechanical engineering prior to attending medical school and residency at UT Southwestern.