Evaluating the performance of AI-based large language models in radiation oncology

In a brand new research published within the journal AI in Precision OncologyNikhil Thaker, from Capital Well being and Bayta Techniques, and co-authors, evaluated the efficiency of assorted LLMs, together with OpenAI’s GPT-3.5-turbo, GPT-4, GPT-4-turbo, Meta’s Llama-2 fashions, and Google’s PaLM-2-text-bison. The LLMs got an examination together with 300 questions, and the solutions have been in comparison with Radiation Oncology trainee efficiency.

The outcomes confirmed that OpenAI’s GPT-4-turbo had the very best efficiency, with 74.2% appropriate solutions, and all three Llama-2 fashions under-performed. The LLMs tended to excel within the space of statistics, however to underperform in scientific areas, excluding GPT-turbo, which carried out comparably to upper-level radiation oncology trainees and superiorly to lower-level trainees.

“Future analysis might want to consider the efficiency of fashions which can be fine-tune educated in clinical oncology,” concluded the investigators. “This research additionally underscores the necessity for rigorous validation of LLM-generated data in opposition to established medical literature and knowledgeable consensus, necessitating knowledgeable oversight of their software in medical education and apply.”

“The research highlights the potential of generative AI to revolutionize radiation oncology schooling and apply. OpenAI’s GPT-4-turbo demonstrates that AI can complement medical trainingsuggesting a future the place AI aids in enhancing affected person outcomes. It is important, although, to validate these applied sciences rigorously and contain specialists to make sure their dependable and efficient use in well being care,” says Douglas Flora, MD, Editor-in-Chief of AI in Precision Oncology.