Towards Expert-Level Medical Question Answering with Large Language Models

Towards Expert-Level Medical Question Answering with Large Language Models

In recent years, artificial intelligence (AI) systems have made significant advancements in various domains, including the ability to play complex games like Go and solve intricate problems like protein-folding. However, one grand challenge that has remained elusive is the capability to retrieve medical knowledge, reason over it, and answer medical questions on par with physicians.

Large language models (LLMs) have emerged as a powerful tool in the field of medical question answering. One such model, Med-PaLM, achieved a "passing" score in US Medical Licensing Examination (USMLE) style questions with a score of 67.2% on the MedQA dataset. While this was a remarkable achievement, it became apparent that there was still significant room for improvement when comparing the models' answers to those provided by clinicians.

To bridge this gap, the researchers developed Med-PaLM 2, which combined several improvements to LLMs, medical domain finetuning, and novel prompting strategies, including an ensemble refinement approach. The results were impressive, with Med-PaLM 2 scoring up to 86.5% on the MedQA dataset, surpassing Med-PaLM by over 19% and setting a new state-of-the-art. Additionally, the performance of Med-PaLM 2 approached or exceeded state-of-the-art across other clinical topics datasets.

To evaluate the clinical utility of Med-PaLM 2, detailed human evaluations were conducted on long-form questions. In a comparative ranking of 1066 consumer medical questions, physicians preferred the answers generated by Med-PaLM 2 over those produced by physicians on eight out of nine evaluation axes pertaining to clinical utility. These findings highlight the potential of LLMs in reaching physician-level performance in medical question answering.

In another exciting development, the LLaMA project aimed to address the challenges faced by the LLM research community. The project focused on unifying the interfaces of instruction-tuning data, multiple LLMs, and parameter-efficient methods. This unified platform, known as PhoebusSi/Alpaca-CoT, facilitates easy access and use of instruction-tuning data, various LLMs, and efficient training methods.

One notable achievement of LLaMA is its zero-shot and few-shot ability, which significantly reduces the cost of training, finetuning, and utilizing large language models. For instance, LLaMA-13B outperforms GPT-3(175B), and LLaMA-65B is competitive with PaLM-540M. Additionally, Stanford Alpaca, a branch of LLaMA, has finetuned LLaMA-7B on 52K instruction-following data generated by the Self-Instruct techniques, further enhancing its instruction-following ability.

However, the LLM research community still faces three major challenges. Firstly, even with advancements like LLaMA-7B, there are still high requirements for computing resources, limiting accessibility. Secondly, there is a scarcity of open-source datasets for instruction finetuning, hindering the progress in this area. Lastly, there is a lack of empirical studies on the impact of different types of instructions on model abilities, such as responding to Chinese instructions and CoT reasoning.

In conclusion, the combination of large language models and medical question answering has shown remarkable progress towards expert-level performance. Med-PaLM 2, with its improved scores and preference from physicians, demonstrates the potential of LLMs in clinical applications. Likewise, projects like LLaMA and PhoebusSi/Alpaca-CoT aim to address the challenges faced by the LLM research community, making these powerful models more accessible and efficient. As the field continues to evolve, it is crucial to focus on refining LLMs, expanding instruction-tuning datasets, and conducting empirical studies to unlock their full potential in various domains.

Actionable Advice:

1. Researchers should continue refining large language models for medical question answering by leveraging advancements in the field and incorporating medical domain finetuning.
2. Efforts should be made to expand open-source datasets for instruction finetuning, enabling researchers to explore and enhance the instruction-following abilities of large language models.
3. Empirical studies should be conducted to investigate the impact of different types of instructions on model abilities, such as responding to instructions in different languages and reasoning over complex scenarios like CoT reasoning.