Prompt Injection Attack on GPT-4: Towards Expert-Level Medical Question Answering with Large Language Models

Prompt Injection Attack on GPT-4: Towards Expert-Level Medical Question Answering with Large Language Models

In recent years, artificial intelligence systems have made significant strides in various domains, from playing complex games like Go to solving complex problems such as protein-folding. However, one grand challenge that has long been elusive is the ability to retrieve medical knowledge, reason over it, and answer medical questions at a level comparable to physicians. This challenge is now being addressed with the help of large language models (LLMs).

One such LLM, Med-PaLM, achieved a milestone by exceeding a "passing" score in the US Medical Licensing Examination (USMLE) style questions. However, there was still room for improvement as the answers generated by the model were not on par with those of clinicians. This led to the development of Med-PaLM 2, which combines base LLM improvements (PaLM 2), medical domain finetuning, and novel prompting strategies.

The results of Med-PaLM 2 are impressive, with scores of up to 86.5% on the MedQA dataset, surpassing Med-PaLM by over 19% and setting a new state-of-the-art. The model also performed exceptionally well on other medical question answering datasets such as MedMCQA, PubMedQA, and MMLU clinical topics datasets. This indicates that Med-PaLM 2 is approaching or even exceeding the performance of physicians in answering medical questions.

To further validate the efficacy of Med-PaLM 2, detailed human evaluations were conducted. In a pairwise comparative ranking of 1066 consumer medical questions, physicians preferred the answers generated by Med-PaLM 2 over those provided by their fellow physicians on eight out of nine axes pertaining to clinical utility. This demonstrates the potential of LLMs in improving clinical decision-making.

Moreover, significant improvements were observed when evaluating Med-PaLM 2 on newly introduced datasets of 240 long-form "adversarial" questions designed to challenge the limitations of LLMs. These results indicate that Med-PaLM 2 is not only effective in standard medical question answering but also shows promise in handling more complex and nuanced queries.

While the progress made by Med-PaLM 2 is remarkable, it is important to acknowledge the limitations and potential risks associated with large language models. The GPT-4 System Card published by OpenAI highlights the vulnerability of such models to prompt injection attacks, which are considered one of the most effective methods of "breaking" the model. This raises concerns about the integrity and reliability of the answers generated by LLMs, especially in critical domains like healthcare.

To address these concerns, it is crucial to develop robust defenses against prompt injection attacks and ensure the transparency and explainability of LLMs. Additionally, ongoing research and collaborations between AI researchers and domain experts are necessary to validate the real-world efficacy of these models and address any biases or limitations they may possess.

In conclusion, the development of Med-PaLM 2 represents a significant step towards achieving expert-level medical question answering with large language models. The impressive performance of Med-PaLM 2 on various datasets and the preference of physicians for its answers highlight the potential of LLMs in improving clinical decision-making. However, it is important to address the vulnerabilities and limitations of these models, such as prompt injection attacks, to ensure their reliability and safety in real-world applications.

Actionable Advice:

1. Invest in robust defenses: Organizations and researchers should prioritize developing robust defenses against prompt injection attacks and other vulnerabilities associated with large language models. This will help ensure the integrity and reliability of the answers generated by these models, especially in critical domains like healthcare.

2. Foster interdisciplinary collaborations: To validate the real-world efficacy of large language models in healthcare and address potential biases or limitations, it is essential to foster collaborations between AI researchers and domain experts. This interdisciplinary approach will help bridge the gap between technical advancements and practical applications.

3. Enhance transparency and explainability: As large language models become more prevalent in various domains, including healthcare, it is crucial to enhance their transparency and explainability. This will enable clinicians and end-users to understand how the models arrive at their answers and make informed decisions based on the generated outputs.

By following these actionable advice, we can harness the power of large language models like Med-PaLM 2 while mitigating the risks and ensuring their responsible and ethical use in medical question answering and other critical applications.