OpenAI DevDay 2024 | Community Spotlight | Tortus

TL;DR

Clinicians use an LLM application to reduce burnout and improve documentation efficiency while ensuring clinical safety.

Transcript

hi everyone my name is Nina I'm a research engineer at tauris and today with s we're going to present our journey of evaluating LMS in a clinical application seven minutes every time a clinician uses our llm powered application called toris they get seven minutes to be to do what they best out which is being a doctor and today this is really critic... Read More

Key Insights

• 🧑‍⚕️ Clinician burnout is a critical issue due to the excessive administrative workload generated by electronic health records and data entry tasks.
• 👻 Toris empowers clinicians by allowing them to design their workflows, cutting down the reliance on developers and increasing system efficiency.
• 🚫 The application effectively decomposes complicated workflows into manageable blocks that can be shared and reused among clinicians, fostering collaboration and efficiency.
• 🦺 The platform closely monitors LLM outputs for hallucinations and emissions, ensuring that clinical outputs remain safe for patient treatment.
• 👻 Experimentation is a core component, allowing clinicians to test new workflows in controlled settings before implementation, reducing risk in production.
• 🔁 Utilizing a feedback loop significantly improves the quality of clinical documentation, providing essential insights for refining LLM capabilities.
• 😨 The focus on clinician-centered design has positively impacted job satisfaction and workflow speed, benefiting patient care.

Questions & Answers

Q: What challenges do clinicians face that LLMs like Toris aim to address?

Clinicians face significant burnout due to excessive computer-related tasks, which consume up to 60% of their time. This includes data entry and order placements, leading to frustration and inefficiency. The Toris application intends to alleviate this burden by allowing clinicians to focus more on patient care instead of administrative tasks.

Q: How does the Toris platform work in terms of documentation?

The Toris platform generates documentation by using LLMs to analyze clinician-patient consultations and create structured outputs usable in electronic health records. The design centers around the clinicians, enabling them to actively participate in workflow creation, which helps ensure that documents meet clinical needs accurately.

Q: What are hallucinations and emissions in the context of LLMs?

Hallucinations refer to instances where an LLM generates information not present in the input data, potentially leading to clinical misjudgments. Emissions, on the other hand, denote data points that the model omits, which can lead to incomplete documentation. The Toris platform actively tracks and categorizes these to ensure effective error monitoring.

Q: How does Toris ensure clinician safety while using LLMs?

Safety is prioritized through the iterative evaluation of workflows and clinician involvement in design. Each workflow is tested by comparing outputs to ensure accuracy, and mistakes are classified as major or minor errors. This rigorous evaluation prevents unsafe outputs from being introduced into clinical practice.

Q: What role do clinicians play in the development of Toris workflows?

Clinicians are crucial in designing and reviewing workflows. They share and modify components known as blocks, allowing for more personalized and efficient output generation. This collaborative approach gives clinicians a significant voice in the system, increasing their engagement and satisfaction with the tool.

Q: How does the experiment and feedback process work within the Toris platform?

The experiment phase involves clinicians providing input on generated data, labeling outputs for hallucinations and omissions. This information feeds back into the workflow development, allowing for rapid adjustments and refinement by utilizing clinician expertise, ultimately creating a safer and more accurate system.

Q: Why is minimizing major errors a priority in the documentation process?

Major errors can significantly impact patient care, leading to potentially harmful clinical decisions. By closely monitoring LLM outputs for major hallucinations and omissions, the Toris platform aims to ensure that all patient documentation remains accurate and reliable, safeguarding patient health.

Q: What future developments does Toris plan to implement with their data set?

Toris plans to use its labeled dataset of errors to automate error detection in clinical documentation further. This will enhance safety by allowing for real-time monitoring of output, aiding clinicians in identifying and rectifying potential mistakes before they affect patient care.

Summary & Key Takeaways

• The presentation discusses clinician burnout caused by excessive computer tasks, emphasizing the need for effective LLM tools like Toris to streamline workflows.

• Toris allows clinicians to create, modify, and share distinct workflow blocks to enhance collaboration and clinical documentation, reducing reliance on developers.

• The process includes rigorous testing for safety, involving clinician feedback to minimize errors, particularly hallucinations and emissions in clinical outputs.