How to evaluate LLMs with MLflow and Dagshub

TL;DR

MLflow provides evaluation for LLMs by logging model outputs and metrics, enabling comparison across prompts, models, and data. The video demonstrates creating an experiment, using an OpenAI model, and storing results in Dagshub for remote access, with attention to ground truth data and relevant QA metrics.

Transcript

hello guys in this video I'm going to talk about MLF flow with generative AI now if you don't know about MLF flow uh it really helps you to do a lot of things like uh run ml uh projects that solve complex real world challenges with experiment tracking visualization evaluation model model registry and serving right so you'll be able to track you'll ... Read More

Key Insights

• MLflow can track experiments, logs, and artifacts for generative AI workflows.
• LLM evaluation requires metric types such as question answering and text summarization.
• Dagshub serves as a remote repository to store data, experiments, and model outputs.
• Ground truth for LLMs can be generated or sourced, enabling comparison against model outputs.
• The workflow includes initializing an experiment, logging a model, and using a completion API.
• Evaluation data is matched to targets and model type to select appropriate metrics.
• MLflow supports additional metrics beyond basic accuracy, including readability and style indexes.
• The overall goal is to enable reproducible, trackable evaluation of LLMs in a practical project setup
• More videos from this channel:
  • How to build AI agents with Microsoft AutoGen for agentic AI
  • How to use AI for coding with wipe coding
  • How to build agents with OpenAI Agent Toolkit
• More from this channel: Krish Naik

Questions & Answers

Q: What is the purpose of using MLflow with LLMs in this video?

The video shows how MLflow can manage the full lifecycle of a generative AI project by tracking experiments, logging models and artifacts, and evaluating outputs from LLMs. This approach helps compare performance across different prompts and data, and it demonstrates how to integrate evaluation with a remote repository like Dagshub for reproducibility and collaboration. It emphasizes how to monitor metrics and artifacts to determine which configurations work best.

Q: How is ground truth used in LLM evaluation according to the video?

Ground truth is used as the reference against which model outputs are evaluated. In the example, ground truth texts are paired with the corresponding prompts to enable direct comparison with the LLM generated responses. This setup allows the evaluation to measure alignment with expected answers, and it helps identify gaps where the model may need improvement or retraining.”

Q: What role does Dagshub play in the workflow?

Dagshub acts as the remote repository for storing the entire evaluation workflow, including datasets, prompts, model outputs, and evaluation results. This enables easy sharing and collaboration, ensuring that all experiments and artifacts are accessible for review and reproducibility. It also helps in organizing projects so teams can track progress over time.

Q: Which metrics are highlighted for LLM evaluation in the video?

The video mentions metrics relevant to QA tasks such as exactness and readability as part of the MLflow evaluation suite. It explains that LLM evaluation requires specialized metrics beyond traditional accuracy since the ground truth can be nuanced. The metrics help quantify how well the model’s responses meet the defined targets and quality criteria for tasks like question answering and text summarization.

Q: How does the video suggest using OpenAI with MLflow?

The video demonstrates calling the OpenAI completion API within an MLflow run to generate responses from the LLM. The approach logs the outputs, system prompts, and user prompts as artifacts, while recording metrics that reflect the quality of the responses. This integration shows how to experiment with different prompts and configurations in a structured, trackable way.

Q: What is the recommended starting point for someone new to this setup?

The recommended starting point is to install MLflow, OpenAI API libraries, and Dagshub integration, then create a simple experiment that logs a basic LLM completion. By following the steps shown, including setting up a ground truth dataset and defining the evaluation metrics, a learner can understand the end-to-end flow and gradually add complexity.

Q: Why are QA metrics important for LLM evaluation?

QA metrics are important because they directly reflect how accurately the model can answer specific prompts, which is critical for many practical applications. They help quantify alignment with ground truth, measure the quality of generated content, and guide improvements in prompts and model configurations. Using a defined metric set prevents subjective judgments and supports reproducible evaluation.

Q: What is the overall takeaway regarding reproducibility and tracking?

The overall takeaway is that combining MLflow with Dagshub enables reproducible, trackable evaluation of LLMs by logging experiments, artifacts, and metrics in a centralized manner. This setup supports comparing multiple models and prompts, sharing results with collaborators, and iterating on improvements with a clear audit trail and consistent data management.

Summary & Key Takeaways

• MLflow evaluation of LLMs is shown through an example that logs model outputs, metrics, and artifacts to track performance across prompts and data. The workflow includes setting up an experiment, logging a model, and evaluating with predefined QA metrics. Dagshub is used as the remote repository for artifacts and results.

• The video emphasizes that LLM evaluation differs from traditional ML evaluation because ground truth can be nuanced or multi-answer, requiring a structured approach to metrics in MLflow. It demonstrates how to integrate OpenAI completions with MLflow tracking.

• The practical takeaway is to leverage MLflow with open source and cloud tools to manage the lifecycle of a generative AI project, including experiments, evaluation, and deployment considerations, while preserving reproducibility and traceability across the workflow.