Autonomous Open Source LLM Evaluator (Ollama) - Full Guide

TL;DR

The autonomous evaluator compares Ollama models by asking each one to plan and solve the same problem, collecting their answers, and having GPT-4 Turbo select the best performer using configurable criteria. The tool supports ordinary text problems, while a second version can execute code and evaluate the returned results. Read on to understand the workflow, model selection, prompts, evaluation process, and manual-review option.

Transcript

today I wanted to show you a small tool I have built for myself that I'm actually using quite a lot to test out different models so basically it's more of an autonomous evaluator so you can kind of set these specific problems you want to solve and then you can kind of just see what models that perform best right so basically how this works you can ... Read More

Key Insights

• 🅰️ The autonomous evaluator workflow effectively demonstrates how AI can be utilized in model testing by systematically addressing specific problem types.
• 👻 The tool’s capability to assess models on both logical reasoning and coding execution allows for comprehensive comparisons across various tasks.
• 🔁 The evaluation process is supported by a feedback loop, enabling continuous improvement in model performance assessment and selection.
• 👤 Users can access a GitHub community to share and improve scripts, fostering collaboration and collective advancements in AI problem-solving techniques.
• 👤 The speaker emphasizes the importance of user engagement and community, hinting at future live streams for deeper interactions and demonstrations.
• 👨‍💻 Incorporating the ability to execute code in one version of the tool adds practical value for developers looking for accurate coding solutions.
• 👤 The adaptability of the tool to changing user needs and model selections makes it a versatile asset in AI development.

Questions & Answers

Q: How does the autonomous Ollama LLM evaluator work?

The evaluator loops through a user-defined list of Ollama models and gives every model the same problem. Each model first creates a detailed step-by-step plan and then produces an answer, which the tool stores for a final comparison by GPT-4 Turbo.

Q: How does the tool determine which model performed best?

GPT-4 Turbo receives the original problem, a correct answer generated for evaluation, and the collected model results. It evaluates each model's plan and answer using configurable parameters such as correctness, then identifies the best performer, explains why, and prints that model's answer.

Q: Which models can the evaluator compare?

The demonstrated list includes models such as Mistral, Llama 3, Gemma 7B, Magicoder, and a 14B model. The list can be made as long as desired, and the loop changes the selected model for each evaluation run.

Q: Why does the evaluator ask each model to create a plan first?

The first prompt asks every model to create a detailed plan for solving the problem step by step. That plan is then supplied alongside the problem in the prompt that instructs the model to produce its answer, and both can be considered during evaluation.

Q: What kinds of problems can the autonomous evaluator test?

The presenter built one version for normal text problems and another for code problems. The code version can execute code and return results that can then be evaluated, although the guide focuses mainly on the text version.

Q: What logical problem is used in the demonstration?

The example asks: Kaye has three brothers, and each brother has two sisters, so how many sisters does Kaye have? The stated answer is one sister because there are two sisters in the family, including Kaye herself.

Q: Why is GPT-4 Turbo used in the evaluation process?

The presenter uses GPT-4 Turbo as the preferred model for evaluating the collected answers. It is prompted with related context, the problem, the correct answer, the model results, and instructions to decide which model performed best and explain why.

Q: Can users review or customize the evaluation?

Yes. Users can define the problem, change the list of Ollama models, and set the evaluation parameters themselves; they can also manually inspect the answers after the automated comparison.

Summary & Key Takeaways

• The speaker introduces an autonomous evaluator tool designed to test different models efficiently by assessing their problem-solving capabilities in various contexts.

• The tool facilitates the evaluation process by storing problem details, running multiple model simulations, and allowing a final assessment to determine the best-performing model based on predefined criteria.

• The presentation includes examples, showing how models handle logical and coding problems, illustrating the practicality of using AI models for problem-solving in real-time evaluations.