Is Vicuna the ChatGPT Killer?

TL;DR

The Urkunya open source chatbot claims to be as good as ChatGPT, with impressive performance in various tasks.

Transcript

since the release of alpaca meta's large language model and the leak of its weights there has been an explosion of Open Source models first we had alpaca from Stanford group followed by Dolly from databricks contain a bunch of models from cerebris and most recently GPT for all from nomec AI but there is a new model which claims to be as good as cha... Read More

Key Insights

• 🤗 Open source language models like Urkunya have gained popularity since the release of Alpaca and Dolly.
• 👋 Urkunya claims to be as good as ChatGPT, with impressive performance on various tasks.
• 👊 It was trained on conversations collected from ShareGPT, a platform where users share their chat conversations.
• 😫 The training data set for Urkunya consisted of 70,000 conversations and was trained on 13 billion parameters.
• ❓ Evaluation results show that Urkunya can achieve response quality comparable to that of ChatGPT.
• 😘 Urkunya's training cost is significantly lower than other models, thanks to using the ShareGPT API for accessing training data.
• 👻 Urkunya allows for longer context in conversations, with a maximum context length of 2048 tokens.

Questions & Answers

Q: How was the Urkunya chatbot model trained?

The Urkunya model was trained using fine-tuning on conversations collected from ShareGPT, a website where users share their chat conversations.

Q: How does Urkunya compare to other open source chatbot models?

Urkunya claims to be as good as ChatGPT, with 90% of ChatGPT's quality. It outperforms other models like Alpaca and Lama.

Q: How many conversations were used to train the Urkunya model?

The Urkunya model was trained on 70,000 conversations collected from ShareGPT.

Q: What was the evaluation strategy for comparing Urkunya with other models?

The evaluation strategy involved selecting 80 diverse questions and using GPT-4 to judge the output of different models, including Urkunya, Alpaca, and Lama.

Q: Is the training code available for Urkunya?

The training code for Urkunya is available, unlike Lama's code, which is not accessible.

Q: How much does it cost to train the Urkunya model?

The training cost for the 7 billion parameter model was around $140, and for the 13 billion parameter model, it was only $300.

Q: How does Urkunya handle longer context compared to other models?

Urkunya has a maximum context length of 2048, much higher than Alpaca's 512. This requires more GPU memory but allows for more extensive context in conversations.

Q: Can the quality of Urkunya's responses be assessed accurately?

The quality of Urkunya's responses was assessed using GPT-4's evaluation, which rated the answers based on helpfulness, relevance, accuracy, and detail.

Summary & Key Takeaways

• The Urkunya chatbot model claims to have similar performance to ChatGPT, with 90% of ChatGPT's quality.

• The model was trained using fine-tuning on user-shared conversations collected from ShareGPT, a website where people can share their chat conversations.

• The model was trained on 70,000 conversations from ShareGPT and has 13 billion parameters.

• Evaluation results show that Urkunya can achieve response quality comparable to that of ChatGPT in various tasks.