XGen 7B: Salesforce's 8k LLM for long sequence modeling

TL;DR

Salesforce has released XGen, a 7 billion parameter language model trained on 1.5 trillion tokens, with an 8K context window, that can be used for text summarization and other tasks.

Transcript

So we've got a new model out from Salesforce, and this is pretty interesting model. This is basically, trying to be a similar style to the LLaMA 7 billion model except one of the big things that they've done here is they've made the sequence length of the context window instead of 2K like LLaMA they've taken it right out to 8K. , so if we look at t... Read More

Key Insights

• 😲 Salesforce has released a new language model called XGen, with 7 billion parameters and an 8K context window, making it larger and more powerful than previous models like GPT-2.
• 😊 Salesforce has a track record of releasing open-source models, contributing to the community and allowing for experimentation and innovation.
• 🌍 XGen is trained on 1.5 trillion tokens and is available under the Apache 2.0 license, enabling commercial use without worrying about restrictions.
• 🚀 Salesforce has released different versions of XGen, including base models in both 4K and 8K versions, as well as an instruct model specifically aimed at summarization and text writing tasks.
• 💡 The XGen model is benchmarked against other models and shows promising performance in many tasks, such as multi-modal language understanding (MMLU), but may not perform as well in code generation tasks.
• 🌐 The XGen model is trained on the Red Pajama dataset, a widely-used and valuable resource in the AI community, and is also multilingual, although it supports only 22 languages at present.
• 📈 XGen shows potential for long sequence tasks, outperforming other models in benchmarks. It leverages a dense attention mechanism for the 8K context window, which may have implications for memory usage.
• 🎯 While XGen performs well in summarization tasks, it may benefit from being fine-tuned on better distilled datasets in the future, potentially leading to even better performance in areas like reasoning tasks.

Questions & Answers

Q: How does XGen compare to other language models on MMLU benchmarks?

XGen performs well on MMLU benchmarks, showing higher accuracy than open source models like Falcon 7B and MPT 7B but being outperformed by LLaMA models.

Q: What are the limitations of XGen's 8K context window?

While the 8K context window allows for long-range dependencies in text, it may consume more memory and is less easily extendable compared to some other models.

Q: Can XGen be used for code generation tasks?

XGen is not the ideal choice for code generation tasks, as it performs better in reasoning tasks and text summarization.

Q: What datasets were used to train XGen?

XGen was trained using the Red Pajama dataset, which has become a standard in the community, and is multilingual, although limited to 22 languages.

Q: How does XGen compare to larger language models like GPT-4?

XGen is a step towards larger models like GPT-4, but it is not distilled from GPT-4 and does not achieve the same level of performance. However, it may be fine-tuned on better datasets in the future.

Q: Can XGen be used for commercial purposes?

Yes, XGen is open source and available under the Apache 2.0 license, allowing for commercial use without any restrictions.

Summary & Key Takeaways

• Salesforce has released XGen, a language model with 7 billion parameters and an 8K context window, trained on 1.5 trillion tokens.

• The model is open source and available under the Apache 2.0 license, allowing for commercial use.

• XGen shows promising results in text summarization and performs well on MMLU benchmarks, but struggles with code generation tasks.