Inside a radical new project to democratize AI: Do You Have Lightning In a Bottle? How to Benchmark Your Social App

Inside a radical new project to democratize AI: Do You Have Lightning In a Bottle? How to Benchmark Your Social App

Artificial intelligence (AI) has become increasingly prevalent in our lives, with large language models (LLMs) like OpenAI's GPT-3 and Google's LaMDA gaining significant attention. These models have been primarily developed by big tech companies, restricting their use and keeping the inner workings a secret. However, a new project called BigScience aims to change this by creating an open and transparent LLM called BLOOM (BigScience Large Open-science Open-access Multilingual Language Model). This project, coordinated by AI startup Hugging Face and funded by the French government, involves over 1,000 volunteer researchers who have worked tirelessly to develop BLOOM.

One of the key features of BLOOM is its ease of access. Unlike other LLMs, BLOOM can be downloaded and used by anyone free of charge on Hugging Face's website. This allows AI developers to utilize the model as a foundation for their own applications. With 176 billion parameters, BLOOM surpasses the size of OpenAI's GPT-3, but offers similar levels of accuracy and toxicity. What sets BLOOM apart is its transparency. The researchers behind BLOOM have shared details about the data it was trained on, the challenges faced during its development, and how its performance was evaluated.

The release of BLOOM is a significant departure from the norm in the AI industry. While companies like Meta have released their own large language models, they are only available upon request and have limited use to research purposes. Hugging Face takes it a step further by introducing a Responsible AI License, which acts as a terms-of-service agreement. This license discourages the use of BLOOM in high-risk sectors such as law enforcement or healthcare, as well as any actions that may harm or deceive people. The project's ethical guidelines have been in place since the beginning, ensuring that BLOOM's development aligns with ethical principles.

The success of a social app can often feel unpredictable, but there are certain metrics that can help assess its performance and potential. One of the key metrics for consumer social apps is daily active users (DAUs). To benchmark growth, it's important to see consistent growth in DAUs, indicating that more people are using the app every day. Monthly user growth is also crucial, with percentages around 20% considered okay, 35% good, and 50% great. It's ideal for this growth to come organically, showcasing the app's inherent virality.

Engagement ratios, such as DAU/MAU (daily active users divided by monthly active users), provide insights into how frequently users are using the app. A ratio of 25% is considered okay, 40% good, and 50%+ great. Another metric to consider is the L-ness curve, which examines the distribution of users by the number of days they are active on the app. Best-in-class social apps have an L-ness curve that "smiles" or skews right, indicating regular usage. A percentage of 30% is okay, 40% good, and 50%+ great for L5+ performance.

Retention is another crucial aspect to evaluate the success of a social app. N-day retention, which measures the percentage of users retained after a certain number of days, is an important metric. For example, d1 (day 1) retention of 50%, d7 (day 7) retention of 35%, and d30 (day 30) retention of 20% are considered okay. Good retention would be d1 retention of 60%, d7 retention of 40%, and d30 retention of 25%. Great retention would be d1 retention of 70%, d7 retention of 50%, and d30 retention of 30%. It's also important to analyze how quickly the retention curve flattens out, as a significant degradation of retention over time can be a cause for concern.

In order to benchmark your social app, it's essential to track these metrics and compare them against the industry standards. Understanding the growth, engagement, and retention of your app can help identify areas for improvement and potential for success. Additionally, it's crucial to iterate and make necessary adjustments based on the data and user feedback.

To further enhance the performance and potential of your social app, here are three actionable pieces of advice:

1. Foster organic growth: Aim for your app's growth to come primarily from organic sources rather than paid acquisition. This indicates that users find value in your product and are more likely to invite their friends, creating a viral effect.

2. Prioritize engagement and retention: Focus on creating a product that users want to incorporate into their daily lives. Regular usage and high retention rates indicate that your app is meeting the needs and expectations of its users.

3. Continuously analyze and iterate: Use the data and feedback from your users to make informed decisions and iterate on your app's features and user experience. Regularly track and benchmark your app's performance to identify areas for improvement and capitalize on opportunities for growth.

In conclusion, the BigScience project's development of BLOOM represents a significant step towards democratizing AI. By providing an open and transparent language model, AI developers can freely access and utilize BLOOM to build their own applications. On the other hand, benchmarking the growth and performance of social apps requires careful analysis of metrics such as daily active users, engagement ratios, and retention rates. By understanding and optimizing these metrics, social app developers can increase their chances of catching that elusive lightning in a bottle.