Open (For Business): Big Tech, Concentrated Power, and the Political Economy of Open AI

Open (For Business): Big Tech, Concentrated Power, and the Political Economy of Open AI

In recent years, there has been a growing interest in the concept of 'open' AI. The idea behind it is that by making AI models transparent, reusable, and extensible, third parties can deploy and build upon these powerful off-the-shelf models. However, it is important to note that even the most open AI systems do not guarantee democratic access or meaningful competition in AI. Additionally, openness alone does not solve the problem of oversight and scrutiny.

One of the most notable developments in AI is the rise of large language models (LLMs). These models are trained to predict the next word and require massive amounts of text to do so. While the inner workings of LLMs may seem like a deep mystery, they can be understood without complex math or jargon.

Language models represent words using word vectors, which are essentially long lists of numbers. These vectors are built based on how humans use words, which means they often reflect the biases present in human language. For instance, some word vector models may yield biased results, such as "doctor minus man plus woman equals nurse." Researchers are actively working on mitigating these biases in language models.

The transformer architecture, which is commonly used in LLMs like GPT-3, has a two-step process for updating hidden states. In the attention step, words "look around" for relevant context and share information with each other. In the feed-forward step, each word considers the information gathered in previous attention steps and tries to predict the next word. GPT-3, the largest version of the transformer, performs thousands of attention operations each time it predicts a new word.

Training powerful models like GPT-3 requires a large amount of labeled data. However, manually labeling data is time-consuming and expensive. To overcome this challenge, researchers have developed techniques like unsupervised learning, where the model learns from unlabeled data. This approach can be likened to adjusting a faucet to achieve the right temperature. The more the model is trained, the smaller the adjustments it makes to improve its predictions.

It's worth noting that GPT-3 was trained on a corpus of approximately 500 billion words, which far exceeds the number of words a typical human child encounters by age 10. This vast amount of training data allows the model to make accurate predictions and navigate the complexities of language.

Interestingly, the concept of prediction is not limited to artificial intelligence. Philosophers like Andy Clark argue that the human brain can be seen as a "prediction machine." Our brains constantly make predictions about the environment to successfully navigate and adapt to it. Good predictions rely on accurate representations, just as a map helps us navigate the world more efficiently.

While open AI and large language models have their merits, it is crucial to ensure that power is not concentrated solely in the hands of big tech companies. Democratizing access to AI and fostering meaningful competition are essential for a more equitable and diverse AI landscape. Additionally, oversight and scrutiny should be embedded in the development and deployment of AI systems to address potential biases and ethical concerns.

To achieve these goals, here are three actionable pieces of advice:

1. Promote transparency and collaboration: Encourage companies and researchers to openly share their AI models and methodologies. This will allow for better understanding, scrutiny, and collective improvement of AI systems.

2. Invest in diverse datasets: To mitigate biases in AI, it is crucial to train models on diverse datasets that represent different perspectives and contexts. This will help create more inclusive and fair AI systems.

3. Embrace regulation and oversight: Governments and regulatory bodies should actively participate in shaping AI policies and frameworks. This will ensure that AI technologies are developed and deployed responsibly, with proper consideration for privacy, security, and ethical concerns.

In conclusion, the concept of 'open' AI holds promise for transparency and collaboration in the development of AI models. However, it is important to recognize that openness alone is not sufficient to address the challenges of access, competition, and oversight. By taking proactive measures to promote transparency, diversify datasets, and embrace regulation, we can strive for a more equitable and responsible AI ecosystem.