Aligning Language Models to Follow Instructions: 4 Metrics Every Growth Hacker Should Be Watching

Aligning Language Models to Follow Instructions: 4 Metrics Every Growth Hacker Should Be Watching

As technology continues to advance, language models have become increasingly sophisticated. However, there is a crucial flaw in these models - they are not aligned with their users. This lack of alignment means that the models are not effectively following instructions and may produce inaccurate or misleading information. To address this issue, researchers have turned to reinforcement learning from human feedback (RLHF) to make the models safer, more helpful, and more aligned.

One significant finding in this research is that InstructGPT models, which have been trained using RLHF, are much better at following instructions than GPT-3 models. In fact, when comparing prompts submitted to both InstructGPT and GPT-3 models on the API, InstructGPT models are significantly preferred. This preference is due to the fact that InstructGPT models make up facts less often and show small decreases in toxic output generation.

To achieve these improvements, the researchers fine-tuned the models using a small curated dataset of human demonstrations. This curated information played a crucial role in reducing harmful outputs and enhancing the quality of the generated content. It is interesting to note that the labelers involved in the evaluation process preferred the outputs from the 1.3B InstructGPT model over those from the much larger 175B GPT-3 model.

Despite these advancements, there is still work to be done to fully align and make the models safe. InstructGPT models still generate toxic or biased outputs, make up facts, and even produce sexual and violent content without explicit prompting. This poses a significant risk as these models may be susceptible to misuse if instructed to produce unsafe outputs. Consequently, the researchers are actively exploring ways to make the models refuse certain instructions reliably, which is an ongoing research problem.

Another challenge that needs to be addressed is the cultural bias present in the models. Currently, InstructGPT is trained to follow instructions in English, making it biased towards the cultural values of English-speaking people. To overcome this bias, the researchers are conducting research to understand the differences and disagreements between labelers' preferences. This understanding will enable the conditioning of the models on the values of more specific populations, making them more inclusive and representative.

Moving away from language models, growth hacking is another area that requires careful monitoring and analysis. While metrics such as total users, daily active users (DAU), and monthly active users (MAU) might seem important, they are often considered vanity metrics. These metrics do not provide real insight into the growth rate or the quality of the users being acquired.