"Unlocking the Power of Human Feedback and Long-Term Thinking"

"Unlocking the Power of Human Feedback and Long-Term Thinking"

Introduction:
In recent developments, Humanloop has partnered with Stability AI to build the first open-source InstructGPT. However, the challenges with language models trained through next word prediction have become evident, as they often produce inaccurate or offensive output. This article explores the potential of Reinforcement Learning from Human Feedback (RHLF) in improving the alignment and usability of language models. Additionally, it delves into the concept of long-term thinking and how humanity's ability to plan and strategize over extended timeframes can address pressing issues like the climate emergency.

The Power of Reinforcement Learning from Human Feedback:
Reinforcement Learning from Human Feedback (RHLF) has proven to be a valuable technique employed by organizations like OpenAI, DeepMind, and Anthropic. By incorporating human feedback, these models can better follow instructions and act as helpful assistants. The collaboration between Humanloop, Carper AI, and Scale aims to collect and apply human feedback data to enhance the underlying language model. This approach ensures that the model is fine-tuned to align with human values and produces more reliable and accurate results.

Long-Term Thinking and the Acorn Brain:
While humans have struggled to respond effectively to long-term crises, such as the climate emergency, we possess a unique cognitive ability known as the Acorn Brain. Located in the frontal lobe, the Acorn Brain allows us to think, plan, and strategize over extended timeframes. This remarkable capability, which is only two million years old, sets us apart from other species.

Evolutionary Explanations for Long-Term Thinking:
Several explanations shed light on how our brains evolved this capacity for long-term thinking. Firstly, the survival skill of "wayfinding" enabled our ancestors to navigate and orient themselves in physical space during hunting or foraging activities. This ability to plan and execute complex sequences of actions laid the foundation for future-oriented thinking.

Secondly, the "grandmother effect" highlights the importance of older post-reproductive females in reducing infant and child mortality. These individuals provide crucial childcare, knowledge, and support, contributing to the survival and well-being of the young. This intergenerational bond fosters relationships of trust and reciprocity, where help given in the present is expected to be returned in the future.

Lastly, our genius for toolmaking played a significant role in the development of long-term thinking. As our brains expanded, we gained the ability to plan and execute complex actions, such as making stone tools. This capacity for planning enabled us to engage in forward-looking activities with long time horizons, such as crop rotation and construction projects.