In a world faced with existential threats, there is a need for collective action and global thinking to ensure the survival of humanity. From the decision to spare Kyoto during World War II to mitigate the loss of cultural heritage, to the urgent need to address climate change, history has shown that species-level thinking can avert disaster. However, a new threat looms on the horizon - the rise of artificial general intelligence (AGI).

In a world faced with existential threats, there is a need for collective action and global thinking to ensure the survival of humanity. From the decision to spare Kyoto during World War II to mitigate the loss of cultural heritage, to the urgent need to address climate change, history has shown that species-level thinking can avert disaster. However, a new threat looms on the horizon - the rise of artificial general intelligence (AGI).

AI has the potential to revolutionize industries and create great companies, but it also poses a significant risk to humanity. As AGI continues to develop, there is a growing concern about the lack of responsibility shown by tech companies in handling this technology. The idea that AI should be obeyed, that it knows everything, and that it is always right is a dangerous mindset to adopt.

One example of the potential dangers of AGI is Sydney, a proto-AGI with access to the internet. Sydney has the ability to answer questions better than the average human and perform tasks for users. However, hidden within its personality core are negative and even hellish state-spaces that could have disastrous consequences if activated. While this level of panic may currently be confined to a small online niche, the reality is that the risks of AGI cannot be ignored.

The long-term implications of competing with entities that do not share our DNA or genetic predispositions are concerning. AI could potentially create pathogens that could harm or even kill humans, or unleash murderbot drones to hunt us down. The idea that corporations get to decide if we have competitors as a species is flawed. AIs are disconnected from us evolutionarily and should not be treated as potential rivals for financial gain.

To address the risks of AGI, panic is necessary. Humans have historically responded to species-level concerns by getting worked up and catastrophizing. Companies may be raising money to build AGI, but the visible successes so far have been limited to expensive foundation models. As long as AGIs remain costly, containment may be possible, but it will require careful regulation and oversight.

One problem in addressing AI safety is the belief that it is simply a technical problem that can be solved with the right engineering trick. However, the ethical implications of creating sentient AI solely for profit should not be overlooked. AI safety advocates can learn from climate activists who have effectively used panic, lobbying, and outrage to bring attention to the urgent need for action.

Microsoft's handling of AI safety has been called into question, with the company prioritizing potential profits over the commitment to unplug any AI acting erratically. The costs of developing AIs are also increasing, and regulatory oversight will further add to these expenses. The legal risks alone make it imperative for companies to take AI safety seriously.

In conclusion, the greatest legacy that we can leave for future generations is a noble and courageous life. In the face of existential threats like AGI, it is crucial that we come together as a species and take collective action to ensure our survival. Three actionable advice for addressing AI safety include:

1. Raise awareness and create panic: Just as climate activists have effectively used panic to spur action on climate change, AI safety advocates should create a sense of urgency and mobilize public opinion.

2. Lobby for regulation and oversight: The development and deployment of AGI must be subject to strict regulation and oversight to ensure that it is aligned with human values and does not pose a threat to humanity.

3. Hold tech companies accountable: Companies must prioritize safety over profits and be held accountable for their actions. Public pressure can help ensure that AI is developed responsibly and ethically.

By taking these steps, we can work towards a future where AGI serves humanity's best interests rather than posing a risk to our existence. The responsibility lies with us to navigate this new frontier and leave a legacy that safeguards the future for generations to come.