The end of software is coming | Peter Wang and Lex Fridman

TL;DR

As software evolves, the concept of correctness is changing, with cybernetic systems becoming more prominent, raising questions about ethics, governance, and human involvement.

Transcript

given the semantics of the real world and the messiness of that what does the word correctness mean when you're talking about code there's a lot of dimensions to correctness historically and this is one of the reasons i say that we're coming to the end of the era of software because for the last 40 years or so software correctness was really define... Read More

Key Insights

• 🔠 Software correctness is evolving from merely functional correctness to considering input values and performance requirements.
• 😮 The rise of cybernetic systems signifies the end of the era of traditional software systems.
• 💁 Cybernetic systems operate without human intervention and make independent decisions to gather more information.
• ❓ Concerns arise from the unintended consequences and unpredictability of cybernetic systems.
• 💢 Ethics, governance, and questions of correctness become more critical in the cybernetic era.
• 🏛️ The collapse or dramatic effect of cybernetic systems may vary based on economic class and geographic location.
• 🏛️ The impact of algorithms, whether in social media or robotics, may affect certain classes more than others.

Questions & Answers

Q: What is meant by "correctness" in the context of software?

Historically, software correctness referred to functional correctness, ensuring that a function produced the correct outputs given certain inputs. However, with modern data-driven systems, correctness now includes considering input values, performance requirements, and the ability to return predictions within specified time constraints.

Q: What defines a cybernetic system?

A cybernetic system is one in which software autonomously closes the observe-orient-decide-act loop. It operates without human intervention, making decisions to gather more information. The absence of humans in the loop distinguishes cybernetic systems from traditional software systems.

Q: Can you give an example of a cybernetic system?

Yes, a prime example of a cybernetic system is the use of automated killer drones without human input. When drones are programmed to identify and target specific vehicles or objects without human intervention, it exemplifies a cybernetic system in action.

Q: How does the absence of humans in cybernetic systems lead to concerns?

The less human involvement in a cybernetic system, the more potential there is for unintended consequences. These systems can produce outcomes that even their designers couldn't have predicted. This lack of human oversight raises questions about ethics, governance, and the potential risks associated with such systems.

Summary & Key Takeaways

• Software correctness has traditionally focused on functional correctness, but with the rise of machine learning and data-driven systems, the concept of correctness now includes considering the values of inputs and meeting performance requirements.

• Cybernetic systems, where software independently closes the observe-orient-decide-act loop, are becoming more significant, marking the end of the era of traditional software systems and introducing new challenges related to ethics and governance.

• The absence of humans in the loop characterizes cybernetic systems, leading to concerns about unintended consequences and the ability to predict the outcomes of such systems.