The physics of music instruments explained | Neil Gershenfeld and Lex Fridman

TL;DR

The computational capacity of a musical instrument lies largely in measuring a performer’s control with sufficient resolution and bandwidth, then mapping those controls into sound. Neil Gershenfeld’s work with Todd Machover and Yo-Yo Ma showed that instrumented playing connected to almost anything still sounded like Yo-Yo. Their cello-sensing research unexpectedly led to a $100 million-a-year auto-safety sensor business. Read on to trace the path from musical expression to airbag control.

Transcript

the path that led me to create CBA with colleagues was I was what's called a junior fellow at Harvard I was visiting MIT through Marvin because I was interested in the physics and musical instruments I this will be another slight aggression I uh and Cornell I would study Physics and and then I would cross the street and go to the music department w... Read More

Key Insights

• 🎼 The speaker's background in physics and music led to the exploration of the computational capacity of musical instruments.
• 🧑‍🔬 Collaborations with musicians and scientists contributed to the development of instruments that could extract data and translate it into computational environments.
• 🥺 Exploring instrument control led to unexpected collaborations with magicians and the development of applications such as Auto Safety sensors.

Questions & Answers

Q: What is the computational capacity of a musical instrument?

Gershenfeld initially considered operations per second, but Yo-Yo Ma’s playing shifted the focus to resolution and bandwidth. The central questions became how quickly a performer’s actions can be measured, how precisely the controls can be detected, and how those measurements can be mapped into sounds.

Q: What did Neil Gershenfeld learn from instrumenting Yo-Yo Ma’s cello playing?

The researchers found that when they instrumented everything Yo-Yo Ma did and connected those controls to almost anything, it still sounded like Yo-Yo. Gershenfeld concluded that the magic was in the performer’s control rather than in ineffable details of how the cello’s wood vibrated.

Q: How did Todd Machover and Yo-Yo Ma contribute to the musical-instrument research?

Marvin introduced Gershenfeld to Todd Machover at the Media Lab as Machover was preparing a project with Yo-Yo Ma. Their collaboration instrumented the cello to extract data from Yo-Yo’s performance and bring it into computational environments.

Q: Why was Yo-Yo Ma willing to replace his Stradivarius with the experimental system?

Yo-Yo Ma viewed the Stradivarius primarily as a high-performance controller that let him manipulate the interface between human and sound. He said the obstacle was logistics: the experimental setup required a rack of electronics, many cables, and graduate students, but he would use it once the technology became as invisible as the Stradivarius.

Q: How did cello sensing lead to auto-safety sensors?

Gershenfeld used local electromagnetic fields to detect how Yo-Yo Ma’s bow moved, but the system also sensed Ma’s nearby hand. Investigating that interference led to Josh Smith’s thesis on seeing in three dimensions with electric fields, a Penn and Teller magic trick, and eventually sensors that helped cars identify occupants for airbag control.

Q: What problem did the auto-safety sensing technology address?

Airbags were killing infants in rear-facing child seats, so cars needed to distinguish among a front-facing adult, a bag of groceries, and a rear-facing infant. The seat effectively needed to see in three dimensions to determine whether firing the airbag would save a life, was unnecessary, or could kill the occupant.

Q: What role did Penn and Teller play in developing the sensing technology?

Through Todd Machover and research scientist Joe Paradiso, the electric-field research led to a collaboration with Penn and Teller. They used the sensing fields in a Las Vegas magic trick about contacting Houdini, and that demonstration later helped connect the underlying technology to an automotive application.

Q: How commercially successful was the auto-safety application?

After the sensing system was taken to an auto show, the car companies asked when they could buy it. Gershenfeld says it became a $100 million-a-year business producing sensors and developed into a leading auto-safety sensor, although it received little publicity because the technology was hidden inside cars.

Summary & Key Takeaways

• The speaker's interest in the intersection of physics and music led them to discover David Borden, the first electronic musician, which sparked their curiosity about the computational capacity of musical instruments.

• Collaboration with musician Yo-Yo Ma and scientist Todd Mackover led to the development of instruments that could extract data and translate it into computational environments.

• The research on instrument control and resolution of detecting controls led to a collaboration with Penn and Teller, which ultimately resulted in the creation of Auto Safety sensors.