Lecture 9.2: Haim Sompolinksy - Sensory Representations in Deep Networks

TL;DR

This analysis discusses the capacity and limitations of deep cortex-like architectures and explores the transformation of sensory information through multiple stages.

Transcript

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high-quality educational resources for free. To make a donation or view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu. HAIM SOMPOLINSKY: My topic today is discussing sensory rep... Read More

Key Insights

• 🥺 Random projections in deep cortical architectures amplify noise and introduce excess correlations, leading to poor performance with noisy inputs.
• 🆘 Unsupervised learning can help suppress noise and reduce correlations in the representation, improving performance.
• ⛔ The size of the cortical layer has a limit to its impact on performance, beyond which further expansion provides no additional benefit.

Questions & Answers

Q: How does the expansion of sensory representations in deep cortical architectures affect noise and correlations?

The expansion of representations through random projections amplifies noise and introduces excess correlations between centers of clusters. This can lead to poorer performance when dealing with noisy inputs.

Q: Can unsupervised learning improve the performance of deep networks with random projections?

Yes, unsupervised learning can help suppress noise and reduce correlations in the representation. By associating randomly chosen representations with actual cluster centers, the noise can be quenched and correlations reduced.

Q: Does the size of the cortical layer in deep networks affect performance?

Increasing the size of the cortical layer in deep networks may lead to a better performance initially. However, beyond a certain point, the performance saturates, and further expansion provides no additional benefit.

Q: How can top-down information and recurrent connections be incorporated in deep networks?

By incorporating contextual knowledge through mixed representations and utilizing recurrent connections, the performance of deep networks can be significantly improved. This approach allows for selective amplification of specific states in the distributed representation.

Summary & Key Takeaways

• The analysis focuses on the transformation of sensory information in deep cortical architectures, with examples from various sensory systems such as vision and audition.

• It examines the effect of random projections on noise and correlations in the representation and highlights the limitations of random projections in dealing with noisy inputs.

• The article also explores the inclusion of recurrent connections and top-down information in deep networks to improve performance.