Learning by Doing: Building and Breaking a Machine Learning System -- Johann Rehberger | Summary and Q&A

TL;DR

In this talk, the speaker shares their machine learning journey, demonstrating the process of building, testing, and attacking a machine learning system.

Key Insights

• 🤖 Machine learning is a fascinating field that can be connected to offensive security and red teaming.
• 🔎 Resources like Andrew Ng's machine learning course and TensorFlow in Practice are helpful for learning and understanding machine learning concepts.
• 🏆 Participating in machine learning security evasion competitions can be a great learning experience and may yield surprising results.
• 📊 Machine learning involves steps like getting data, preprocessing, defining the model, training, and deployment.
• 🚀 Adversarial training can help make machine learning systems more resilient against adversarial attacks.
• ⚠️ It is important to consider the security vulnerabilities and threats when building machine learning systems.
• ❓ The trustworthiness and integrity of machine learning models should be carefully evaluated to ensure their reliability.
• 🔐 Protecting machine learning models from backdoors and attacks, such as through hash validation or rate limiting, is crucial.
• 🌟 Generative adversarial networks (GANs) can be used to create fake images and test the robustness of machine learning models.
• ⚙️ Deploying machine learning systems requires considering security aspects like SSH keys and agents to prevent unauthorized access.
• ⚠️ Building trust and addressing ethical considerations in machine learning is essential for responsible and beneficial use of the technology.

Transcript

Read and summarize the transcript of this video on Glasp Reader (beta).

Questions & Answers

Q: What is the main objective of the speaker's talk?

The speaker's main objective is to share their machine learning journey, from building a system to testing and attacking it, in order to demonstrate the vulnerabilities and considerations in machine learning systems.

Q: How does the speaker suggest building resilience against adversarial attacks in machine learning systems?

The speaker suggests using adversarial training, where the system is trained with adversarial examples to improve its resilience. They also recommend implementing rate limiting, interpreting results differently, and continuously improving the model's accuracy to enhance the system's defense against attacks.

Q: What is the purpose of the image rescaling attack discussed by the speaker?

The purpose of the image rescaling attack is to demonstrate how resizing an image can fundamentally change its content, potentially leading to misclassification by the machine learning model. This highlights the need for caution when processing and resizing images in machine learning systems.

Q: How does the speaker propose mitigating the risk of backdooring in machine learning models?

The speaker suggests implementing hash validation to verify the integrity of the model file, as any modifications to the file can be detected. They also recommend conducting outside client checks to ensure consistent predictions and monitoring for repudiation attempts.

Summary & Key Takeaways

• The speaker introduces their experience in offensive security and their journey into machine learning.

• They discuss the resources they used to learn about machine learning and their participation in a machine learning security evasion competition.

• The speaker presents their machine learning system, Husky Eye, which allows users to upload pictures of huskies and receives predictions on whether there is a husky in the image. They then explore various attacks on the system, including adversarial examples, image rescaling, and backdooring.