How Do Adversarial Examples Fool Neural Networks?

TL;DR

Adversarial examples fool neural networks by adding carefully chosen perturbations that may be nearly invisible or unobtrusive to people but push a model toward an incorrect output. Attackers can calculate these changes efficiently with derivatives, transfer attacks from locally trained copies to production models, manipulate image and audio classifiers, and potentially extract private training data through exposed model APIs.

Transcript

So I'm gonna spend this session talking instead about how you can use machine learning to make security and privacy better, about how we can actually attack the machine learning algorithms themselves to do things like make them produce incorrect outputs or extract training data out of them once they've been trained. So this is sort of a different k... Read More

Key Insights

• Neural networks can perform at human levels or above across many object categories while remaining vulnerable to carefully designed inputs. Strong performance on ordinary test images does not establish that a classifier will behave reliably when an attacker deliberately manipulates what the model receives.
• Adversarial examples are inputs modified to produce incorrect model outputs while preserving their apparent meaning to people. A golden retriever image can remain visually recognizable to humans yet be classified as guacamole after a change that makes the two versions appear essentially indistinguishable.
• Physical adversarial markings can create safety-relevant classification errors. Researchers added black and white dots to a stop sign, causing a state-of-the-art machine learning system to identify it as a forty-five-mile-an-hour sign despite its red color, octagonal shape, and visible stop text.
• Adversarial attacks are not limited to computer vision. A piece of audio perceived by people as music can be manipulated so that a machine learning system transcribes a complete sentence, demonstrating that the gap between human and model perception also affects speech recognition.
• Hidden audio commands can control smart devices without making nearby people aware of the instruction. The demonstrated attack caused a smartphone to browse to facebook.com, and the same general technique could embed an inaudible or disguised command within content such as a YouTube video.
• Gradient-based attacks work by calculating how changes to an input affect a model's loss. The derivative identifies a direction that maximizes classification error, allowing an attacker to add a small perturbation efficiently instead of testing possible changes individually through a slow trial-and-error process.
• Iterative perturbation can make adversarial generation more practical. Rather than applying the full calculated change in one step, an attacker can make several smaller changes, repeatedly moving the input toward a desired incorrect classification while keeping the resulting modification difficult for people to notice.
• Black-box attacks can work without direct access to a production model. An attacker can train a local substitute, generate an adversarial example against that copy, and rely on the attack transferring to the deployed classifier, potentially improving the result with a small number of queries.

Questions & Answers

Q: What is an adversarial example in machine learning?

An adversarial example is an input deliberately changed so that a machine learning model produces an incorrect output, even though the input still appears normal or retains its original meaning to people. The examples presented include nearly indistinguishable dog images receiving completely different labels and a visibly recognizable stop sign being classified as a forty-five-mile-an-hour sign.

Q: How are adversarial examples generated with gradients?

A gradient-based attack calculates the derivative of a model and its input with respect to a loss function. This derivative indicates the direction in which the input should move to maximize the model's error. The attacker adds a small amount of the resulting perturbation to the original input, often through repeated smaller steps, until the classification changes.

Q: Why are adversarial examples a security problem?

Adversarial examples are a security problem because systems expected to make trustworthy decisions can be manipulated into producing incorrect outputs. The stop-sign demonstration illustrates the risk clearly: a sign that people recognize from its color, shape, and text was classified as a forty-five-mile-an-hour sign after researchers added a few black and white markings.

Q: Can adversarial attacks work against speech recognition?

Adversarial attacks can affect speech recognition as well as image classification. The talk presents audio that sounds like music to human listeners but is transcribed by a state-of-the-art machine learning system as a complete sentence. This mismatch shows that attackers can manipulate what an audio model perceives without creating equivalent speech for human listeners.

Q: How can hidden audio commands control smart devices?

A concealed or disguised audio signal can contain a command that a smart device recognizes even when nearby people do not understand or notice it. In the demonstration, speakers played audio that caused a smartphone to browse to facebook.com. The talk notes that such commands could also be embedded in a YouTube video and triggered when someone watches it.

Q: Can attackers fool a model without accessing it directly?

Attackers can target a production model even when they lack direct access to its internal parameters. They can train a local copy that behaves similarly, generate an adversarial input against that substitute, and then submit the same input to the deployed service. The talk says these attacks generally transfer and can be improved with a small number of queries.

Q: What does the Google image classifier example demonstrate?

The production classifier correctly labeled an ordinary mountain image with two people as skiing and ski. Researchers then added a small perturbation that left the scene looking essentially unchanged to people, but the returned labels became dog and dog-like mammal. The example demonstrates that practical black-box services can inherit the same adversarial vulnerability as locally accessible models.

Q: How can exposed neural network APIs threaten privacy?

Exposing an API to a pretrained neural network can create a privacy risk because attackers may attempt to extract sensitive information from the data used to train it. The talk identifies private training-data extraction as a major concern alongside adversarial examples, so organizations must consider whether model outputs or API access could reveal information that should remain private.

Summary & Key Takeaways

• Neural networks can classify ordinary objects accurately yet fail dramatically on inputs modified by an attacker. Examples include a golden retriever labeled as guacamole and a stop sign identified as a forty-five-mile-an-hour sign after adding black and white markings, showing why model reliability becomes a security concern in physical settings.

• Adversarial examples are created by making small input changes that increase the model's error. Instead of testing pixels individually, an attacker can calculate the derivative of a loss function with respect to the input, identify the direction that maximizes misclassification, and apply the resulting perturbation once or through repeated smaller steps.

• The threat extends beyond images and direct model access. Modified audio can be transcribed as speech that people do not hear, allowing concealed commands to control smart devices. In black-box settings, attackers can train a local copy, attack that substitute, and often transfer the resulting adversarial input to a deployed production classifier.