When Is Attacking Machine Learning Illegal?

TL;DR

Legality depends on the attacker’s intent, conduct, and the surrounding context, not merely on whether a machine learning model was fooled. Existing criminal laws may address harmful manipulation, while prosecutors can examine motives, prior behavior, communications, and records to distinguish deliberate attacks from accidental changes or legitimate research.

Transcript

Good afternoon, everyone, and welcome back to the Law Track. Um, our next session is Am I Allowed to Subvert Machine Learning for Fun and Profit? Our moderator today is Ram Shankar Siva Kumar, data cowboy from Microsoft and Harvard. Um, before we get started, I just wanted to make a quick announcement if anyone is new to the Law Track. Um, the Law ... Read More

Key Insights

• Machine learning failures can result from sophisticated perturbations, basic image transformations, or ordinary-looking inputs. A slightly altered five became a seven, a cropped and rotated bird became an orangutan, and an unchanged red tube was identified as a hotdog.
• Adversarial perturbation works by testing whether a tiny change to each pixel moves an image toward a desired classification. Although every modification is small, changes across a million pixels can accumulate until the model produces the attacker’s chosen result.
• Direct access to a machine learning model is not always necessary for an evasion attack. An attacker can query a remotely hosted service, observe its responses, and use that information to develop inputs that fool the system.
• Evasion techniques can target more than image classifiers. The panel identifies spam classifiers and malware detectors as systems that may be fooled through repeated queries and observed responses, while an audio demonstration shows that speech can be embedded in music.
• The Computer Fraud and Abuse Act is only one possible part of the legal analysis. The panel advises against treating it as the exclusive remedy because prosecutors can examine the broader offense and consider other provisions of the United States criminal code.
• Criminal liability depends significantly on intent and conduct. The legal concepts discussed are mens rea, meaning an intent to commit a crime, and actus reus, meaning the presence of an act that can support prosecution.
• Intent cannot be inferred from a changed pixel alone. Investigators may examine the full context, including motivation, previous malicious behavior, communications with other people, and records that help establish whether manipulation was deliberate or accidental.
• Machine learning attacks create policy questions beyond cybercrime enforcement. The panel’s stated scope includes copyright, product liability, organizational impact, social impact, legal remedies, attacker risk, and whether terms of service have been updated to address these attacks.

Questions & Answers

Q: How do adversarial machine learning evasion attacks work?

An evasion attack can work by changing an input in very small increments. For an image, an attacker may test whether darkening or otherwise changing one pixel makes the model classify the image more like a desired category. Repeating that process across many pixels allows individually small modifications to accumulate until the classifier returns a different result.

Q: Can attackers fool a model without accessing it directly?

Yes. Direct access to the model is helpful for many attacks, but the panel states that it is not always required. When a model is hosted by a remote cloud service, an attacker can send queries, observe the returned responses, and use that feedback to construct inputs capable of fooling machine learning classifiers or detectors.

Q: What kinds of machine learning systems can evasion attacks target?

The discussion identifies image classifiers, spam classifiers, malware detectors, facial recognition systems, and systems used by major companies as possible targets or examples. It also demonstrates an audio attack in which speech can be embedded in music. These examples show that adversarial behavior is not limited to a single type of input or classification task.

Q: Why can tiny image changes cause a major classification error?

A model’s classification can shift when many individually small pixel changes point toward another category. The researcher describes testing pixels one by one and retaining changes that increase the desired classification. Because an image may contain a million pixels, the combined effect can transform the model’s output even when each separate adjustment remains very small.

Q: Does the Computer Fraud and Abuse Act govern every machine learning attack?

No. The legal perspective presented cautions against focusing too heavily on the Computer Fraud and Abuse Act. It is one law within the broader United States criminal code. A prosecutor may instead analyze the complete offense, determine the attacker’s intent and actions, and consider whatever criminal provisions best fit the particular manipulation and resulting harm.

Q: How can prosecutors distinguish malicious manipulation from an accidental change?

Prosecutors can examine the entire context rather than relying only on the altered input. Relevant evidence may include the person’s motivation, any history of malicious behavior, communications with others, and available records describing what the person said or intended. Together, those details can help establish whether a perturbation was purposeful, accidental, or otherwise innocent.

Q: What legal concepts matter when evaluating a machine learning attack?

The panel highlights mens rea and actus reus. Mens rea concerns whether the person intended to commit a crime, while actus reus concerns whether a relevant act occurred. A criminal prosecutor can break the incident into these elements, examine the surrounding evidence, and decide how the alleged offense should be charged under existing law.

Q: What policy issues do attacks on machine learning systems create?

Machine learning attacks create definitional and enforcement challenges across cybercrime, copyright, and product liability law. Organizations must consider what remedies may be available after an attack and whether their terms of service address adversarial conduct. Policy analysis must also consider the risks assumed by attackers and the wider effects on organizations and society.

Summary & Key Takeaways

• Machine learning systems can misclassify inputs after tiny pixel changes, simple cropping or rotation, or even without deliberate manipulation. Demonstrations included a five classified as seven, a bird classified as an orangutan, a red tube classified as a hotdog, and altered audio designed to produce a targeted transcription.

• Evasion attacks work by repeatedly making small input changes that move a model toward an attacker’s chosen classification. Direct access to the model can help, but it is not always required. Attackers may instead query a remotely hosted service, observe its responses, and use that feedback to fool classifiers and detectors.

• The legal analysis should examine the complete conduct rather than focus exclusively on the Computer Fraud and Abuse Act. Prosecutors may consider intent, the prohibited act, motivation, prior malicious behavior, communications, and records. The panel also identifies broader questions involving cybercrime, copyright, product liability, terms of service, organizations, and society.