How to Use Machine Learning to Fight Online Abuse

TL;DR

Deep learning can strengthen fraud and abuse defenses because its performance improves as more data and computation become available. Effective systems generalize from known examples, adapt to evolving attacks, combine many signals nonlinearly, and help human reviewers focus on risky content, but teams must account for non-stationary threats that attackers continually modify.

Transcript

Bonjour. My name is Elie Bursztein. I work at Google, where I lead the anti-abuse research team. Today, I'm going to tell you how to successfully harness AI for fraud and abuse purposes. Uh, before I get started, uh, I would like a show of hand. How many of you read The New York Times? That's it? Yeah. The New York Times at The New York Times websi... Read More

Key Insights

• Conventional abuse defenses are failing because user expectations, content volume, content richness, and attacker sophistication have all increased. Defenses created several years earlier cannot reliably address the larger attack surface or the rapidly evolving behavior of adversaries.
• Machine learning is useful for ill-defined abuse concepts because data generalization allows a classifier to learn from previous examples and apply those patterns to unseen cases. The talk uses hate as an example of a concept that is difficult to define with rigid rules.
• Temporal extrapolation helps defenses recognize new attacks as variations of earlier attacks. This capability matters because fraud and abuse are non-stationary problems, meaning attackers continually alter their techniques instead of presenting an unchanging target like ordinary object recognition.
• Data maximization works by combining many input signals in nonlinear ways. According to the presentation, humans cannot practically perform this analysis across millions of signals, and traditional systems are also unable to combine the available information in the same manner.
• Deep learning improves as more data and computation are supplied. This scaling property changes the defender's relationship with expanding data because increasing volumes can improve detection performance instead of serving only as an additional operational burden.
• Perspective API helps human moderators focus on comments that may be offensive. It processes the 11,000 comments submitted daily to The New York Times, enabling the review team to scale and allowing comments to reopen across more articles.
• Gmail uses deep learning as one component of its defenses against spam, phishing, and malware. The presented breakdown attributes an additional 3.5 percent of coverage to deep learning, which catches substantial amounts of spam and phishing attacks.
• Deep learning tools are accessible beyond large organizations because free, mature frameworks and cloud APIs are available. The presentation identifies TensorFlow and Keras and states that teams can begin quickly or train an image classifier with a few lines of code and several hours.

Questions & Answers

Q: Why are conventional online abuse defenses failing?

Conventional defenses are failing for three stated reasons. Users now expect systems to prevent even a single offensive image, troll comment, or spam email. Online services must process dramatically more content, including richer formats that expand the attack surface. Attackers have also learned alongside defenders, making their methods more sophisticated and reducing the effectiveness of systems built several years earlier.

Q: How does machine learning detect new forms of abuse?

Machine learning detects new forms of abuse through data generalization and temporal extrapolation. A classifier learns patterns from earlier examples and applies them to unseen cases, even when the underlying concept, such as hate, is difficult to define precisely. It can also recognize new attacks as modified versions of earlier ones, helping defenses respond as adversaries change their techniques over time.

Q: Why is deep learning valuable for anti-abuse systems?

Deep learning is valuable because it can scale with both data and computation. As an organization supplies more of each, the algorithms can become better. For abuse defenders, this reverses the usual concern that expanding data creates only more problems. The growing volume of available information can instead support stronger defenses that keep pace with user expectations and evolving attacks.

Q: How does Perspective API support comment moderation?

Perspective API processes the 11,000 comments submitted to The New York Times each day and helps the editorial review team concentrate on comments that may be offensive. This prioritization lets human reviewers scale their work more effectively. The system helped the publication reopen comments and make them available on more articles, and it was also adopted by Wikipedia and The Guardian.

Q: How does Gmail use deep learning against malicious email?

Gmail scans billions of messages and attempts to remove spam, phishing, and malware so inboxes remain safe. Its defense combines an older generation of systems, reputation mechanisms, and deep learning. The presentation attributes an additional 3.5 percent of coverage to deep learning and says this classifier catches substantial amounts of spam and phishing, helping Gmail stay ahead of spammers.

Q: What is data maximization in machine learning?

Data maximization is the ability to take available input signals and combine them in a nonlinear manner. This is especially useful for anti-abuse work because systems may have millions of signals to consider. The presentation argues that humans cannot practically combine information at that scale and that traditional systems cannot perform the same kind of nonlinear integration, giving machine learning an important advantage.

Q: Can smaller teams start using deep learning for abuse detection?

Deep learning is presented as accessible rather than limited to Google or other large companies. Free, mature frameworks are available, including TensorFlow and Keras, which is described as a wrapper that makes the process easier. Cloud APIs also let teams begin within minutes, while available online tools can support training an image classifier with a few lines of code and several hours.

Q: Why is abuse detection a non-stationary problem?

Abuse detection is non-stationary because attackers deliberately change their behavior and techniques. A classifier recognizing cats deals with a comparatively stable subject, since cats are expected to look similar far into the future. Fraud and abuse classifiers face a moving target instead. Their training and defenses must therefore account for attacks that evolve after earlier patterns have been identified.

Summary & Key Takeaways

• Conventional abuse defenses are struggling because user expectations have risen, online content has expanded in volume and richness, and attackers have become more sophisticated. Even a single offensive image, troll comment, or spam email can make users feel unprotected, while service providers must evaluate far more content and a broader attack surface.

• Machine learning helps anti-abuse systems through data generalization, temporal extrapolation, and data maximization. These capabilities let classifiers extend lessons from known examples to new cases, recognize variations of earlier attacks, and combine numerous input signals nonlinearly. Deep learning adds the ability to improve as organizations supply more data and computation.

• Practical examples include Perspective API, which processes 11,000 daily comments submitted to The New York Times and helps reviewers prioritize potentially offensive material. Gmail also uses deep learning alongside older defenses and reputation systems, gaining 3.5 percent more coverage and catching substantial amounts of spam and phishing attempts.