This Neural Network Learned The Style of Famous Illustrators

TL;DR

GANILLA learns the styles of famous children’s-book illustrators while preserving the content of the original image. Its skip connections retain content information deeper into the neural network, addressing CycleGAN’s heavy-handed transformations and DualGAN’s weaker style transfer. A 48-person user study favored GANILLA over previous techniques, and it can even render images in the style of Hayao Miyazaki. Read on to see how the approaches differ.

Transcript

Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér. In the last few years, we have seen a bunch of new AI-based techniques that were specialized in generating new and novel images. This is mainly done through learning-based techniques, typically a Generative Adversarial Network, a GAN in short, which is an architecture wh... Read More

Key Insights

• ⚾ AI-based techniques, like CycleGAN and DualGAN, have improved image style transfer in recent years.
• 🖤 CycleGAN excels in style transfer but sacrifices content preservation, while DualGAN focuses on content preservation but lacks strong style transfer.
• 😒 GANILLA combines the advantages of both techniques with the use of skip connections, achieving impressive results in preserving content and transferring style.
• 👤 User studies have favored GANILLA over previous techniques.
• ❓ GANILLA can accurately reproduce distinct artistic styles, including the style of Hayao Miyazaki.

Questions & Answers

Q: How does GANILLA preserve image content while transferring an illustrator’s style?

GANILLA uses skip connections that help preserve content information as processing travels deeper into the neural network. This enables it to keep the source image’s content intact while transferring the selected illustrator’s style prominently.

Q: What is GANILLA designed to do?

GANILLA is designed to apply the styles of famous children’s-book illustrators to photographs. Its goal is to preserve the photograph’s content while reproducing a distinct artistic style.

Q: How do GANs generate new images?

A Generative Adversarial Network has a generator that creates images and a discriminator that learns to distinguish real photos from generated ones. The two networks learn and improve together, producing progressively better images.

Q: What is the main weakness of CycleGAN for this type of style transfer?

CycleGAN transfers style strongly, but it can be too heavy-handed. In the illustrated examples, the style changed completely while the original content was not left intact.

Q: How does DualGAN perform image-to-image translation?

DualGAN uses two GANs that learn opposite translations, such as day to night and night to day. It preserves image content well and makes the process efficient, but the transferred style may not appear prominently enough.

Q: How does GANILLA compare with CycleGAN and DualGAN?

CycleGAN is strong at transferring style but weaker at preserving content, while DualGAN preserves content but may add too little style. GANILLA combines both desired qualities by keeping the content intact and transferring the style clearly.

Q: Did users prefer GANILLA over earlier techniques?

Yes. The authors conducted a user study with 48 people, and the participants favored GANILLA over previous techniques.

Q: Can GANILLA reproduce the style of Hayao Miyazaki?

Yes, the presented results include GANILLA drawing in the style of Hayao Miyazaki. The transcript describes such distinct artistic styles as difficult even for humans to reproduce.

Summary & Key Takeaways

• AI-based techniques, such as CycleGAN and DualGAN, have been developed for image style transfer.

• CycleGAN excels in transferring style but compromises the content, while DualGAN preserves the content but lacks in style transfer.

• GANILLA, a new technique, successfully combines content preservation and style transfer, thanks to the use of skip connections.