Tips N Tricks # 8: Using automatic mixed precision training with PyTorch 1.6

TL;DR

Learn how to use automatic mixed precision in PI Taj 1.6 for faster and more memory-efficient model training.

Transcript

hello everyone and welcome to this new short video in this one I'm going to show you how you can use automatic mixed precision from PI Taj natively so PI dot 1.6 is going to have native support for automatic expression in training and mixer engine helps you in many different ways one of the things is your model will occupy less memory you so you ca... Read More

Key Insights

• 🥳 PI Taj 1.6 natively supports automatic mixed precision, eliminating the need for third-party libraries like Nvidia's apex.
• 😒 Automatic mixed precision reduces memory consumption, allowing for the use of larger batch sizes and faster training.
• 😓 The implementation of automatic mixed precision in PI Taj 1.6 involves importing the necessary libraries, defining a gradient scaler, and using the auto-casting context.
• ❓ The GPU memory usage decreased by 2 gigabytes when using automatic mixed precision.
• ⌛ The training time was reduced from 32 minutes to 18 minutes with automatic mixed precision.
• ❓ Data parallelism can also be used in combination with automatic mixed precision to further optimize training.
• 💨 Automatic mixed precision is a recommended optimization for faster training, provided the GPU supports mixed precision training.

Questions & Answers

Q: What are the benefits of using automatic mixed precision in PI Taj 1.6?

Automatic mixed precision in PI Taj 1.6 reduces memory usage, allows for larger batch sizes, and enables faster model training.

Q: How can you use automatic mixed precision in PI Taj 1.6?

To use automatic mixed precision, you need to import the necessary libraries, define a gradient scaler, and use the auto-casting context for the forward and backward passes in your training function.

Q: What is the impact of using automatic mixed precision on training time?

Training time can be significantly reduced by using automatic mixed precision. In the example shown, the training time decreased from 32 minutes to 18 minutes.

Q: Which GPU architectures support mixed precision training?

Mixed precision training is supported on GPUs with Pascal architecture or newer.

Summary & Key Takeaways

• PI Taj 1.6 natively supports automatic mixed precision, which reduces memory usage, allows for larger batch sizes, and enables faster training.

• Using automatic mixed precision can be done by importing the required libraries, defining a gradient scaler, and using the auto-casting context for forward and backward passes.

• Automatic mixed precision training reduced training time from 32 minutes to 18 minutes and decreased GPU memory usage by 2 gigabytes.