Why Do We Add Losses Before Back Propagating in PyTorch

TL;DR

Actor and critic networks in an actor-critic architecture share lower layers, requiring the sum of losses before back propagation to ensure gradients are properly calculated and retained.

Transcript

hello everybody i just want to do a really quick video today so i got a question in the discord chat which if you are not on our discord you should be there are quite a few very smart individuals on there answering questions driving discussions it's a great place to be check it out but the question is why uh why is it that we have to do the sum of ... Read More

Key Insights

• 🧑‍🏭 Actor and critic networks in an actor-critic architecture share lower layers, requiring loss combination before back propagation for proper gradient calculation.
• 🥺 PyTorch conventionally throws out gradients after each back propagation, leading to the need to retain the graph or combine losses.
• 👻 Separate architectures for actor and critic networks allow sequential back propagation without loss combination.
• 🐢 Retaining the graph during back propagation can slow down future calculations due to gradient accumulation.
• 😷 Discord chat is a valuable resource for asking questions and engaging in discussions on the topic.
• 🍵 PyTorch may handle back propagation differently compared to TensorFlow.
• 🧑‍🏭 Testing separate networks for actor and critic functions in PyTorch confirmed the ability to back propagate through separate graphs without loss combination.

Questions & Answers

Q: Why do we need to add the losses of the actor and critic networks before back propagation?

The actor and critic networks share lower layers, and the derivative of the actor with respect to the critic's parameters will be zero without adding the losses together. This ensures shared gradients in the lower layers are properly calculated.

Q: Can we calculate gradients in any order in PyTorch?

Yes, it is valid to calculate gradients in any order in PyTorch. However, PyTorch conventionally throws out gradients once one loss function is back propagated, leading to the need to combine losses in actor-critic networks.

Q: What happens if we don't retain the graph when back propagating in PyTorch?

If the retain_graph parameter is not specified, PyTorch throws out gradients associated with the second network when calculating back propagation for the first network. This can result in incorrect gradient calculation.

Q: Can separate actor and critic networks be back propagated sequentially without combining losses?

Yes, if the actor and critic networks have separate architectures and do not share input layers, it is valid to back propagate through them sequentially without combining losses.

Summary & Key Takeaways

• Actor and critic networks in actor-critic architectures share lower layers, serving as the interface with the environment, before branching out into separate outputs.

• Back propagation in PyTorch conventionally requires adding the losses of the actor and critic networks together to ensure proper gradient calculation.

• If actor and critic networks have separate architectures, it is valid to back propagate through them sequentially without combining losses.