How Do You Diagnose Bias and Variance? (C2W1L02)

TL;DR

Diagnose bias and variance by comparing training-set error with development-set error: high training error indicates high bias, while a large increase from training to development error indicates high variance. For example, 1% training error and 11% development error suggests high variance. This analysis assumes Bayes error is small, so read on to understand each error pattern and its limitations.

Transcript

I've noticed that almost all the really good machine learning practitioners tend to have a very sophisticated understanding of buyers invariant but in various one of those concepts as easy to learn but difficult to master even if you think you've seen the basic content advisor variants is often more nuanced to attend you'd expect in the deep learni... Read More

Key Insights

• 🎰 Machine learning practitioners should have a sophisticated understanding of bias and variance.
• ✋ High bias and high variance have distinct effects on model performance.
• 😫 Assessing the training set error and development set error can help diagnose bias and variance issues.

Questions & Answers

Q: What are bias and variance in machine learning?

High bias means an algorithm is underfitting the training data. High variance means it fits the training set much better than the development set and therefore does not generalize well.

Q: How can bias and variance be diagnosed?

Compare the training-set error with the development-set error. Training error indicates how well the algorithm fits the training data, while the increase in error on the development set indicates how well it generalizes.

Q: What does 1% training error and 11% development error indicate?

This pattern indicates high variance. The algorithm performs well on the training set but relatively poorly on the development set, suggesting that it overfit the training data.

Q: What does 15% training error and 16% development error indicate?

Assuming human-level or optimal error is nearly zero, this indicates high bias. The algorithm performs poorly even on its training set, although the one-percentage-point increase on the development set suggests that it generalizes at a reasonable level.

Q: Can an algorithm have both high bias and high variance?

Yes. A training error of 15% and a development error of 30% indicates high bias because training performance is poor and high variance because development performance is substantially worse.

Q: What indicates low bias and low variance?

A training error of 0.5% and a development error of 1% indicates low bias and low variance in the example. The algorithm fits the training data well and experiences only a small increase in error on the development set.

Q: How do underfitting and overfitting relate to model complexity?

A straight-line or logistic-regression fit may be too simple for the data and exhibit high bias or underfitting. An incredibly complex classifier, such as a large neural network, may fit the training data perfectly yet exhibit high variance or overfitting, while medium complexity may provide a more reasonable fit.

Q: Why does Bayes error matter when diagnosing bias and variance?

The diagnosis assumes that Bayes error, or optimal error, is nearly zero. If Bayes error were 15%, then a 15% training error could be perfectly reasonable rather than evidence of high bias, as may happen when images are too blurry for any system to classify well.

Summary & Key Takeaways

• Bias and variance are concepts that are easy to learn but difficult to master in machine learning.

• High bias refers to underfitting the data, while high variance refers to overfitting the data.

• Evaluating the training set error and development set error can help diagnose whether the algorithm has high bias or high variance.