Sizeof Dev and Test Sets (C3W1L06)

TL;DR

Set up larger training sets and smaller development and test sets in the deep learning era, and adjust the size of the test set based on the required confidence level.

Transcript

in the last video you saw how your death and test then should come from the same distribution but how long should they be these guidelines the house set up your dev intent says are changing in the deep learning era let's take a look at some best practices you might have heard of the rule of thumb in machine learning of taking all the data you have ... Read More

Key Insights

• 🥶 The old rule of thumb for training and test set splits in machine learning no longer applies in the deep learning era.
• 👻 Larger dataset sizes in modern machine learning allow for more data to be used for training.
• 😫 The test set should be set to be big enough to confidently evaluate the final performance of the system.
• 😫 For some applications, having only a training and development set might be sufficient, but having a separate test set is reassuring.
• 😫 Adjusting the size of the test set based on the required confidence level is recommended.
• 😫 It is important to distinguish between a test set and a development set, and not to iterate on the test set.

Questions & Answers

Q: What was the rule of thumb for setting up training and test sets in machine learning?

The old rule of thumb was to use a 70/30 split for training and test sets, or a 60/20/20 split for training, development, and test sets. This was reasonable for smaller dataset sizes.

Q: How has the trend changed in the deep learning era?

With larger dataset sizes, the trend is to use more data for training and less for development and testing. For example, with a million training examples, it might be reasonable to have 98% in the training set and 1% each in the development and test sets.

Q: How should the test set size be determined?

The test set should be big enough to evaluate the overall performance of the system. Depending on the application, having 10,000 or 100,000 examples might be enough, which could be much less than 20% of the dataset.

Q: Is it acceptable to not have a test set?

Not having a test set is not recommended, as it provides an unbiased estimate of how well the system is performing before it is deployed. However, if the dataset is very large and overfitting is not a major concern, having a training and development set might be sufficient.

Summary & Key Takeaways

• In the modern era of deep learning and larger dataset sizes, the old rule of thumb of a 70/30 split for training and test sets no longer applies.

• The trend is to use more data for training and less for development and testing, especially with very large datasets.

• The size of the test set should be big enough to evaluate the final performance of the system, but it can be much smaller than 20% of the overall dataset.