How Does Automated Testing Improve Gameplay in Sea of Thieves?

TL;DR

Automated testing significantly enhances gameplay quality in Sea of Thieves by ensuring faster build verification, reducing manual testing efforts, and minimizing bug counts. Rare adopted automated tests to handle the complexity of their open world game, implementing a framework that combines unit tests, integration tests, and actor tests for effective coverage and efficiency.

Transcript

foreign I'm a software engineer from rare because we had such bad experiences with testing on our previous projects when we started work on our latest game sea of Thieves instead of relying on manual testing we decided to completely change our approach and use automated testing on every part of the code base including on gameplay features which are... Read More

Key Insights

• 🥺 Automated testing in game development can lead to faster build verification and reduced manual testing effort.
• 🏆 A mix of unit tests, integration tests, and actor tests can provide comprehensive test coverage while optimizing testing time and resources.
• 😤 It is important to consider team buy-in, invest time in building knowledge and robust infrastructure, and be pragmatic in test coverage and prioritization.

Questions & Answers

Q: What challenges did Rare face in their previous projects that led them to adopt automated testing for Sea of Thieves?

Rare had bad experiences with manual testing in their previous projects. With Sea of Thieves, they faced challenges such as an open world game with complex interactions and the need for quick releases and bug fixing.

Q: How did Rare optimize their testing process during development?

Rare created actor tests, which are similar to unit tests but treated Unreal Engine concepts as first-class dependencies. They merged related features in integration tests and used World Travel to reduce test initialization time. They also handled intermittent and consistently failing tests efficiently.

Q: What benefits did Rare gain from their automated testing approach?

Rare experienced reduced build verification time, a decrease in bugs, and lower manual testing efforts. They also saw improvements in build stability, enabling quicker releases and reducing the need for crunch.

Q: Did Rare prioritize automated testing over gameplay iteration?

No, Rare's approach was pragmatic. They separated prototype branches for gameplay iteration, allowing developers to work quickly without the burden of testing. They focused on test coverage during production to ensure build stability and reduced crunch.

Key Insights:

• Automated testing in game development can lead to faster build verification and reduced manual testing effort.
• A mix of unit tests, integration tests, and actor tests can provide comprehensive test coverage while optimizing testing time and resources.
• It is important to consider team buy-in, invest time in building knowledge and robust infrastructure, and be pragmatic in test coverage and prioritization.
• Automated testing can contribute to reduced crunch, improved build stability, and lower bug count in game development.

Summary & Key Takeaways

• Rare, a Microsoft first-party studio, adopted automated testing on their game Sea of Thieves to overcome challenges in manual testing and improve build quality.

• They implemented a test framework using unit tests and integration tests to cover different aspects of their gameplay features.

• By optimizing their testing process and using a mix of automated and manual testing, they achieved faster build verification, reduced manual testing team, and lower bug count.