comma ai | Adeeb Shihadeh | How We Test openpilot | openpilot testing | COMMA_CON talks

TL;DR

Open Pilot uses automated regression testing, hardware in the loop tests, offline anomaly detection, and online checks to ensure quality and reduce errors in its software.

Transcript

so i'm indeed i work on open pilot and i'm going to talk about uh mostly testing but uh as i've said there's going to be a couple fun announcements at the end so we're going to talk about quality and how we don't break stuff because uh opel is getting to be a big project and we support two hardware platforms now with two different operating systems... Read More

Key Insights

• 🔸 Open Pilot has a small team but supports a wide range of car models and hardware platforms.
• ✅ The testing process for Open Pilot involves automated checks, hardware in the loop tests, offline anomaly detection, and online checks.
• ⌛ By investing time in testing and quality assurance, Open Pilot has improved the reliability and performance of its devices.
• 😒 Open Pilot actively involves the community in testing through the use of GitHub Actions for forked repositories.

Questions & Answers

Q: How does Open Pilot ensure software quality and reduce errors?

Open Pilot uses automated regression testing, hardware in the loop tests, offline anomaly detection, and online checks to catch errors and ensure software quality. These tests cover various aspects of the software and hardware compatibility.

Q: How does Open Pilot handle testing for different car models and hardware platforms?

Open Pilot has a small team but supports 132 car models from various brands. They rely on contributors to provide testing data for different car models. The team also utilizes hardware in the loop tests to check compatibility with different hardware platforms.

Q: How does Open Pilot use anomaly detection in testing?

Open Pilot uses offline anomaly detection to analyze data logs from users and identify any unexpected errors or issues. They have a pipeline stage called "find interesting" that looks for anomalies in the incoming data and helps identify potential problems.

Q: How does Open Pilot involve the community in testing?

Open Pilot allows popular fork maintainers to use their testing infrastructure by enabling GitHub Actions in their forked repositories. This allows them to run a large portion of Open Pilot's testing suite on their own supported car models.

Summary & Key Takeaways

• Open Pilot is a large project that supports two hardware platforms (Android and Ubuntu) and releases monthly updates.

• The testing goals for Open Pilot are to enforce quality, reduce time spent in cars, and efficiently scale a small team.

• Open Pilot uses automated regression testing, hardware in the loop tests, offline checks, and online checks to ensure quality.