comma ai | Learning a Driving Simulator | Yassine Yousfi | COMMA_CON talks | Research | ML

TL;DR

comma ai is learning a driving simulator that predicts future driving scenes from a few context frames, helping driving models experience deviations and errors during training. The machine-learning model produces rollouts of about one minute with learned physics, dynamics, lighting, and temporal consistency. Yassine Yousfi also explains why classical and small-offset simulators fall short, so read on for the approach, limitations, and next steps.

Transcript

foreign my name is Yasin I work in research comma and who's excited about some machine learning yes I heard that I haven't started presenting yet and I already have some questions so I guess I'll answer them later don't forget so today I'm going to talk to you about our progress in learning a driving simulator so it's been a few months of of this p... Read More

Key Insights

• 🪛 ML-driven simulator predicts driving scenes with minute-long rollouts, demonstrating impressive realism.
• 🌍 Need for simulation in training autonomous models to encounter real-world noise and errors not present in controlled environments.
• ❓ Challenges like flickering in image prediction addressed through improved tokenizer and smoothing decoder.
• 💨 ML research aims to revolutionize driving simulation, paving the way for scalable and realistic training for autonomous driving models.

Questions & Answers

Q: How is comma ai learning a driving simulator with machine learning?

The team gives a machine-learning model a few frames of context and asks it to predict the next frames of a driving scene. Its imagined rollouts last about a minute and show learned physics, dynamics, lighting, and temporal consistency.

Q: Why does comma ai need a driving simulator?

Without simulation, a driving model is not exposed to significant noise, deviations, or errors caused by its own decisions. A model that moves away from the lane center may keep postponing recovery until it drifts outside the lane.

Q: Why not use Unity, GTA 5, or another classical simulator for training?

Yassine Yousfi says classical simulators are useful for deterministic testing and manually constructed scenarios. For training, however, they must match the real-world distribution of driving data and require many scenarios to be hard-coded, which the team does not consider scalable.

Q: How does comma ai’s small offset simulator work?

The small offset simulator shifts an image slightly right, left, forward, or backward to represent small movements from the original position. Closed-loop training in this simulator produces driving models that recover and stop for stop signs and red lights.

Q: What is the cheating problem in the small offset simulator?

The simulator depends on highly accurate ground truth, height estimation, road-plane estimation, and pose estimation. Imperfections create artifacts that can reveal where the model came from or how it should drive, allowing it to learn from artifacts instead of driving normally.

Q: Why can’t the small offset simulator handle large deviations well?

It is designed for small positional shifts, and larger offsets make its generated images look unrealistic. This is especially important for longitudinal changes at highway speeds, where a difference of plus or minus 10 miles per hour creates a large distance.

Q: What qualities appear in comma ai’s predicted driving videos?

The predicted videos exhibit learned physics and driving dynamics while maintaining good temporal consistency over roughly one-minute rollouts. Night scenes also preserve convincing lighting, which the team says is important.

Q: Where did the title “Learning a Driving Simulator” come from?

Yassine Yousfi says he borrowed the presentation title from a 2016 paper named “Learning a Driving Simulator.” The paper was written by Edward and George.

Summary & Key Takeaways

• ML researcher presents progress in developing a driving simulator using machine learning to predict realistic driving scenarios.

• Simulator visually predicts minute-long driving scenes with impressive accuracy and fidelity.

• Focus on the need for simulation in training autonomous models to handle real-world noise and scenarios effectively.