Matroid
4 HN recommenders · recommended in 12 months, 2020–2022
30:09 · 374K views · Apr 20, 2020
Andrej Karpathy, then Tesla's head of AI, presents at ScaledML 2020 on how Tesla builds full self-driving using vision-based neural nets and fleet data collection. He covers training custom detectors and neural nets gradually replacing traditional code.
HN repeatedly points to it as the best public technical overview of Tesla's FSD approach, used both to defend and to critique Tesla's progress.
16 comments from Hacker News
You can learn about their systems by watching talks by Andrej Karpathy. As a robotics engineer interested in vision, their architecture is inspiring. This talk [1] is a good overview but each talk he gives is a little different so search for more if you want to know as much as possible. But the big thing is that their autonomy computer can be programmed to look for odd scenarios and send them back home. Tesla uses their fleet of hundreds of thousands of cars to collect edge cases like this, and then they have a kind of compartmentalized neural network system that breaks apart disparate tasks. With their collected examples they can create unit tests to ensure that the moon stops activating the stoplight detector. Once trained, the unit tests presumably help ensure they don't end up with future regressions. So basically every time you see a Tesla do a weird thing, there is a good chance it will stop doing it soon enough. At least if it's hitting hacker news. [1] this video
Removing the cameras may be a good idea, but note that their extensive fleet of camera-equipped vehicles is an important part of their potentially VERY lucrative self driving software development. EDIT: I highly recommend watching head of AI at Tesla Andrej Karpathy's talk on their self driving architecture. "The Fleet" as a source of data is a vital functional part of their self driving software development. They may be able to calculate that including cameras and a computer on a car that will ship in high volumes would increase sales of their self driving option across all Tesla vehicles (including that low cost model) enough to justify the inclusion of those parts. this video
If you actually believe that Tesla will fail to deliver full self-driving in the coming years, I have two questions. 1. have you watched this entire technical presentation made by Andrej Karpathy, Senior Director of AI at Tesla? this video 2. if you understand what you've seen in that video, why do you think Tesla will fail?
Here's the most recent talk by Karpathy about this topic: this video (there are more if you google for "karpathy talk") Here's the summary via 2 examples. 1. Example of gathering data that needs further labeling To implement neural network (NN) to recognize stop signs they program the cars to recognize things that look like stop signs and deploy that to the fleet of over 600 thousand cars on the road. The cars send those "might or might not be a stop sign" images back to tesla and they get manually labelled and added to a training set. They gather enough images and the "recognize stop signs" feature is done. Apply the same logic to other recognition tasks: recognize cars, pedestrians, animals, speed signs, traffic lights etc. Yes, labeling is expensive but everyone else (including Waymo) has the same cost. But Waymo has even bigger cost of driving around to collect those images. Tesla makes gross profit on every car they sell and they got 600 thousand people driving for them for free. Waymo has 600 cars, each of them reportedly costing over $200k and they have to pay drivers at least minimum wage for each hour of driving. That's why Google is reportedly is spending $1 billion a year on Waymo and why they need $3 billion of additional investment to keep going. 2. Example of gathering data that doesn't need labeling Consider implementing NN to recognize cut in i.e. other cars entering your lane in front of you. They deploy a first version trained on small sample, running in shadow
Better yet, let's get rid of cars altogether from city streets. This is coming along very nicely... [LINK] Autonomous driving is just around the corner, too, which means that there is no need to waste space on parking structures, either. Here's a Tesla engineer that gave a great technical talk demonstrating their progress as of early this year: this video EDIT: yes, there are clear reasons why having an three-dimensional underground structure will absolutely affect the social dynamics above-ground in a city, yet it remains true that it is possible to banish cars from downtown cores in the near future (i.e., say by 2030 or so), given the present shift to electrification and autonomous driving. These issues are worthy of debate, and I'm just trying to shift the conversation to what's current. I'm not trying to rain on anyone's parade! I love the goal of city streets for people, as exemplified by the beautiful laneways in Melbourne (11 minutes): [LINK]