McLovin
8 HN mentions · mentioned in 6 months, 2017–2020
0:32 · 62K views · Mar 4, 2017
A short amateur clip of a Tesla Model S on Autopilot failing to follow a lane shift in a construction zone and hitting a concrete barrier. The uploader uses it to argue against driverless trucks.
Cited as a concrete example of Autopilot's limits whenever threads debate self-driving safety and Tesla's claims.
8 comments from Hacker News
Tesla's system demands more from drivers than manual driving. The driver has to detect automation failures after they occur, but before a crash. This requires faster reaction time than merely avoiding obstacles. Here's a good example - a Tesla on autopilot crashing into a temporary road barrier which required a lane change.[1] This is a view from the dashcam of the vehicle behind the Tesla. At 00:21, things look normal. At 00:22, the Tesla should just be starting to turn to follow the lane and avoid the barrier, but it isn't. By 00:23, it's hit the wall. By the time the driver could have detected that failure, it was too late. Big, solid, obvious orange obstacle. Freeway on a clear day. Tesla's system didn't detect it. By the time it was clear that the driver needed to take over, it was too late. This is why, as the head of Google's self driving effort once said, partial self driving "assistance" is inherently unsafe. Lane following assistance without good automatic braking kills. This is the Tesla self-crashing car in action. Tesla fails at the basic task of self-driving - not hitting obstacles. If it doesn't look like the rear end of a car or truck, it gets hit. So far, one street sweeper, one fire truck, one disabled car, one crossing tractor trailer, and two freeway barriers have been hit. Those are the ones that got press attention. There are probably more incidents. Automatic driving the Waymo way seems to be working. Automatic driving the Tesla way leaves a trail of bl
That being said, Toyota's (and Subaru's) approach is the smart way forward - add sensors and capabilities that augment the human, but leave the human always responsible and in control. In order for a crash to happen, the human AND the machine BOTH have to miss it. That's the approach we know won't work. That was Tesla's first-round autopilot, the self-crashing car. The one that would happily run into obstacles protruding into a lane.[1][2][3] There are two big flaws with that approach. First, if it's good enough that people can tune out, they will tune out. Second, if the driver waits until the self-driving system has clearly made a mistake before taking over, they will be slower at reacting than if they were driving, and may be too late. See especially [3]. Volvo's CEO takes the position that if one of their cars crashes in self-driving mode, it's Volvo's fault. Urmson, while at Google, pointed out the first problem coming up even with their test drivers. [1] [LINK] [2] [LINK] [3] this video
I agree crypto is actually hard. But there are lots of domains where details matter and small errors have big consequences. Vehicles, medicine, and manufacturing are all places where errors have killed people. Obviously severity depends on the situation, but generally speaking I'd rather my encryption get broken than die. Software defects in the Therac-25 radiation therapy machine killed several people [1]. Software defects in Toyota's acceleration killed 37 people [2]. "We've demonstrated how as little as a single bit flip can cause the driver to lose control of the engine speed in real cars due to software malfunction that is not reliably detected by any fail-safe," Michael Barr, CTO and co-founder of Barr Group, told us in an exclusive interview. Barr served as an expert witness in this case." [3] Self-driving autopilot has got to be pretty hard. [4] [1] [LINK] [2] [LINK] [2] [LINK] [4] this video
I'll be shocked if it's not next year at the latest that having having a human intervene will decrease safety (22:32) For certain interpretations of "intervene", that's true now. See this video of a Tesla hitting a construction barrier.[1] This is a dashcam view from a car behind the Tesla. At 00:21, it's too early for the vehicle to start turning, and at 00:22, the vehicle has already hit the barrier. The driver has to detect the failure and then take over. Much of the driver's allowed reaction time is taken up by having to wait until it's almost too late. Control handoff inherently implies a slower reaction time than manual driving. (Not noticing the lane change warning and orange construction 45MPH speed limit sign back at 00:05 didn't help, either.) This is the problem with shared control, and why Waymo rejected it early on. [1] this video
Here's autopilot swerving into oncoming traffic. [LINK] Here's autopilot following a lane straight into a concrete barrier. this video You can't assume that autopilot won't screw up lane following and swerve into a large obstacle. In those situations it's not as simple as making sure the lane ahead of you is clear, you might only have a split second warning between Autopilot going into "casual murder mode" and a collision.