The Linux Foundation
1 HN recommender · recommended in 17 months, 2019–2024
32:02 · 6.9K views · Dec 20, 2018
Will Wilson explains autonomous testing: using deterministic simulation and fuzzing, building on FoundationDB's approach, to test software continuously. It sets out the ideas behind his later company Antithesis.
HN recommends it as an excellent talk on generalizing deterministic simulation testing, and Antithesis hiring posts link it to explain the work.
17 comments from Hacker News
I very much agree with this sentiment, but it is difficult to endorse this line of reasoning without offering a better alternative. As folks have pointed out in other comments, there are other values to tests in the broad sense that the author doesn’t address. That said, if you are interested in improving the state of testing, I would encourage you to listen to Will Wilson’s excellent talk on the subject of Autonomous Testing: this video Indeed, FoundationDB (which Wilson worked on) has a very unique and sophisticated approach to testing that seems to have served them very well.
Congratulations to the Antithesis team! I actually interviewed with them when they were just starting, and outside of being very technically proficient, they are also a great group of folks. They flew my wife and I out to DC on what happened to be the coldest day of the year that year (we are from California) so we didn’t end up following through but I’d like to think there is an alternative me out there in the multiverse hacking away on this stuff. I highly recommend Will’s talks (which I believe he links in the blog post): [LINK] this video
FoundationDB is a truly one-of-a-kind bit of technology. Others have already linked to the testing methodology that allows them to run orders of magnitude more database hours in test than have run in production: [LINK] A less known but also great talk is the follow which talked about what the a few of the team worked on next, effectively trying to generalize the methodology to any computer program: this video I liken the approach to being able to fuzz the execution space of the program, not just the inputs.
Really unique product with a super-strong founding team! I’m not a working with them, but am cheering from the sidelines. One of the founders gives a great talk here: this video
FWIW I think that DOES happen, but it happens on the "wrong" computers! Google has at least tens of thousands of cores running builds and tests 24/7. And they're utilized to the hilt. Travis and other continuous build services do essentially the same thing, although I don't know how many cores they have running. From a larger perspective, Github does significant things "in the background" to make me more productive, like producing statistics about all the projects and making them searchable. (Admittedly, it could do MUCH more.) I think part of the problem is that it's cheaper to use "the cloud" than to figure out how to use the developer's own machine! There is a lot of heterogeneity in developer machines, and all the system administration overhead isn't worth it. And there's also networking latency. So it's easiest to just use a homogeneous cloud like Google's data centers or AWS. There's also stuff like [LINK] which improves productivity. I do think that most software will be tested 24/7 by computers in the future. Foundation DB already does this: "Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson [LINK] Autonomous Testing and the Future of Software Development - Will Wilson this video They have sort of an "adversarial" development model where the tests are "expected" to find bugs that programmers introduce. So it's basically like having an oracle you can consult for correctness, which is a productivity aid. Not exactly, but that would be the ideal.