Antithesis Blog

Thanks for a great question!

When we made the decision:

1. We were thinking long term. For example, a distributed database is a significant investment. I love the spirit and orthogonality of C, but we didn't want to pay the safety tax of C over the next 20 years. At the same time, Zig has that same spirit—it's not only a perfect replacement, but a leap forward in developer velocity, power, performance and ergonomics.

2. We realized that TigerBeetle would take 2 years to get to a production release. This meant that our timeline would intersect with Zig's stability—like skating to where the puck is going to be at, or catching the swell as it breaks, rather than riding out the last of an old wave. The number of surfers (quantity) was not a concern. Rather, we were impressed by the sheer quality and early maturity of Zig's compilation story, not to mention the quality of the Zig community in general. For example, it would be hard to find a better std lib crypto than what Zig already has right now, thanks to Frank Denis of libsodium.

3. The design of TigerBeetle is also a single-threaded control plane. We use io_uring for the data plane to eliminate multi-threaded context switches, so multi-threading for async I/O is less of a necessary evil than it used to be. All memory is statically allocated at startup. We never call free() so there are no UAFs. You can start to see why Rust's borrow checker made less sense for our domain than it would have for others. Of course,…

Author here. Let me know if you have any questions!

Let me take a shot at answering what I think will be the most common question: how can I trust this? Our #1 priority right now is correctness. Over the past two weeks, I've done a mix of formal verification and differential fuzz testing. We've been able to prove over 1000 user facing functions have the exact same logic in both pgrust and postgres (see the proofs directory if you're curious). For cases where formal verification is not easy, we've taken the c implementation of a function and the rust implementation of a function and ran millions of inputs through each of them and confirmed they gave the same results every time.

We've only covered about 15% of the surface area so far, but in the process, we've discovered ~100 bugs in pgrust and ~20 bugs in Postgres itself. My favorite postgres bug we found is this one[0]. Postgres has a quadtree implementation. Due to floating point rounding, it was possible for a point to be neither above, nor below, nor even with the center point of the quadtree.

We've also entered engagements with Antithesis[1] to do Jepsen style fault testing and Aretta[2] to do more serious formal verification.

If you want to support the project, the easiest way is to give us a star on GitHub[3]

[0] https://www.postgresql.org/message-id/19597-39c532e61d78dff6%40postgresql.org

[1] https://antithesis.com/

[2] https://aretta.ai/

[3] https://github.com/malisper/pgrust

> Isolated QA should not exist because anything a QA engineer can do manually can be automated.

I almost entirely agree. You'd certainly want a majority of QA responsibilities to fall to devs since that keeps the feedback loop tight. Also with the amount of compute we have you can trivially outpace a typical QA team with a single dev working on fuzz tests and property based tests. If you really still feel the need to outsource then there are cool tools like Antithesis [1] which let you trade money for compute in their complicated fuzz testing suite.

Where that falls apart imo is when your software interacts with hardware in any non-trivial way. Fuzz testing doesn't really work when your system is spinning big and heavy things for example. Real hardware that interacts with the world will always create this gap where the devs don't know how to fully test it. Your systems engineers are best equipped to handle this and ensure certain controls follow the right curves, but they typically don't have the bandwidth or are very inexperienced with software. I think a QA organization _can_ fill this gap and deliver value. But I'd almost always prefer hiring another 1 or 2 system engineers who can work towards this problem full time instead. It's much easier to train someone in software than it is to train them in motor control or sealed systems.

[1] https://antithesis.com/

Antithesis | Software Engineers, ML Research Scientists | Northern VA | ONSITE | https://antithesis.com/

Antithesis is using deterministic simulation, fuzzing, and novel ML techniques to improve software testing and increase developer velocity. For an idea of what we're doing, see Will's talk here: https://www.youtube.com/watch?v=fFSPwJFXVlw or read the Simulation Testing section of https://www.foundationdb.org/files/fdb-paper.pdf. Fully funded medical, dental, and vision insurance. Unlimited vacation (with mandatory MINIMUM vacation time). Flexible hours. Incredible, cohesive team with the hardest and most interesting problems to solve. We work together onsite in a highly collaborative environment.

We're doing things nobody has done before, so we don't need familiarity with any particular language, framework, domain, or anything like that. Don't care. (But if you love Nix...we should talk.) What we do care about: you're smart, get things done, a joy to work with, and are comfortable dealing with open-ended problems.

Apply at https://antithesis.breezy.hr/ (Don't worry too much about which role you choose. We're looking for engineers who are ready to learn, not those that know everything already!)

We're also hiring for professional services roles. Feel free to ask about those.

It is difficult to use by itself: the "foundation" in the name describes it quite well. It is a foundation that you build a database on. It fits my use case very well, for example, because I know my data model and usage patterns very well and I can integrate deeply with the database, but it's not a good match by itself for quick-and-dirty apps.

It provides fantastic (strict serializable) consistency guarantees in a distributed database, which is extremely rare. It is a huge advantage, but sadly most people do not understand how badly most distributed databases are broken and don't even understand the concepts (https://antithesis.com/resources/reliability_glossary/) well enough to talk about the issues involved. See every discussion where someone mentions ACID.

It's hard to compete for mindshare when the concepts are difficult and every other database has a warm-and-fuzzy-feeling website saying that everything will be great (it usually won't).

Personally, I hope more people will start using it, and I hope to see more easy-to-use databases built on top of it (that's what it was designed for, really). In my experience with it, working with a fast distributed database that gives you strict serializable semantics right in your code is fantastic.

Oh, I just realized I am guilty of the drive-by-free-association comment without actually saying anything about the subject of the post - sorry!

Very cool to see a team use Jepsen for super early pre-release testing of the system.

I wonder if you wish you had waited for the runtime to be a bit more stable, or you feel this was already well worth the effort, even with some of the identified failures being in "known incomplete" areas? (I could see either side of the argument - waiting longer might give you more valuable failures, but testing early gives you a chance to catch problems before they become baked into the foundation and become more difficult to fix...)

Another tool that feels like sci-fi to me any time I hear a mention of it, is Antithesis [1] - written by the people who built FoundationDB. Could be another interesting integration to investigate in the future to help bulletproof the language runtime?

[1]: https://antithesis.com