VoltDB Blog

Thank you, it was FoundationDB I was thinking of, post-mortem analysis from the VoltDB guys here[1]. I'll contend they failed though because Apple bought them as an acqui-hire and killed the product. They were acquiring good engineers with a track record of executing well at a premium. It's a common recruiting strategy for tech companies. It doesn't mean that FoundationDB was a successful product - certainly it means it was not successful enough for the founders to risk turning down a lucrative acqui-hire deal.

> his is a rigid and narrow view. There's lots of other design possibilities such as fractured mirrors, column decommission only within pages, etc.

Yes definitely I over-simplified to make my point, but I'll stand by it. OLAP and OLTP are different enough workloads and the engineering tradeoffs for being good at one of the other make it hard to excel at both. VoltDB is a row store, best suited for OLTP. Vertica is a column store best suited for OLAP. VoltDB can also make a decent OLAP database. Vertica does not make a good OLTP database. Both databases come out of Michael Stonebraker's work (although given the number of people required to make complex products like VoltDB and Vertica happen, it seems unfair to credit him unduly.)

But especially in the case of a sharded, distributed database where communication overhead is likely to dominate many types of queries (VoltDB, TiDB), one might not notice the inefficiency of using a poorly-suited design, like a row store…

Yes, this is the argument that VoltDB made as well:

https://www.voltdb.com/blog/foundationdbs-lesson-fast-key-value-store-not-enough

My feelings on this topic are mixed. On the one hand, I think many of the specific examples chosen in that post are false (and have told John as much in person). On the other hand, the general point that you can squeeze out constant factor performance improvements by violating abstraction boundaries is obviously usually true.

Nevertheless, I still think this is a bad argument. While it's true that abstractions are rarely costless, they can often be made so cheap that the low-hanging performance fruit is elsewhere. And in particular, cheap enough that they're worth it when you consider all the other benefits that they bring.

When I built a query language and optimizer on top of FoundationDB, my inability to push type information down into the storage engine was about the last thing on my mind. Perhaps someday when I'd made everything else perfect it would've become a big pain (and perhaps someday we would've provided more mechanisms for piercing the various abstractions and providing workload hints), but in the meantime partitioning off all state in the system into a dedicated component that handled it extremely well made the combined distributed systems problem massively more tractable. The increased developer velocity and reduced bugginess in turn meant that I (as a user of the key-value store) could spend scarce engineering resources on…

FaunaDB uses Calvin, a transaction protocol developed by Daniel Abadi. Their blog post explains it nicely, after a bit of a slow start:

https://fauna.com/blog/consistency-without-clocks-faunadb-transaction-protocol

But in summary:

1. A 'transaction' is a self-contained blob of code which reads input, does deterministic logic, and writes output (so not like a traditional RDBMS transaction, where the application opens a transaction and then interleaves its own logic between reads and writes)

2. When a transaction arrives, the receiving node runs it, and captures the inputs it read, and the outputs it wrote

3. The transaction, with its captured inputs and outputs, is written to a global stream of transactions - this is the only point of synchronisation between the nodes

4. Each node reads the global stream, and writes each transaction into its persistent state; to do that, it repeats all the reads that the transaction did, and checks that they match the captured input - if so, the outputs are committed, and it not, the transaction is aborted, and retried

The key idea is that because the process is deterministic, the nodes can write transactions to disk independently without drifting out of sync.

It's pretty neat. And it's exactly what Abadi wrote about a couple of months ago:

http://dbmsmusings.blogspot.com/2019/01/its-time-to-move-on-from-two-phase.html

This is also what VoltDB does (which Abadi worked on, along with Michael Stonebraker):

As an operational store,…

Quick things

1) I expected that thread to look like a Cultural Revolution struggle session". Thankfully it wasn't.

2) As I am sure many others have already said it, durability has very little to do with CAP. CAP is about the A and I in in ACID, D is an orthogonal concern.

3) Durability doesn't necessarily mean losing high performance. Most databases let the user choose how much data they're willing to lose and for what latency decreases -- the standard approach (used even in main-memory databases like RAMCloud[1] and recent versions of VoltDB[2]) is to keep a separate write-ahead log (WAL) and let the end-user choose how frequently to fsync() it to disk as well as how frequently to flush a snapshot of main memory data structures to disk.

There are many papers (e.g., http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.174.6205) that talk about various challenges of building WALs, but fundamentally users who want strongest possible single-machine durability can choose to fsync() on every write (and usually use battery backed raid controllers or separate log devices like SSDs with supercapacitors or even NVRAM if the writes are going to be larger than what fits into the RAID controller's write-back cache). Others can choose to live with possibility of losing some writes, but use replication[3] to protect against power supply failure and crashes -- idea being that machines in a datacenter are connected to a UPS, replicas don't all live on the same rack (to protect…

From their whitepaper:

"Conventional databases experience disk and user stalls within transactions. Rather than let the CPU be idle during the stalls, those DBMSs interleave SQL execution from multiple transactions during the waits so the CPU is always busy. This is what requires much of the complex latching and locking overhead.

VoltDB doesn’t experience user stalls (since transactions happen within stored procedures) or disk stalls (because VoltDB processes data in main memory). Therefore, it is able to eliminate the overhead associated with multi-threading (latching) and locking. Each VoltDB execution engine is single-threaded and contains a queue of transaction requests, which it executes sequentially—and exclusively—against its data. Elimination of stalls and the need for locking and latching overhead and allows typical VoltDB SQL operations to complete in microseconds.

For single-partition transactions, each VoltDB engine operates autonomously. For multi-partition transactions, one engine distributes and coordinates work plans for the other engines. VoltDB assumes that an application designer can construct a partitioning/cloning scheme and a transaction design that makes a large majority of the transactions local to a single virtual node. Many common applications such as order-fulfillment, software as a service (SaaS), Web 2.0 and trading systems have this property."

http://www.voltdb.com/_pdf/VoltDBTechnicalOverviewWhitePaper.pdf

Sounds like event sourcing [1] applied to databases.

The server-side transactions reminded me a lot of VoltDB [2], which also has server-side transactions and turns out was a previous system Abadi was involved in.

My naive impression is that VoltDB is more about being in-memory, single-threaded, and lock-free, where as this Calvin approach is more about deterministic ordering of events.

Sounds really cool, but there is a reason people don't like writing stored procedures. Besides (historically) meaning you're forced into a non-general programming language, it means you can't do anything else while executing the logic (call out to a 3rd party, read a file from somewhere else, etc.).

While SQL has its flaws, the semantics of writing your application logic freely over multiple queries ("make a query, do some work, make a query, do some work, ... commit!") and having the database just make that magically work for you is pretty darn nice and not something I'd easily give up.

Maybe a heavy dose of optimistic locking and a pleasant/mostly automated process of coding/integrating/deploying the stored procedures would make it possible to get all the performance benefits of their approach. It does sound pretty smart.

[1]: http://martinfowler.com/eaaDev/EventSourcing.html

[2]: http://voltdb.com/