The TiKV project team writes about its distributed key-value database, covering Raft, performance tuning, releases, and Rust internals.
HN commenters point to TiKV as a ready-made RocksDB plus Raft option and as a reference design for sharded replicated systems.
It's just as desirable as with a simple database since it avoids certain anomalies (write skew). For many operations it's strong enough, but it adds the risk that you run into problematic anomalies you did not anticipate.
> In a write skew anomaly, two transactions (T1 and T2) concurrently read an overlapping data set (e.g. values V1 and V2), concurrently make disjoint updates (e.g. T1 updates V1, T2 updates V2), and finally concurrently commit, neither having seen the update performed by the other. Were the system serializable, such an anomaly would be impossible, as either T1 or T2 would have to occur “first”, and be visible to the other. In contrast, snapshot isolation permits write skew anomalies.
FoundationDB has a workaround, but I don't know how well that works in practice.
> In TiDB, you can use SELECT … FOR UPDATE statement to avoid write skew anomaly. In this case, TiDB will use locks to serialize writes together with MVCC to gain some of the performance gains and still support the stronger “serializability” level of isolation.
Serializability is more expensive of course, since the database needs to track what was read and not only what was modified. It is possible to achieve this in a distributed database, FoundationDB is one example that supports it.
https://tikv.org/deep-dive/distributed-transaction/isolation-level/
Hi Diggsey, great question. We are currently focussed on functionality and stability, and then will draw our attention to performance. Coming this week we have a RocksDB storage implementation. We've only just launched our initial beta version, and we know there is a lot of improvement and work to be done (some of these performance issues we know about already and are on our Github issues list).
With regards to the consistency/isolation model, SurrealDB sits on top of a number of key-value stores. By using the distributed highly-available TiKV storage backend, https://tikv.org, (and we have a FoundationDB integration in the works), the database is designed to be highly-scalable and highly-available. The same guarantees (albeit just single-node, so no high-availability or scalability) will be available with the RocksDB implementation coming this week. By sitting on top of these key-value stores, SurrealDB ensures that all transactions are ACID compliant. We don't want to go for speed (for instance by writing to /dev/null) over anything, but want SurrealDB to be a reliable and performant backend for any application. Obviously we have a way to go to catch up with PostgreSQL (launched in 1996), but we will strive to get there!
Restate is built as a sharded replicated state machine similar to how TiKV (https://tikv.org/), Kudu (https://kudu.apache.org/kudu.pdf) or CockroachDB (https://github.com/cockroachdb/cockroach) are designed. Instead of relying on a specific consensus implementation, we have decided to encapsulate this part into a virtual log (inspired by Delos https://www.usenix.org/system/files/osdi20-balakrishnan.pdf) since it makes it possible to tune the system more easily for different deployment scenarios (on-prem, cloud, cost-effective blob storage). Moreover, it allows for some other cool things like seamlessly moving from one log implementation to another. Apart from that the whole system design has been influenced by ideas from stream processing systems such as Apache Flink (https://flink.apache.org/), log storage systems such as LogDevice (https://logdevice.io/) and others.
We plan to publish a more detailed follow-up blog post where we explain why we developed a new stateful system, how we implemented it, and what the benefits are. Stay tuned!
data structures making use of underlying hardware may have better performance than generic b-tree.[0][1][2][3]
[0] LSM (relies on ability to cache reads) vs. b-tree : https://tikv.org/deep-dive/key-value-engine/b-tree-vs-lsm/
[1] alternatives such as c-tree (lower memory usage/query time at cost of flexibility): https://en.wikipedia.org/wiki/C-trie
[2] radix variants : https://en.wikipedia.org/wiki/Radix_tree
[3] striped trie (distributed/parallel searching of shards) : https://www.agdresearch.com/striped-trie
> Many,many smart people…
True and if you look inside the code you can see the stated features are a result of underlying engine from those smart people (TiKV [0] in C and rust from pingcap). Surrealdb is standing on shoulders of giants at present, they are TiKV, FoundationDB and rocksdb. The feature set they mentioned mostly coming from TiKV at present.
[0] https://tikv.org/
This is an example of a system built like you're describing at the end: https://www.pingcap.com/blog/how-an-open-source-distributed-newsql-database-delivers-time-services/
Also see https://tikv.org/deep-dive/distributed-transaction/timestamp-oracle/
Cockroachdb describes their approach here: https://www.cockroachlabs.com/blog/living-without-atomic-clocks/