PingCAP Blog

Brian Anderson wrote up his thoughts here, and it's a good intro to the topic: https://www.pingcap.com/blog/rust-compilation-model-calamity/

Let's dig into this bit of that, to give you some more color:

> Split compiler/package manager — although it is normal for languages to have a package manager separate from the compiler, in Rust at least this results in both cargo and rustc having imperfect and redundant information about the overall compilation pipeline. As more parts of the pipeline are short-circuited for efficiency, more metadata needs to be transferred between instances of the compiler, mostly through the filesystem, which has overhead.

> Per-compilation-unit code-generation — rustc generates machine code each time it compiles a crate, but it doesn’t need to — with most Rust projects being statically linked, the machine code isn’t needed until the final link step. There may be efficiencies to be achieved by completely separating analysis and code generation.

Rust decided to go with the classic separate compilation model that languages like C use. Let's talk about that compared to Zig, since it was already brought up in this thread.

So imagine we have a project, A, and it depends on B. B is a huge library, 200,000 lines of code, but we only use one function from it in A, and that function is ten lines. Yes, this is probably a bad project management decision, but we're using extremes here to make a point.

Cargo will compile B first, and then A, and then link…

I do agree with you in that regard, however, that's also a dangerous line of thinking.

There are attempts to provide horizontal scalability for RDBMSes in a transparent way, like TiDB https://pingcap.com/ (which is compatible with the MySQL 5.7 drivers), however, the list of functionality that's sacrificed to achieve easily extensible clusters is a long one: https://docs.pingcap.com/tidb/stable/mysql-compatibility

There are other technologies, like MongoDB, which sometimes are more successful at a clustered configuration, however most of the traditional RDBMSes work best in a leader-follower type of replication scenario, because even those aforementioned systems oftentimes have data consistency issues that may eventually pop up.

Essentially, my argument is that the lack of good horizontally scalable databases or other data storage solutions is easily explainable by the fact that the problem itself isn't solvable in any easy way, apart from adopting eventual consistency, which is probably going to create more problems than it will solve in case of any pre-existing code that makes assumptions about what ways it'll be able to access data and operate on it: https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing

To that end, i'd perhaps like to suggest an alternative: use a single vertically scalable RDBMS instance when possible, with a hot standby if you have the resources for that. Let the architecture around it be horizontally scalable instead, and let it deal…

> So can someone please explain? Maybe what he actually wants is madvise or hugetlbfs ?

I think you're getting warm. It's hard to know anything from this confused article but I'll assume they're correct that there was only one process, the pages were filled prior to the load test, and there were many minor page faults later. My best guess is that the page faults are due to the kernel's moving stuff to transparent huge pages in the background (and struggling to find/make unfragmented physical address space). The behavior change was due to the mmap size change they observed: before with the extra bytes the kernel didn't initially use huge pages. After, it did, so no background operation was necessary. (Or maybe MAP_SHARED helped because they had them backed by a non tmpfs filesystem, where huge issues aren't supported. Or maybe some behavior changes were just because they tested multiple times without rebooting and the physical page space got fragmented. Again, hard to know much from this article.)

If I were them, I'd confirm this by "perf record" or eBPF and/or by disabling transparent huge page compaction. Then I'd switch to using dedicated huge pages, because huge pages really do speed things up by like 15% for a memory heavy workload; they're worth it if you can avoid THP's sometimes pathological behavior.

Edit to add: Here is a much clearer article explaining the same problem. https://pingcap.com/blog/why-we-disable-linux-thp-feature-for-databases I don't agree with…

One of the founder of TiDB/TiKV here from PingCAP (https://pingcap.com)

I have been thinking about this problem with my peers when I started to build TiDB (https://github.com/pingcap/tidb) seven years ago. At that time, nearly all of us were familiar with Go language, so we decided to use Go to build the SQL layer of TiDB. Thanks to Go, we could develop TiDB very quickly and released the first MVP in half a year. I remembered clearly the sense when we ran TPC-C successfully, although the TPMC was just 1 at that time, this was a good start for us.

But Go had some problems, e.g. the GC was not good before, the fair scheduling might cause some latency problem, or data racing may happen sometimes. So when we decided to build a distributed storage (aha, TiKV (https://githbu.com/tikv/tikv), we wanted use another language to guarantee safety. I really admire our courage - we chose Rust which was just released 1.0 and missed lots of libraries at that time. Now it seems that this is an awesome choice, TiKV has been graduated from CNCF, and been used as building block not only for TiDB, but also for other distributed systems. Thanks Rust.

When TiDB started being used in many companies, we found that our customer not only ran lots of online transactions in TiDB, but also they wanted to ran some realtime analytic queries directly because the data has been in TiDB already. So we decided to build a HTAP database, to introduce a column storage beside TiKV, this is TiFlash…

TiDB developer here.
Yes, I think TiKV is an alternative to FDB. Compare to FDB Record Layer, TiKV aims to provide a more atomic primitive, just including Get/Set/Transaction in key-value layer, so users can build customized distributed system around it. The main differences between TiKV/TiDB and FDB are:

1. TiKV uses Multi-Raft architecture, I think Raft provides more HA.

2. TiKV's transaction model is inspired by Google Percolator, it's a classical optimistic 2PC transaction model with MVCC support. I'm not a expert of FDB, but I think different transaction models fit for different application scenarios, TiKV's transaction model is good when your workload is mainly small transactions and with a low conflict rate.

3. TiDB is a full-featured SQL layer on top of TiKV, aims to provide a MySQL compatible solution, you know, most of the TiDB users are migrated from the MySQL, so the focus of TiDB will be how to be compatible with these legacy MySQL-based applications. For example, how to read MySQL binlog and then replay on TiDB in real time, let TiDB become a MySQL active replica, or how to support complex SQL queries like distributed join or groupby, you know, building a full-featured SQL optimizer is a huge project.

There are some case studies:

https://pingcap.com/success-stories/

https://pingcap.com/success-stories/tidb-in-meituan-dianping/

There are some quick-start documents you can start…

I do want to clarify a few points, on the project page it does provide the following information:

> Distributed Transactions: TiDB uses a two-phase commit protocol to ensure ACID compliance, providing strong consistency. Transactions span multiple nodes, and TiDB's distributed nature ensures data correctness even in the presence of network partitions or node failures.

> High Availability: Built-in Raft consensus protocol ensures reliability and automated failover. Data is stored in multiple replicas, and transactions are committed only after writing to the majority of replicas, guaranteeing strong consistency and availability, even if some replicas fail. Geographic placement of replicas can be configured for different disaster tolerance levels.

See https://github.com/pingcap/tidb?tab=readme-ov-file#key-features

Correctness has been a focus for a long time for TiDB, including working on passing Jepsen Tests back in 2019, see https://www.pingcap.com/blog/tidb-passes-jepsen-test-for-snapshot-isolation-and-single-key-linearizability/ and https://jepsen.io/analyses/tidb-2.1.7

Disclosure: Employee of PingCAP the company behind TiDB