MemSQL's company blog on its in-memory distributed relational database, covering indexes, lock-free algorithms, BPF performance analysis and geospatial features.
HN commenters cite it for explanations of skiplist indexes, BPF, and NoSQL versus relational design, and to announce MemSQL features.
This post is missing a key trade-off that you get with an in-memory row store (tl;dr ability to build lock-free), but also since 6/25/2012 several other innovations in MemSQL that have shipped over the past 2+ years.
The main advantage of having a main-memory row store is the ability to leverage random-access data in memory. This unlocks you to use random access aggressively and build lock-free indexes (atomic instructions are available only for data that's in memory). With these tradeoffs you can build extremely concurrent data structures that simply aren't possible with a system that must manage a buffer cache. See http://blog.memsql.com/the-story-behind-memsqls-skiplist-indexes/ for more details.
The article also suggests that MemSQL is not durable in the event of a power outage. This is simply not the case - we've had durability since the first ever public release of MemSQL, including both asynchronous and synchronous modes.
MemSQL now also has distributed transactions and MPP, supports intra and inter data-center replication, semi-structured (JSON) data, and even has a column store that works as a hybrid between disk and in-memory.
And BTW, I actually agree with this mentality in the context of column stores. There's no real advantage to having an "in-memory" column store because of the append-only nature of columnar blobs of data. That's why ours is a hybrid.
I've been thinking about a hypothetical CPU that has explicit cache and RAM access. It's pretty close to OP's alternative universe.
What would happen is people would write wrappers for such a system so that you could think about your RAM in a linear way. And you're now doing what a Haswell CPU does.
Of course, this approach has advantages for specific tasks Let's say, we want to implement constant-timing crypto primitive and put all of the required data in cache. We can now do it.
This would also help with implementing an efficient VM, OS primitives, or many other interesting things. But it would not change the way people write C/Rust/JS/Python.
Of course, it's a useful model for your performance, and you can now think about cache misses as "internet download", and synchronization primitives as "LAN share", etc, etc.
And, of course:
http://blog.memsql.com/cache-is-the-new-ram/
Hi @jhugg, a performant implementation of a counter usually does not read and update the value in separate statements within a transaction. Generally, people use UPDATE or INSERT...ON DUPLICATE KEY UPDATE (upsert) to implement this workload. In fact, transactional, high-throughput counters is an extremely common use-case for MemSQL [1].
As a matter of fact, even Oracle and MS SQL Server offer READ-COMMITTED as the default isolation level. Moreover, there are known issues with using SERIALIZABLE isolation in Oracle [2].
[1] - http://blog.memsql.com/high-speed-counters/
[2] - http://stackoverflow.com/questions/11826368/oracle-select-immediately-after-insert-in-serializable-transaction
I respect the author's experience, but "Database Engines Cannot Handle Real-Time Geospatial" is probably not true. We (MemSQL) have developed geospatial on top of our existing skiplist indexes, and can deliver thousands of intersection queries per second, over billions of points, in milliseconds per query, on very modest hardware. If you want more performance, just add more hardware.
We previewed our geo code at the Esri Developer's Conference last week. The demo delivers queries in millisecond time over 170M records without any tuning. The naive version worked fine.
http://blog.memsql.com/geospatial-intelligence/
https://en.wikipedia.org/wiki/FoundationDB
designers haven't solved all problem around transaction and performance, but that doesn't mean they have stopped trying
"As various NoSQL databases matured, a curious thing happened to their APIs: they started looking more like SQL. This is because SQL is a pretty direct implementation of relational set theory, and math is hard to fool."[1]
[1]http://blog.memsql.com/cache-is-the-new-ram/
> Thus lock-free data structures were developed which magically did not need locking and thus could operate blazingly fast.
This is wrong assumption. With lock-free data structures you don't skip waiting for your right to access data, you just avoid using lock. It may or may not be faster depending on conditions, and whole lot harder to get it right:
http://blog.memsql.com/common-pitfalls-in-writing-lock-free-algorithms/