Domas Mituzas

> I would take 100x increase in compilation times for a release build over a 2x increase in development time due to manual optimization.

If you're referring to super-optimization, it can take far more than 100x and isn't something that is being "classically automated" as humans likely can't do it in the first place. In addition, super-optimization will yield better results if you give it a better starting point. It's nothing at all like the optimization discussed in the blog post. Either way, it can be done against LLVM IR[4] and I'm sure it will creep into clang at some point.

So far as using profiling data to optimize code in the way that humans would: MSVC supports it[1], clang supports it[2] and gcc supports it[3]. Supposedly Microsoft saw a 30% performance increase [with a very specific workload] when they tried POGO on an internal build of SQL Server. Under normal circumstances, PGO should net you around 7% free perf.

[1]: https://msdn.microsoft.com/en-us/library/e7k32f4k.aspx
[2]: http://clang.llvm.org/docs/UsersManual.html#profile-guided-optimization
[3]: http://dom.as/2009/07/27/profile-guided-optimization-with-gcc/
[4]: https://github.com/google/souper

Note that Stonebraker makes some good points, but there are many ways to build scalability and Stonebraker is too fast to dismiss many.

In particular, his criticism of traditional databases seems based more on philosophy rather than evidence.

I'd advise reading both sides of the story:

http://lemire.me/blog/archives/2009/09/16/relational-databases-are-they-obselete/

http://lemire.me/blog/archives/2009/07/03/column-stores-and-row-stores-should-you-care/

http://architects.dzone.com/articles/stonebraker-talk-triggers

http://gigaom.com/2011/07/11/amazons-werner-vogels-on-the-stonebrakernewsql-debate/

http://dom.as/2011/07/08/stonebraker-trapped/

The date on some of those posts in interesting. 2009 is quite a while ago now, and I'd suggest that columnar datastores haven't exactly taken over. Some implementations have made some progress (eg Cassandra), but OTOH many non-traditional datastores have added traditional-database like features (eg, Facebook's SQL front end on their NoSQL system), and traditional databases have added NoSQL features too.

Yeah, I am the engineer mentioned in the article, and I agree the explanation doesn't really work. The pieces of the explanation that ended up in the article itself don't add up to a coherent explanation. The fault for that is mostly mine. In hindsight, my original explanation was too long and too elaborate to be helpful. It's a good reminder that it is easy to go to far with an analogy and end up complicating the thing you were trying to simplify. Oh well, live and learn :)

There are more (coherent and to-the-point) details about PoolCounter in the prologue to PoolCounter.php in MediaWiki's source tree:

https://github.com/wikimedia/mediawiki/blob/1617e7822eaf74261687d2d2c3148ce10ab5da69/includes/poolcounter/PoolCounter.php#L3-L43

And in a short blog post by Domas Mituzas, who is the original author of PoolCounter:

https://dom.as/2009/06/26/embarrassment/

You're right that memSQL absoutely crawls when it is set to "fully-durable" mode where it has to sync write to the disk. But it was never built for the use-case, and I guess that's missing the point.

memSQL, AFAIK, was built for highly concurrent reads and writes. It uses lock-free in-memory data structures, couple that with an eventually-durable mode, in addition to optimizing queries by the way of re-compiling SQL to CPP (something similar to what HHVM does for PHP), I think it would result in performance gains in certain (and not all) use-cases.

memSQL, if we dare simplify, is really a high-throughput cache front-ending an eventually-durable datastore.

I wonder how would RocksDB + a disk-backed DB measure upto memSQL.

Ref: http://dom.as/2012/06/26/memsql-rage/

Facebook guys as I understand it did look carefully at PostgreSQL; at least some of the engineers are very familiar with its internals. They found MySQL satisfactory for a variety of reasons including ability to engineer better performance but also replication which was much better in MySQL for a long time. In fact, for an interesting perspective on the "Facebook stuck on MySQL" topic see [1].

Interestingly much Facebook database work has been outside the relational orbit on systems like Cassandra and HBase, as well as RocksDB. The choice for most web properties today is not MySQL vs. PostgreSQL but RDBMS vs. "other" systems.

[1] https://dom.as/2011/07/08/stonebraker-trapped/

Nice benchmark script. With EXPLAIN (ANALYZE, BUFFERS), I see that

* normalized/join version needs to read 5600 pages

* normalized/join version with an additional UNIQUE INDEX .. INCLUDE (type) needs to read 4500 pages

* denormalized version only needs to read 66 pages, almost 100x fewer

Related to this pagination use case, when using mysql, even the denormalized version may take minutes: https://dom.as/2015/07/30/on-order-by-optimization/