Baron Schwartz writes about databases, MVCC, time-series systems, and engineering management, drawing on his MySQL performance background.
HN cites his posts on immutability and MVCC, and on why automated database failover often causes more downtime than it prevents.
This can be flipped into a simple heuristic known as the "square root staffing rule" (sometimes "... law"). The name comes from call centres.
Basically, the number of servers you need to serve X amount of demand with Y probability of queuing is not linear with X. It is proportional to the square root of X.
The intuition is that a call centre agent is either talking to a customer, or they are not. If they are not talking to a customer, then they can immediately serve a customer. The odds that a given agent is talking to a customer is on a probability distribution, usually assumed to be the normal distribution.
In a normal distribution, larger samples lead to more and more appearances of outliers in the sample. More agents means more "outliers", in this case, more idle agents. And that curve is not linear, it follows the shape of the normal distribution.
Where it sucks for call centres is that demand shows seasonality and shift planning has relatively inflexible lead times. But in a software scenario we can typically acquire additional capacity quickly, so precision in forecasts is less of a problem.
A decent explanation: https://www.networkpages.nl/the-golden-rule-of-staffing-in-contact-centers/
A short scenario: https://www.xaprb.com/blog/square-root-staffing-law/
A simple calculator linked from the short scenario: https://www.desmos.com/calculator/8lazp6txab
The biggest problem is the query language. Without a query language, a company must hire and keep on-staff a developer who can help express problems to the database. Time after time people told me that they ended up with giant databases and needed to run lots of ad-hoc queries or generate reports from them. Once a time-series database becomes the system of record for important information, it needs to be queried for everything from invoicing to analytics to troubleshooting. In practice, larger companies end up with at least two full time developers who write applications to produce answers from the database. These are answers that the business/marketing/ops/whoever should be able to get themselves by just running queries. As time passes, this is both a huge cost and a bottleneck.[0] Link [1] goes to the archive mirror.
[0]: https://www.xaprb.com/blog/2014/03/02/time-series-databases-influxdb/
[1]: https://web.archive.org/web/20170917033402/https://www.xaprb.com/blog/2014/03/02/time-series-databases-influxdb/
> It is difficult to imagine anything resembling a "general" intelligence that isn't attached to a life-form[...]
How is that argument any stronger than these ones that (non-animal) heavier than air flight is impossible?
https://www.xaprb.com/blog/flight-is-impossible/
I think these are actually stronger arguments, because I can think of at least three very least significant material-science advantages that birds have even over modern technology that would, from a physics point of view, let me entertain the idea that a 19th century physicist could believe you'd at least need to be made of flesh to fly. I can't think of any comparable case for intelligence being restricted to biology.
Immutability is hardly a cure-all, see the discussion here for why RethinkDB moved away from it: http://www.xaprb.com/blog/2013/12/28/immutability-mvcc-and-garbage-collection/
The reality is shared, mutable state is the most efficient way of working with memory-sized data. People can rant and rave all they want about the benefits of immutability vs mutability, but at the end of the day, if performance is important to you, you'd be best to ignore them.
Actually, to be more honest, reality is more complicated still. MVCC that many databases use to get ACID semantics over a shared mutable dataset is really a combination of mutable and immutable.
MVCC is not versioning. MVCC is concurrency control.
I would have to be convinced that the DB is an appropriate place to handle versioning of the kind you are speaking of.
For instance, see this article: http://www.xaprb.com/blog/2013/12/28/immutability-mvcc-and-garbage-collection/
Upshot is that RethinkDB started out append-only with rewindability etc. and then ran into real-world problems.
Every system I've ever seen that did versioning does it at the software level and stores multiple records in the database.
Rails gem "paper trail" is an example.
I'd like to welcome the github ops/dbas to the club of people who've learned the hard way that automated database failover usually causes more downtime than it prevents.
Here's sortof the seminal post on the matter in the mysql community: http://www.xaprb.com/blog/2009/08/30/failure-scenarios-and-solutions-in-master-master-replication/
Though it turns into an MMM pile-on the tool doesn't matter so much as the scenarios. Automated failover is simply unlikely to make things better, and likely to make things worse, in most scenarios.