Werner Vogels

Disclaimer: ex-AWS as well, worked very closely with MemoryDB, but not paid to shill!

You're entitled to your opinion but your line of reasoning for how MemoryDB for Redis came to exist or the reasoning about why it isn't in upstream Redis is not factual. MemoryDB's architecture uses Amazon's home grown log replication services as pointed out by Werner Vogel in his blog post about MemoryDB[0]. This architecture is fundamentally incompatible with upstream Redis. The real reason why MemoryDB for Redis exists is far less juicy: MemoryDB came about by meeting customers where they were. Customer's love Redis, especially for its non-caching features, but replication is a headache with all the existing solutions today.

Also as far as I know one of the lead committers to Redis is from AWS.

0 - https://www.allthingsdistributed.com/2021/11/amazon-memorydb-for-redis-speed-consistency.html

The marketing message could be clearer here but if you read the aurora paper

https://www.allthingsdistributed.com/files/p1041-verbitski.pdf

And the docs

There are actually two backup/restore methods.

1) Point In Time - streamed to S3. Guaranteed restore to a new cluster with less than 5 minutes data loss

2) Aurora Backtrack - On cluster buffer, up to 24 hours retention (no guarantee). "Instant" restore to the same cluster

So you should feel safe using the service (up to the 5 minute worst case scenario).

Obviously no one would ever want to use either backup method. However previously disaster recovery would involve spinning up a new cluster, deploying application changes with the new endpoint (or in a best case repointing a dns entry), waiting for the cache to warm up again. Now it becomes a 1 click fix through the UI with no application changes.

I think a lot of things. CRDTs predate the term "CRDT", they seem to be a mathematical construct/pattern found in the wild, that smart people codified and formalised.

IME CRDTs just lay out the rules you need to follow for syncing eventually consistent nodes properly, ie without weird surprises or data loss.

As for concrete examples:

- in the famous Amazon Dynamo Whitepaper, they perfectly describe a CRDT in section 4.4. (https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf), though I don't think this made it to modern Dynamo DB

- I believe CouchDB follows all the CRDT laws

- TomTom uses them

I think the TL:DR is if you're going to sync two data stores that can be written to without coordination, learn the CRDT laws and make sure you follow them. They're fairly simple mathematical properties at the end of the day.

That's right: You upload an object, and the HTTP 200 response includes a header that is the object ID. Your imagination is almost certainly correct: creating a mapping from user-defined key to storage location is really best done by a metadata store so that data doesn't need to be reshuffled with topology changes.

A classic distributed key-value store (such as Dynamo[1]) spends a lot of its design on how to consistently map user-keys to storage locations. Most of these solutions are focused on performance (so avoiding a middle-layer lookup) and accept the tradeoff that data will need to be re-balanced when doing things like expanding the cluster. This isn't an acceptable tradeoff for object stores that have much higher data density (100+ TBs per NIC)

[1]: https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf

Werner Vogels has a recent blog post [1] entitled "Modern Applications at AWS" where he says "To succeed in using application development to increase agility and innovation speed, organizations must adopt five elements, in any order: microservices; purpose-built databases; automated software release pipelines; a serverless operational model; and automated, continuous security."

I see the devops role at large cos evolving into dev-sec-ops for the "automated, continuous security" tooling and processes. (Besides the usual tooling for delivering and supporting a reliable network service, cloud engineering, system-level issue resolution in dev and test, load and performance and containerized test automation in the pipelines, failover fire-drills, etc.)

[1] https://www.allthingsdistributed.com/2019/08/modern-applications-at-aws.html

Amazon Aurora does something similar in the sense that it decouples InnoDB functions including buffer cache and storage access from the overlying MySQL engine. The decoupled InnoDB lives on separate VM(s) and includes optimized data replication to support DBMS high availability. For example, with Aurora you can failover a large DBMS in seconds to a new site and have it come up with a heated buffer cache since that replicates too.

This approach makes sense for Aurora because it leverages cloud storage to provide data high availability across availability zones. Just putting InnoDB in a file system seems less interesting unless you add features that are strikingly different from what it does now.

See http://www.allthingsdistributed.com/files/p1041-verbitski.pdf for a full description.