Amazon Web Services
2 HN recommenders · recommended in 6 months, 2019–2023
55:44 · 37K views · Nov 27, 2018
An AWS re:Invent 2018 talk on how AWS limits the blast radius of failures. It covers design techniques such as cell-based architecture and shuffle sharding.
It is the go-to reference whenever HN discusses cell-based architecture and outage isolation, as an explanation straight from AWS engineers.
9 comments from Hacker News
Multi-AZ doesn't protect against a software/OS issue like this, Multi-AZ would be relevant if it was an infrastructure failure (e.g. underlying EC2 instances or networking). The relevant resiliency pattern in this case would be what they refer to as cell-based architecture, where within an AZ services are broken down into smaller independent cells to minimize the blast radius. They specifically mention in the write-up that this was a gap they plan to address, the "backend" portion of Kinesis was already cellularized but that step had not yet been completed on the "frontend". Celluarization in combination with workload partitioning would have helped, e.g. don't run Cloudwatch, Cognito and Customer workloads on the same set of cells. It is also important to note that celluarization only helps in this case if they limit code deployment to a limited number of cells at a time. This YouTube video[1] of a re:invent presentation does a great job of explaining it. The cell-based stuff, starts around minute 20. 1. this video
This is a great outline. Martin Kleppmann's Designing Data Intensive Applications [0] is very much in the same domain as what's in these notes. Speaking about scale and distributed systems: AWS specific resources I find interesting from their EdgeEngineering/NetEng teams (I haven't seen many other service teams in AWS openly share as much about design as them): - [LINK] series of articles on Route53's 100% data-plane availability architecture [1]. - [LINK] colmmacc [2] (seems to be the eng behind AWS HyperPlane [3][4]?) on 10 design patterns for building resilient systems. - this video Peter Vosshall (co-creator of Dynamo [5]) presenting "cell-based" design in-use at AWS. ---- [0] [LINK] [1] [LINK] [2] [LINK] [3] [LINK] [4] [LINK] [5] Decandia, G.; Hastorun, D.; Jampani, M.; Kakulapati, G.; Lakshman, A.; Pilchin, A.; Sivasubramanian, S.; Vosshall, P.; Vogels, W. (2007). "Dynamo: Amazon's Highly Available Key-value Store". [6] Bonus: The SRE Book [LINK]
From what you describe the 'AWS cells' are very different than Borg's. My view is that at a high level aws cells are isolated, fuzzy-partitioned, migratable, heterogeneous instances of a complete service stack that are managed such that they always fail independently of each other; contact points among these cell instances, if required, are established via highly available intermediate services (like ReplicatedStateMachinesOrStorage); the clients are routed to assigned cells by a highly available routing layer (like NetworkTrafficShapers/DNS), which can also help facilitate migration of clients from one/more cell to another. I encourage you to view the re:invent presentation on it [0] if resiliency in distributed systems is something that interests you. [0] this video
> [LINK] > Almost all updates to Google’s services proceed gradually, according to a defined process, with appropriate verification steps interspersed. [..] The first stages of a rollout are usually called "canaries” — an allusion to canaries carried by miners into a coal mine to detect dangerous gases. Our canary servers detect dangerous effects from the behavior of the new software under real user traffic. > Canary testing is a concept embedded into many of Google’s internal tools used to make automated changes, as well as for systems that change configuration files. [..] If the change doesn’t pass the validation period, it’s automatically rolled back. Any Googlers around to explain why Canaries didn't catch configuration causing this side effect and stopping the deployment before the issue cascaded globally? The outage here reminds me of issues CloudFront faced wrt configurations in their earlier years. Here's master presenter Harvo Jones going into a bit more detail: [LINK] Kind of vindicates AWS' decision (after major outages to DynamoDB and S3) to not just isolate regions or zones but resources within those regions/zones further into what they call "cells" this video Tumblr spoke abt "cells" too in the context of achieving isolation at scale, but in a quite different way: [LINK] As these known-knowns (and the Google sre-book is teeming with these) get designed around, it'd be interesting to see why/how future outages would happen.
root-cause tldr: ...[adding] new capacity [to the front-end fleet] had caused all of the servers in the [front-end] fleet to exceed the maximum number of threads allowed by an operating system configuration [number of threads spawned is directly proportional to number of servers in the fleet]. As this limit was being exceeded, cache construction was failing to complete and front-end servers were ending up with useless shard-maps that left them unable to route requests to back-end clusters. fixes: ...moving to larger CPU and memory servers [and thus fewer front-end servers]. Having fewer servers means that each server maintains fewer threads. ...making a number of changes to radically improve the cold-start time for the front-end fleet. ...moving the front-end server [shard-map] cache [that takes a long time to build, up to an hour sometimes?] to a dedicated fleet. ...move a few large AWS services, like CloudWatch, to a separate, partitioned front-end fleet. ...accelerate the cellularization [0] of the front-end fleet to match what we’ve done with the back-end. [0] this video and [LINK]