dotconferences
1 HN recommender · recommended in 6 months, 2015–2018
25:18 · 6.8K views · Jun 25, 2015
John Graham-Cumming of Cloudflare talks at dotScale 2015 about handling roughly 10 trillion log lines a month. The talk touches on Cloudflare's hardware, its logging stack and its use of Lua in Nginx.
Mostly cited as a source for details of Cloudflare's architecture, such as LuaJIT-based request inspection and request volumes.
6 comments from Hacker News
I discovered a moderately old video from CloudFlare hiding in there, and had a rather big TIL moment. It's one thing to read "LuaJIT is one of the fastest dynamic languages" etc blah blah on [LINK] ... ...it's quite another to learn that EVERY SINGLE CLOUDFLARE REQUEST - all 4 million/sec of them - goes through a LuaJIT-based attack detection system. Oh. Video: this video (from [LINK] ) Found in the linked thread here: [LINK]
> Even putting aside that the computer tech part of the business isn't their core competency, custom hardware is hard. Hmmm. Good point. (Maybe I scared OP away, heh.) > Is there an example of any large company, other than Google, who has mastered this? Well, I certainly don't know. I've heard Cloudflare gets their systems made, for some definition of "from scratch", by Quanta. Presumably there's a bit of a decision-making process in there, and presumably Quanta handles everything related to the "why is the devkit sample broken again #51383" side of things and ships a more or less finished product (maybe even with finished enclosures). My reference point for this knowledge is this video (2015). Regarding SAS, maybe past a certain ordering quantity (or anything similar that would reveal a massive operation) the market would expect certain behavior like requesting value-adds, ordering with only one supplier, or other things that would generate lock-in. I of course have absolutely no idea [if this sort of thinking is reasonable]. Your points about commodity at scale sound quite reasonable. (You clearly have more experience than I do, which is kind of zero, heh.) As for suspicion about summaries and conclusions, well, (tangential example) the radial graph on [LINK] (the one from Forrester) doesn't show GitHub :D Yeah, building own hardware is sadly still within the realm of over-specialization at this point. I guess this is because 4+-layer PCB printing is still eye-watering so the
You make incredibly fair points and I don't have the data to answer what you've raised. This could be lack of research or the fact that CF hasn't released it. I expect a 20/80 non-researched/non-available mix. My understanding is that the WAF detects trivial stuff like SQL injection and XSS, and the video also mentions it detected ShellShock and so forth (yeah, this was a while ago). "Thousands of regular expressions" is mentioned verbally in the video around ~11:25. That's where I got that from. There are some bits mentioned about the hardware around 19:30: - E5-2630v3 (16 cores, 32 w/ HT which is turned off), 128GB RAM, 10Gbps - Custom built by Quanta: 4 nodes per 2U (each node is like half-width and 1U high), redundant 2xPSU in the middle - 40 machines run Kafka storing 400TB of log data (ingest @ 15Gbps), 50TB disk per node - 5 machines run CitusDB (sharded PostgreSQL) to power realtime graph generation, 12TB SSD per node - 100+ machines for realtime analytics running Go (and presumably LuaJIT) - LuaJIT seems to be the realtime "hmm, interesting" flagging system, while Go is used for in-depth intelligent analysis that's (I assume) slightly behind realtime. Also, the video is actually over here - this video --- I'm curious to understand what tail latency is, and very interested to learn more about language latency and performance in general. What subjects would you recommend I learn about / read? (I would describe myself as interested from a hobbyist-intermediate standpont, b
Rewriting the hodge podge of C, Perl and other custom bits of Nginx almost entirely in Lua and hiring its maintainer Yichun in the process where important steps for scaling up the CloudFlare's CDN and make sure each node is running as hot as possible. More on this from dotScale 2015: this video
For reference, two years ago Cloudflare was serving more than three million requests per second[0]. I can only imagine this figure has gone up a lot. 0: this video