Roblox's engineering and company blog covers outage postmortems, avatar technology, Luau language work and machine learning at scale.
HN commenters often cite its postmortem of the 73-hour 2021 outage caused by Consul as a great read, and praise its layered clothing technology.
(this got kind of long, might make it into a blog post later)
> Well, in non-competitive "markets" like government services, your failure will just manifest itself in different ways.
It is true that failure might manifest in different ways, but this possibly conflates not using the newest tech with failure, which isn't necessarily true. Back when the COVID tracing apps were being implemented, my country managed to mostly get it done with using a pretty old tech stack: some Ruby, some Node here and there, as well as OCI containers without anything like K8s operators or CRDs.
It worked fine, it scaled decently, there were no loud snafus in the media or glaring deficiencies. I personally got to write the homepage with a pretty boring tech stack: React with no hooks, Bootstrap (instead of something like Tailwind), Nginx and Node for some read-only API components and S3 for files: https://apturicovid.lv/#en
Though maybe the difference here was that it was a volunteer project by people who actually cared about what they were developing, instead of just getting a fat paycheck and padding their CVs (more about that below).
Sometimes you just work at a scale where you don't need millions of writes/reads per second, given that your country literally doesn't have enough citizens for that sort of a load. You also don't always need to explore something like CQRS, pub-sub and event driven design and so on. Dare I say it, it would have probably worked just fine with Apache, PHP,…
Well, here's the link to the post-mortem of said "horror" story on the Roblox blog:
https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2021/
TL;DR The outage was caused by (a) they enabled a new streaming feature in Consul under unusualy high read-and-write load, (b) the load conditions triggered a pathological issue in the third-party BoltDB system upon which Consul relies and (c) all of that was exacerbated by having one consul cluster supporting multiple workloads.
I'd use quotes to catalog this as a "horror" story, because this was clearly a very specific issue triggered by a specific and complex set of circumstances.
The blogpost also mentions that Roblox worked closely with Hashicorp engineers to mitigate the issue, and work towards structural solutions; the post also affirms their choice to manage their infra themselves rather then moving into a public cloud solution.
Sure, Kubernetes covers loads of territory. But there definitely are niches where products like Consul & Nomad do add value.
ROBLOX | Senior/Principal Software Engineers | San Francisco Bay Area | Onsite Preferred | Full Time | http://www.roblox.com
The Studio Collaboration team at ROBLOX is looking for Engineers! We’re developing new ways to collaborate on art, assets, and code. We regularly release new versions and new tools to our pool of millions of active creators and directly iterate based on feedback from them.
We have opportunities for Game Engine collaboration using C++ and Lua and Back End Services using C#.
Join us in making collaborative game development possible for everyone! 52.2 Million Daily Active Users, 11.3 Billion Hours Engaged, 188 Countries, 32 Million Experiences, and 12 million active creators.
Learn More Here - https://blog.roblox.com/
Apply Here,
Principal: https://jobs.roblox.com/careers?gh_jid=3705571
Senior: https://jobs.roblox.com/careers?gh_jid=3528023
Principal Version Control: https://jobs.roblox.com/careers?gh_jid=4528630
https://blog.roblox.com/2008/02/the-future-of-open-source/
> I learned to program when I was 7. I started with LOGOWriter and QBASIC. What did I make? Games. It should be obvious. All kids want to write games. If your kid wants to write insurance software at age 7, you should stop wandering around aimlessly on the internet and find a good psychiatrist. Do it. Do it now.
> At its heart, ROBLOX is a game development platform. You can do a lot with it without writing a line of code. But if you really get into it, you’re going to want more power. You’re going to be very motivated to figure out how to program.
[...]
> I don’t care what fancy private school you send your kids to. The only place your 13 year old is going to encounter a PID-Controller is in ROBLOX’s Body(Position/Velocity/Thrust) objects, which can be used to script motion for parts and models. That’s just one example.
WhatsApp stored no data and ran everything as through service, that's why they did it with so few engineers.
Almost no games ever reach scale close to what you just wrote. The few that do are known with large technical teams.
Same goes to everything else. The reason why you can name them almost all is because it's so rare.
You are grossly overblowing the numbers. Only a very few services ever get to these numbers and once they do, they talk about the challenges of scale constantly [0] [1] [2]
[0] https://blog.discordapp.com/using-rust-to-scale-elixir-for-11-million-concurrent-users-c6f19fc029d3
[1] https://blog.roblox.com/2012/04/supporting-millions-of-players-at-roblox/
[2] http://highscalability.com/blog/2014/2/26/the-whatsapp-architecture-facebook-bought-for-19-billion.html
You won't find it.
Code grows, decisions get made, and practices are adopted.
I am working on my fourth iteration of what turns out to be my largest undertaking.
Step
1. Idea
2. Hack something
3. Commit, comment.
4. Hit a wall
5. Rewrite, commit, repeat.
6. Grow until new problems are discovered
7. Modules, tests, etc to avoid future problems.
8. New problems caused by complexity.
9. Go to 5.
Read this from Roblox's outage.
https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2021/
My recent example - I thought I could get away with 3 VMs to run my SaaS. Yes, I could, although my storage capacity guesses were 100x less than required.