Instagram's engineering team writes about running Python and Django at scale, including sharding, garbage collection, and static analysis.
HN commenters cite it as evidence of Python at scale, noting the tricks Instagram needed, such as copy-on-write GC and static analysis.
"Ideas that diverged too much from the existing successful tech had a huge obstacle in their way: they had to recreate all of the existing solutions in their new model.”
You’re so right! There are great examples of companies that preserved the existing ecosystem of solutions (Facebook with PHP, Microsoft with C++, Google with Java and C++, maybe also Amazon with Perl 5 and C++).
There are also great examples of successful companies that in a certain sense did or had to do things in an entirely new model (WhatsApp with the Open Telecom Platform, F5 Networks with their data center FPGA-powered hardware load balancers, Tesla and SpaceX, and so on).
There are also great examples of companies that did both, where they preserved an existing ecosystem but also added another ecosystem on top to fix flaws (NeXT Computer with Smalltalk and C, Stripe with the modern version of that as Ruby and Go, Twitter with Ruby and Java, so on).
Microsoft is the best example of a company that can succeed even with what seems like a crazy choice. Almost everything being written in C++ sounds insane to the point of being business suicidal, but they worked hard enough to make it work. If Microsoft can succeed in the ways that they have with C++, and if Facebook could succeed for as long as they did with PHP, and Instagram can allegedly succeed with Django[1], then anyone can succeed with anything as long as you can endure the stress that a seemingly peculiar decision might cause you. And those…
Color is really, really hard.
I love the clarity of black-and-white imaging, but the thing that keeps me coming back to it more than anything else is the ability to successfully render my intended image on more electronic platforms.
When working in color, I either gravitate toward images with unambiguous color or accept the fact that only those with calibrated screens will see what I intended to create. Moreover, those calibrated screens need to be calibrated to the same reference I'm using, or they'll see something different.
Even with prints, without controlling the viewer's ambient light, you'll also get different perception. It is really hard.
Agreed that debugging is hard, too. I've recently come across the notion of 'canary images' [1] , which can make some of the debugging effort into binary search.
[1] https://instagram-engineering.com/bringing-wide-color-to-instagram-5a5481802d7d
The primary reason is of course encapsulation. Many things you don't want to leave publically accessible. For instance if you're creating a thread safe api, you need to be mindful of references/mutexes/locks which can easily be enforced with wrappers. If you expose the underlying objects directly you could easily dead lock, corrupt memory, create race conditions, etc. It also greatly simplifies testing and guarantees/provability since you can guard against access patterns or certain inputs or states.
Personally, I have run into enough horror caused by mutating a global state that I write code which is quite strict about what it exposes. If you're a python dev, I'd suggest reading this (posted on hn recently but I can't find the hn link currently): https://instagram-engineering.com/python-at-scale-strict-modules-c0bb9245c834
I had asked a similar question on Quora [1] few years back. Here is the summary:
- Facebook: https://www.facebook.com/Engineering?sk=notes
- Google: http://google-engtools.blogspot.com/
- Twitter: http://engineering.twitter.com/
- Linkedin: http://engineering.linkedin.com/blog
- Dropbox: https://tech.dropbox.com/
- Instagram: http://instagram-engineering.tumblr.com/
- Netflix: http://techblog.netflix.com/
- Flickr: http://code.flickr.com
- Etsy: http://codeascraft.etsy.com/
- Yelp: http://engineeringblog.yelp.com/
- Heroku: https://blog.heroku.com/
- Airbnb: http://nerds.airbnb.com/
- Bitly: http://word.bitly.com/
- Evernote: https://blog.evernote.com/tech/
- Highscalability: http://highscalability.com/
[1] https://www.quora.com/What-are-the-companies-that-have-some-of-the-best-engineering-blogs
Instagram's backend is a Django monolith "with several million lines of code and a few thousand Django endpoints" as of 2019, and the same architecture was used for Threads! They now run Django on a JIT-enabled fork of CPython, and leverage a ton of internal tooling for static analysis and strong typing.
https://instagram-engineering.com/types-for-python-http-apis-an-instagram-story-d3c3a207fdb7
https://engineering.fb.com/2023/09/07/culture/threads-inside-story-metas-newest-social-app/
https://engineering.fb.com/2022/05/02/open-source/cinder-jits-instagram/
https://engineering.fb.com/?s=django
Django is also excellent for e-commerce with projects like https://github.com/saleor/saleor - or you can roll your own e-commerce system quite easily, which we've done at my travel startup.
If you know which type of uuid you have (v1, v4, etc) then you can take a look at how many bits of randomness it has, how many total items you have, and compute the probability of a collision if you just take a subset of the bits and use that as an ID.
In theory it's definitely possible. The 128 bits you get in a UUID is a LOT of randomness for an identifier. Postgres BIGINTs are just 64 bits. Instagram's sharded IDs are just 64 bits. (See below.)
You can test it. If you're using uuidv4 (which is 100% random bits, minus a few for the version), you could make a new column in your table in Snowflake, populate it with the first 64 random bits of your existing uuid column, then see if you have any collisions.
https://instagram-engineering.com/sharding-ids-at-instagram-1cf5a71e5a5c