Yelp Engineering Blog
Yelp's engineers write about infrastructure and open-source tools, including HAProxy, Docker init, data pipelines, and image compression.
What HN says
HN links it for tools like mrjob and PaaSTA and for posts on its data pipeline, image size reduction, and HAProxy reloads.
Stats for nerds
129HN people shared it
175linked in comments
2.6Kstory points
May 2025last shared
Top posts on HN
- 1Undebt: How We Refactored 3M Lines of Code2 people · 258 points · first shared 2016 · 138 comments on HN
- 2Introducing Yelp's GraphQL API1 person · 202 points · first shared 2017 · 51 comments on HN
- 3Billions of Messages a Day – Yelp's Real-Time Data Pipeline3 people · 177 points · first shared 2016 · 47 comments on HN
- 4Introducing dumb-init, an init system for Docker containers6 people · 169 points · first shared 2016 · 37 comments on HN
- 5Making Photos Smaller Without Quality Loss3 people · 154 points · first shared 2017 · 50 comments on HN
- 6True Zero-Downtime HAProxy Reloads11 people · 129 points · first shared 2015 · 49 comments on HN
- 7Open-sourcing Clusterman, a cluster autoscaler for Kubernetes and Mesos1 person · 109 points · first shared 2019 · 44 comments on HN
- 8Streaming Messages from Kafka into Redshift in Near Real-Time1 person · 104 points · first shared 2016 · 10 comments on HN
- 9Pyleus: An Open-source Framework for Building Storm Topologies in Pure Python1 person · 104 points · first shared 2014 · 5 comments on HN
- 10Yelp’s Style Guide1 person · 102 points · first shared 2014 · 18 comments on HN
What HN people wrote when they linked it
- GooseyOct 2016
(disclaimer: I work on the Data Pipeline project @ Yelp - but these opinions are my own and not necessarily representative of Yelp)
@jack9 - If I am understanding your point correctly, it is that you could just use S3 directly as the 'unified buffer' and the Kafka part is unnecessary. I'll try to shed some light on why we made Kafka part of this infrastructure.
- Regarding "near real-time" this is our cheeky way of saying we internally haven't set an SLA or a formal definition of the maximum latency in the system for us to declare it 'real time'. Hard numbers wise from the time a MySQL[0] event happens to a transformed version of it being in redshift is roughly in the realm of 10-30 seconds (unofficial and not a guarantee of course, just what I am anecdotally seeing at this point, and we are certain we can bring that down).
- As ora600 points out, we not only have multiple data sources, but also multiple data targets. So it's important to us to reduce the IN*OUT down to IN+OUT. It's important to note that nearly all of these connection points more naturally 'speak' in streaming than batch processing.
- Kafka has been fantastic tech for this use case for us: aside from this s3/redshift OUT connector we generally are dealing with connectors which want to be streaming. In-fact if it were performant to do so we would love to stream message-by-message into redshift as well (it's not).
- We already have a lot of infrastructure for batch processing - specifically a service for…
- jolynchOct 2015
The most significant issue with HAProxy and large SOAs is the lack of really good dynamic re-configuration. You don't want to have to manually change your configuration every time someone wants to add or change load balancing to a service, and if you do it automatically you run into HAProxy reloads being a reliability problem as they drop traffic on the floor due to Linux being goofy in how it implements SYN handling.
It looks like there is a lot of good advice on this thread for how to mitigate the configuration problem (bamboo, consul template, etc ...) but then you run into reloads, and even if you use zero downtime restarts as I described in my HAProxy deep dive (http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html), you still end up with periodic ~50ms blips in your timings.
We've found that Synapse (https://github.com/airbnb/synapse) is best in class at managing HAProxy for us in a fairly large SOA. It is really intelligent about doing as much as possible dynamically over the HAProxy stats socket as opposed to reloading for every configuration change, and we can finely tune how often we are reloading HAProxy. The best part is that it is fully pluggable. Use DNS, marathon, zookeeper, etcd, etc...: Synapse can manifest that service registration information into HAProxy configurations.
- justinc-yelpJul 2016
Most of the stream processing in the Data Pipeline happens inside of an internal project called PaaStorm, which is storm-like. It was built to take advantage of our platform as a service (http://engineeringblog.yelp.com/2015/11/introducing-paasta-an-open-platform-as-a-service.html), which handles process scheduling really well. Architecturally, it's pretty similar to Samza, with distributed processes communicating using Kafka.
We do use Spark streaming, and are starting to use Kafka Streams and Data Flow, where they're a better fit. I'm personally most excited about Beam/Flink. We'll probably end up replacing the PaaStorm internals with some other tool, when one with good python support matures. Beam's event-time handling and windowing seem really promising at this point. https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102 is a great overview of the different concerns for stream processing.
- thebostikDec 2018
This announcement makes sense to me, having done something similar before personally (1). The difference between a screenshot of a photo saved as a PNG and a JPG is huge, usually over 1MB, and I'm sure Twitter gets plenty of these.
Publishing their thresholds for the JPG vs PNG size comparison instead of referring to it just an internal metric would clear up some disconnect. Plus, I'd love to improve the static values I chose if there is better data available. I doubt the people working on this project wanted to change any PNG unless it's clear cut but the line has to go somewhere. These will be better handled by JPG.
Numbers about avg file size would be awesome too, how much data will this save in a typical user session?
1 - https://engineeringblog.yelp.com/2017/06/making-photos-smaller.html (Section: Large PNG Detection)
- JayeshSidhwaniApr 2017
I had asked a similar question on Quora [1] few years back. Here is the summary:
- Facebook: https://www.facebook.com/Engineering?sk=notes
- Google: http://google-engtools.blogspot.com/
- Twitter: http://engineering.twitter.com/
- Linkedin: http://engineering.linkedin.com/blog
- Dropbox: https://tech.dropbox.com/
- Instagram: http://instagram-engineering.tumblr.com/
- Netflix: http://techblog.netflix.com/
- Flickr: http://code.flickr.com
- Etsy: http://codeascraft.etsy.com/
- Yelp: http://engineeringblog.yelp.com/
- Heroku: https://blog.heroku.com/
- Airbnb: http://nerds.airbnb.com/
- Bitly: http://word.bitly.com/
- Evernote: https://blog.evernote.com/tech/
- Highscalability: http://highscalability.com/
[1] https://www.quora.com/What-are-the-companies-that-have-some-of-the-best-engineering-blogs
- SmerityJun 2015
I disagree with the "big data" line in the sand being whether "a kid doing a paper run could afford your servers". Using spot instances on AWS, you can have a 100 machine cluster with 1.5TB RAM for only $3 per hour.
Working at Common Crawl, some of the coolest projects I've seen have been side projects or weekend projects by interested volunteers. WikiReverse[1] cost only $64USD to parse the metadata of all 3.6 billion pages (even cheaper if you avoided EMR fees) whilst Yelp extracted 748 million US phone numbers from 2 billion pages for $10.60USD.
These days, big data (regardless of how you define the term) is within the reach of the kid with a paper route.
[1]: https://wikireverse.org/
[2]: http://engineeringblog.yelp.com/2015/03/analyzing-the-web-for-the-price-of-a-sandwich.html