Lyft Engineering is the ride-sharing company's engineering blog, covering microservices, data pipelines, infrastructure tooling and performance tuning.
Commenters share its posts on gevent tuning, Lyft Line matchmaking and IAM, and say good company blogs signal interesting work.
Lyft is hiring for all positions, including mobile, frontend, backend, infrastructure, and product roles. We're working on interesting and challenging engineering and product problems to improve people's lives with great transportation.
I'm specifically looking for several information security roles:
• An application security engineer (https://www.lyft.com/jobs/software-engineer-application-security) who’s excited about helping Lyft ship safe software quickly. You’ll help product and engineering teams at Lyft develop new features and products that are innovative and protect our users, build systems and tools to make it easy for engineers at Lyft to develop safe software by default, and promote security throughout the company.
• A detection & response engineer (https://www.lyft.com/jobs/software-engineer-security-detection-response) who’s excited about protecting Lyft and our users through swift detection of and response to security incidents. You’ll develop and automate techniques to hunt for and alert on attacker behavior with high signal-noise ratio, test our detection and response capabilities to improve our tools and processes, and act as a calm and rational incident commander when incidents do occur.
• A data privacy engineer (https://www.lyft.com/jobs/software-engineer-data-privacy) who’s excited about protecting our users by ensuring the data we collect is used appropriately without inadvertent or malicious misuse or disclosure. You’ll develop tools to enable…
We only use KubernetesOperators, but this has many downsides, and it's very clearly a 2nd thought of the Airflow project. It creates confusion because users of Airflow expect features A, B, and C, and when using KubernetesOperators they aren't functional because your biz logic is separated. Eg., if your biz logic knows what S3 it talks to in an external task, how can Airflow? So now its Dataset feature is useless.
There are a number of blog posts echoing a similar critique[1].
Using KubernetesOperators creates a lot of wrong abstractions, impedes testability, and makes Airflow as a whole a pretty overkill system just to monitor external tasks. At that point, you should have just had your orchestration in client code to begin with, and many other frameworks made this correct division between client and server. That would also make it easier to support multiple languages.
According to their README: https://github.com/apache/airflow#approach-to-dependencies-of-airflow
> Airflow has a lot of dependencies - direct and transitive > The important dependencies are: SQLAlchemy, Alembic, Flask, werkzeug, celery, kubernetes
Why should biz logic that just needs to run Spark and interact with S3 now need to run a web server?
[1] Anecdotes from various posts - https://medium.com/bluecore-engineering/were-all-using-airflow-wrong-and-how-to-fix-it-a56f14cb0753 - https://eng.lyft.com/orchestrating-data-pipelines-at-lyft-comparing-flyte-and-airflow-72c40d143aad -…
I definitely become more attracted to companies with high quality blog posts on things I'm interested in. The right blog is a really big signal that I'd be interested in the work at the company.
Besides picking the next company, great blog posts can be a real help in sanity checking the work that's happening in our team. I've noticed even senior engineers treating good blog posts from big companies as "See, we're not crazy".
Right now I'm operating in the data science platform space at my company, and have been reading a lot more company blogs than usual. Wayfair is a company I now know of after an engineer I really respect shared on Twitter "Bayesian Product Ranking at Wayfair"[1].
Lyft, on the other hand, disappointed me with "How Lyft Designs the Machine Learning Software Engineering Interview"[2]. That post sounds like it could have been victim to what Dan notes as:
> Approval/editing process mainly de-risks posts, removes references to specifics, makes posts vaguer and less interesting to engineers
1. https://tech.wayfair.com/data-science/2020/01/bayesian-product-ranking-at-wayfair/
2. https://eng.lyft.com/how-lyft-designs-the-machine-learning-software-engineering-interview-bbbb9fc8bb28
I'm losing count of all the metadata management systems being open sourced. First there was Wherehows out of LinkedIn [1]. Then there was Amundsen out of Lyft, which is now stemma.io [2]. Then there was Marquez out of WeWork [3]. Now we have DataHub out of LI. It's a little crazy how underserved this problem was several years ago and how over served it's becoming.
I use Amundsen and am pretty happy with it.
[1] https://engineering.linkedin.com/blog/2016/03/open-sourcing-wherehows--a-data-discovery-and-lineage-portal
[2] https://eng.lyft.com/amundsen-lyfts-data-discovery-metadata-engine-62d27254fbb9?gi=2f233e51dc79
[3] https://marquezproject.github.io/marquez/
It would appear that Lyft is making some strong ties with Google as of late. Between this and the announcement of the Lyft partnership with IBM and Google to work on some sort of Kubernetes router mesh [1] I am starting to see a trend.
"We are excited to announce that we are working in partnership with both Google and IBM to bring Envoy to Kubernetes. Fun fact: there are now more people working on Envoy at Google than there are at Lyft! We have a lot of other things planned with Google that we will be able to share more about in the coming months."
[1] https://eng.lyft.com/envoy-7-months-later-41986c2fd443
Thanks for the feedback! One of the creators here :)
Our initial thought is to create a solution similar to what Lyft has done (https://eng.lyft.com/scaling-productivity-on-microservices-at-lyft-part-1-a2f5d9a77813?gi=e9ed3e63c4ee) to enable using staging as a dev environment and sharing it with many developers at once.
We do want people to use at us production later on, but right now we know there might be compliance/security issues.