LinkedIn Engineering

There seems to be a trend where companies are creating tools to monitor data quality...

* Great Expectations: https://github.com/great-expectations/great_expectations

~2 years old, does data profiling

* Data Sentinel: https://engineering.linkedin.com/blog/2020/data-sentinel-automating-data-validation

It's not open sourced however.

* Data Anomaly Detector: https://www.slideshare.net/iblaine/using-airflow-for-tools-development

Something I built at One Medical that is not open sourced...but this deck has some interesting concepts in it. The approach here is to minimize the effort, lines of code, number of clicks needed, to test your data.

* Amundsen: https://github.com/lyft/amundsen

This is open sourced, and some contributes are building features for data quality analysis. Amundsen seems like it has a bright future.

> now they're stumped as to how they're supposed to quickly get data out

I'd argue that (given a large enough business) "reporting" ought to be its own own software unit (code, database, etc.) which is responsible for taking in data-flows from other services and storing them into whatever form happens to be best for the needs of report-runners. Rather than wandering auditor-sysadmins, they're mostly customers of another system.

When it comes to a complicated ecosystem of many idiosyncratic services, this article may be handy: "The Log: What every software engineer should know about real-time data's unifying abstraction"

[0] https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying

That blog post is a great read as well. Truely, the log abstraction [1] and "Turning the DB inside out" [2] have been hugely influential.

In a way this article here suggests to extend that

(1) from a log that represents data (upserts, cdc, etc.) to a log of coordination commands (update this, acquire that log, journal that steo)

(2) have a way to link the events related to a broader operation (handler execution) together

(3) make the log aware of handler execution (better yet, put it in charge), so you can automatically fence outdated executions

[1] https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying

Facebook definitely does it. Thats where we first learned about this way of working. Some bigger tech companies have actually written about it (see links below) but a list of ones that we've seen with this kind of infra include LinkedIn, Shopify, Dropbox, Stripe, Slack, DataDog, and Palantir - to name a few.

Eng blog links:

LinkedIn: https://engineering.linkedin.com/blog/2021/building-in-the-cloud-with-remote-development

Shopify: https://shopify.engineering/modelling-developer-infrastructure-teams

Slack: https://slack.engineering/development-environments-at-slack/

Doximity: https://technology.doximity.com/articles/github-codespaces-is-great-but-mutagen-is-better

The web part of it is pretty easy to scale. You simply add more web servers. The problem is in the storage/db layer. I imagine the feed was the primary challenge scaling Twitter.

Other components such as the search were probably also quite tricky. One thing i've never figured out is how Facebook handles search on your timeline. That a seriously complex problem.

Linkedin Recently published a bunch of papers about their feed tech:

https://engineering.linkedin.com/blog/2016/03/followfeed--linkedin-s-feed-made-faster-and-smarter

And Stream's stackshare is also interesting: https://stackshare.io/stream/stream-and-go-news-feeds-for-over-300-million-end-users

Nothing in that diagram has anything to do with Kafka, it's just showing that you can connect different systems together using a single message bus. If you are new to that concept then I recommend reading the seminal blog post about using logs as the data backbone: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying

After that, you can choose whichever implementation you want, of which Kafka is just one option. And yes it does require some work to setup and run, as most software does, but it's no more or less than any other distributed system at this point.