Slack Engineering

I was Chief Architect at Slack from 2016 to 2020, and was privileged to work with the engineers who were doing the work of migrating to Vitess in that timeframe.

The assumption that tenants are perfectly isolated is actually the original sin of early Slack infrastructure that we adopted Vitess to migrate away from. From some earlier features in the Enterprise product (which joins lots of "little Slacks" into a corporate-wide entity) to more post-modern features like Slack Connect (https://slack.com/help/articles/1500001422062-Start-a-direct-message-with-someone-outside-your-company) or Network Shared Channels (https://slack.com/blog/news/shared-channels-growth-innovation), the idea that each tenant is fully isolated was increasingly false.

Vitess is a meta-layer on top of MySQL shards that asks, per table, which key to shard on. It then uses that information to maintain some distributed indexes of its own, and to plan the occasional scatter/gather query appropriately. In practice, simply migrating code from our application-sharded, per-tenant old way into the differently-sharded Vitess storage system was not a simple matter of pointing to a new database; we had to change data access patterns to avoid large fan-out reads and writes. The team did a great write-up about it here: https://slack.engineering/scaling-datastores-at-slack-with-vitess/

I contacted support to opt out.. Here is the answer.

"Hi there,

Thank you for reaching out to Slack support. Your opt-out request has been completed.

For clarity, Slack has platform-level machine learning models for things like channel and emoji recommendations and search results. We do not build or train these models in such a way that they could learn, memorize, or be able to reproduce some part of customer data. Our published policies cover those here (https://slack.com/trust/data-management/privacy-principles), and as shared above your opt out request has been processed.

Slack AI is a separately purchased add-on that uses Large Language Models (LLMs) but does not train those LLMs on customer data. Slack AI uses LLMs hosted directly within Slack’s AWS infrastructure, so that customer data remains in-house and is not shared with any LLM provider. This ensures that Customer Data stays in that organization’s control and exclusively for that organization’s use. You can read more about how we’ve built Slack AI to be secure and private here: https://slack.engineering/how-we-built-slack-ai-to-be-secure-and-private/.

Kind regards, Best regards,"

Good luck folks! I'm glad there are projects trying to solve enterprise search.

I guess the main problem is the "private" aspect, if I've understood your goals correctly. Since most SaaS products lock down the private data unless you pay enterprise fees for compliance tooling.

For instance, if you want to ingest data from private Slack channels or Notion groups, you have to get the users in those groups to add your bot to them, otherwise there's no way of your service getting access to the data. It's possible, just a bad UX for users.

That said, built-in search for most SaaS products built after 2015 is generally quite good (e.g. Slack has an internal Learning to Rank service for a while now, which makes their search excellent: https://slack.engineering/search-at-slack/), so you'd be solving for companies like Webex and Confluence where their internal search is not great. At companies like Google they have internal search across products, which is the ideal end state, but have the benefit that they own the source code for most of their internal products.

Super interesting post. Following blog links, the timeline in https://slack.engineering/all-hands-on-deck-91d6986c3ee also offers a look at the play by play.

However, as far as I can read it, they have somewhat different views on the root cause?

"Soon, it became clear we had stale HAProxy configuration files, as a result of linting errors preventing re-rendering of the configuration."

vs.

"The program which synced the host list generated by consul template with the HAProxy server state had a bug. It always attempted to find a slot for new webapp instances before it freed slots taken up by old webapp instances that were no longer running. This program began to fail and exit early because it was unable to find any empty slots, meaning that the running HAProxy instances weren’t getting their state updated. As the day passed and the webapp autoscaling group scaled up and down, the list of backends in the HAProxy state became more and more stale."

Maybe a combination of the two?

Use the language your team is familiar with! If it is PHP don't hesitate just because it is not "cool"..

PHP was a great tool back in the day and today!

PHP with a decent framework like Laravel [1] or Zend Framework [2] really works well and addresses all the features that you have listed (except the last 2).

Uniquely among modern languages, * PHP was born in a web server *. Its strengths are tightly coupled to the context of request-oriented, server-side execution.

We've managed to develop several complex apps over the last 10 years on PHP. For Web Socket support, we use Socket.IO or Firebase and it both really well together with PHP.

If you choose PHP now, you are not alone. Many companies still choose PHP for their Web Application. Slack uses PHP for most of its server-side application logic [3]

[1] https://laravel.com/
[2] https://framework.zend.com/
[3] https://slack.engineering/taking-php-seriously-cf7a60065329

Yeah, although mostly variations on that basic premise.

Airbnb Himeji: https://medium.com/airbnb-engineering/himeji-a-scalable-centralized-system-for-authorization-at-airbnb-341664924574
Carta's AuthZ system: https://medium.com/building-carta/authz-cartas-highly-scalable-permissions-system-782a7f2c840f
Slack's architecture is a bit different, but solves some of the same challenges: https://slack.engineering/role-management-at-slack/

I've also talked to a number of teams who just implemented pattern 3 internally with a custom service. Generally they've determined it's worth it to centralize all authorization data (like roles, groups, etc) into one place and perform ALL permission checks there.

There are also some companies building essentially Zanzibar clones, like Auth0, Authzed, Ory Keto, and a few more.