Spotify Engineering

Disclaimer, I'm a developer working on the Vespa.ai project. One reason is that we simply don't have DevRel teams or marketing teams, but we still have decent interest, from large companies like Spotify using Vespa in production for semantic search (at scale) https://engineering.atspotify.com/2022/03/introducing-natural-language-search-for-podcast-episodes/

On topic: IMHO a better title of the blog post would be:

Do you actually need to enable approximate vector search?

Which is an excellent question, because, introducing approximation also introduces accuracy degradation, plus that you need an ANN algorithm, which also has tradeoffs.

In Vespa, you can choose between exact and approximate, with related tradeoffs such as recall (accuracy), search speed, indexing throughput and resource footprint. Just start easy with exact search, then it's just change the schema definition of the tensor field to include `index` and Vespa will build the necessary data structures for enabling approximation.

At my last company I helped build the experimentations platform that processes millions of requests per day. I have some thoughts:

- The most useful resource we've found was from Spotify, of all places: https://engineering.atspotify.com/category/data-science/

- For hashing, md5 hash on (user-id + a/b-test-id) is sufficient. In practice we had no issues with split bias. You should not get too clever with hashing. You'll want to stick to something reliable and widely supported to make any post-experiment analysis easier. You definitely want to log the user-id to version mapping somewhere.

- As for in-house vs external, I would probably go in-house, though that depends on the system you're A/B testing. In practice the amount of work needed to integrate a third-party tool was roughly the same as building the platform, but building the platform meant we could test more bespoke features.

>"Spotify was a stand-out, almost no large tech company bought into cloud like they did, and everyone said the same thing: "It's not our core competence, it would require more people".

I'm not sure what you mean by this as Spotify was actually quite late buying into cloud. They ran on bare metal through at least 2015. From their engineering blog:

">Thus, in early 2015 we started exploring what a cloud strategy would look like for Spotify."[1]

So they would have been running on bare metal through some of the biggest spikes in growth i.e launching in the US market. I don't think they bought in any more or any less than anyone else who transitioned from bare metal to cloud for the same reasons.

[1] https://engineering.atspotify.com/2019/12/views-from-the-cloud-a-history-of-spotifys-journey-to-the-cloud-part-1-2/

> especially from tech company blogs,

https://engineering.fb.com/

https://netflixtechblog.com/

https://stripe.com/blog/engineering

https://eng.uber.com

https://engineering.linkedin.com/

https://engineering.atspotify.com/

https://tailscale.com/blog

https://careersatdoordash.com/engineering-blog/

https://dropbox.tech/

--

Aggregators:( https://engineering.fyi/ ; https://diff.blog/ )

+ https://hn.algolia.com/?query=engineering%20blog

---

create a public engineering-blog SKILL.md.
( ~ collect the writing patterns that work on HN )

Regarding inventorization of assets: my team also did the same at Spotify last year for Google Cloud Platform.
https://engineering.atspotify.com/2019/06/04/painting-a-picture-of-your-infrastructure-in-minutes/

At the time Cartography support for GCP was lacking, so we just ingested Cloud Asset Inventory data in neo4j.

If you are on GCP I highly suggest using CAI as a backbone for IAM security controls: hands down the best service on the platform for security teams.

Spotify Desktop is kinda different from the website because of how it uses APIs locally to control things https://engineering.atspotify.com/2021/04/07/building-the-future-of-our-desktop-apps/

VSCode is also appreciably different both in capability and implementation, https://franz-ajit.medium.com/understanding-visual-studio-code-architecture-5fc411fca07 but interesting that V8 debug protocol is becoming more common, so maybe that will come to a browser some day :)