Algolia Blog

Search is a fascinating topic because it's such a fundamental problem and every search engine is based around the same extremely simple data structure (Posting list/inverted index). Despite that, search isn't easy and every search engine seems to be quite unique. It also seems to get exponentially harder with scale.

You can write your own search engine that will perform very well on a surprisingly large amount of data, even doing naive full-text search. A search tool I came across a while back is a great example of something at that scale: https://pagefind.app/.

For anyone who doesn't know anything about search I highly recommend reading this (It's mentioned in the blog post as well): https://swtch.com/~rsc/regexp/regexp4.html.

Algolia also has a series of blog posts describing how their search engine works: https://www.algolia.com/blog/engineering/inside-the-algolia-engine-part-1-indexing-vs-search/.

---

It's interesting that GitHub seems to have quite a few shards. Algolia basically has a monolithic architecture with 3 different hosts which replicate data and they embed their search engine in Nginx:

"Our search engine is a C++ module which is directly embedded inside Nginx. So when the query enters Nginx, we directly run it through the search engine and send it back to the client."

I'm guessing GitHub probably doesn't store repos in a custom binary format like Algolia does though:

"Each index is a binary file in our own format. We put the information in a specific…

Hi naiv,

Nick Vlku, VP of Growth at Algolia, here. Wanted to dive in a bit more on Ben's answer below. To be honest, I was surprised at this limitation too when I first starting using Algolia. Upon some research, it turns out this is really complicated and Algolia errs on doing what yields the best experience for end users (even if it might require a little bit more work).

Basically, we are trying to avoid the phenomenon when you apply a filter search (like "sort by price") and returning something that has nothing to do with your query or intent. For example, a customer searching for an "Eames chair" and then sorting by lowest price would return things like $2 postcards of Eames chairs.

So, in order to capture the right kind of intent for that sort (and also allow our customers to further customize results), we require a separate replica of the data. (Maybe if somebody sorts for cheapest Eames chair, you would bump up a similar cheaper chair by another company instead of a postcard?)

The good news is, for most of these cases, there is no additional cost. You basically can create a virtual replica which points to your same records with different ranking strategies. So no doubling up the records! See here for info on this: https://www.algolia.com/doc/guides/managing-results/refine-results/sorting/in-depth/replicas/#what-are-virtual-replicas

It's a balancing act between "magical sorting just working" and "doing what looks right for the customer". We landed on this…

Algolia | multiple roles | SF, Paris, NY, Atlanta, London | Onsite only, Full time

Hey HN, at Algolia we are facing a major challenge of scaling our team to follow our growth and our series B investment. We are currently looking for multiple dev roles, but looking for other areas such as marketing and sales as well :)

We are a hosted search API, that allows anyone to quickly build good search within hours. https://www.algolia.com/

Responsibilities depend on the role, but we normally have teams that are responsible for different areas like the dashboard, infrastructure, our open source libs etc.

Technical requirements are as per role. You don't need to be a superstar, but it helps if you want to become really good and enjoy the environment.

I'm biased, I work there, but I can say that the past year has been good to me both technically and life wise (I relocated to Paris). My co-workers are extremely nice and knowledgable people who seek to bring the best out of everyone. My colleague wrote an article about his experience -
https://medium.com/@liamboogar/celebrating-one-year-at-algolia-bc22c8c76bb3

As usual, there are job perks but I'll let those be discussed if you apply -> https://www.algolia.com/careers

Natural language search and answer generation _are_ completely separate.

Search is often (but not exclusively) performed using cosine similarity over semantic vectors. Such vectors are produced using embedding models, which represent the meaning of the document via an arbitrary length vector called an embedding, and 768 is a common vector size for this.

You calculate the embedding for all documents in your database ahead of time (during insertion), and you calculate the embedding for the user's query, then search for documents closest to the query using a similarity metric of choice such as cosine.

Nothing prevents you from serving documents found this way directly, instead of using them to generate answers. Part of the Google search pipeline involves something like this. Many full-text search products also do this (Algolia is such an example https://www.algolia.com/blog/ai/what-is-vector-search/)

LLMs and generation are used to synthesize the answer and tie it back into the question using a layer of soft judgment based on the LLMs prior knowledge. This does work out great in some contexts, less so in others, as you pointed out. But these components aren't coupled in any way.

Disclaimer: I work at Algolia (https://www.algolia.com/), a hosted search as a service API.

While I agree that building a great and relevant search experience with a Lucene-based engine requires lots of extra time and effort to get right, there are other non-TFIDF based solutions that provide a much faster path to great relevance with far less effort (https://blog.algolia.com/inside-the-algolia-engine-part-1-indexing-vs-search/), and it's possible to have semantic ranking without too much machine learning (https://blog.algolia.com/promote-search-results-query-rules/). Not to discount the value of machine learning - we're finding that for specific usecases ML can be a very valuable way to help surface more pertinent content for individuals based on their profile/preferences etc. (https://blog.algolia.com/personalization-announcement/).

This may be along the lines of what you mentioned around "commoditizing" complex traditional search workflows. I'd be curious to hear more about what kind of use-cases you think are trickiest without SS.

Algolia (YC W14) | San Francisco | Paris | Full-Time On-Site

Frontend, Full-Stack, iOS Engineers; Site Reliability Engineer; Customer Solutions Engineer (Paris)

Customer Solutions Engineer (SF)

Algolia is "search at the speed of thought." A hosted search service, that hits the sweet spot between "wrestle for months with Elastic" and "just throw some subpar search up." We power the search for sites and apps like Genius, Periscope, and many, many others.

Relocation and visa sponsorship (Paris) can potentially be offered for the right person.

The hiring process differs between teams, but involves a small number (around 2) of video calls, followed by a take-home test, and a day on-site.

More information and can apply here: https://www.algolia.com/careers. Also feel free to email jen (for SF) or marion (for Paris) @algolia.com.