OpenSource Connections consultants write about search relevance, Solr and Elasticsearch, BM25, BERT, and algorithms like HyperLogLog.
HN commenters point to its posts as strong comparisons of Solr and Elasticsearch, and recommend its free Relevance Slack community.
People have been predicting the death of keyword search for at least as long as I've been working in the field (23 years and counting!). We've seen outrageous marketing claims, buzzwords like "concept search", "insight engines", many new companies promising a step change in search quality, some of which are still in business but many who shone briefly then vanished. The concept of an easy to use, fire-and-forget, scalable search engine that gives you great relevance out of the box isn't new. Yet the bag-of-words model remains the standard across the sector, it's well understood with many powerful and scalable open & closed source implementation options.
To make search work in practice however is hard. It's as much about process and people as it is about technology: many companies aren't even measuring search quality, recording search issues correctly or have an active search team (bigger than one poor overworked search person). No matter how clever the tech, these problems aren't going away: they're compounded by bad source data quality, misunderstandings of user search intent and bad search UX. Martin White, author of many books on search, describes search as a 'wicked problem'. Getting all these parts working in harmony so you can truly own your search is what we do here at OSC and it takes time, investment and commitment.
I think Vectara is very interesting and the people involved have impressive track records (there's also some other great engines like Vespa, Pinecone,…
In 2015, they evaluated Apache Solr and Elasticsearch and decided to build their own (Firefly). They said, other solutions did not scale. So, instead of contributing to scaling (like Apple and Bloomberg and Cloudera did), they went the other way. Now, they seem to be doing it again (at least they are using Tika).
In a meanwhile, Solr implemented most of the features they are describing in their architecture document.
Specifically:
1)General scaling: https://lucene.apache.org/solr/guide/7_5/introduction-to-scaling-and-distribution.html (using ZooKeeper and SolrCloud)
2) Search Ranking and click-data training: https://lucene.apache.org/solr/guide/7_5/learning-to-rank.html (Contributed by Bloomberg)
3) Offline builds with substitution into production: https://lucene.apache.org/solr/guide/7_5/collections-api.html#createalias
4) Near-Real-Time: https://lucene.apache.org/solr/guide/7_5/near-real-time-searching.html
5) Sharding specifically: https://lucene.apache.org/solr/guide/7_5/shards-and-indexing-data-in-solrcloud.html
6) Extraction pipeline, they are doing all together. We have:
a) pre-Solr extraction (usually done in a stand-alone client, though we do include Tika and DataImportHandler for quick start),
b) in-Solr pre-schema processing with Update Request Processors https://lucene.apache.org/solr/guide/7_5/update-request-processors.html
c) Actual per-field text processing pipelines, separate both for index and query (they call query part later "query…
(I'm working at Algolia) The answer I'm used to provide when people ask me this question is the one our CTO posted on Quora a few years ago:
https://www.quora.com/How-does-Elasticsearch-relate-and-or-compare-to-Algolias-Search-as-a-service
tldr; Algolia's focus is on consumer-grade & user-facing search (searchbar like you would have on Amazon, IMDB, Twitch, Medium.com, quora, ...) while ES's sweet spot is on indexing billions of documents (or lines of logs) and provide rich dashboarding & aggregation capabilities (like a monitoring dashboard with their ELK stack). Recently, ES released a beta of "App Search" which seems to target more and more the user-facing search bars.
Here are also a few resources to help get you an idea:
* from Doug Turnbull, author of "Relevant Search" & ES expert: https://opensourceconnections.com/blog/2016/06/01/thoughts-on-algolia/
* from an Algolia & ES user: https://medium.com/@matayoshi.mariano/elasticsearch-vs-algolia-96364f5567a3
* from the Algolia FAQ: https://www.algolia.com/doc/faq/why/what-makes-algolia-different-than-elasticsearch-or-solr/
* from the Algolia blog: https://blog.algolia.com/?s=comparing+algolia+elasticsearch
HyperLogLog is an algorithm that allows you to find an estimate of the distinct elements of a dataset with very little memory. Here's a simple explanation:
http://opensourceconnections.com/blog/2015/02/04/its-log-its-log-its-big-its-hyper-its-good/
HyperLogSandwich solves a similar problem. From the README:
> Unlike the HyperLogLog, which estimates the cardinality of unique items in a set, and unlike a CountMinSketch, which estimates the frequency of a specific item, the HyperLogSandwich estimates the number of unique items that have occurred for any frequency.
Real world examples include:
- How many people viewed my Tweets 4+ times this month?
- How many users watched this video 2 times this week?
- How many users visited my website 3 times during an arbitrary time range?
- How many log exceptions happened 10+ times last week?
https://opensourceconnections.com/blog/2023/02/27/hybrid-vigor-winning-at-hybrid-search/, https://jina.ai/news/hype-and-hybrids-multimodal-search-means-more-than-keywords-and-vectors-2/ are two I've been looking at
Basically, get the BM25 results and normalize them to be between 0 and 1, then take the (potentially weighted) average of them and the cosine similarity results (already between 0 and 1) to get the final ranking.
BTW, I highly recommend the Relevance Slack as a good space to discuss this and search-related topics generally. It's run by OpenSource Connections, but it's free and anyone can join, even folks who compete with them. It's the best community I have found for search developers.
https://opensourceconnections.com/blog/2021/07/06/building-the-search-community-with-relevance-slack/