Sujit Pal

Hello, author of the post here. Thanks for all your valuable comments. I noticed that there was some talk about neural network driven models. In my follow up post to this one:

http://sujitpal.blogspot.com/2016/06/comparison-of-image-search-performance.html

I describe how I use transfer learning to generate image vectors for my butterfly images from the Caffe reference model trained on ImageNet. There are some other approaches too, but probably not as interesting to this audience.

@GrantS - thanks for the intro and the links, and for pointing out that I was using BoVW incorrectly. Not an image person, trying to get into it from a search/NLP background.

@infinitone and @deckar01 - I measured the goodness of search by checking how many times the query butterfly was the #1 result. I agree its pretty hard to distinguish butterflies from one another, but I needed something to work with, and that was as good as any. I did not want one that crosses subject areas, such as flowers and cars, so a search for a red flower might bring back a red car.

@chrischen - if you got rid of the bucketing portion in my pipeline, I think you might achieve what you are after. But if you are only looking for exact match on color, then hashing might be cheaper.

@rcarmo - thanks for the reference to pHash, I think it might work better in near-duplicate detection kind of cases. Will check it out.

"Asynchronious message passing is old and great idea. Instead of synchronizing objects and dealing with deadlocks we try to isolate objects and let them exchange messages. Erlang proved it to be extremly powerful tool applicable to wide range of highly scalable mission critical applications.

In JVM landscape big interest to this approach started after introducing Scala actors. "

That statement is so very poorly informed! JMS ("surprise"?) has been a sign of "interest" in "Asynchronious message passing" in the "JVM landscape".

I am not up on my Scala, but the author fails to convey (based on what I do know ;) the competence level that would lend weight to his attempt to address (much less support) the topic.

For those interested in other JVM alternatives, Kilim seems quite promising and seems to hold its own quite well in benchmarks against Erlang.

http://www.malhar.net/sriram/kilim/

http://www.youtube.com/watch?v=37NaHRE0Sqw

http://sujitpal.blogspot.com/2009/01/more-java-actor-frameworks-compared.html [scroll to end to see the impressive results for Kilim.]

http://osl.cs.uiuc.edu/docs/pppj09/paper.pdf

[edit: added pdf link]

(Author here) Case in point where Elasticsearch shines is one of yesterday's hacker news articles

http://sujitpal.blogspot.com/2016/05/elasticsearch-based-image-search-using.html?m=1

Here Elasticsearch is used as a framework where "search" really means some kind of distributed feature similarity. But even text search where your incorporating a lot of data science, external signals, or semantic awareness can fall more into the category of distributed feature similarity.

Algolias strength is in a simpler path to straightforward and easily understood search. I'm not sure it's the path to amazing and deeply customized search (or search-driven features). Algolia gets far more right out of the box at a configuration level many need when they can't put a team around building an amazing Solr/ES experience.

My hope is Solr/ES can learn a thing or two in the ease of use dept with relevance!

Sure:

My algorithm is something I made up, and from memory it works like this:

1) Remove HTML, stem, remove stopwords etc

2) Sort unique words by popularity in the text

3) Split the original text on sentence boundaries.

4) Include each sentence that first mentions the next most popular word, until the summary is the maximum length requested.

http://news.ycombinator.com/item?id=1803020

Googling turns up http://sujitpal.blogspot.com.au/2009/02/summarization-with-lucene.html which compares a few approaches.

Edit: Also http://stackoverflow.com/questions/2829303/given-a-document-select-a-relevant-snippet, which I think is by Dn_Ab who wrote the OP.