The engineering diary for BitFunnel, an open source search engine based on algorithms from Microsoft's Bing.
Commenters point to it as an open source take on Bing algorithms and like the idea of a development diary alongside the code.
1. What's the license of TurboPFor,TurboRLE ?
GPL (see license in the source files).
2. Have you tried comparing TurboPFor that with roaring bitmaps ?
No, but because of the size overhead of roaring bitmaps they can be only used for few integer arrays.
For example when collecting intermediate results for queries. Whereas there are no size limits for TurboPFor functions (ex. millions of tiny or big arrays).
3. Have you tried if it makes sense putting it in lucene (on the postings list)?
Some expriments with SIMD posting lists in lucene are reporting ~20% gains.
see http://blog-archive.griddynamics.com/2015/06/lucene-simd-codec-benchmark-and-future.html
Posting lists are only one part in a search engine. The whole architecture must be taken into account,
like parallel indexing, parallel query processing, particularly intersections.
4. What forums would you suggest for low level stuff on compression,indexes,search-engine,db-engine,mechanical-sympathy ?
I'm not aware of special forums for SE, but there are excellent academic papers on this subject.
Some pointers: https://github.com/lemire, https://twitter.com/powturbo, arxiv.org, Papers about in memory database like about SAP-HANA,...
5. What do you work on in real-life? Or licensing the libs?
?
6. Is it possible to make lucene 10X+ faster and what would need to be done (in high level) ?
Multicores parallel shared nothing architecture like the on in the TurboPFor inverted index app and
a ram resident inverted…
1. What's the license of TurboPFor,TurboRLE ?
2. Have you tried comparing TurboPFor that with roaring bitmaps ?
3. Have you tried if it makes sense putting it in lucene (on the postings list)?
4. What forums would you suggest for low level stuff on compression,indexes,search-engine,db-engine,mechanical-sympathy ?
5. What do you work on in real-life? Or licensing the libs?
6. Is it possible to make lucene 10X+ faster and what would need to be done (in high level) ? (like what scylladb did to cassandra, like they didn't change the language but architecture explained here: http://www.scylladb.com/technology/architecture/)
7. What do you think of bing/bitfunnel using bloom-filters instead of posting-lists http://bitfunnel.org/strangeloop/ ?
8. And about http://research.microsoft.com/en-us/um/people/trishulc/papers/Maguro.pdf ? Distributing documents by 'term' instead of 'id' ?
I searched for GTGACCTTGGGCAAGTTACTTAACCTCTCTGTGCCTCAGTTTCCTCATCTGTAAAATGGGGATAATA and found another post by the same author on another website http://bitfunnel.org/strangeloop/
I've seen some other posts by Dan before but didn't know what kind of things he works on because I generally don't check that because if I did then I'd end up just checking people instead of reading what people were writing. Anyhow, from the article linked above -- for which a video is available at https://www.youtube.com/watch?v=80LKF2qph6I -- it turns out that Dan has done quite a bit of work with implementing search related technology, and I think that was interesting. His contributions to BitFunnel can be seen at https://github.com/BitFunnel/BitFunnel/graphs/contributors
Take a look at http://bitfunnel.org/, an open-source version of some of the algorithms from Bing. It's not ready for production use but you may enjoy following along in the implementation of a large search engine. The design notes and blog are also worth reading.
Anyone following IR related news, do you know what happened to BitFunnel (opensourced rewrite of Bing search engine).
https://bitfunnel.org/categories/blog/
https://github.com/bitfunnel/nativejit/
https://github.com/BitFunnel/BitFunnel
Bing uses bloom filters for the most-recent index:
https://dl.acm.org/doi/pdf/10.1145/3077136.3080789
https://bitfunnel.org/strangeloop/