Mining of massive datasets

What Hacker News says

1. samscullyMar 2023
   The book Mining of Massive Datasets [1] has useful information on building an efficient similarity index using Jaccard/minhash. I would also recommend Otmar Ertl's papers on extensions of minhash that approximate Jaccard better in certain situations, e.g. superminhash [2].

   [1] http://www.mmds.org/ Chapter 3 [2] https://arxiv.org/abs/1706.05698

   thread
2. mindcrimeNov 2021
   Maybe Clever Algorithms: Nature-Inspired Programming Recipes[1]. These techniques might not be useful to everybody, all the time, but they can be very handy in places.

   Edit:

   A couple of other suggestions.

   Managing Gigabytes: Compressing and Indexing Documents and Images[2]

   Mining of Massive Datasets[3]

   Algorithm Design Manual[4]

   Network Algorithmics[5]

   Neural Network Design[6]

   I think all of these fall into the category of "Won't be applicable to everyone, but can be good for those who need this kind of stuff."

   [1]: https://github.com/clever-algorithms/CleverAlgorithms

   [2]: https://people.eng.unimelb.edu.au/ammoffat/mg/

   [3]: http://www.mmds.org/

   [4]: https://www.amazon.com/Algorithm-Design-Manual-Steven-Skiena...

   [5]: https://www.amazon.com/Network-Algorithmics-Interdisciplinar...

   [6]: https://hagan.okstate.edu/NNDesign.pdf

   Show morethread
3. andyxorJun 2021
   great post and my favorite CS topic, LSH is particularly relevant to machine learning because of its use in indexing of embeddings, which are now omnipresent in ML (from word2vec to transformers to image and graph embeddings etc, etc.)

   it is now supported in ElasticSearch KNN index (they use HNSWLIB but you can call it a descendant of original LSH in a way)

   check out ANN benchmarks [0] for comparison of LSH performance to other state of the art methods like proximity graphs/HNSWLIB [1] and quantization/SCANN [2]

   As an introduction LSH (with MinHash) is also described in detail in the book "Mining Of Massive Datasets", ch.3, "Finding Similar items", highly recommended [3]

   if you want to play with LSH, python "annoy" library is the best place to start [4]

   [0] https://github.com/erikbern/ann-benchmarks

   [1] https://github.com/google-research/google-research/tree/mast...

   [2] https://github.com/nmslib/hnswlib

   [3] http://infolab.stanford.edu/~ullman/mmds

   [4] https://github.com/spotify/annoy

   Show morethread
4. ignoramousMay 2021
   See also: http://mmds.org/ and their youtube channel https://www.youtube.com/channel/UCmLB71obuOBLMdWCQUI-DiA
   thread
5. snidaneDec 2020
   Mining Massive DataSets (MMDS) from Stanford professors.

   Videos: https://www.youtube.com/playlist?list=PLLssT5z_DsK9JDLcT8T62...

   Book: http://www.mmds.org/

   thread
6. shawnAug 2018
   Great! I was just reading through the first edition this week. It's excellent.

   Also see Mining of Massive Datasets: http://infolab.stanford.edu/~ullman/mmds/book.pdf

   thread