RARE Technologies, the company behind Gensim, writes on machine learning, word embeddings, benchmarks, and interviews with researchers.
HN links it for ML tutorials and benchmarks; one commenter faulted its podcast page for not linking an RSS feed.
I run a company that specializes in design & implementation of kick-ass ML solutions [1]. We've had successful projects in quite a few industries at this point:
LEGAL INDUSTRY
Aka e-discovery [2]: produce digital documents in legal proceedings.
What was special: stringent requirements on statistical robustness! (the opposing party can challenge your process in court -- everything about way you build your datasets or measure the production recall the has to be absolutely bullet proof)
IT & SECURITY
Anomaly detection in system usage patterns (with features like process load, frequency, volume) using NNs.
What was special: extra features from document content (type of document being accessed, topic modeling, classification).
MEDIA
Built tiered IAB classification [3] for magazine and newspaper articles.
Built a topic modeling system to automatically discover themes in large document collections (articles, tweets), to replace manual taxonomies and tagging, for consistent KPI tracking.
What was special: massive data volumes, real-time processing.
REAL ESTATE
Built a recommendation engine that automatically assembles newsletters, and learns user preferences from their feedback (newsletter clicks), using multi-arm bandits.
What was special: exploration / exploitation tradeoff from implicit and explicit feedback. Topic modeling to get relevant features.
LIBRARY DISCOVERY
Built a search engine (which is called "discovery" in this industry), based on Elasticsearch.
What…
Like a lot of work in NLP at the moment, this is a reasonably straight-forward mash-up of existing techniques. This particular idea is pretty obvious. What wasn't obvious was whether it would work well, which was why nobody else got around to trying it yet. The experiments are nicely conducted, with strong baselines across multiple evaluations. The authors also include experiments with variations of the idea, to further validate the approach.
To understand the technique, first understand word2vec:
http://rare-technologies.com/word2vec-tutorial/
http://colah.github.io/posts/2014-07-NLP-RNNs-Representations/
Now understand part-of-speech tagging:
http://spacy.io/blog/part-of-speech-POS-tagger-in-python/
By default word2vec gives you clusters for each word, this paper is giving you clusters for word_POS, e.g. The_DT apple_NNP employee_NN is_VBZ eating_VBG an_DT apple_NN. The same trick is done with named entity labels as well.
The following papers explain how the new word vectors are used in a dependency parser:
Collobert and Weston (2011): http://arxiv.org/pdf/1103.0398.pdf
Wang and Manning (2014): http://cs.stanford.edu/~danqi/papers/emnlp2014.pdf
Yoav Goldberg (2015): http://u.cs.biu.ac.il/~yogo/nnlp.pdf Survey/review, aimed at grad students
This isn't a real downstream task, but one of the researchers at RaRe compared FastText to word2vec/gensim/skipgram word embeddings on the original testsets for the 'semantic' and 'syntactic' analogy tasks from the word2vec papers here:
https://rare-technologies.com/fasttext-and-gensim-word-embeddings/
The conclusion:
These preliminary results seem to indicate fastText embeddings are
significantly better than word2vec at encoding syntactic information. This is
expected, since most syntactic analogies are morphology based, and the char
n-gram approach of fastText takes such information into account. The original
word2vec model seems to perform better on semantic tasks, since words in
semantic analogies are unrelated to their char n-grams, and the added
information from irrelevant char n-grams worsens the embeddings.
Personally I think those analogy testsets are not very good, because they just test all pairs of relations between a very small number of words from very limited domains (like capital and country names).
One advantage of FastText should be better learning on small amounts of data like Wikipedia.
I don't think you are correct here. The advantage of using sparse storage and sparse matrix multiplication is that you can get savings in both storage and runtime. There would be no point in using sparse storage if runtime still scaled with the cube of vocab size. It would be that way if the best way of getting sparse SVD is by materializing a dense matrix product, but people have discovered smarter ways using sparse matvec. The time complexity for obtaining k eigenvalues seems to be O(dkn) where d is the average number of nonzeroes per row, and n is the vocab size [1]. Therefore, one can assert that sparse SVD too is linear in vocab size just like word2vec.
This is corroborated by the link elsewhere in this thread that shows SVD enjoying lower wall clock time on the 1.9B word Wikipedia dataset [2].
[1] https://en.wikipedia.org/wiki/Lanczos_algorithm#Application_to_the_eigenproblem
[2] https://rare-technologies.com/making-sense-of-word2vec/
> However, I think this just calls out all the open questions about this style of consulting and doesn't answer them. How does one get clients, how does one keep a steady income, how to invest for retirement, how does one convince clients you have the right skills, how do you know you have the rights skills, etc.?
He touches on some of these issues in a podcast interview with Radim Rehurek (of Gensim fame).
https://rare-technologies.com/rrp-2-john-d-cook-consulting/
Blender Python lib by default is not optimized much. It has nothing to do with Python as a language.
Use numpy for matrices. If you have to implement an algo with a hot inner loop, use cython or numba.
I've never seen 100x difference in Python-C++ rewrite if Python was optimized already.
Here is a good article about some of the options:
https://rare-technologies.com/word2vec-in-python-part-two-optimizing/