Christian S. Perone writes technical posts on machine learning, PyTorch, LLVM, geometry, and data pipelines.
HN commenters cite his article on Google's S2 geometry library, cells, and the Hilbert curve as a useful explainer when S2 documentation is scarce.
Infact someone has run word2vec on the Voynich manuscript: http://blog.christianperone.com/2016/01/voynich-manuscript-word-vectors-and-t-sne-visualization-of-some-patterns/ (web archive while it's down: http://web.archive.org/web/20160205003817/http://blog.christianperone.com/2016/01/voynich-manuscript-word-vectors-and-t-sne-visualization-of-some-patterns/) Such methods could someday completely decode the thing, but for now they just show the relationships between words and different clusters of words, not their meaning.
Of course we have no idea how the Voynich manuscript is encrypted (which would make the assumptions of word2vec wrong), or if it even has any meaning at all. And it's an incredibly small dataset compared to modern text corpuses, so there is probably significant uncertainty and overfitting. And other problems like inconsistent spellings, many errors in transcriptions, etc. But in principle this is a good strategy.
>how would a person do if they were locked in a room with lots of books written in a language unknown to them?
If you spent all day reading them, for years, and you somehow didn't get bored and kept at it, eventually you would start to see the patterns. You would learn how "slithy toves" are related to "brillig", even if you have no idea how that would translate to English. Study it long enough, and you may even be able to produce text in that language, indistinguishable from the real text. You may be able to predict the next word in a sentence,…
Thanks for the info. I'm looking at this from the perspective of designing a backend datastore and query engine for a knowledge system. The idea is to encode a spatial data structure (similar to Google's S2 Geometry Library [0]) that enables content-based addressing of non-spatial data types for data fusion.
One idea is to make a lattice of unicode characters that builds up to combination of words a la Formal Concept Analysis [1] -- on one level, the characters compose into words that represent properties (key/value pairs), and then the KV pairs compose into higher-level objects. Each property and higher-level object is encoded with an integer derived from its constituent objects/properties, and each object is encoded in such a way that its constituent objects/properties can be determined algorithmically from the integer without having to traverse the structure [2] -- ANS encoding [3] embedded into a space with a VI metric (https://en.wikipedia.org/wiki/Variation_of_information) [4] might make this work. Have you played with this type of design?
[0] http://blog.christianperone.com/2015/08/googles-s2-geometry-on-the-sphere-cells-and-hilbert-curve/
[1] https://www.youtube.com/watch?v=Xuxm929tIRY
[2] http://www09.sigmod.org/sigmod/record/issues/0506/p47-article-tropashko.pdf
[3] https://en.wikipedia.org/wiki/Asymmetric_Numeral_Systems
[4]…
>You could probably do this with other systems, but the goal is to allow everyday people to be able to pack a short string in a tweet and have everybody else know exactly where they mean.
89c25855b9b
That's the S2 string for a section of Times Square, NY.
Usefully "89c25855b" is a well-defined, less precise location comprising a larger area. You can go up or down in levels of precision easily and everything can be stored as a single 64-bit integer with lots of extremely useful properties, (e.g. the use of the Hillbert Curve with all of it's benefits, like localization in address space and real space - see http://blog.christianperone.com/2015/08/googles-s2-geometry-on-the-sphere-cells-and-hilbert-curve/)
If your system adds more benefits than this, I'd love to hear more about it!
Library also available in Java[0] and to an extent in Golang[1], though the latter is incomplete.
S2 is fantastic for geofencing. Arbitrary regions can be covered by a set of S2 cells of varying level. A point (lat/lon) can be converted to a S2 cell of the smallest level (around a cm^2 of area on the sphere). Checking to see if one S2 cell is contained in another amounts to searching an integer range, since all children of a given S2 cell have IDs that fall within a fixed ranged. See also [2]
[0] https://github.com/google/s2-geometry-library-java
[1] https://github.com/golang/geo
[2] http://blog.christianperone.com/2015/08/googles-s2-geometry-on-the-sphere-cells-and-hilbert-curve/
I was playing around with s2 a year or two ago, and it was surprisingly difficult to find information on it, so FWIW here are some links I was able to find at the time if you're looking for some historical reading on s2:
Google Presentation on S2: https://docs.google.com/presentation/d/1Hl4KapfAENAOf4gv-pSngKwvS_jwNVHRPZTTDzXXn6Q/view#slide=id.i122
Christian Perone's blog post on it: http://blog.christianperone.com/2015/08/googles-s2-geometry-on-the-sphere-cells-and-hilbert-curve/?a=2
Google Code Archive: https://code.google.com/archive/p/s2-geometry-library/
Java Implementation: https://github.com/rschreijer/s2-geometry-library-java
TensorFlow: https://www.tensorflow.org/
TensorFlow Serving: https://github.com/tensorflow/serving
ReCeption (actually they call in Inception v3. Not sure where I got the ReCeption name - though I'm sure I read it somewhere?): https://www.tensorflow.org/versions/r0.7/tutorials/image_recognition/index.html
Using a SVN on neural network extracted features: http://blog.christianperone.com/2015/08/convolutional-neural-networks-and-feature-extraction-with-python/
If you want a quick and dirty version here's some Python to create a web service that calls a Caffe based Image recognizer: https://gist.github.com/nlothian/c3519adb81b3452c1938