Cohere Blog

Yes -- this is a good article on it: https://txt.cohere.com/sentence-word-embeddings

As a blackbox, it is a generalization of word2vec to sequence2vec. For example, simply summing or averaging word vectors in a sentence can give you a fast & cheap sentence embedding.

But natural language sentences have more structure than natural language words. Ex: it matters precisely where "not" goes in a sentence. So a lot of impressive scientific experimentation went into making these models smarter, with many evolutions. Impressively, this so blackboxed now that doesn't super matter.

Implicit to my post here... that's powerful, and easy to use... but not necessarily a great knowledge representation for someone who wants good Q&A over enterprise-scale data. One of our customer scenarios: "What is known vs believed about incident X." We can index each paragraph as multiple sentence embeddings, so if any phrase matches a query, the full paragraphs can get thrown into GPT as part of our answer. Easy. However, if information in the paragraph may lead to wanting to get information from elsewhere in the system (mention of another team, project, incident, ...), that means either a Planning agent needs to then realize that and recursively generate more vector search queries (mini-AutoGPT)... or we need to index on more than the sentence embedding.

Again, super interesting problems, and we're hiring for folks interested in helping work on it!

Project Aya is a multi-institution collaborative project to accelerate multi-lingual AI and make sure diverse languages are properly represented in this wave of large language models.

The first release is targeting 101 languages and we're 7 weeks away from starting to package it.

I'm one of the contributor to the project and all data and model will be open-sourced. We're looking for more contributors across many languages! It’s quite easy to contribute.

No knowledge of ML/AI required and you can make a big difference for the languages you’re fluent in! Please consider contributing!

More details about the project also at: https://txt.cohere.com/aya-multilingual/

This technique had a very recent resurgence via https://txt.cohere.com/int8-binary-embeddings/. Hugging face also covered the technique here https://huggingface.co/blog/embedding-quantization. It seems like a very good tradeoff compared to the shorter embeddings which require fine tuning via the matryoshka technique. On the other hand, Nils Reimers suggests that trivial quantization of the full precision embeddings is not as good as using “compression friendly” embeddings like Cohere’s Embed V3. Does anyone know what’s the difference in precision between trivial quantization and optimized embeddings?

Since publishing this I've found a few additional resources that are really useful for understanding embeddings at a lower level (my article is deliberately very high level and focuses mainly on their applications).

Cohere's Text Embeddings Visually Explained: https://txt.cohere.com/text-embeddings/

The Tensorflow Embedding Projector tool: https://projector.tensorflow.org/

What are embeddings? by Vicki Boykis is worth checking out as well: https://vickiboykis.com/what_are_embeddings/

Actually I'll add those as "further reading" at the bottom of the page.

There's a lot you can do with the vectors themselves without needing to embed any more text (e.g., clustering, exploration, visualization after dimensionality reduction...etc). Here's a previous embeddings exploration of top HN posts: https://txt.cohere.com/combing-for-insight-in-10-000-hacker-news-posts-with-text-clustering/ A lot of that code can be used here as well.

If you want to query for a search term, you can use a trial API key which is free to use for prototyping. The embedding model itself is not open source, though. [co-author of the post here]