Towards AI

I'd be interested to see lossy text compression. That would also give much more scope for AI methods. It turned out that recently Stable Diffusion gives excellent compression

https://pub.towardsai.net/stable-diffusion-based-image-compresssion-6f1f0a399202

Funny thing is, on a per pixel basis, Stable Diffusion and jpeg are just as good. Stable diffusion looks much better, though. The reason is that when Stable diffusion lacks the information it just invents what it expects to see there. If you zoom in on the background entire appartement buildings are invented, moved or disappeared.

So, one wonders what GPT-3 could do for text compression. On a per-character basis, I would not expect any miracles. On a generally, the same, kind-of, basis, I'd would expect something special.

Look at OpenAssistant (https://github.com/LAION-AI/Open-Assistant); they are trying to give a chatbot the ability to trigger actions in other systems. I fleshed out a scenario in more detail here: https://news.ycombinator.com/item?id=34808674.

But in brief, the short-term evolution of LLMs is going to involve something like letting it `eval()` some code to take an action as part of a response to a prompt.

A recent paper, Toolformer: https://pub.towardsai.net/exploring-toolformer-meta-ai-new-transformer-learned-to-use-tools-to-produce-better-answers-1b8c042e9c46 which is training on a small set of hand-chosen tools, rather than `eval(<arbitrary code>)`, but hopefully it's clear that it's a very small step from the former to the latter.

In another adjacent thread, people are talking about the implications of a neural network conforming to the training data with some error margin with regards to copyright.

Many textbooks on information theory already call out the content-addressable nature of such networks[1], and they’re even used in applications like compression due to this purpose[2][3], and therefore it’s no surprise that the NYT prompting OpenAI models with a few paragraphs of their articles reproduced them nearly verbatim.

[1] https://www.inference.org.uk/itprnn/book.pdf

[2] https://bellard.org/nncp/

[3] https://pub.towardsai.net/stable-diffusion-based-image-compresssion-6f1f0a399202

It doesn't need to be lossless to be useful. A lossy LLM can just throw all the redundant stuff away and focus on the actual information in the text, instead of the text word by word. It wouldn't need to keep dozens of languages around for if it could translate on the fly. And it wouldn't need to keep a full text when it can just have the raw facts and generate the text on the fly.

Not sure what the state of the art is with LLM text compression, but with StableDiffusion compression ratios of up to 155:1 have been achieved[1] compared to 10:1 for JPEG.

[1] https://pub.towardsai.net/stable-diffusion-based-image-compresssion-6f1f0a399202

I like the idea. Note that LLMs have some skill at decoding sequential dense vectors in the human brain

https://pub.towardsai.net/ais-mind-reading-revolution-how-gpt-llms-decode-thoughts-and-what-it-means-for-our-future-a24001a25d8

so why not have them decode sequential dense vectors of their own activations?

As for the majority scoring poorly, they suggest that most neurons won't have clear activation semantics so that is intrinsic to the task and you'd have to move to "decoding the semantics of neurons that fire as a group"

Volga is an open-source feature calculation engine for real-time ML systems aimed to remove dependency on complex Spark/Flink setups and managed feature platforms like Tecton.ai, Fennel.ai, FeatureForm, Chalk.ai

Github - https://github.com/anovv/volga Medium - https://pub.towardsai.net/volga-open-source-data-engine-for-real-time-ai-part-1-b8d7b16cb9d2

Would love to hear feedback, especially if you have experience building real-time feature calculation pipelines.