Matt Mahoney, a data compression researcher, hosts the Large Text Compression Benchmark, compression programs and a book, plus ultramarathon writing.
HN links it for the compression benchmark and its tie to the Hutter Prize, though a commenter notes it is updated less often now.
When people claim that there is such a thing as "X% accuracy in reasoning", it's really hard to take anything else seriously, no matter how impressive.
AI (and humans!) aside, claiming that there was an oracle that could "answer all questions" is a solved problem. Such a thing cannot exist.
But this is going already too deep IMO.
When people start talking about percentages or benchmark scores, there has to be some denominator.
And there can be no bias-free such denominator for
- trivia questions
- mathematical questions (oh, maybe I'm wrong here, intuitively I'd say it's impossible for various reasons: varying "hardness", undecidable problems etc)
- historical or policital questions
I wanted to include "software development tasks", but it would be a distraction. Maybe there will be a good benchmark for this, I'm aware there are plenty already. Maybe AI will be capable to be a better software developer than me in some capacity, so I don't want to include this part here. That also maps pretty well to "the better the problem description, the better the output", which doesn't seem to work so neatly with the other categories of tasks and questions.
Even if the whole body of questions/tasks/prompts would be very constrained and cover only a single domain, it seems impossible to guarantee that such benchmark is "bias-free" (I know AGI folks love this word).
Maybe in some interesting special cases? For example, very constrained and clearly defined classes of questions, at…
I'm doing my PhD in compression-based machine learning, wanted to contribute a few clarifying points.
The relationship between probabilistic modeling and lossless compression is very direct. A model that predicts the next symbol with probability p can on average losslessly compress that symbol with the help of an entropy coder (e.g. arithmetic coding) in -log(p) bits. Therefore improved probabilistic models immediately translate into improved lossless compressors.
There are two ways people have used compressors for ML: the first is based on the Minimum Description Length (MDL) principle [0] which says that the best model is the one which provides the shortest description of the data, counting the size of the model itself. This is similar to the technique used in this blog post (argmin code length across class-conditioned compressors) except for counting the model size. Basically you can train a compressor per class and then choose the class compressor which best compresses the test data. It is a maximum likelihood argument because of Shannon's source coding theorem: a code length L corresponds to a probability 2^-L. The second way is the Normalized Compression Distance (NCD) [1], which uses code lengths to calculate information-theoretic distances which can then be plugged into distance-based algorithms like kNN. MDL interprets compressed lengths as likelihoods, while NCD interprets them for distance calculations. The theoretical foundation is Kolmogorov complexity which…
jsaxton86's Test Result's:
Input File:
Title: The Simpsons Movie - 1080p Trailer.mp4
Kind: MPEG-4 movie
Size: 147,399,359 bytes (147.4 MB on disk)
Video Dimensions: 1920 x 800
Codecs: H.264, AAC
Duration: 02:17
Audio Channels: 6
Output Cram:
Title: The Simpsons Movie - 1080p Trailer.mp4.cram
Kind: Document
Size: 113,620,340 bytes (113.6 MB on disk)
Time to Cram File:
Seconds: 3313.306152 (55 Minutes ~13.3 Seconds)
Memory Usage:
GB (Rounded): 24.67 GB
Hardware Overview:
Model Name: iMac
Model Identifier: iMac11,1
Processor Name: Intel Core i7
Processor Speed: 2.8 GHz
Number of Processors: 1
Total Number of Cores: 4
L2 Cache (per Core): 256 KB
L3 Cache: 8 MB
Memory: 16 GB
Processor Interconnect Speed: 4.8 GT/s
Boot ROM Version: IM111.0034.B02
SMC Version (system): 1.54f36
We're really proud to try our best on your test jsaxton86, and we know that test results from the person showing you something always come with a healthy pinch of salt. So we'll ask the community to give it a try, and back us up here. Scientific testing means repeatable parameters, so we urge others to take his test too. (--EDIT: Turns out this file is too big 50MB in the limit of Github push-- We're also making our Cram file available on github, check it out with the decompressor or in a Hex editor: https://github.com/Cramcore/Crams)
From these results you can clearly see areas we aren't happy with either. Time and memory, but this is just the beginning for…
It's actually not the best at enwik8 or 9.
The results at https://www.mattmahoney.net/dc/text.html explicitly add the size of the compressor itself to the result. Note the "enwik9+prog" column. That's what it's ranked on.
The reason to do this is that it's trivial to create a compressor that 'compresses' a file to 0 bytes. Just have an executable with a dictionary of enwik9 that writes that out given any input. So we always measure what is effectively the Kolmogorov complexity. The data+program as a whole that produces the result we want.
So those results add in the compressor size. The programs there generally have no dictionary built in or in the case of LLM based compressors, no pre-trained data. They effectively build the model as they process data. Not compressing much at all at the start and slowly compressing better and better as they go. This is why these programs do better and better with larger data sets. They start with 0 knowledge. After a GB or so they have very good knowledge of the corpus of human language.
This program here however is pre-trained and shipped with a model. It's 150MB in size! This means it has 150MB of extra starting knowledge over those models in that list. The top models in that list are the better compressors, they'll quickly out learn and overtake this compressor but they just don't have that headstart.
Of course measuring fairly this should be listed with that 150MB program size added to the results when doing a comparison.
Impressive! But an order of a magnitude slower then non-LLM compression techniques.
I try to compare with the results from the top 9 of this enwik8 compression test by Matt Mahoney:
https://www.mattmahoney.net/dc/text.html
The durilca compressor by Dmitry Shkarin (what happened to him?) has been the fastest compressor in the top since it's debut in 2006.
The original size, enwik8, is 10^8 bytes. The compressed size is the compressend enwik8 plus the size of a zip archive containing the decompressor.
The bpb value of rwkv_430M on enwik8 is 0.948 bpb, the 7B models will be lower, perhaps around 0.800. So if my calculations are correct, the LLM's can perform 50% better then the best performing conventional compressors excluding the zipped LLM decompressor code (I am unsure about the size).
The bpb ratio for each existing program can be calculated as:
bpb ratio=(Original size (enwik9)Total size (enwik9+prog) )×8
nncp v3.1:
Given compressed size is not provided, we cannot calculate bpb for nncp v3.1 using enwik8.
cmix v19:
bpb=(14,837,987+223,485108)×8bpb=(10814,837,987+223,485 )×8
bpb≈1.205
tensorflow-compress v4:
bpb=(15,905,037+55,283108)×8bpb=(10815,905,037+55,283 )×8
bpb≈1.272
cmix-hp 10 Jun 2021:
Given compressed size is not provided, we cannot calculate bpb for cmix-hp using enwik8.
fast-cmix:
Given compressed size is not provided, we cannot calculate bpb for fast-cmix using enwik8.
starlit 31 May 2021:
bpb=(15,215,107108)×8bpb=(10815,215,107…
Interesting, I've recently spent an unhealthy amount of time researching archival formats to build the same setup of using SQLite with ZStd.
My use case is extremely redundant data (specific website dumps + logs) that I want decently quick random access into, and I was unhappy with either the access speed, quality/usability or even existence of libraries for several formats.
Glancing over the code this seems to use the following setup:
- Aggregate files
- Chunk into blocks
- Compress blocks of fixed size
- Store file to chunk and chunk to block associations
What I did not see is a deduplication step for the chunks, or an attempt to group files (and by extend, blocks) by similarity in an attempt improve compression.
But I might have just missed that due to lack of familiarity with Pascal.
For anyone interested in this strategy, take a look at ZPAQ [1] by Matt Mahoney, you might know him from the Hutter Prize competition [2] / Large Text Compression Benchmark. It takes 14th place with tuned parameters.
There's also a maintained fork called zpaqfranz, but I ran into some issues like incorrect disk size estimates with it. For me the code was also sometimes hard to read due to being a mix of English and Italian. So your mileage may vary.
[1]: http://mattmahoney.net/dc/zpaq.html [2]: http://prize.hutter1.net [3]: https://github.com/fcorbelli/zpaqfranz