Answer.AI

Yes, correct, and that fetching operation is a non-blocking operation, once we dequantize the weights we discard it before moving to the next layer.

Technically, you can do it for the weights as well. But that wouldn't work in many situations. For example, when training with FSDP: the quantized weights stay on the device but you can still offload the meta-data (https://www.answer.ai/posts/2024-03-06-fsdp-qlora.html)

I would like to re-iterate that larger models, which would be more interesting to run at low-bits, are much less sensitive to quantization compared to a 7B. So you could potentially use a larger group-size and just keep it on device, like what is done with 4-bit and 3-bit now using a group size of 64. We just started running some experiments with a 13B llama2 and it looks very good so far (outperforming some full-precision llama2-13B-based models), let's see how far we can push it, ideally get-rid of the reshaping all together will be great.

I would like to throw our project in the ring that solves this problem: https://github.com/tjmlabs/ColiVara

1. Uses half-vecs, so you cut down everything by half with no recall loss 2. Uses token pooling with hierarchial clustering at 3, so, you further cut down things by 2/3rd with <1% loss 3. Everything is on Postgres and pgvector, so you can do all the Postgres stuff and decrease corpus size by document metadata filtering 4. We have a 5000+ pages corpus in production with <3 seconds latency. 5. We benchmark against the Vidore leaderboard, and very near SOTA

You can read about half-vecs here: https://jkatz05.com/post/postgres/pgvector-scalar-binary-quantization/

Hierarchical token pooling: https://www.answer.ai/posts/colbert-pooling.html

And how we implemented them here: https://blog.colivara.com/

From the post describing llms.txt (https://www.answer.ai/posts/2024-09-03-llmstxt.html):

"The problem this solves is that today, constructing the right context for LLMs based on a website is ambiguous — do you:

1. Crawl the sitemap and include every page, trying to automatically format into an LLM-friendly form?

2. Selectively include external links in addition to the sitemap?

3. For specific domains like software documentation should you also try to include all the source code?

Site authors know best, and can provide a list of content that an LLM should use."

(There's quite a bit more info there that answers this question in more detail.)

I get this eerie feeling she has no idea what a bot even is in a technical sense and most likely is being puppeteered by weasels like State Senator Scott Wiener having introduced gems like the AI Models Act [1][2][3]. If the bill passes wouldn't then she have to be the first to disclose she's a bot?

[1] https://www.answer.ai/posts/2024-04-29-sb1047.html (SB-1047 will stifle open-source AI and decrease safety)

[2] https://safesecureai.org/responseletter (Response to inaccurate, inflammatory statements by Y Combinator & a16z regarding Senate Bill 1047)

[3] https://news.ycombinator.com/item?id=41690302 (Gavin Newsom vetoes SB 1047)

chef's kiss

This guy and the person that quit the bullshit industrial complex 6 months ago should get together and launch a startup.

Better yet, we should all go and join Jeremy Howard's answer.ai pro bono. Besides being miraculously headed by a guy who is Not An Asshole, it incidentally also had the most refreshing launch post (in the warm and fuzzy way) this side of the AI bubble.

The launch post concluded with this heading:We Don’t Really Know What We’re Doing.[0]

I mean, for the finest minds in our respective fields, what else is there left to say really?

[0] https://www.answer.ai/posts/2023-12-12-launch.html

Some laws are just bad. When the API-mediated/closed-weights companies agree with the open-weight/operator-aligned community that a law is bad, it’s probably got to be pretty awful. That said, though my mind might be playing tricks on me, I seem to recall the big labs being in favor at one time.

There are a number of related threads linked, but I’ll personally highlight Jeremy Howard’s open letter as IMHO the best-argued case against SB 1047.

https://www.answer.ai/posts/2024-04-29-sb1047.html