Hamel Husain, a machine learning engineer, writes about evaluating LLM products, LLM-as-a-judge, inference latency, and practical AI engineering.
Commenters recommend his writing as some of the best on LLM evals and his notes as a reliable source on applied work and inference benchmarks.
That's a big topic :D
First thing about building AI or LLM powered apps is that 90% of the time you don't need to train models from scratch. LLMs are models that have the world's knowledge, so you can "train them" by prompting them and giving them the right context.
Where to start? I think the first thing is to play around with the OpenAI playground to get an idea of what's possible with prompts.
You'll find tons of frameworks out there: LangChain, LlamaIndex, Agno, and the list goes on. But in my opinion, if you have a use case in mind, start very very simple. Almost a single prompt using the OpenAI SDK or compatible SDK like LiteLLM.
The core challenge isn't technical complexity, it's the stochastic nature of LLMs. Your prompt works perfectly on one example, breaks on another. You fix it for the second case, and it stops working for the first.
The whole trick of building an LLM powered application is finding how to deal with this in an iterative approach. You need to have a workflow that allows you to iterate quickly. For this, start very simple and then add complexity. Don't use frameworks at the beginning.
I think Hamel's approach of error analysis (https://hamel.dev/notes/llm/officehours/erroranalysis.html) is a very good starting point. The idea is that you create a prompt, run it against a test set, then identify clusters of error types and systematically fix them.
What tools do you need? In the beginning, just a playground and a Jupyter Notebook. At some point…
About .ml
The first demo Jeremy put out was called "Build Applications For LLMs in Python",¹ as part of the "Mastering LLMs" conference by Hamel Husain and Dan Becker.² (You can see a few PoC demos by the end of that video when Johno takes over, it looks a lot like what Gradio or Streamlit can do).
So I think your .ml angle is definitely part of the original ethos of FastHTML (which isn't surprising coming from the founder of fast.ai & answer.ai, among other things).
The FastHTML team explicitly recommends would-be contributors to consider making reusable components, the likes of Gradio's, to facilitate all the things notably relating to AI workflows.
----
About WordPress & CMS
That part is admittedly much larger in scope. I'd expect it to rise in correlation with the success of FastHTML itself in the Python web ecosystem writ large (beyond data / AI) but no sooner—unless someone makes a killer case for a FastHTML-based Python CMS that becomes a driver of popularity, but that's admittedly a much taller and wider order than 'simply' becoming the go-to #1 Python/ML prototype-to-market-at-scale one-stop shop. I mean, just that is huge, and yet nowhere near WordPress.
But tbh, I really like your idea, and I think it may eventually prove true, having used FastHTML first-hand for a few weeks now (and web dev being far from my turf). The fact is can ship with FastHTML, fast & well-behaved web apps, more than I ever could. If I ever get the time I'll play a bit to see what a…
- open-source vs closed fork of vscode
- transparent assistant panel vs opaque composer. you control your own prompts (cf. [0])
- in Zed the assistant panel is "just another editor", which means you can inline-assist when writing prompts. super underrated feature imo
- Zed's assistant is pretty hackable as well, you can add slash commands via native Zed extensions [1] or non-native, language-agnostic Context Servers [2]
- Zed's /workflow is analogous to Cursor's composer. to be honest it's not quite as good yet, however it's only ~1 week old. we'll catch up in no time :)
- native rust vs electron slop. Zed itself is one of the larger Rust projects out there [3], can be hard to work with in VS Code/Cursor, but speedy in Zed itself :)
[0]: https://hamel.dev/blog/posts/prompt/
[1]: https://zed.dev/docs/extensions/slash-commands
[2]: https://zed.dev/docs/assistant/context-servers
[3]: https://blog.rust-lang.org/inside-rust/2024/08/15/this-development-cycle-in-cargo-1.81.html#optimizing-git-sources
Hamel Husain hasn't done testing vs llama.cpp, but this still might be of interest (includes mlc which is roughly in line w/ llama.cpp batch=1 perf): https://hamel.dev/notes/llm/inference/03_inference.html
He has benchmarks on an A6000 which should be roughly in line w/ a 3090 if you want to compare to my numbers (I test mlc as well, although my 3090 results are slower since I'm testing a llama2-7b @ 4K context and mlc currently slows down significantly w/ longer context): https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYpb63e1ZR3aePczz3zlbJW-Y4/edit#gid=1788227831
> Can anyone direct me towards how to ... make one?
https://hamel.dev/blog/posts/evals/
> What are "obvious" things that are important to get right - temperature set to 0? At least ~10 or 20 attempts at the same problem for each llm?
LLMs are actually pretty deterministic, so there is no need to do more than one attempt with the exact same data.
> Finally, any known/commonly used frameworks to do this, or any tooling that can call different LLMs would be enough?
https://github.com/vercel/ai
https://github.com/mattpocock/evalite
Lots of people can get impressive demos up and running, but if you want to run AI products in production, you're going to have to do system evals. System evals make sure your product is doing what it says on the box with unquantifiable qualities.
We wrote a zine on system evals without jargon: https://forestfriends.tech
Eugene Yan has written extensively on it https://eugeneyan.com/writing/evals/
Hamel has as well. https://hamel.dev/blog/posts/evals/