Mlearning.ai is a Medium publication with machine learning tutorials on prompts, NLP, language models, and search.
HN commenters link its posts as explainers, such as on softmax temperature and the attention mechanism, and on an AI art contest win.
Discrete diffusion with rewriting can work well. It feels loosely similar to backtracking, if you assume n_steps large enough - need to be able to rewrite any non-provided position though I think. Downside is the noise in discrete diffusion (in simplest case randomizing over all vocabulary space) is pretty harsh and makes things very difficult. Don't have an exact reference on the relationship, but feels similar to backtracking type mechanics in my experience. I found things tend to "lock in" quickly once a good path is found, which feels a lot like pathfinding.
Some early personal experiments with adding "prefix-style" context by a cross-attention (in the vein of PerceiverAR) seemed like it really helped things along, which would kind of point to search-like behavior as well.
Probably the closest theory I can think of is orderless NADE, which builds on the "all orders" training of https://arxiv.org/abs/1310.1757 , which in my opinion closely relates to BERT and all kinds of other masked language work. There's a lot of other NAR language work I'm skipping here that may be more relevant...
On discrete diffusion:
Continuous diffusion for categorical data shows some promise "walking the boundary" between discrete and continuous diffusion https://arxiv.org/abs/2211.15089 , personally like this direction a lot.
If you have a pre-made embedding space, SSD-LM is a straightforward method https://arxiv.org/abs/2210.17432
SUNDAE worked well for translation…
I was thinking more of areas like early stage cancer detection[1], Fraud Detection[2] or the energy sector[3]
As for the glorified chatbot is of course one of the applications of AI, but i refuse to believe it has a very complex model, and the computational power required to use the model is also modest enough that most smartphones can execute it without any noticeable impact on battery life.
On the smaller scale, most of Apple's AI stuff is running on-device[4]. From iOS 15 all of Siri's "voice to text" happens on-device as well. While it's probably still a glorified chatbot, i for one enjoy being able to enter "pet autumn 2019" into my photo search bar, and be shown pictures of pets taking during autumn 2019, regardless of location.
[1]: https://engineering.berkeley.edu/news/2021/08/using-machine-learning-to-detect-early-stage-cancers/
[2]: https://medium.com/mlearning-ai/machine-learning-in-fraud-detection-use-cases-2130bd151b15
[3]: https://www.sciencedirect.com/science/article/pii/S0268401219302968
[4]: https://developer.apple.com/machine-learning/api/
Amazing results!
I did something similar with RoBERTa and my own Kindle library to graph (with D3.js) all mentions/citations between my books (which books cites another books I have). I sorted the final graph by publication date to see some cool historical patterns of books citing another older books [1]
I also manually annotated ~1000 book mentions, but I combine RoBERTa with string search (I list all titles I want to search a priori) to reduce the number of false positives. I also augumented the dataset with thousands of books titles and metadata from goodreads.
I explain all the process on a blog post[2]
[1] https://thiagolira.blot.im/_projects/book_graph/main.html
[2] https://medium.com/mlearning-ai/graphing-citations-between-books-using-roberta-and-d3-js-1747360805ca
The image is the one that won the contest a while ago, see here, e.g.,
https://medium.com/mlearning-ai/ai-art-wins-fine-arts-competition-and-sparks-controversy-882f9b4df98c
My 2 cents
https://medium.com/mlearning-ai/understanding-and-coding-the-attention-mechanism-the-magic-behind-transformers-fe707a85cc3f
https://arxiv.org/pdf/2210.10749.pdf
https://medium.com/mlearning-ai/softmax-temperature-5492e4007f71