BAIR Blog

Just some context here:

* The raw data in nuscenes ( https://www.nuscenes.org/ ) is about 5x larger than this dataset from Lyft. 300GB train vs 60GB train.
Argoverse ( https://www.argoverse.org/ ) is is about 3.x larger at 200GB. The Waymo dataset will (allegedly) be an order of magnitude larger than nuscenes ( https://i2.wp.com/syncedreview.com/wp-content/uploads/2019/06/image-55.png?resize=1024%2C768&ssl=1 ). BDD100k ( https://bair.berkeley.edu/blog/2018/05/30/bdd/ ) is the "largest" public dataset to date, but lacks lidar, and labels are inconsistent; most of the 100,000 scenes only have one labeled frame.

* The Lyft sensor suite has bumper-mounted lidar, which is absent from other existing datasets. Point cloud data in these areas is critical for pedestrians, bikes, and various road hazards. So this dataset alone is useful for validating work trained through other means.

* The current Lyft Level 5 release has no explicit test / validation set, which is crucial for properly measuring performance of any experiment one might do with the data. In nuscenes and Argoverse, there's a small snippet dataset that helps you prepare your pipeline. Feels like Lyft might have rushed things a little here-- they could have posted a "teaser" and then the full train and test/validation set a couple weeks later.

Great to see more public data (especially from a more modern sensor suite), plus investment into a contest with prizes.

There are tons of research projects that do exactly what you're talking about, initializing an agent with zero knowledge in an environment which just provide rules of discovery and reward. The agent then takes actions in that space and learns from its own experiences. It's just that it is hard to compare with humans, who have had the benefit of evolution over millions of years. A basic example is that of AlphaGo Zero[1], which learns to play a top level game by being given enough time and just the rules of the game. This is similar to how a child learns to walk, it is just that we only have the capability to model toy situations (a board game in this instance) right now, and access to harder instances (movement in the real world) will slowly come about. There are cases of robots being programmed to poke / move / pick objects to try and learn about their shapes[2], in case you are interested in another such example.

[1] - https://en.wikipedia.org/wiki/AlphaGo_Zero
[2] - https://bair.berkeley.edu/blog/2019/03/21/tactile/

Some are, some aren't. See Koala for instance. The problem with Koala is that it fine-tunes on open sourced data, but makes no claims about the data for the base LLaMA models. https://bair.berkeley.edu/blog/2023/04/03/koala/

The irony is that openAI and Meta themselves might be in flaky ground for having trained models on other people data with dubious rights to do so in many instances, and then using it to produce output commercially.

But this is a new frontier and enforcement might be effectively not possible unless new legislation requires reproducibility and audits on the data sets or something like that.

But without that, how do you know exactly how did they arrive at a given set of weights with Montecarlo algorithms and arbitrary fine tuning? You basically don't know what was there and you cannot prove they didn't achieve those results with perfectly clean data.

PS: https://medium.com/geekculture/list-of-open-sourced-fine-tuned-large-language-models-llm-8d95a2e0dc76

I found a few more, if you want a page which aggregates all of the RSS feeds, look here: https://news.aiapipro.com/

Feeds that I found:

- https://openai.com/blog/rss.xml

- https://hnrss.org/newest

- https://towardsdatascience.com/feed

- http://feeds.feedburner.com/blogspot/gJZg

- https://machinelearningmastery.com/feed/

- http://distill.pub/rss.xml

- https://bair.berkeley.edu/blog/feed

- http://karpathy.github.io/feed.xml

- http://fastml.com/atom.xml

- https://aws.amazon.com/blogs/machine-learning/feed/

- https://www.analyticsvidhya.com/blog/feed

- http://news.mit.edu/rss/topic/artificial-intelligence2

- https://www.marktechpost.com/feed/

- https://deepmind.com/blog/feed/basic/

- https://qudata.com/en/news/rss.xml

- https://machinelearning.apple.com/rss.xml

- https://www.oreilly.com/radar/topics/ai-ml/feed/index.xml

- https://blogs.rstudio.com/ai/index.xml

You could possibly fine tune it for narrow domain tasks like they did with tiny-agent https://bair.berkeley.edu/blog/2024/05/29/tiny-agent/

I like the approach that Apple seems to be taking with fine tuned small models that handle routine tasks and then defer to larger off device models for things they can’t confidently do. I imagine you could construct a training set that contains examples that should produce low confidence answers where you could add an output that is essentially a “call for help” option so you could train it to choose that. Smaller models also means you could have more running in parallel and use another to route requests to the appropriate expert.

This is why the "AI learns from materials just like a human does so it's not copyright infringement" argument always bothered me. A person won't recite full pages of word-for-word copies [1] from their head when you ask them something.

When I first tried Copilot, I asked it to write a simple algorithm like FizzBuzz and it ripped off a random repo verbatim, complete with Korean comments and typos. Image models will also happily generate near-identical [2] copies (usually with some added noise) of copyrighted images with the right prompt.

[1] https://bair.berkeley.edu/blog/2020/12/20/lmmem/

[2] https://www.theregister.com/2023/02/06/uh_oh_attackers_can_extract/