Robert Miles AI Safety
3 HN recommenders · recommended in 5 months, 2019–2023
17:52 · 259K views · Dec 13, 2019
Robert Miles explains reward modelling, in which an AI learns what humans want from their preference comparisons instead of a hand-written reward function. He walks through the 2017 paper on deep reinforcement learning from human preferences.
Seen as the clearest introduction to the idea behind RLHF, and also borrowed by commenters as a metaphor for human motivation.
6 comments from Hacker News
I would highly recommend this video as a way of framing the problem. It seems like our options for ML are currently: self-play, dataset fitting, or human feedback. It's possible to solve any problem that can fit into one of those molds. (Unfortunately the opposite is true: if a problem doesn't fit into those categories, it seems very hard to solve with any form of ML.) But, now that I think about it a bit more, the sentence "use a trained net as a tool for building a controller with known continuity properties" seems neither obvious nor straightforward to do. I'm decently well-versed in AI techniques at this point, and I can't really think how I'd sit down and write a program that would generate such a system. Anyone have any leads? Basically, the goal is to make a neural net that helps you design a system that can be predictable. One way would be to write a loss function that penalizes unpredictability, and then let it run until it converges on a predictable control system. But that's a bit like saying "Just draw an owl." Not so easy to think of the actual code to do that.
Specifically about RLHF, I find this video by Rob Miles still the best presentation of the ingenious original 2017(!) paper: this video RLHF is actually older than GPT-1, which came out in 2018. It didn't get applied to language models until 2022 with InstructGPT, an approach which combined supervised instruction fine-tuning with RLHF.
ADHDer here. Please steal some of the motivation-techniques I've learnt over the years: 1. Find a friend who is a software engineer or data analyst and cares about you as a person. Talk about this with them. Humans evolved being able to talk things over with their aunt or best friend. When you are perplexed by something, a rubber duck probably won't cut it -- you should talk with a human. 2. Make sure you're taking care of your body and brain: exercise, good food, good sleep (this is easier if you're getting exercise), meditation, etc. 3. Understand what your "reward model" is. Under what conditions do you recognize "Aha! I've made progress!"? To really understand what I'm getting at here, watch this video: this video 4. Find internal things to strengthen your reward model's signals. Go back to when you originally started learning ML and Re-connect to your "Why". Do you still find those reasons compelling? If so, take some time to bring some vividness to your picture. -- Start With Why. 5. Find external punishments to remove. Now that you've spent 8 months working on it, you've noticed the things which are annoying you about your workflow. - After 8 months, it is totally healthy to take a week or two and improve your development environment to sand the edges off of things that give you friction. For a stranger's recommendation: [LINK] has made it dramatically easier for me to write little scripts that take away annoying tasks. - Maybe you feel like you've been "in the weeds" for
A. Meet your basic biological needs. Your brain is a part of your body. A person with an iron deficiency can feel compelled to chew ice[1]. Ice does not cures anemia, brains are weird parts of the body and do weird things when the body is not getting what it needs. So, get better sleep, food, and exercise. Prof Andrew Huberman has some good tips[2]. B. Decide what few things you really value and allocate your time to those on purpose. If you find yourself repeatedly drawn to the news, maybe you are genuinely give a fuck about civic engagement. Your personality has a need for it. Trying to meet that need by scrolling twitter is like eating 3 bags of tortilla chips for dinner -- quick-access, but not healthy. If you be better off deciding that you value Mexican food, opening your calendar and honestly deciding to spend the time to learn to cook Mexican food. Look for the virtue underneath your vice -- What information are you most drawn to and what healthy activities does it point towards? Pick 3 and pursue those deeply. Why 3? You can only honestly give a fuck about a few things[3], so that requires deciding to not do other reasonable fun activities. If shallow engagement with your interests has been your primary mode of procrastination, then honest and deep engagement might deceptively feel like a waste of time. That makes this hard. Why arbitrarily avoid those other things? You are choosing to be the arbiter of your time. Take pride in grappling with the fear of procrastinat
I think your URL is a fantastic choice and hints at what a lot of "accountability coaching" really is: Camaraderie An easily-imaginable person who acts as a reward model[1] you can imagine saying "yea! Make progress on the thing! You're on the right path[2]! Keep going." When you anticipate [social] rewards for an action, that action is motivating. If you watch Dr. Andrew Huberman's talks about the roles of Dopamine in the brain, he'll say it is not a signal of reward, but of anticipated reward. It does spike when you recognise a surprisingly strong reward, (like when you hit a jackpot or an unexpectedly interesting HN link) to help you learn to repeat the path to that reward[2], but mostly it signals "Yep, you're on the right path. Now go make that test green". [1] this video [2] [LINK]