AI Weirdness

> Are you saying that GPT-6 would be less intelligent than a human, because we could give it an arbitrary deadline within which to answer the question, while not restricting a human that way? That doesn't sound like a fair test.

The basic design of an ML model is just one really big matrix multiplication that always takes the same time - I'm saying we haven't given GPT the ability to not do that, so it always has that arbitrary default.

It does have a system where it does beam search through the model instead of simply running it once, but it's not fully recurrent.

> That's a reasonable intuition and a fair experiment, but I think it hinges on the question of "What does a human do when they go away and think about a problem?". Could it be that they are just clearing their mind of other distractions, and trying to look at the problem from multiple perspectives, which is something that a machine could do more efficiently?

Well, some problems just take longer to think about or need knowledge that isn't already written down.

> Putting it another way, if you met someone who had never performed an experiment before, would you think they were not intelligent?

I think a prisoner or someone like that could demonstrate they're intelligent. If you're in the room with them, I mean, not like the Chinese room problem. But I think pure "intelligence" is just going to be less interesting that it looks, because it can't accomplish many tasks you'd expect it to if it can't actively…

Oh man, good question. I'm always up for swapping these stories. A lot of these came from a paper on weird AI tricks, and resulting best-of list on a blog collecting these stories.[1][2] Suffice to say, the people who think the orthogonality thesis is a weird hypothetical aren't keeping up with the state of things.

- The aforementioned Tetris story: an undirected learner was set to maximize score at Tetris learned normal play techniques, but also learned to pause the game immediately before losing so that the score wouldn't "decline" at game over.

- In the same vein as interns quitting, proxy detection of all sorts. Identify "field with sheep" by finding green fields with grey skies, or letting heuristics like "humans pick up dogs and cats" override correct identifications. (It's a goat until you pick it up, then it's a dog!)

- An agent playing Q*bert found a known bug for infinite lives, then escalated to an unknown bug which disabled the game while overflowing the score counter.

- Agents in a physics sim tasked with jumping as high as possible instead learned to 'fly' by abusing collision detection bugs, hitting themselves in ways that created upward momentum.

- Another "maximize jump height" task demonstrated that "highest" is an extremely fuzzy term. Initially measured by highest point, they became incredible tall. Measured by lowest point, they stayed tall and grew topheavy to 'kick' their base upwards.

- Number-handling bugs of all kinds. In one case, small…

One of the things I'm rather worried about here is AI's ability to learn on the most unlikely (to human cognition) features. Take the image recognizer that finds "sheep" by seeing pastures where it thinks sheep might graze http://aiweirdness.com/post/172894792687/when-algorithms-surprise-us , or take any of the adversarial perturbing examples like https://blog.openai.com/robust-adversarial-inputs/ .

You're not allowed to discriminate on race in housing. But is an AI that determines your creditworthiness for mortgages allowed to discriminate on what you eat, where you go to church, what your favorite music is, etc.? Maybe it doesn't have that data, but it will have one level higher - what store credit cards you have and how much you use them and where you opened them.

If you train an AI on a segregated city with a history of actively discriminatory citizens, where the few people of race X who moved into a not-race-X neighborhood got harassed out and sold before they paid off their mortgages, how easily will the AI conclude that people born in certain neighborhoods are more likely to pay off their mortgage if they avoid certain other neighborhoods?

Is that illegal? (My guess is there's no way to prove to a court, to the court's usual standards, that the AI happened to learn the city's racial tensions.) Should it be illegal, if outright racial discrimination is illegal?

> Sometimes after a night’s sleep, we wake up with an insight on a topic or a solution to a problem we encountered the day before.

The current crop of models do not "sleep" in any way. The associated limitations on long term task adaptation are obvious barriers to their general utility.

> When conversing with LLMs, I never get the feeling that they have a solid grasp on the conversation. When you dig into topics, there is always a little too much vagueness, a slight but clear lack of coherence, continuity and awareness, a prevalence of cookie-cutter verbiage. It feels like a mind that isn’t fully “there” — and maybe not at all.

One of the key functions of REM sleep seems to be the ability to generalize concepts and make connections between "distant" ideas in latent space [1].

I would argue that the current crop of LLMs are overfit on recall ability, particularly on their training corpus. The inherent trade-off is that they are underfit on "conceptual" intelligence. The ability to make connections between these ideas.

As a result, you often get "thinking shaped objects", to paraphrase Janelle Shane [2]. It does feel like the primordial ooze of intelligence, but it is clear we still have several transformer-shaped breakthroughs before actual (human comparable) intelligence.

1. https://en.wikipedia.org/wiki/Why_We_Sleep 2. https://www.aiweirdness.com/

You can watch the lecture referred to in the article here:

https://www.youtube.com/watch?v=rGgGOccMEiY

The paper you quote is from 2022; it's also the last one Hinton posted to the arXiv:

https://arxiv.org/search/cs?searchtype=author&query=Hinton%2C+G

His resignation from Google and public warning about AI happened last month:

https://www.bbc.com/news/world-us-canada-65452940

As LeCun is quoted saying in the article, "Geoff only started thinking about these things a couple of months ago".

Finally, as already mentioned by others, agents finding unexpected ways to maximize their reward is a well known problem of reinforcement learning:

https://www.aiweirdness.com/when-algorithms-surprise-us-18-04-13/

Fun quote from that 2018 article: "there was an algorithm that was supposed to figure out how to apply a minimum force to a plane landing on an aircraft carrier. Instead, it discovered that if it applied a huge force, it would overflow the program’s memory and would register instead as a very small force. The pilot would die but, hey, perfect score."

This reminds me of Janelle C. Shane's work on AI[1]. Computer vision's been learning based on the photos that people take, not the visions that people see, and this is giving AI a preference for photogenics instead of reality.

People generally don't take pictures of rolling green hillsides. But they very often take pictures of rolling green hillsides with sheep on them. So if you ask the robot to draw a picture of rolling green hillsides, it will include sheep. Or, if you ask it to draw a picture of the savanna, it will want to include giraffes.

Now you're being asked to find a bus in a photo without a bus because it's a street scene, and every street scene has a bus in it.

I haven't read her book, yet, but her Twitter[2] is often full of amusing anecdotes like this.

[1] https://aiweirdness.com/

[2] https://twitter.com/JanelleCShane