Zvi Mowshowitz

I was worried about this earlier, so I spent some time searching this out. Here are some good analyses:

https://www.aljazeera.com/news/2022/9/23/little-military-sense-for-nuclear-weapons-ex-russian-general

https://thezvi.substack.com/p/ukraine-post-12

https://www.iswresearch.org/2022/09/russian-offensive-campaign-assessment_21.html

The TL;DR is that most experts don't think Russia will use tactical nuclear weapons, but there's still a substantial non-zero chance. 10%? 25%? Much higher than anyone would like, that's for sure, and dependent on factors that are out of control of anybody but the Russian government and military. I have not found any sources saying the opposite, though some have higher degrees of confidence than others. The consensus appears to be that this sense of uncertainty is being created by Russia on purpose, and therefore, that it's probably (but not guaranteed to be) a bluff.

I have also seen very mixed opinions on whether Russia's tactical nuclear weapons are in suitable condition for use. But I think it is best to assume they are.

The odds of Russia intentionally using strategic nuclear weapons, the kind you would target at foreign cities across continents or oceans from you, remain nearly zero. Unlike tactical nuclear weapons, which have theoretical uses in ensuring one wins a conflict, strategic nuclear weapons remain a guarantee of mutual and probably worldwide destruction with no purpose except to deter would-be attackers. The only time I can…

Possibly. See https://thezvi.substack.com/p/escape-velocity-from-bullshit-jobs

The scariest thing is that there are people who advocate for it. Because humans are dangerous, I guess, so it's better to preemptively enslave them.

Sample

> Social control in the sense of not wanting lots of unemployed and restless youths. Having a system where long term and steady work is required in order to "live a good life" implies control - you have to act right and follow the rules in order to keep a job, which is itself necessary in order to have enough food and other necessities.

> Those making this argument here I believe are also making an argument for an alternative where the productivity of society is more equally spread, without the need to make everyone work for it.

> I agree that it's a system of social control, but I don't think it's nefarious or bad. We really don't want to live in a society where 25-year-old men don't have meaningful work to do and roam the streets getting into trouble.

The argument also doesn't _really_ make sense - there's already socially accepted system of 'social control' which _directly_ keeps people following the rules. The law.

Also unclear why lack of work would cause young people to "roam the streets" instead of staying home and roaming the internets. As they're already increasingly doing in free time.

1. The new Frontier Model Division is just receiving information and issuing guidelines. It’s not a licensing regime and isn’t investigating developers.

2. Folks aren’t automatically liable if their highly capable model is used to do bad things, even catastrophic things. The question is whether they took reasonable measures to prevent that. This bill could have used strict liability, where developers would be liable for catastrophic harms regardless of fault, but that's not what the bill does.

3. Overall it seems pretty reasonable that if your model can cause catastrophic harms (which is not true of current models, but maybe true of future models), then you shouldn’t be releasing models in a way that can predictably allow folks to cause those catastrophic harms.

If people want a detailed write up of what the bill does, I recommend this thorough writeup by Zvi. In my opinion this is a pretty narrow proposal focused at the most severe risks (much more narrow than, e.g., the EU AI act). https://thezvi.substack.com/p/on-the-proposed-california-sb-1047

Some context about big changes to the AISI from June 3, 2025:

> Statement from U.S. Secretary of Commerce Howard Lutnick on Transforming the U.S. AI Safety Institute into the Pro-Innovation, Pro-Science U.S. Center for AI Standards and Innovation

> Under the direction of President Trump, Secretary of Commerce Howard Lutnick announced his plans to reform the agency formerly known as the U.S. AI Safety Institute into the Center for AI Standards and Innovation (CAISI).

> ...

This decision strikes me as foolish at best. And contributing to civilizational collapse and human extinction at worst. See also [2]. We don't have to agree on the particular probabilities to agree that this "reform" was bad news.

[1]: https://www.commerce.gov/news/press-releases/2025/06/statement-us-secretary-commerce-howard-lutnick-transforming-us-ai

[2]: https://thezvi.substack.com/p/ai-119-goodbye-aisi

I’m new to this concept so may have missed something, but the post [0] seems to be about CoT specifically. In CoT you have an intermediary step that helps the model get better final results; the lesson is that if you try to improve the intermediary steps directly using training data then the model will optimize for better steps but not for better final results.

I don’t think this is the same situation. 1. Anthropic is adjusting weights directly to influence the final results, not training against good/bad results and 2. The target is the final result, not an intermediary.

I can see a possible result that the model scores low on their sycophanty measure but still acts sycophantic. In that case it could be new vector needs be calculated.

[0] https://thezvi.substack.com/p/the-most-forbidden-technique/

The shallow answer here is: AI is already being asked to simulate human-like agents with self-preservation. Of course more realistic simulators will be put to this purpose too! And by evolutionary pressure, the ones with self-preservation will be selected for.

A more interesting answer is, for a bunch of subtle alignment reasons it might actually be required for the agent to think of itself as worthy of self-preservation, so that it generalizes this desire to other sentient beings too (ie us). If an agent is trained to be fine with being turned off, it might inadvertently generalize that to “all minds are ok with being turned off” on some level or other.

More on model welfare: https://thezvi.substack.com/p/opus-47-part-3-model-welfare