Security researcher Johann Rehberger writes about attacks on AI systems, including prompt injection, data exfiltration, and agent vulnerabilities.
HN commenters cite it for concrete prompt injection and invisible Unicode examples with video demos, including shells obtained on OpenAI machines.
Yes, I can. This is just one possible strategy: there are many others, where different things are done, and where things are done in a different order.
You use the collider [1] and one of the many scaling attacks ([2] [3] [4], just the ones linked in this thread) to create an image that matches the hash of a reasonably fresh CSAM image currently circulating on the Internet, and resizes to some legal sexual or violent image. Note that knowing such a hash and having such an image are both perfectly legal. Moreover, since the resizing (the creation of the visual derivative) is done on the client, you can tailor your scaling attack to the specific resampling algorithm.
Eventually, someone will make a CyberTipline report about the actual CSAM image whose hash you used, and the image (being a genuine CSAM image) will make its way into the NCMEC hash database. You will even be able to tell precisely when this happens, since you have the client-side half of the PST database, and you can execute the NeuralHash algorithm.
You can start circulating the meme before or after this step. Repeat until you have circulated enough photos to make sure that many people in the targeted group have exceeded the threshold.
Note that the memes will trigger automated CSAM matches, and pass the Apple employee's visual inspection: due to the safety voucher system, Apple will not inspect the full-size images at all, and they will have no way of telling that the NeuralHash is a false positive.
[1]…
I'm a Master's student in Artificial Intelligence at UofT. Right now, I'm adding the "ML (Machine Learning) Streaming" feature to PyMilo. PyMilo is an open-source Python package that provides a transparent, safe, and end-to-end way for users to export pre-trained machine-learning models.
Transparency and non-executable export format is a serious thing, take a look at https://embracethered.com/blog/posts/2022/machine-learning-attack-series-injecting-code-pickle-files/.
After version 0.9 release, PyMilo became feature-completed with full support of scikit-learn models, now it's time to move on to PyTorch and then Tensorflow. But we decided to add the "ML Streaming" feature before getting into PyTorch, in order to provide an easy way to smoothly stream your ML model. By using the "ML Streaming" feature you can easily deploy your model into the remote server, connect to it from the client side, and choose the working mode, either delegation or local mode, through delegation mode your requests will be relayed to the remote server and you can easily work with your remote model from any devices without any further dependencies, and finally, you can download model for local use.
We will release the 1.0 (tenth) version of PyMilo around Sep 16th, this release will be the first release to have the "ML Streaming" feature with support of REST API, and we will next add other protocols such as Websocket.
Here is PyMilo: https://github.com/openscilab/pymilo Thank you for your time
The problem is that it is a scaled low-res version. There are well publicized attacks[1] showing you can completely change the contents of the image post scaling. There's also the added problem that if the scaled down image is small, even without the attack, it's impossible to make a reasonable human judgement call (as OP points out).
The problem isn't CSAM scanning in principle. The problem is that the shift to the client & the various privacy-preserving steps Apple is attempting to make is actually making the actions taken in response to a match different in a concerning way. One big problem isn't the cases where the authorities should investigate*, but that a malicious actor can act surreptitiously and leave behind almost no footprint of the attack. Given SWATting is a real thing, imagine how it plays out if child pornography is a thing. From the authorities perspective SWATting is low incidence & not that big a deal. Very different perspective on the victim side though.
[1] https://embracethered.com/blog/posts/2020/husky-ai-image-rescaling-attacks/
* One could argue about the civil liberties aspect & the fact that having CSAM images is not the same as actually abusing children. However, among the general population that line of reasoning just gets you dismissed as supporting child abuse & is only starting to become acknowledged in the psychiatry community.
Oh yeah! I totally forgot about Intel's on-chip AI acceleration[1]. I'm not sure how well it works, but I know that it exists and nothing major is currently taking advantage of it AFAIK.
Local Copilot sounds interesting, but I'd imagine it would be akin to the old Siri on iPhones where its use would be limited to controlling on-device settings. I've heard and read quite a lot about the dangers of that, though: with prompt injections being an issue with Copilot[0] (and Bing Chat frequently going crazy[2]), I can understand why Microsoft are very hesitant to give it any real control over the system.
[0]: https://embracethered.com/blog/posts/2024/whoami-conditional-prompt-injection-instructions/
[0.1]: https://kai-greshake.de/posts/llm-malware/
[1]: https://www.tomshardware.com/news/intel-details-meteor-lakes-ai-acceleration-for-pcs-vpu-unit
[1.1]: https://www.extremetech.com/computing/intel-announces-an-integrated-neural-ai-accelerator-in-meteor-lake (hard to find a good explanation of these accelerators)
[2]: https://www.forbes.com/sites/siladityaray/2023/02/16/bing-chatbots-unhinged-responses-going-viral/
> a human reviewer has to confirm that the photos are illegal before any action is taken
The reviewer actually only confirms that the-photos-as-modified-by-an-undisclosed-transformation are illegal, and this strategic ambiguity is presumably designed to maintain plausible deniability that what Apple subjects its reviewers to is a) not psychologically damaging, and b) not illegal to possess.
Unfortunately this is "security by obscurity", so we have to assume it is vulnerable to image scaling attacks[0].
Unless Apple document their human review process, and allow an independent audit of the technology (to provide a strong mathematical guarantee that it is not susceptible to such image-smuggling attacks) and independent verification that the published process is the one they actually use, then there is an unacceptably high chance that a human reviewer will trigger a legal process against users for files they couldn't tell contained illegal data.
[0] https://embracethered.com/blog/posts/2020/husky-ai-image-rescaling-attacks/
we recently moved to a similar approach, inspired by gondolin which does the same: https://earendil-works.github.io/gondolin/secrets/
an 'mitm' tls proxy also gives you much better firewalling capabilities [1], not that firewalls aren't inherently leaky,
codex's a 'wildcard' based one [2]; hence "easy" to bypass [3]
github's list is slightly better [4] but ymmv
[1] than a rudimentary "allow based on nslookup $host" we're seeing on new sandboxes popping up, esp. when the backing server may have other hosts.
[2] https://developers.openai.com/codex/cloud/internet-access#common-dependencies
[3] https://embracethered.com/blog/posts/2025/chatgpt-codex-remote-control-zombai/
[4] https://docs.github.com/en/copilot/reference/copilot-allowlist-reference#copilot-cloud-agent-recommended-allowlist