Researchers, notably Nicolas Papernot, write about attacks on machine learning, adversarial examples, privacy and differential privacy.
HN commenters cite it for adversarial example transferability and for a post showing federated learning can be exploited by an untrusted server.
Clever Hans was a real comically smart horse!
https://en.wikipedia.org/wiki/Clever_Hans
>Clever Hans (German: der Kluge Hans; c. 1895 – c. 1916) was a horse that was claimed to have performed arithmetic and other intellectual tasks. After a formal investigation in 1907, psychologist Oskar Pfungst demonstrated that the horse was not actually performing these mental tasks, but was watching the reactions of his trainer. He discovered this artifact in the research methodology, wherein the horse was responding directly to involuntary cues in the body language of the human trainer, who was entirely unaware that he was providing such cues. In honour of Pfungst's study, the anomalous artifact has since been referred to as the Clever Hans effect and has continued to be important knowledge in the observer-expectancy effect and later studies in animal cognition. Pfungst was an assistant to German philosopher and psychologist Carl Stumpf, who incorporated the experience with Hans into his further work on animal psychology and his ideas on phenomenology.
NLP's Clever Hans Moment Has Arrived (thegradient.pub)
https://thegradient.pub/nlps-clever-hans-moment-has-arrived/
https://news.ycombinator.com/item?id=20861586
Finding and removing Clever Hans: Using explanation methods to debug and improve deep models:
https://www.sciencedirect.com/science/article/pii/S1566253521001573
Is your AI a “Clever Hans”?
https://medium.com/high-stakes-design/is-your-ai-a-clever-hans-736214670e00
The…
Yes:
> Adversarial training seeks to improve the generalization of a model when presented with adversarial examples at test time by proactively generating adversarial examples as part of the training procedure. This idea was first introduced by Szegedy et al. [SZS13] but was not yet practical because of the high computation cost of generating adversarial examples. Goodfellow et al. showed how to generate adversarial examples inexpensively with the fast gradient sign method and made it computationally efficient to generate large batches of adversarial examples during the training process [GSS14]. The model is then trained to assign the same label to the adversarial example as to the original example—for example, we might take a picture of a cat, and adversarially perturb it to fool the model into thinking it is a vulture, then tell the model it should learn that this picture is still a cat. An open-source implementation of adversarial training is available in the cleverhans library and its use illustrated in the following tutorial.
http://www.cleverhans.io/security/privacy/ml/2017/02/15/why-attacking-machine-learning-is-easier-than-defending-it.html
https://arxiv.org/abs/1312.6199
https://arxiv.org/abs/1412.6572
This seems to be a newly hot topic, whether to fight adversarial inputs or looking for efficiencies. I bookmarked a few relevant blogs (but really need to read more deeply)
http://composition.al/blog/2017/05/31/proving-that-safety-critical-neural-networks-do-what-theyre-supposed-to-where-we-are-where-were-going-part-2-of-2/
http://www.cleverhans.io/security/privacy/ml/2017/06/14/verification.html
https://blog.foretellix.com/
and another preprint: https://arxiv.org/abs/1706.10268
That's not entirely true:
> If we transfer adversarial examples from one model to another model trained with one of these defenses, the attack often succeeds, even when a direct attack on the second model would fail [PMG16].
http://www.cleverhans.io/security/privacy/ml/2017/02/15/why-attacking-machine-learning-is-easier-than-defending-it.html
AFAIK Federated Learning is not a magic solution. This post explains how you can exploit FL if there's some untrusted server (http://www.cleverhans.io/2022/04/17/fl-privacy.html)