Blaise Agüera y Arcas, a Google researcher, writes essays on AI, machine learning ethics, and the nature of intelligence.
Commenters recommend his essays as rebuttals to AI claims about detecting sexual orientation and physiognomy, and discuss his piece on whether language models understand.
You're absolutely right to be concerned about this stuff, but be aware that it is generally acknowledged as a problem and that the "ethics of machine learning" is quite an interesting and active research topic.
One of the best articles I've read on the topic, if you're interested: https://medium.com/@blaisea/physiognomys-new-clothes-f2d4b59fdd6a
Necessary link to an article by Blaise Aguera y Arcas, the lead of an ML team at Google: "Physiognomy’s New Clothes" [0]
He gives several examples, and explains how these type of statements are extremely misleading (and dangerous!), and not something new at all...
https://medium.com/@blaisea/physiognomys-new-clothes-f2d4b59fdd6a
This is probaly the best article that you need to read after reading another one of those 'detecting sexual orientation with dating profile pictures/prison inmates/etc' papers:
https://medium.com/@blaisea/do-algorithms-reveal-sexual-orientation-or-just-expose-our-stereotypes-d998fafdf477
It's not really a complaint about the author's magical expectations that weren't met. If anything, it's a complaint about other people's claims like
https://medium.com/@blaisea/do-large-language-models-understand-us-6f881d6d8e75
This response to the research is quite good: https://medium.com/@blaisea/do-algorithms-reveal-sexual-orientation-or-just-expose-our-stereotypes-d998fafdf477
The takeaway: AIs can learn our stereotypes.