Robb Knight writes about personal websites, RSS, bots, nginx, macOS fixes, and small apps like TrackerZapper.
HN commenters cite his finding that Perplexity misstates its user agent, which sparked debate, and link his posts on styling RSS feeds.
> What is this post about https://rknight.me/blog/blocking-bots-with-nginx/
He is asking Perplexity to summarize a single page. This is simply automation for opening a browser, navigating to that URL, copying the content, pasteing the content into Perplexity.
This is not automated crawling or indexing. Since the person is driving the action. An automated crawler is driven into action by a bot.
Nor is this article added into the foundational model. It's simply in a person's session context.
If for some reason, the community deems this as automated crawling or indexing. One could write an extension to automate the process of copying the article content & pasting the content into an LLM/Rag like Perplexity.
AI scrapers are pretty widely ignoring robots.txt, and plenty lie about their user agents. https://rknight.me/blog/perplexity-ai-is-lying-about-its-user-agent/
I'd fully expect OpenAI to do some checks that their bot isn't getting different responses than a seemingly real request.
it works, i think the most visible ones are where people style their atom / rss feeds instead of rendering separate xml / html pages https://rknight.me/blog/styling-rss-and-atom-feeds/
Because that's literally what the author does in TFA and then complains about when Perplexity complies.
> What is this post about https://rknight.me/blog/blocking-bots-with-nginx/
Don't forget TrackerZapper to strip tracking codes from copied URLs: https://rknight.me/apps/tracker-zapper/