Anubis

If the rest of my post is accurate, that's not the actual concern, right? Since I'm not sure if the check itself is meaningful. From what is described in the documentation [1], I think the practical effect of this system is to block users running old mobile browsers or running browsers like Opera Mini in third world countries where data usage is still prohibitively expensive. Again, the off-the-shelf scraping tools [2] will be unaffected by any of this, since they're all built on top of Puppeteer, and additionally are designed to deal with the modern SPA web which is (depressingly) more or less isomorphic to a "proof-of-work".

If you are open to jumping on a call in the next week or two I'd love to discuss directly. Without going into a ton of detail, I originally started looking into this because the group I'm working with is exploring potentially funding a free CDN service for open source projects. Then this AI scraper stuff started popping up, and all of a sudden it looked like if these reports were true it might make such a project no longer economically realistic. So we started trying to collect data and concretely nail down what we'd be dealing with and what this "post-AI" traffic looks like.

As such, I think we're 100% aligned on our goals. I'm just trying to understand what's going on here since none of the second-order effects you'd expect from this sort of phenomenon seem to be present, and none of the places where we actually have direct data seem to show this…

> Work functions don't make sense as a token tax; there's actually the opposite of the antispam asymmetry there. Every bot request to a web page yields tokens to the AI company. Legitimate users, who far outnumber the bots, are actually paying more of a cost.

Agreed, residential proxies are far more expensive than compute, yet the bots seem to have no problem obtaining millions of residential IPs. So I'm not really sure why Anubis works—my best guess is that the bots have some sort of time limit for each page, and they haven't bothered to increase it for pages that use Anubis.

> with a content protection system built in Javascript that was deliberately expensive to reverse engineer and which could surreptitiously probe the precise browser configuration a request to create a new Youtube account was using.

> The next thing Anubis builds should be that, and when they do that, they should chuck the proof of work thing.

They did [0], but it doesn't work [1]. Of course, the Anubis implementation is much simpler than YouTube's, but (1) Anubis doesn't have dozens of employees who can test hundreds of browser/OS/version combinations to make sure that it doesn't inadvertently block human users, and (2) it's much trickier to design an open-source program that resists reverse-engineering than a closed-source program, and I wouldn't want to use Anubis if it went closed-source.

[0]: https://anubis.techaro.lol/docs/admin/configuration/challenges/preact

[1]:…

> yes, it's a cartoon character; get over it

I think it's cute. It wouldn't be a problem if it wasn't cute. It's inoffensive and doesn't get in the way of the software serving its function, the only critique I could do if I wanted to is people unfamiliar with the project would get a bit confused if they only saw the character for a split second: https://anubis.techaro.lol/

I think even Cloudflare gives you a second or two to see what's going on before forwarding you to the page behind it, that would be a nice UX improvement.

Otherwise, agreed about self-hosting and the ability to choose whatever platform aligns with your goals the closest to be a good thing!

> And now everyone has to learn GitHub Actions, Gitlab pipelines, Jenkins pipelines, and more.

A part of me doesn't like the churn of new CI solutions coming in all the time. On the other hand, after learning GitLab CI, I'd generally prefer it to something like Jenkins, it just feels more pleasant. But with something like Drone CI or Woodpecker CI my non-work needs are also covered wonderfully, especially with a lot of the software I build being packaged in Docker containers, which further simplifies and somewhat standardizes things! A lot of the time I can also encapsulate most of the logic in shell scripts that are easy to run regardless of the environment (CI server or locally), which makes it even more portable.

So first, let me prefix this by saying I generally don't accept cookies from websites I don't explicitly first allow, my reasoning being "why am I granting disk read/write access to [mostly] shady actors to allow them to track me?"

(I don't think your blog qualifies as shady … but you're not in my allowlist, either.)

So if I visit https://anubis.techaro.lol/ (from the "Anubis" link), I get an infinite anime cat girl refresh loop — which honestly isn't the worst thing ever?

But if I go to https://xeiaso.net/blog/2025/anubis/ and click "To test Anubis, click here." … that one loads just fine.

Neither xeserv.us nor techaro.lol are in my allowlist. Curious that one seems to pass. IDK.

The blog post does have that lovely graph … but I suspect I'll loop around the "no cookie" loop in it, so the infinite cat girls are somewhat expected.

I was working on an extension that would store cookies very ephemerally for the more malicious instances of this, but I think its design would work here too. (In-RAM cookie jar, burns them after, say, 30s. Persisted long enough to load the page.)

In the past 6 months, I've seen tons of news about the Anubis PoW. Also on HN, like https://news.ycombinator.com/item?id=43427679

So I decided to take a deeper look into it, and it turned out that not only does Anubis have a fragilely designed PoW (Native accel + Concurrent solver can solve it within 100ms), it also tends not to use PoW by default.

As a PoC, I've created a solver in Python, which can be used like:

- import requests from anubis_solver import solve cookie = solve("https://anubis.techaro.lol/") print(cookie) content = requests.get("https://anubis.techaro.lol/", headers={"Cookie": cookie}, verify=False).text -

I don't really like to think this way, but it made me doubt whether it's really possible to make something, especially FOSS ones, that can stop crawlers or similar stuff. Anubis, at least, doesn't seem "right" to me.

Yeah that’s a classic problem, and it's why good tests are such closely guarded secrets: to keep them from becoming training fodder for the next generation of models. Regarding the "model date" vs "benchmark date" - that's an interesting point... I'll definitely look into it!

I don't have any captcha systems in place, but I wonder if it might be worth putting up at least a few nominal roadblocks (such as Anubis [1]) to at least slow down the scrapers.

A few weeks ago I actually added some new, more challenging tests to the GenAI Text-to-Image section of the site (the “angelic forge” and “overcrowded flat earth”) just to keep pace with the latest SOTA models.

In the next few weeks, I’ll be adding some new benchmarks to the Image Editing section as well~~

[1] - https://anubis.techaro.lol