The Archive.today blog is where the operator of the web archiving service answers questions about how it works.
HN links it for the owner's answers on topics like the scraper it uses, and when people wonder who runs the service and why.
Brave got a tap on the shoulder from the government and was told they had to collect all that data in order to release crypto payments to the owners of the websites they'd been collecting micropayments for:
https://preview.redd.it/52bkflf0cld31.jpg?width=462&format=pjpg&auto=webp&s=8b37760e51e638b965ca2ad0b37f4de00d77b885
https://support.brave.com/hc/en-us/articles/360032158891-What-is-KYC-
I disagree with a lot of Brave's approach. However their experience shows that this problem isn't theoretical. One company has already managed to deliver frictionless micropayments, and was told by the government that if they didn't add some friction back in (AML/KYC), executives would be going to prison.
Not cool.
This also led to a real mess with the operators of archive.today, who don't live in the USA and apparently aren't supported by the AML/KYC process being used. So a fairly large amount of money collected in order to support them is stuck and can't be paid out:
https://blog.archive.today/post/626174398020403200/please-provide-all-the-details-about-how-brave
No, they run their own crawler against the URL you provide (IIRC, it's a headless Google Chrome [1]). For some special / often-requested (?) sites the people behind the project have:
- implemented post-processing to strip ads and tracker beacons
- bought legitimate accounts that they use to bypass paywalls (some of them show up as remnants in the archived pages' code, some of them show up when you request an archive of the site's account management portal, some show up as HTTP traces during the archival process - the crawler will dump every URL it requests on a "WIP" page, and for some, auth information like JWTs is passed in the URL)
- implemented anti-nagwall features (some sites are "vulnerable" in their paywall/"you read X articles, end of free limit" by pressing the ESC key to disrupt the JS of the site)
[1] https://blog.archive.today/post/618635148292964352/what-scraper-or-headless-browser-are-you-using-it
Archive.today has a different approach to the baseline archive technology (executing javascript at archival time and saving the DOM instead of saving and replaying server responses verbatim). Additionally, Archive.today employs a number of site specific mitigations which aren't visible to the end user. In some cases, for instance, they use accounts, but then retroactively modify the DOM to mask this mitigation. [0] While the exact strategy they use for Twitter isn't known to me, they are doing something by their own admission. [1]
[0] https://blog.archive.today/post/708008224368001024/why-isnt-instagram-archive-working compounded with personal observation.
[1] https://blog.archive.today/post/708565142782246912/pretty-please-remove-the-recent-restrictions-for
To add, apparently another reason is that he believes using Cloudflare as your recursive resolver could lead to phishing[0]:
> the same entity which answers your DNS queries is able to issue SSL certs for any domain, so using CloudFlare DNS you never know whether you access the original website or a fishing one
Generally this is protected via certificate transparency+CAA records. If CF's CA were to issue a bad certificate, it'd be blocked by the browser and, should it get out, jeopardize the entire company, likely DigiCert as well given they cross-signed Cloudflare's issuing CA.
0: https://blog.archive.today/post/634795612966125568/when-will-your-site-be-accessible-from-cloudflare#notes
archive.today or archive.is - Wikipedia: https://en.wikipedia.org/wiki/Archive.today
Help:Using archive.today - Wikipedia: https://en.wikipedia.org/wiki/Help:Using_archive.today
archive.today - FAQ : https://archive.md/faq
archive.today - wiki : https://wiki.archiveteam.org/index.php/Archive.today
Archive Team wiki : https://wiki.archiveteam.org/
archive.today - Blog : https://blog.archive.today/
Tumblr : https://archive-is.tumblr.com/
Twitter : https://twitter.com/archiveis
archive.today
archive.ph
archive.is
archive.li
archive.vn
archive.fo
archive.md
archiveiya74codqgiixo33q62qlrqtkgmcitqx5u2oeqnmn5bpcbiyd.onion
Launched May 16, 2012; 11 years ago
>What scraper or headless browser are you using? it works so well.
>Before 2019 - PhantomJS, after - ordinary (not headless) Chromium/80 with few small patches.
https://blog.archive.today/post/618635148292964352/what-scraper-or-headless-browser-are-you-using-it (2020)
>Archive.today launches real browsers (not even headless) and tries to load lazy images, unroll folded content, login into accounts if prompted with login form, remove “subscribe our maillist” modals
https://blog.archive.today/post/642952252228812800/people-often-compare-various-features-of