ScrapingBee Blog

I'm working on a personal project that involves A LOT of scraping, and through several iterations I've gotten some stuff that works quite well. Here's a quick summary of what I've explored (both paid and free):

* Apify (https://apify.com/) is a great, comprehensive system if you need to get fairly low-level. Everything is hosted there, they've got their own proxy service (or you can roll your own), and their open source framework (https://github.com/apify/crawlee) is excellent.

* I've also experimented with running both their SDK (crawlee) and Playwright directly on Google Cloud Run, and that also works well and is an order-of-magnitude less expensive than running directly on their platform.

* Bright Data nee Luminati is excellent for cheap data center proxies ($0.65/GB pay as you go), but prices get several orders of magnitude more expensive if you need anything more thorough than data center proxies.

* For some direct API crawls that I do, all of the scraping stuff is unnecessary and I just ping the APIs directly.

* If the site you're scraping is using any sort of anti-bot protection, I've found that ScrapingBee (https://www.scrapingbee.com/) is by far the easiest solution. I spent many many hours fighting anti-bot protection doing it myself with some combination of Bright Data, Apify and Playwright, and in the end I kinda stopped battling and just decided to let ScrapingBee deal with it for me. I may be lucky in that the sites I'm scraping don't really use JS…

This is almost cool, unfortunately the product itself (a network of bots to allow websites to be scraped when they obviously don't want to be) seems a little shifty. For example, put these three exhibits together:

Exhibit 1: The ScrapingBee terms and conditions state "We assume that you use the Website Platform and Services legally and ethically and that you have obtained permission, if necessary, to use it on the targeted websites and/or other data sources." This is even backed up with an indemnity clause in which the user has to cover ScrapingBee for any third-party legal claim arising out of their use of the product.

Reference: https://www.scrapingbee.com/terms-and-conditions/

Exhibit 2: ScrapingBee explicitly advertises a feature allowing you to get Google search results via an API call. These results are presumably generated by scraping Google's search pages:

Reference: https://www.scrapingbee.com/features/google/

Exhibit 3: Google's own documentation explicitly states that automated querying is prohibited, so if you use this advertised ScrapingBee service, you are naturally violating Google's terms, and could be liable to cover ScrapingBee's legal costs if Google decide to come after them.

Reference: https://developers.google.com/search/docs/advanced/guidelines/automated-queries

$1MM in ARR is all well and good, but there's a limit to how large this business can grow without being pursued by the websites whose scraping they are enabling, and in the case of…

There's services specialized in crawling + VPN. Often used to crawl shopping websites or Google result pages. One example is https://www.scrapingbee.com and you select the country https://www.scrapingbee.com/documentation/country_codes/ They can do screenshots, too.

If the concern is uptime there's a couple of service that also do regular screenshots and alerts when the website is slow or unresponsive. https://www.debugbear.com/ for example.

For a super quick check (every couple of months I need this) I usually use https://www.webpagetest.org/ Under 'advanced configuration' there's a dropdown of locations around the world. I gives a screenshot but not access to the full HTML.

> Cloudflare can always spot crawlers

I mention in the post a scraping service that Cloudflare isn't spotting: https://www.scrapingbee.com/blog/how-to-bypass-cloudflare-antibot-protection-at-scale/

Plenty of open-source ones as well that could bypass, eg maybe this one that came up in search https://github.com/VeNoMouS/cloudscraper Combine with residential proxies and you're just not going to find them.

> Comprehensive crawl of LinkedIn, FB, instagram, IMDB, Amazon, would be worth a lot.

Just from a quick Google search:

- LinkedIn: https://brightdata.com/products/datasets/linkedin

- Amazon: https://www.junglescout.com/features/product-database/

Really depends on how big your scraping operation is going to be. These days there's a lot of "managed" providers that give you headless browsers / proxy rotators through an easy API so it's relatively easy to plug them into your code. Examples of these would be https://www.browserless.io or https://www.scrapingbee.com for headless browsers to render JS.

From my work experience of working on a large scraping stack with thousands of integrations, I can say that we are very happy with our own custom framework, written in Go (https://github.com/PuerkitoBio/goquery for HTML parsing) and using headless Chrome for JS rendering.

Theoretically yes! Unfortunately there's no API with more restrictive permissions :P

Good point around releasing the source code for self-hosting. The only painful parts are that I two commercial services here and there for stealth proxying requests to fetch articles like (https://www.scrapingbee.com/) and Puppeteer on Cloudflare Workers (https://developers.cloudflare.com/browser-rendering/platform/puppeteer/) to generate PDFs, so it's like $50/month to self-host something that actually works

But maybe if I open source this others can contribute too and find ways to make this cheaper to run