Browserless Blog

Thanks for sharing your experience! I'm quite opinionated on this topic so buckle up :D

We avoided the fork & patch route because it's both labor intensive for a limited return on investment, and a game of catching up. Updating the forked framework is challenging on its own right, let alone porting existing customer payloads to newer versions, locking you de-facto to older versions. I did maintain a custom fork at a previous workplace that was similar in scope to Browserless[0] and I can tell you it was a real pain.

Developing your own framework (besides satisfying the obvious NIH itch) allows you to precisely control your exposure (reduce the attack surface) from a security perspective, and protects your customers from upstream decisions such as deprecations or major changes that might not be aligned with your customer requirements. I also have enough experience in this space to know exactly what we need to implement and the capabilities we want to enable. No bloat (yet)

> you will burn the user's fingerprint depending on scale

It's relative to your activity. See my other comment about scale and use cases, for personal device usage this is not an issue in practice, and users can automate several websites[1] using their personal agents without worrying about this. For more involved scenarios we have appropriate strategies that avoid this issue.

> we fix leaks and bugs of automation frameworks

Sounds interesting! I'd love to read a write up, or PRs if you have…

Hey justhw, I launched something similar [1] a few months back. I’d love to talk with you about your stack. Here are the three biggest problems I ran into.

1. Request timeouts

Even "fast" blogs sometimes have hiccups or rate-limit scrapers/crawlers. The more I scale the more often I see requests timing out. I've tried to mitigate this with retry buttons and trying the request again, but it doesn't always work.

2. Scaling Puppeteer

I saw from another of your comments that this is built on top of Puppeteer. From my experience, trying to scale to even 3 concurrent renders starts to fall apart when you have to spin up a new Chrome instance for each request.

I've looked into Browserless [2] but they charge per second of usage, and I'm not at the point where I want to have that big of an expense.

3. Getting a relevant image

I love that your tool adds an image to the thumbnail as well! I struggled with searching Unsplash using the text in the title of the article but they never came out that well.

I love it when people make these massively useful services stitching together a few simple tools. Congrats on the launch!

[1] https://mugshotbot.com
[2] https://www.browserless.io

Weird question: Since you mention 'bare-metal' and this is "enables you to run Postgres in the browser"

UH, could you do:

bare_metal --> HeadlessGPU-Chrome --> PGLite with whatever orchesrtration and you have a an automatable 'browser' that can scrape/store/logic/log/be-config'd from/lookup directly to your browserDB

and maybe run a fastAPI ontop of such that you have a full stack, baremetal, headless, browserDB/fastAPI/endpoint processing doohicky?

Or does this sound dumb?

Because this is exactly what I actually need and am attempting to create - but its beyond my capabilities in any functional/elegant implementation...

https://www.browserless.io/blog/browserless-gpu-instances

https://github.com/dhamaniasad/HeadlessBrowsers

I'm not sure if I'd call it simple, but the product itself is fairly straightforward (at least to me :). I run a SaaS product called browserless (https://browserless.io). Goal here was to make it really simple to run headless browser work in a production environment, without worrying about all the provisioning and maintenance.

However, like anything simple on the exterior, there's still layers of complexity behind it. How do you charge users? Where do you host? How do you acquire users? How do you interact with customers?

Don't mean to be cynical, but even $500/mo is going to have some cost and time complexity. Best of luck!

Really depends on how big your scraping operation is going to be. These days there's a lot of "managed" providers that give you headless browsers / proxy rotators through an easy API so it's relatively easy to plug them into your code. Examples of these would be https://www.browserless.io or https://www.scrapingbee.com for headless browsers to render JS.

From my work experience of working on a large scraping stack with thousands of integrations, I can say that we are very happy with our own custom framework, written in Go (https://github.com/PuerkitoBio/goquery for HTML parsing) and using headless Chrome for JS rendering.

For more generic web indexing you need to use a browser. You do not index pages served by a server anymore, you index pages rendered by javascript apps in the browser. So as a part of the "fetch" stage I usually let parsing of title and other page metadata to a javascript script running inside the browser (using https://www.browserless.io/) and then as part of the "parse" phase I use cheerio to extract links and such. It is very tempting to do everything in the browser, but architecturally it does not belong there. So you need to find the balance that works best for you.