Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we run Firecracker VMs inside EC2 and start browsers in less than 1s

browser-use.com

Post reply on HN
Print view View original

231–240 of 251 posts

Re: How we run Firecracker VMs inside EC2 and start browsers in less than 1s

#231

Post by gck1 »

Reformedot wrote: post #84
Dibby053 wrote: post #76

No mention of the tools/methods used to do the profiling, I think that would be the most interesting part. Also a bit surprising that a checkpoint with the browser running wouldn't just work. Is this some quirk of firecracker?

Checkpoint with Chromium running is possible and will be our next step. Main blockers right now is fingerprint injection and profile injection, solved already. It's always a balance of engineering effort & gains. Post-Chromium snapshot let's us save 200ms, which is not that important for 99% of use-cases, but that will come soon since it brings some other benefits (like CPU footprint) Profiling and tools used are alr…

> Main blockers right now is fingerprint injection and profile injection, solved already.

Do you do this at the chromium/V8 level or CDP?

I've been having mixed success with CDP and was thinking of going to the level below, but it feels like just getting Chromium itself to baseline chrome detection profile is significant work

Re: How we run Firecracker VMs inside EC2 and start browsers in less than 1s

#232

Post by pocksuppet »

zx8080 wrote: post #225
doublerabbit wrote: post #186

Earlier quoted context omitted.

I can't use soundcloud because I am on FreeBSD. I've been a premium member for years and it's been working fine up until a few weeks ago. Mobile App works, Windows Works. Changing user-agent doesn't work neither. When I try to load a track I get an CF XHR 403. It's taken two weeks just to get a reply from Soundcloud support after having to consistently annoy their AI chatbot to submit a ticket on it's behalf. If anyo…

Agriculture looks more and more appealing.

Billionaires own all the land and the equipment and although food is valuable, the money from the food you sell barely covers the cost of getting the land and the equipment from them.

Re: How we run Firecracker VMs inside EC2 and start browsers in less than 1s

#233

Post by pocksuppet »

losteric wrote: post #9

> Plain headless Chromium is easy to detect by websites with anti-bot measures. Plain headless Chromium avoided getting blocked by websites only 2% of the time, according to our stealth benchmark. > Our browsers avoid blocks 81% of the time on our stealth benchmark, and 84.8% on Halluminate BrowserBench, the highest of any provider. Seems very unethical, no? Who uses service providers like this? The whole point of an…

Which won't work, obviously, because as a bot operator, I'd just have my users provide their own IDs, or run another website to harvest IDs.

Re: How we run Firecracker VMs inside EC2 and start browsers in less than 1s

#234

Post by Reformedot »

iririririr wrote: post #222

> Plain headless Chromium is easy to detect by websites with anti-bot measures. what a disgusting business model. they are central to the main bot problem of our time. any one with morals would expect those systems to respect robots.txt and announce themselves via user agent strings. disgusting and i hope they crash and burn. i will actively spend time today looking for open source projects detecting their browsers a…

Im super interested on that, please let me know if you manage to detect their browsers!

Re: How we run Firecracker VMs inside EC2 and start browsers in less than 1s

#235

Post by Reformedot »

Natfan wrote: post #195
Reformedot wrote: post #83

Earlier quoted context omitted.

We decided to maintain Chromium as engine for stealth purposes. Browsers like LightPanda lack stealth at all, they are trivial to detect. There are ways to make Chromium more performant, by removing everything that you don't need. We believe that Chromium can reach that performance without starting an entire engine from scratch, and without losing stealth, a top priority for us. The language is not the problem, C++ i…

why are you making the internet worse for everyone with this "stealth" initiative? you are effectively lying to website operators.

I get why it can look like we're “lying” to website operators, that’s a fair concern.

But stealth, as we see it, isn’t about deception for abuse. It’s about making automated access behave closer to how real users interact with the web, in an ecosystem where most anti-bot systems default to blocking everything that isn’t explicitly whitelisted.

Right now, the model is broken. Unless you have a direct partnership, you're often locked out, even for legitimate use cases like research, monitoring, or building user-facing tools on top of public data.

We’re not supporting harmful behavior (credential stuffing, DDoS, piracy, etc.). The goal is to enable responsible access to publicly available information without forcing every use case into closed-door agreements.

There’s also a real tradeoff happening. Increasingly aggressive anti-bot measures (like harder CAPTCHAs) degrade the experience for actual users, while not necessarily stopping sophisticated automation, robots solve CAPTCHAs better than humans.

So the question isn’t “bots vs no bots” — it’s what kinds of automated access should exist, and under what norms. Right now, that line is blurry, and we think there’s room for better balance.

Happy to engage on where that line should be drawn.

Re: How we run Firecracker VMs inside EC2 and start browsers in less than 1s

#236

Post by Reformedot »

gck1 wrote: post #231
Reformedot wrote: post #84

Earlier quoted context omitted.

Checkpoint with Chromium running is possible and will be our next step. Main blockers right now is fingerprint injection and profile injection, solved already. It's always a balance of engineering effort & gains. Post-Chromium snapshot let's us save 200ms, which is not that important for 99% of use-cases, but that will come soon since it brings some other benefits (like CPU footprint) Profiling and tools used are alr…

> Main blockers right now is fingerprint injection and profile injection, solved already. Do you do this at the chromium/V8 level or CDP? I've been having mixed success with CDP and was thinking of going to the level below, but it feels like just getting Chromium itself to baseline chrome detection profile is significant work

> Do you do this at the chromium/V8 level or CDP?

Deepest level possible, harder but required for some workflows

Re: How we run Firecracker VMs inside EC2 and start browsers in less than 1s

#237

Post by pocksuppet »

sroussey wrote: post #41
joatmon-snoo wrote: post #28

Earlier quoted context omitted.

An example I ran into recently: I wanted to scrape pricing data for used cars, to better inform a friend's decision about what to purchase. I know there's a relationship between mileage and depreciation, but wanted to have a better sense of what that relationship is to know whether a given car was over or underpriced. Similarly, if I was pulling that data to build a service of my own to offer to users... is that unet…

All of these questions are easily answered by the question: can I run the bot on the same PC I use regularly? If so, then do it there. If not, then don’t do it at all.

This is often really good for your bot, because anti-bot providers are loathe to block what, as far as they know, is a residential CGNAT address. Sometimes you get more success scraping from home with Firefox or Chrome, than with an army of proxy networks.

Re: How we run Firecracker VMs inside EC2 and start browsers in less than 1s

#238

Post by baby_souffle »

saltmate wrote: post #166
baby_souffle wrote: post #79

Earlier quoted context omitted.

> Who uses service providers like this? I use change detection to monitor all sorts of websites for changes. Some of my favorite authors don't have RSS. I always set up price monitoring for any big ticket item I'm considering like appliances so I can see how their pricing changes over time. I also use scrapers for websites that don't have an API. I like having all of my purchase history indexed in a database where I…

Did you ask them for an RSS feed? Lots of people are pretty reasonable for such requests if you write a nice email.

> Did you ask them for an RSS feed? Lots of people are pretty reasonable for such requests if you write a nice email.

Yep!

They're busy people or just don't feel the need to do anything beyond hit the "publish" button on their CMS and call it good and that's fine / why I have a robot to make an RSS for me :).

Re: How we run Firecracker VMs inside EC2 and start browsers in less than 1s

#239

Post by baby_souffle »

ssl-3 wrote: post #177
hirsin wrote: post #164

Earlier quoted context omitted.

Look, it wasn't _my_ request that made the server fall over, it must have been one of the other several thousand thoughtless scrapers running on the website that caused it to die.

There was a time when a person could walk through a few department stores every week (or even every day) just to take note of some prices along the way, and ultimately tabulate them to try to identify and snatch up the best deal once it happens. And if everyone did this, it'd be a real problem. The stores would be clogged up by geeks writing notes in little books with Parker Jotters and just basically wasting space a…

> There was a time when a person could walk through a few department stores every week (or even every day) just to take note of some prices along the way, and ultimately tabulate them to try to identify and snatch up the best deal once it happens.

This is how it started! I noticed certain things during my weekly shop that I did a double-take on and thought "wasn't that $cheaper last week!?". Took me ~ 45 min to figure out that the retailer actually has a really nice graphQL endpoint that powers the "view your previous receipts" function on their website. Of course they don't document this / make it available for 3rd parties... so scrape it is!

I wrote a bot to dump every receipt into a sqlite DB and I fire it up ~ weekly to pull down receipts that it doesn't have locally.

Turns out, not _everything_ has gotten more expensive @ my local grocery store over the past few years... just most things have :/.

> But not many people ever did that in stores,

There's a cottage-industry of firms out there that get gig-workers to pop in to $randomStore and take a picture of $randomItem on shelf w/ the price tag in the photo. The firms sell this info to stores that want to know how a competitor might be doing pricing / placing certain items on the more valuable shelf spots.

> and not many individual people are doing that today with the web. It's really not a problem.

That's my point! I scrape a few hundred pages per day across _many_ domains. My bots respect 429s and they have some other backoff/random-jitter strategies baked in to _not_ be the reason anti-scrape proliferates.

Re: How we run Firecracker VMs inside EC2 and start browsers in less than 1s

#240

Post by inigyou »

fc417fc802 wrote: post #217
inigyou wrote: post #199

Earlier quoted context omitted.

This is called "victim blaming". You are saying the blame for a problem shouldn't be on those who directly caused the problem, but on those who failed to prevent them from causing the problem. You're right but in a different way. Scalpers aren't independent, they work for the artists to maximise artist revenue while absorbing the PR hit themselves.

No, I am saying that the people who went and created conditions that they knew would lead to the problem are the ones to blame. They are not victims except perhaps of their own poor decisions. I explicitly do not think that scalpers are doing anything wrong given that in a capitalistic system someone is always going to arbitrage things. You don't get to enact poor policy, stick your fingers in your ears, then blame e…

Policy is enacted by the government and by Cloudflare, not by web scrapers.
Post reply on HN