Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we run Firecracker VMs inside EC2 and start browsers in less than 1s

browser-use.com

Post reply on HN
Print view View original

141–150 of 251 posts

Re: How we run Firecracker VMs inside EC2 and start browsers in less than 1s

#141

Post by baby_souffle »

arianvanp wrote: post #93
baby_souffle wrote: post #79

Earlier quoted context omitted.

> Who uses service providers like this? I use change detection to monitor all sorts of websites for changes. Some of my favorite authors don't have RSS. I always set up price monitoring for any big ticket item I'm considering like appliances so I can see how their pricing changes over time. I also use scrapers for websites that don't have an API. I like having all of my purchase history indexed in a database where I…

The litmus test here is whether they support https://blog.cloudflare.com/introducing-pay-per-crawl/ out of the box or not They do not.

I don't know who 'they' is here but that's not the point? I would bet that a decent chunk of scraping happens because there is no API (or other machine-focused interface, like RSS).

"pay to crawl" sounds like the absolutely laziest possible way that a particular site could bolt on an API.

Re: How we run Firecracker VMs inside EC2 and start browsers in less than 1s

#142

Post by Dylan16807 »

saintfire wrote: post #140
orf wrote: post #50

Earlier quoted context omitted.

Was Rosa Parks unethical for sitting down on a bus? The point is that the context matters: both the users context and the context of the restriction. It’s not as clear cut as “ignoring restrictions = bad”. The restriction itself can be unethical, in the same way that bypassing a restriction can be unethical.

[flagged]

You're taking an analogy way too seriously so: you too.

Re: How we run Firecracker VMs inside EC2 and start browsers in less than 1s

#143

Post by Dylan16807 »

dist-epoch wrote: post #63
sillysaurusx wrote: post #22

Earlier quoted context omitted.

(I haven't tried this out yet.) My use case would be to take a snapshot of each HN story. This is surprisingly hard, because most websites prevent bots from doing that. For example, Claude has a lot of trouble reading HN's front page. HN itself is fine, but the moment you ask it to pick out an article, it often chokes. The website has put up a verification captcha, or it's a paywall, etc. Paywalls can be bypassed by…

> one bot taking a snapshot isn't going to make a difference until half of HN users start asking their agent to do the same, to summarize the top HN articles every day

I don't think half is a realistic number, and any realistic numbers are not going to change the server load much. The real cost of that scenario is all the AI compute.

Re: How we run Firecracker VMs inside EC2 and start browsers in less than 1s

#144

Post by OhSoHumble »

eab- wrote: post #69
mikeocool wrote: post #44

Earlier quoted context omitted.

Whether or not scrapping publically available websites is unethical is probably up for debate. In some cases at least, courts have found it to be legal, even when the site is throwing up technical barriers or issues cease and desists. What is likely unethical is the fact that they offer residential proxies. The residential providers of those proxies are frequently not aware they’ve been opted in to provide such a ser…

> courts have found it to be legal ≠ ethical

I don't know if this falls under good ethical behavior but the one time I needed to mass scrape sites was to build an affordable housing directory for the Bay Area. Whether or not a unit of housing was under rent control was only available through private entities or apartment hosting sites and it was scattered and hard to find.

In my opinion, a directory of subsidized housing should have been provided by local governments and not through a plethora of real estate websites.

Re: How we run Firecracker VMs inside EC2 and start browsers in less than 1s

#145

Post by mannanj »

arianvanp wrote: post #93
baby_souffle wrote: post #79

Earlier quoted context omitted.

> Who uses service providers like this? I use change detection to monitor all sorts of websites for changes. Some of my favorite authors don't have RSS. I always set up price monitoring for any big ticket item I'm considering like appliances so I can see how their pricing changes over time. I also use scrapers for websites that don't have an API. I like having all of my purchase history indexed in a database where I…

The litmus test here is whether they support https://blog.cloudflare.com/introducing-pay-per-crawl/ out of the box or not They do not.

has anyone been using this for success? wondering what kinds of pays they are getting. or are the crawlers just avoiding those sites.

Re: How we run Firecracker VMs inside EC2 and start browsers in less than 1s

#146

Post by Nathanba »

mystifyingpoi wrote: post #34
embedding-shape wrote: post #21

Earlier quoted context omitted.

> Seems very unethical, no? Who uses service providers like this? The whole point of anti-bot measures is to get rid of bots - you are not wanted there. Unethical just because it does something someone else doesn't want? I guess it depends on why and what the intention is. I don't have time to sit 24/7 in front of a computer to get a ticket to some events, does that mean it's unethical for me to use my own bot so I c…

> even if others don't think that thing should be automated It's an interesting thought that can be further explored. Could anything that's considered "unwanted" by a third party considered unethical, if I do it anyway? If the hotel self-service restaurant has a sign "don't take the food out" and I take 1 apple in my pocket for a snack, is it unethical? Or maybe the sign is just for people that would otherwise take $…

We ("society") do things to people that they don't want all the time, namely punishing them via the legal system. Then there are things that other people are doing that are immoral and should be punished for even though they are not illegal. And the whole class of inactions where we don't e.g buy something because it's overpriced and that's certainly something the seller doesn't like.

Re: How we run Firecracker VMs inside EC2 and start browsers in less than 1s

#147

Post by devmor »

baby_souffle wrote: post #79
losteric wrote: post #9

> Plain headless Chromium is easy to detect by websites with anti-bot measures. Plain headless Chromium avoided getting blocked by websites only 2% of the time, according to our stealth benchmark. > Our browsers avoid blocks 81% of the time on our stealth benchmark, and 84.8% on Halluminate BrowserBench, the highest of any provider. Seems very unethical, no? Who uses service providers like this? The whole point of an…

> Who uses service providers like this? I use change detection to monitor all sorts of websites for changes. Some of my favorite authors don't have RSS. I always set up price monitoring for any big ticket item I'm considering like appliances so I can see how their pricing changes over time. I also use scrapers for websites that don't have an API. I like having all of my purchase history indexed in a database where I…

This attitude, and by proxy this business are the epitome of selfish entitlement.

You state that you believe you deserve access to others’ resources, at their cost, despite their clear attempts to stop you from using them, simply because you want it.

Re: How we run Firecracker VMs inside EC2 and start browsers in less than 1s

#148

Post by otterley »

latchkey wrote: post #133
otterley wrote: post #130

Earlier quoted context omitted.

It depends on how you think about spare capacity on your hardware. They're an opportunity cost. Every idle cycle and bit of unallocated memory could be spent doing something else valuable. Consider the most extreme example of GPUs: CoreWeave was founded by repurposing nearly-useless GPUs their founders had on hand to mine Bitcoin (correction: Ethereum) that was obsoleted by ASICs and other specialized mining hardware…

Your CW analogy is wonky since that isn't how it went down. You know my name (I don't know yours. edit: Michael), so you should know a bit more about my history in the space too, right? I can explain it out, but afraid of either being called names, or just not being worth it to you (or me for that matter). EC2 has preemptable and reserved pricing. It is possible to build autosizing solutions, this is what Google did…

You're right to call me out on the facts about CW as the story's a bit different from what I remembered from reading an article about them a few months back (https://www.wired.com/story/coreweave-scrappy-cryptominer-mu...).

That said, I think you get the point about idle capital being waste. EC2 Spot Instances are a great example about how to turn that waste into revenue.

Re: How we run Firecracker VMs inside EC2 and start browsers in less than 1s

#149

Post by keynha »

wewewedxfgdf wrote: post #35

But Firecracker is not compatible with GPU for Chrome, is that right? That means Chrome is slow - quite the tradeoff.

Headless, so there is no screen to composite and no GPU passed into the VM. Firecracker has no GPU passthrough, so GL work falls back to SwiftShader, the software rasterizer. For automation that is fine. The cost is in layout, JS and network, not raster. It only bites on WebGL or canvas heavy pages.
Post reply on HN