How many tabs do you use per server?
How we run Firecracker VMs inside EC2 and start browsers in less than 1s
71–80 of 251 posts
Re: How we run Firecracker VMs inside EC2 and start browsers in less than 1s
#72Earlier quoted context omitted.
> Seems very unethical, no? Who uses service providers like this? The whole point of anti-bot measures is to get rid of bots - you are not wanted there. Unethical just because it does something someone else doesn't want? I guess it depends on why and what the intention is. I don't have time to sit 24/7 in front of a computer to get a ticket to some events, does that mean it's unethical for me to use my own bot so I c…
Its unethical because you're intentionally bypassing restrictions. Just because others do it doesn't mean its okay. If you saw a sign in a store that said "1 per person" or "for registered guests only", would you ignore it?
Look at what Google's doing right now with Chrome. On June 30 Chrome will remove the last flag that let uBlock keep working, and there's no workaround. Google says it's about security and performance, but is it? $239 billion in ad revenue last year seems to be the motivational factor. The "restriction" is a rule written by the company that profits when you can't block its ads, dressed up as protecting you. But... CISA recommends ad blockers as a defense against malware spread through ad networks.
The rules aren't always right and sometimes have unintended consequences. I think a bigger issue than Browser Use is all of the copyrighted material in every LLM. Given that precedent has been set with zero legal consequences, I'm not sure there's much of a leg for you to stand on here.
Re: How we run Firecracker VMs inside EC2 and start browsers in less than 1s
#73have you tried running android browsers ? we run RL workloads using android browsers. We are having to maintain a fork of https://github.com/budtmo/docker-android/ and android chrome on top. We would rather use browser-use if it had that support. P.S. we do maintain our fork of a browser for rubric computation...but that is not relevant for this. The infrastructure is what we are looking for.
Why do this? I can’t see that it would be better than running chrome.
Re: How we run Firecracker VMs inside EC2 and start browsers in less than 1s
#74The article doesn't mention docker at all. I don't understand why containers are not viable solution for headless browsers.
Docker doesn't provide any security. You install Docker on your local laptop, and the container you spin up when you execute `docker run` interacts with your laptop's kernel directly. It provides logical isolation between containers but provides zero protection for your host kernel (assuming you decide to install Docker on a remote server instead).
Firecracker provides an isolation between the host kernel, on the one hand, and the guest microVM, on the other hand. So on AWS, you use an Amazon Machine Image (AMI) to specify the OS and other components and libraries installed on an EC2 server such as c5.metal, or if you're using nested virtualization, you can use c8i, s8i, or m8i instances at a discount of about 80%-90% at some performance and other cost, and you bundle Linux along with the Firecracker binary. Then you compile a build artifact including `rootfs` for the Firecracker baked image which is the microVM image (analogous to a Docker image that results from executing `docker build`). But the microVM process has its own virtual kernel and is a guest on the host machine. So for instance, you can place Docker inside the microVM, then the container is executing against the microVM kernel, not the host EC2 kernel. Communication is achieved securely between the two using `vsock` and probably something like `socat` so that data travels, say, from guest RAM to host RAM directly to an S3 quarantine bucket, for instance, without ever touching the host's kernel or filespace.
Re: How we run Firecracker VMs inside EC2 and start browsers in less than 1s
#75> Plain headless Chromium is easy to detect by websites with anti-bot measures. Plain headless Chromium avoided getting blocked by websites only 2% of the time, according to our stealth benchmark. > Our browsers avoid blocks 81% of the time on our stealth benchmark, and 84.8% on Halluminate BrowserBench, the highest of any provider. Seems very unethical, no? Who uses service providers like this? The whole point of an…
The people who've been in charge of the web (i.e., mostly the browser makers, but also the owners of the most popular sites) have made decisions that are IMHO severely anti-user. Although these anti-user design decisions have been accumulating for 30 years, users have had no alternative because all the content was on the web with way to get it other than to visit web sites with a web browser.
Now that there is an alternative (namely AI) people (including me) are flocking to the alternative. You want frame this as unethical bots versus ethically-acceptable human site visitors, but the main motivation for the use of scraping bots these days is to provide services (i.e, AI-based question answering) that users (like me) consider far superior to going directly to web sites for information because visiting web sites with a web browser is a frustrating tedious experience.
Re: How we run Firecracker VMs inside EC2 and start browsers in less than 1s
#76No mention of the tools/methods used to do the profiling, I think that would be the most interesting part.
Also a bit surprising that a checkpoint with the browser running wouldn't just work. Is this some quirk of firecracker?
Re: How we run Firecracker VMs inside EC2 and start browsers in less than 1s
#77We run a screenshot API (ApiFlash) with Chromium packaged in an AWS Lambda container image instead of Firecracker on EC2. AWS Lambda gives you the isolation and autoscaling for free which is ideal for spiky stateless work like screenshots. I believe we get mostly the same benefits compared to browser-use solution but with a much much simpler architecture. The tradeoff is the AWS lambda cold starts, but in practice sequential AWS Lambda invocations actually reuse a hot function. As a result, with a large enough volume, spikes are smoothed and cold starts are not that frequent.
Re: How we run Firecracker VMs inside EC2 and start browsers in less than 1s
#78[flagged]
> Browser stealth has been a solved problem for some time now
That's not true. Bots can still automate the web and there's demand for products that allow it. It's harder than years ago, but not impossible.
Defenders are always in favor, but the demand for automating the web exists, so research keeps going. There are ways to hide everything, including residential proxies.
For reference, I'm the blog author, and I have another one talking about this topic: https://browser-use.com/posts/bot-detection
Re: How we run Firecracker VMs inside EC2 and start browsers in less than 1s
#79> Plain headless Chromium is easy to detect by websites with anti-bot measures. Plain headless Chromium avoided getting blocked by websites only 2% of the time, according to our stealth benchmark. > Our browsers avoid blocks 81% of the time on our stealth benchmark, and 84.8% on Halluminate BrowserBench, the highest of any provider. Seems very unethical, no? Who uses service providers like this? The whole point of an…
> Who uses service providers like this?
I use change detection to monitor all sorts of websites for changes. Some of my favorite authors don't have RSS. I always set up price monitoring for any big ticket item I'm considering like appliances so I can see how their pricing changes over time. I also use scrapers for websites that don't have an API. I like having all of my purchase history indexed in a database where I can do analysis.
> These kinds of services inevitably make the web more human-hostile and expensive.
I would rather not have to spend more time circumventing stupid bot detection things. I would be more than happy to pay for access to some of this data that I cannot access any other way.. but sure, let's keep burning resources on a cat and mouse game that scrapers will always be able to win.
Re: How we run Firecracker VMs inside EC2 and start browsers in less than 1s
#80> During a burst in traffic, the system, instead of reacting on its own, required humans to adjust it. Isn't this solvable with autoscaling? how is this not an issue with Firecracker as well?
Our previous solution (Unikraft) did not supported auto-scaling
That's why we moved to a fully in-house solution with Firecracker and auto-scaling on EC2