Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Lightpanda, an open-source headless browser in Zig

github.com

Post reply on HN
Print view View original

121–130 of 141 posts

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#121

Post by fbouvier »

cush wrote: post #52
fbouvier wrote: post #18

Earlier quoted context omitted.

The cost of the browser part is still a problem. In our previous startup, we were scraping >20 millions of webpages per day, with thousands of instances of Chrome headless in parallel. Regarding the RAM usage, it's still ~10x better than Chrome :) It seems to be coming mostly from v8, I guess that we could do better with a lightweight JS engine alternative.

> there are hundreds of Web APIs, and for now we just support some of them (DOM, XHR, Fetch) > it's still ~10x better than Chrome Do you expect it to stay that way once you've reached parity?

I don't expect it to change a lot. All the main components are there, it's mainly a question of coverage now.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#122

Post by fbouvier »

zlagen wrote: post #83
corford wrote: post #67

Earlier quoted context omitted.

Not sure what category of ecomm sites you were scraping but I scrape >10million ecomm URLs daily and, honestly, in my experience the compute is not a major issue (8 times out of 10 you can either use API endpoints and/or session stuffing to avoid needing a browser for every request; and in the 2 out of 10 sites where you really need a browser for all requests it's usually to circumvent aggressive anti-bot which means…

fully agree here, using a browser for everything is the dumb way. You just usually use it to circumvent the blocking and then reuse the cookies to call the endpoints directly.

It might works if you need to handle a few websites. But this retro engineering approach is not maintainable if you want to handle hundreds or thousands of websites.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#123

Post by hombre_fatal »

bityard wrote: post #30
fbouvier wrote: post #2

Author here. The browser is made from scratch (not based on Chromium/Webkit), in Zig, using v8 as a JS engine. Our idea is to build a lightweight browser optimized for AI use cases like LLM training and agent workflows. And more generally any type of web automation. It's a work in progress, there are hundreds of Web APIs, and for now we just support some of them (DOM, XHR, Fetch). So expect most websites to fail or c…

Please put a priority on making it hard to abuse the web with your tool. At a _bare_ minimum, that means obeying robot.txt and NOT crawling a site that doesn't want to be crawled. And there should not be an option to override that. It goes without saying that you should not allow users to make hundreds or thousands of "blind" parallel requests as these tend to have the effect of DoSing sites that are being hosted on…

This is HN virtue signaling. Some fringe tool that ~nobody uses is held to a different, weird standard and must be the one to kneecap itself with a pointless gesture and a fake ethical burden.

The comparison to DRM makes sense. Gimping software to disempower the end user based on the desires of content publishers. There's even probably a valid syllogism that could make you bite the bullet on browsers forcing you to render ads.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#124

Post by aniviacat »

emporas wrote: post #104
10000truths wrote: post #55

Earlier quoted context omitted.

The issue is that DOM structure does not correspond one-to-one with perceived structure. I could render things in the DOM that aren't visible to people (e.g. a transparent 5px x 5px button), or render things to people that aren't visible in the DOM (e.g. Facebook's DOM obfuscation shenanigans to evade ad-blocking, or rendering custom text to a WebGL canvas). Sure, most websites don't go that far, but most websites al…

Servers operate on top of tcp/ip not to serve information, rather to serve information plus something else, usually ads. This is usually implemented with websites and captchas n stuff. That's a problem of misaligned economic incentives. If there is a blockchain which enables micro-transactions of 0.000001 cent per request, and in the order of a million tps or a billion tps, then servers have no reason not to accept m…

No need for blockchain, microtransaction functionality should be integrated into our existing payment methods.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#125

Post by calvinmorrison »

gkbrk wrote: post #43
bityard wrote: post #30

Earlier quoted context omitted.

Please put a priority on making it hard to abuse the web with your tool. At a _bare_ minimum, that means obeying robot.txt and NOT crawling a site that doesn't want to be crawled. And there should not be an option to override that. It goes without saying that you should not allow users to make hundreds or thousands of "blind" parallel requests as these tend to have the effect of DoSing sites that are being hosted on…

Please don't. Software I installed on my computer needs to the what I want as the user. I don't want every random thing I install to come with DRM. The project looks useful, and if it ends up getting popular I imagine someone would make a DRM-free version anyway.

i still wont forgive libtorrent for not implementing sequential access.

and also, xpdf for implementing the "you cant select text" feature

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#126

Post by evanjrowley »

I'm interested to see if this could be made to work as a drop-in replacement for the headless Chromium that Hoarder uses to archive web content. I don't have a problem with the current Hoarder solution, but it would be nice to use something that requires less RAM.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#127

Post by GuB-42 »

gkbrk wrote: post #43
bityard wrote: post #30

Earlier quoted context omitted.

Please put a priority on making it hard to abuse the web with your tool. At a _bare_ minimum, that means obeying robot.txt and NOT crawling a site that doesn't want to be crawled. And there should not be an option to override that. It goes without saying that you should not allow users to make hundreds or thousands of "blind" parallel requests as these tend to have the effect of DoSing sites that are being hosted on…

Please don't. Software I installed on my computer needs to the what I want as the user. I don't want every random thing I install to come with DRM. The project looks useful, and if it ends up getting popular I imagine someone would make a DRM-free version anyway.

Who told you about DRM? It is an open source tool.

Simply requiring a code change and a rebuild is enough of a barrier to prevent rude behavior from most people. You won't stop competent malicious actors but you can at least encourage good behavior. If popular, someone will make a fork but having the original refuse to do stuff that are deemed abusive sends a message.

It is like for the Flipper Zero. The original version does not let you access frequency bands that are illegal in some countries, and anything involving jamming is highly frowned upon. Of course, there are forks that let you do these things, but the simple fact that you need to go out of your way to find these should tell you it is not a good idea.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#128

Post by tossandthrow »

concerndc1tizen wrote: post #114
tossandthrow wrote: post #44

Earlier quoted context omitted.

Where do you read DRM? Parent commenter merely and humbly asks the author of the library to make sure that it has sane defaults and support for ethical crawling. I find it disturbing that you would recommend against that.

> I find it disturbing Oh no, someone on the internet found something offensive!

Disturbing, not offensive - it is literally right there in the quote you have been so nice to pass along.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#129

Post by buzzerbetrayed »

bityard wrote: post #30
fbouvier wrote: post #2

Author here. The browser is made from scratch (not based on Chromium/Webkit), in Zig, using v8 as a JS engine. Our idea is to build a lightweight browser optimized for AI use cases like LLM training and agent workflows. And more generally any type of web automation. It's a work in progress, there are hundreds of Web APIs, and for now we just support some of them (DOM, XHR, Fetch). So expect most websites to fail or c…

Please put a priority on making it hard to abuse the web with your tool. At a _bare_ minimum, that means obeying robot.txt and NOT crawling a site that doesn't want to be crawled. And there should not be an option to override that. It goes without saying that you should not allow users to make hundreds or thousands of "blind" parallel requests as these tend to have the effect of DoSing sites that are being hosted on…

Nerfing developer tools to save the "open web" is such a fucking backward argument.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#130

Post by bsnnkv »

bityard wrote: post #30
fbouvier wrote: post #2

Author here. The browser is made from scratch (not based on Chromium/Webkit), in Zig, using v8 as a JS engine. Our idea is to build a lightweight browser optimized for AI use cases like LLM training and agent workflows. And more generally any type of web automation. It's a work in progress, there are hundreds of Web APIs, and for now we just support some of them (DOM, XHR, Fetch). So expect most websites to fail or c…

Please put a priority on making it hard to abuse the web with your tool. At a _bare_ minimum, that means obeying robot.txt and NOT crawling a site that doesn't want to be crawled. And there should not be an option to override that. It goes without saying that you should not allow users to make hundreds or thousands of "blind" parallel requests as these tend to have the effect of DoSing sites that are being hosted on…

Looking at the responses here, I'm glad I just chose to paywall to protect against LLM training data collection crawling abuse.[1]

[1]: https://lgug2z.com/articles/in-the-age-of-ai-crawlers-i-have...

Post reply on HN