Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Lightpanda, an open-source headless browser in Zig

github.com

Post reply on HN
Print view View original

41–50 of 141 posts

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#41

Post by fbouvier »

afk1914 wrote: post #38
fbouvier wrote: post #2

Author here. The browser is made from scratch (not based on Chromium/Webkit), in Zig, using v8 as a JS engine. Our idea is to build a lightweight browser optimized for AI use cases like LLM training and agent workflows. And more generally any type of web automation. It's a work in progress, there are hundreds of Web APIs, and for now we just support some of them (DOM, XHR, Fetch). So expect most websites to fail or c…

I am curious how Lightpanda compares to chrome-headless-shell ({headless: 'shell'} in Puppeteer) in benchmarks.

We did not run benchmarks with chrome-headless-shell (aka the old headless mode) but I guess that performance wise it's on the same scale as the new headless mode.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#42

Post by jjcoffman »

fbouvier wrote: post #40
cratermoon wrote: post #36

So is this the scraper we need to block? https://news.ycombinator.com/item?id=42750420

I fully understand your concern and agree that scrapers shouldn't be hurting web servers. I don't think they are using our browser :) But in my opinion, blocking a browser as such is not the right solution. In this case, it's the user who should be blocked, not the browser.

If your browser doesn't play nicely and obey robots.txt when its headless I don't think it's that crazy to block the browser and not the user.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#43

Post by gkbrk »

bityard wrote: post #30
fbouvier wrote: post #2

Author here. The browser is made from scratch (not based on Chromium/Webkit), in Zig, using v8 as a JS engine. Our idea is to build a lightweight browser optimized for AI use cases like LLM training and agent workflows. And more generally any type of web automation. It's a work in progress, there are hundreds of Web APIs, and for now we just support some of them (DOM, XHR, Fetch). So expect most websites to fail or c…

Please put a priority on making it hard to abuse the web with your tool. At a _bare_ minimum, that means obeying robot.txt and NOT crawling a site that doesn't want to be crawled. And there should not be an option to override that. It goes without saying that you should not allow users to make hundreds or thousands of "blind" parallel requests as these tend to have the effect of DoSing sites that are being hosted on…

Please don't.

Software I installed on my computer needs to the what I want as the user. I don't want every random thing I install to come with DRM.

The project looks useful, and if it ends up getting popular I imagine someone would make a DRM-free version anyway.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#44

Post by tossandthrow »

gkbrk wrote: post #43
bityard wrote: post #30

Earlier quoted context omitted.

Please put a priority on making it hard to abuse the web with your tool. At a _bare_ minimum, that means obeying robot.txt and NOT crawling a site that doesn't want to be crawled. And there should not be an option to override that. It goes without saying that you should not allow users to make hundreds or thousands of "blind" parallel requests as these tend to have the effect of DoSing sites that are being hosted on…

Please don't. Software I installed on my computer needs to the what I want as the user. I don't want every random thing I install to come with DRM. The project looks useful, and if it ends up getting popular I imagine someone would make a DRM-free version anyway.

Where do you read DRM?

Parent commenter merely and humbly asks the author of the library to make sure that it has sane defaults and support for ethical crawling.

I find it disturbing that you would recommend against that.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#45

Post by gkbrk »

tossandthrow wrote: post #44
gkbrk wrote: post #43

Earlier quoted context omitted.

Please don't. Software I installed on my computer needs to the what I want as the user. I don't want every random thing I install to come with DRM. The project looks useful, and if it ends up getting popular I imagine someone would make a DRM-free version anyway.

Where do you read DRM? Parent commenter merely and humbly asks the author of the library to make sure that it has sane defaults and support for ethical crawling. I find it disturbing that you would recommend against that.

Here's what the parent comment wrote.

> And there should not be an option to override that.

This is not just a sane default. This is software telling you what you are allowed to do based on what the rights owner wants, literally DRM.

This is exactly like Android not allowing screenshots to be taken in certain apps because the rights owner didn't allow it.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#46

Post by fbouvier »

jjcoffman wrote: post #42
fbouvier wrote: post #40

Earlier quoted context omitted.

I fully understand your concern and agree that scrapers shouldn't be hurting web servers. I don't think they are using our browser :) But in my opinion, blocking a browser as such is not the right solution. In this case, it's the user who should be blocked, not the browser.

If your browser doesn't play nicely and obey robots.txt when its headless I don't think it's that crazy to block the browser and not the user.

Every tool can be used in a good or bad way, Chrome, Firefox, cURL, etc. It's not the browser who doesn't play nicely, it's the user.

It's the user's responsibility to behave well, like in life :)

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#47

Post by bityard »

gkbrk wrote: post #43
bityard wrote: post #30

Earlier quoted context omitted.

Please put a priority on making it hard to abuse the web with your tool. At a _bare_ minimum, that means obeying robot.txt and NOT crawling a site that doesn't want to be crawled. And there should not be an option to override that. It goes without saying that you should not allow users to make hundreds or thousands of "blind" parallel requests as these tend to have the effect of DoSing sites that are being hosted on…

Please don't. Software I installed on my computer needs to the what I want as the user. I don't want every random thing I install to come with DRM. The project looks useful, and if it ends up getting popular I imagine someone would make a DRM-free version anyway.

I feel like you may have a misunderstanding of what DRM is. Talking about DRM outside the context of media distribution doesn't really make any sense.

Yes, someone can fork this and modify it however they want. They can already do the same with curl, Firefox, Chromium, etc. The point is that this is project is deliberately advertising itself as an AI-friendly web scraper. If successful, lots of people who don't know any better are going to download it and deploy it without a full understanding (and possibly caring) of the consequences on the open web. And as I already point out, this is not hypothetical, it is already happening. Right now. As we speak.

Do you want cloudflare everywhere? This is how you get cloudflare everywhere.

My plea for the dev is that they choose to take the high road and put web-server-friendly SANE DEFAULTS in place to curb the bulk of abusive web scraping behavior to lessen the number of gray hairs it causes web admins like myself. That is all.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#48

Post by nicoburns »

weinzierl wrote: post #10

If I don't need JavaScript or any interactivity, just modern HTML + modern CSS, is there any modern lightweight renderer to png or svg? Something in the spirit of wkhtmltoimage or WeasyPrint that does not require a full blown browser but more modern with support of recent HTML and CSS? In a sense this is Lightpanda's complement to a "full panda". Just the fully rendered DOM to pixels.

We're working on this here: https://github.com/DioxusLabs/blitz See the "screenshot" example for rendering to png. There's no SVG backend currently, but one could be added.

(proper announcement of project coming soon)

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#49

Post by blacksmith_tb »

gkbrk wrote: post #45
tossandthrow wrote: post #44

Earlier quoted context omitted.

Where do you read DRM? Parent commenter merely and humbly asks the author of the library to make sure that it has sane defaults and support for ethical crawling. I find it disturbing that you would recommend against that.

Here's what the parent comment wrote. > And there should not be an option to override that. This is not just a sane default. This is software telling you what you are allowed to do based on what the rights owner wants, literally DRM. This is exactly like Android not allowing screenshots to be taken in certain apps because the rights owner didn't allow it.

Not sure what "digital rights" that "manages"? I don't see it as an unreasonable suggestion that the tool shouldn't be set up out of the box to DoS sites it's scraping, that doesn't prevent anyone who is technical enough to know what they're doing to fork it and remove whatever limits are there by default? I can't see it as a "my computer should do what I want!" issue, if you don't like how this package works, change it or use another?

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#50

Post by slt2021 »

jjcoffman wrote: post #42
fbouvier wrote: post #40

Earlier quoted context omitted.

I fully understand your concern and agree that scrapers shouldn't be hurting web servers. I don't think they are using our browser :) But in my opinion, blocking a browser as such is not the right solution. In this case, it's the user who should be blocked, not the browser.

If your browser doesn't play nicely and obey robots.txt when its headless I don't think it's that crazy to block the browser and not the user.

it is trivial to spoof user-agent, if you want to stop a motivated scraper, you need a different solution that exploits the fact that robots use headless browser
Post reply on HN