Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Lightpanda, an open-source headless browser in Zig

github.com

Post reply on HN
Print view View original

71–80 of 141 posts

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#71

Post by fbouvier »

nwienert wrote: post #58
fbouvier wrote: post #18

Earlier quoted context omitted.

The cost of the browser part is still a problem. In our previous startup, we were scraping >20 millions of webpages per day, with thousands of instances of Chrome headless in parallel. Regarding the RAM usage, it's still ~10x better than Chrome :) It seems to be coming mostly from v8, I guess that we could do better with a lightweight JS engine alternative.

Playwright can run webkit very easily and it's dramatically less resource-intensive than Chrome.

Yes but WebKit is not a browser per se, it's a rendering engine.

It's less resource-intensive than Chrome, but here we are talking orders of magnitude between Lightpanda and Chrome. If you are ~10x faster while using ~10x less RAM you are using ~100x less resources.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#72

Post by codetrotter »

fbouvier wrote: post #22
wiradikusuma wrote: post #20

Earlier quoted context omitted.

But what if the human programmer needs to visually verify that their code works by eyeballing which element got selected, etc?

You're right, the debugging part is a good use case for graphical rendering in a headless environment. I see it as a build time/runtime question. At build (dev) time I want to have a graphical response (debugging, computer vision, etc.). And then, when the script is ready, I can use Lightpanda at runtime as a lightweight alternative.

I was doing a personal side project for a while where I was trying to make my own little Wayback Machine-alike. Mine was very rudimentary, built on top of Firefox and WebDriver plus Squid proxy.

For debugging purposes you could have your headless browser function as a HTTP Proxy Server, maybe? And in your headless browser you could capture a static snapshot of the DOM after your JavaScript runtime has executed the scripts for the page. Similar to how the archive.today guy serves static snapshots of websites. And then developers using your headless browser could point their Firefox or Chrome browser to the HTTP Proxy server hosted by your headless browser program, in order to get a static snapshot view of what the DOM is like after your headless browser has executed JavaScript from the page. And then Firefox or Chrome will render static HTML view of what the page looked like to your headless browser, that the developer can inspect to make decisions about further interactions with the page. As a tool for debugging.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#73

Post by lbotos »

frankgrecojr wrote: post #56

The hello world example does not work. In fact, no website I've tried works. It's usually always panics. For the example in the readme, the errors are: ``` ./lightpanda-aarch64-macos --host 127.0.0.1 --port 9222 info(websocket): starting blocking worker to listen on 127.0.0.1:9222 info(server): accepting new conn... info(server): client connected info(browser): GET https://wikipedia.com/ 200 info(browser): fetch http…

Not OP -- do you have some kind of proxy or firewall?

Looks like you couldn't download https://wikipedia.com/portal/wikipedia.org/assets/js/gt-ie9-... for some reason.

In my contributions to joplin s3 backend "Cannot read properties of undefined (reading 'length')" was usually when you were trying to access an object that wasn't instantiated. (Can't figure out length of )

So for some reason it seems you can't execute JS?

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#74

Post by 867-5309 »

fbouvier wrote: post #2

Author here. The browser is made from scratch (not based on Chromium/Webkit), in Zig, using v8 as a JS engine. Our idea is to build a lightweight browser optimized for AI use cases like LLM training and agent workflows. And more generally any type of web automation. It's a work in progress, there are hundreds of Web APIs, and for now we just support some of them (DOM, XHR, Fetch). So expect most websites to fail or c…

does this work with selenium/chromedriver?

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#75

Post by fbouvier »

867-5309 wrote: post #74
fbouvier wrote: post #2

Author here. The browser is made from scratch (not based on Chromium/Webkit), in Zig, using v8 as a JS engine. Our idea is to build a lightweight browser optimized for AI use cases like LLM training and agent workflows. And more generally any type of web automation. It's a work in progress, there are hundreds of Web APIs, and for now we just support some of them (DOM, XHR, Fetch). So expect most websites to fail or c…

does this work with selenium/chromedriver?

For now we just support CDP. But Selenium is definitely in our roadmap.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#76

Post by bdhcuidbebe »

fbouvier wrote: post #71
nwienert wrote: post #58

Earlier quoted context omitted.

Playwright can run webkit very easily and it's dramatically less resource-intensive than Chrome.

Yes but WebKit is not a browser per se, it's a rendering engine. It's less resource-intensive than Chrome, but here we are talking orders of magnitude between Lightpanda and Chrome. If you are ~10x faster while using ~10x less RAM you are using ~100x less resources.

How well does it compare to specialized headless scraper browsers, like camoufox (firefox based) or secret agent (chrome based)?

Either should reduce your ram usage compared to stock chrome by a lot.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#77

Post by radium3d »

fbouvier wrote: post #18
JoelEinbinder wrote: post #7

Earlier quoted context omitted.

When I've talked to people running this kind of ai scraping/agent workflow, the costs of the AI parts dwarf that of the web browser parts. This causes computational cost of the browser to become irrelevant. I'm curious what situation you got yourself in where optimizing the browser results in meaningful savings. I'd also like to be in that place! I think your ram usage benchmark is deceptive. I'd expect a minimal bro…

The cost of the browser part is still a problem. In our previous startup, we were scraping >20 millions of webpages per day, with thousands of instances of Chrome headless in parallel. Regarding the RAM usage, it's still ~10x better than Chrome :) It seems to be coming mostly from v8, I guess that we could do better with a lightweight JS engine alternative.

As a web developer and server manager AI trainers scraping websites with no throttle is the problem. lol

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#78

Post by keepamovin »

fbouvier wrote: post #2

Author here. The browser is made from scratch (not based on Chromium/Webkit), in Zig, using v8 as a JS engine. Our idea is to build a lightweight browser optimized for AI use cases like LLM training and agent workflows. And more generally any type of web automation. It's a work in progress, there are hundreds of Web APIs, and for now we just support some of them (DOM, XHR, Fetch). So expect most websites to fail or c…

If you support Page.startScreencast or even just capture screenshot we could experiment with using this as a backend for BrowserBox, when lightpanda matures. Cool stuff!

https://github.com/BrowserBox/BrowserBox/

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#79

Post by the__alchemist »

I have a meta question from browsing the repo: Why do C, C++, and Zig code bases, by convention, include a license at the top of every module" IMO it makes more sense to insetead include of an overview of the module's purpose, and how it fits in with the rest of the program, and one license at the top-level, as the project already has.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#80

Post by cchance »

bityard wrote: post #30
fbouvier wrote: post #2

Author here. The browser is made from scratch (not based on Chromium/Webkit), in Zig, using v8 as a JS engine. Our idea is to build a lightweight browser optimized for AI use cases like LLM training and agent workflows. And more generally any type of web automation. It's a work in progress, there are hundreds of Web APIs, and for now we just support some of them (DOM, XHR, Fetch). So expect most websites to fail or c…

Please put a priority on making it hard to abuse the web with your tool. At a _bare_ minimum, that means obeying robot.txt and NOT crawling a site that doesn't want to be crawled. And there should not be an option to override that. It goes without saying that you should not allow users to make hundreds or thousands of "blind" parallel requests as these tend to have the effect of DoSing sites that are being hosted on…

Its literally open source, any effort put into hamstringing it would just be forked and removed lol
Post reply on HN