Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Lightpanda, an open-source headless browser in Zig

github.com

Post reply on HN
Print view View original

31–40 of 141 posts

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#31

Post by fbouvier »

fbouvier wrote: post #28
monkmartinez wrote: post #23

This is pretty neat, but I have to ask; Why does everyone want to build and/or use a headless browser? When I use pyautogui and my desktop chrome app I never have problems with captchas or trigger bot detectors. When I use a "headless" playwright, selenium, or puppeteer, I almost always run into problems. My conclusion is that "headless" scrapping creates more problems than it solves. Why don't we use the chrome, fir…

I guess it depends on the scale of your requests. When you want to browse a few websites from time to time, a local headful browser might be a solution. But when you have thousands or millions of webpages, you need a server environment and a headless browser.

In the past I've run hundreds of headful instances of Chrome in a server environment using Xvfb. It was not a pleasant experience :)

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#32

Post by fbouvier »

Tostino wrote: post #27
fbouvier wrote: post #18

Earlier quoted context omitted.

The cost of the browser part is still a problem. In our previous startup, we were scraping >20 millions of webpages per day, with thousands of instances of Chrome headless in parallel. Regarding the RAM usage, it's still ~10x better than Chrome :) It seems to be coming mostly from v8, I guess that we could do better with a lightweight JS engine alternative.

You may reduce ram, but also performance. A good JIT costs ram.

Yes, that's true. It's a balance to find between RAM and speed.

I was thinking more on use cases that require to disable JIT anyway (WASM, iOS integration, security).

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#33

Post by Tostino »

fbouvier wrote: post #32
Tostino wrote: post #27

Earlier quoted context omitted.

You may reduce ram, but also performance. A good JIT costs ram.

Yes, that's true. It's a balance to find between RAM and speed. I was thinking more on use cases that require to disable JIT anyway (WASM, iOS integration, security).

Yeah, could be nice to allow the user to select the type of ECMAScript engine that fits their use-case / performance requirements (balancing the resources available).

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#34

Post by andrethegiant »

fbouvier wrote: post #14
cropcirclbureau wrote: post #3

Pretty cool. Do you have a list of features you plan to support and plan to cut? Also, how much does this differ from the DOM impls that test frameworks use? I recall Jest or someone sporting such a feature.

The most important "feature" is to increase our Web APIs coverage :) But of course we plan to add others features, including - tight integration with LLM - embed mode (as a C library and as a WASM module) so you can add a real browser to your project the same way you add libcurl

Could it potentially fit in a Cloudflare worker? Workers are also V8 and can run wasm, but are constrained to 128MB RAM and 10MB zipped bundle size

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#35

Post by fbouvier »

kavalg wrote: post #29

Why AGPL? I am not blaming you. I am just curious about the reasoning behind your choice.

We had some discussions about it. It seems to us that AGPL will ensure that a company running our browser in a cloud managed offer will have to keep its modifications open for the community.

We might be wrong, maybe AGPL will damage the project more than eg. Apache2. In that case we will reconsider our choice. It's always easier this way :)

Our underlying library https://github.com/lightpanda-io/zig-js-runtime is licensed with Apache2.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#37

Post by fbouvier »

andrethegiant wrote: post #34
fbouvier wrote: post #14

Earlier quoted context omitted.

The most important "feature" is to increase our Web APIs coverage :) But of course we plan to add others features, including - tight integration with LLM - embed mode (as a C library and as a WASM module) so you can add a real browser to your project the same way you add libcurl

Could it potentially fit in a Cloudflare worker? Workers are also V8 and can run wasm, but are constrained to 128MB RAM and 10MB zipped bundle size

WASM support is not there yet but it's on the roadmap and we had it in our mind since the beginning of the project, and have made our dev choices accordingly.

So yes it could be used in a serverless platform like Cloudflare workers. Our startup time is a huge advantage here (20ms vs 600ms for Chrome headless in our local tests).

Regarding v8 in Cloudflare workers I think we can not used directly, ie. we still need to embed a JS engine in the wasm module.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#38

Post by afk1914 »

fbouvier wrote: post #2

Author here. The browser is made from scratch (not based on Chromium/Webkit), in Zig, using v8 as a JS engine. Our idea is to build a lightweight browser optimized for AI use cases like LLM training and agent workflows. And more generally any type of web automation. It's a work in progress, there are hundreds of Web APIs, and for now we just support some of them (DOM, XHR, Fetch). So expect most websites to fail or c…

I am curious how Lightpanda compares to chrome-headless-shell ({headless: 'shell'} in Puppeteer) in benchmarks.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#39

Post by xena »

fbouvier wrote: post #2

Author here. The browser is made from scratch (not based on Chromium/Webkit), in Zig, using v8 as a JS engine. Our idea is to build a lightweight browser optimized for AI use cases like LLM training and agent workflows. And more generally any type of web automation. It's a work in progress, there are hundreds of Web APIs, and for now we just support some of them (DOM, XHR, Fetch). So expect most websites to fail or c…

How do I make sure that people can't use lightpanda to bypass bot protection tools?

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#40

Post by fbouvier »

cratermoon wrote: post #36

So is this the scraper we need to block? https://news.ycombinator.com/item?id=42750420

I fully understand your concern and agree that scrapers shouldn't be hurting web servers.

I don't think they are using our browser :)

But in my opinion, blocking a browser as such is not the right solution. In this case, it's the user who should be blocked, not the browser.

Post reply on HN