Very cool project, congrats guys!
Show HN: Lightpanda, an open-source headless browser in Zig
101–110 of 141 posts
Re: Show HN: Lightpanda, an open-source headless browser in Zig
#102Author here. The browser is made from scratch (not based on Chromium/Webkit), in Zig, using v8 as a JS engine. Our idea is to build a lightweight browser optimized for AI use cases like LLM training and agent workflows. And more generally any type of web automation. It's a work in progress, there are hundreds of Web APIs, and for now we just support some of them (DOM, XHR, Fetch). So expect most websites to fail or c…
Hi. Can I embed this as library? Is there C API exposed? I can't seem to find any documentation. I'd prefer this to a CDP server.
Re: Show HN: Lightpanda, an open-source headless browser in Zig
#103Why didn't you just fork Chromium and strip out the renderer? This is guaranteed to bitrot when the web standards change unless you keep up with it forever and have perpetual funding. Yes, modifying Chromium is hard, but this seems harder.
It was my first idea. Forking Chromium has obvious advantages (compatibility). But it's not architectured for that. The renderer is everywhere. I'm not saying it's impossible, just that it did look more difficult to me than starting over.
And starting from scratch has other benefits. We own the codebase and thus it's easier for us to add new features like LLM integrations. Plus reducing binary size and startup time, mandatory for embedding it (as a WASM module or as C lib).
Re: Show HN: Lightpanda, an open-source headless browser in Zig
#104Earlier quoted context omitted.
Headless mode skips the visual rendering meant for humans, but the DOM structure and layout still exist, allowing the model to parse elements programmatically (e.g. button locations). Instead of 'seeing' an image, the model interacts with the page's underlying structure, which is faster and more efficient. Our browser removes the rendering engine as well, so it won't handle 100% of automation use cases, but it's also…
The issue is that DOM structure does not correspond one-to-one with perceived structure. I could render things in the DOM that aren't visible to people (e.g. a transparent 5px x 5px button), or render things to people that aren't visible in the DOM (e.g. Facebook's DOM obfuscation shenanigans to evade ad-blocking, or rendering custom text to a WebGL canvas). Sure, most websites don't go that far, but most websites al…
Servers operate on top of tcp/ip not to serve information, rather to serve information plus something else, usually ads. This is usually implemented with websites and captchas n stuff.
That's a problem of misaligned economic incentives. If there is a blockchain which enables micro-transactions of 0.000001 cent per request, and in the order of a million tps or a billion tps, then servers have no reason not to accept money in exchange for information, instead of using ads to extract some eyeball attention.
There is no reason that i cannot invoke a command line program: `$fetch_social_media_posts -n 1000` and get the last thousand posts right there in the console, as long as i provide some valid transactions to the server.
Websites and ads are the wrong solution to the problem of gaining something while serving information, and headless browsers and scraping are the wrong solution to the first wrong solution and the problems it creates.
Re: Show HN: Lightpanda, an open-source headless browser in Zig
#105The hello world example does not work. In fact, no website I've tried works. It's usually always panics. For the example in the readme, the errors are: ``` ./lightpanda-aarch64-macos --host 127.0.0.1 --port 9222 info(websocket): starting blocking worker to listen on 127.0.0.1:9222 info(server): accepting new conn... info(server): client connected info(browser): GET https://wikipedia.com/ 200 info(browser): fetch http…
Lightpanda co-author here.
Thanks for opening the issue in the repo. To be clear here, the crash seems relative with a socket disconnection issue in our CDP server.
> info(events): event handler error try catch: TypeError: Cannot read properties of undefined (reading 'length')
This message relates to the execution of gt-ie9-ce3fe8e88d.js. It's not the origin of the crash.
I have to dig in, but it could be due to a missing web API.
Re: Show HN: Lightpanda, an open-source headless browser in Zig
#106I think this is a really cool project. Scrapping aside, I would definitely use this with playwright for end2end tests if it had 100% compatibility with chrome and ran with a fraction of the time/memory. At my company we have a small project where we are running the equivalent of 6.5 hours of end2end tests daily using playwright. Running the tests in parallel takes around half an hour. Your project is still in very ea…
Thank you! Happy if you use it for your e2e tests in your servers, it's an open-source project!
Of course it's quite easy to spin a local instance of a headless browser for occasional use. But having a production platform is another story (monitoring, maintenance, security and isolation, scalability), so there are business use cases for a managed version.
Re: Show HN: Lightpanda, an open-source headless browser in Zig
#107Author here. The browser is made from scratch (not based on Chromium/Webkit), in Zig, using v8 as a JS engine. Our idea is to build a lightweight browser optimized for AI use cases like LLM training and agent workflows. And more generally any type of web automation. It's a work in progress, there are hundreds of Web APIs, and for now we just support some of them (DOM, XHR, Fetch). So expect most websites to fail or c…
Hi. Can I embed this as library? Is there C API exposed? I can't seem to find any documentation. I'd prefer this to a CDP server.
Not now but we might do it in the future. It's easy to export a Zig project as a C ABI library.
Re: Show HN: Lightpanda, an open-source headless browser in Zig
#108The second social media botters find this.
Re: Show HN: Lightpanda, an open-source headless browser in Zig
#109Earlier quoted context omitted.
That's Zig for you. A ``modern'' systems programming language with no borrow checker or even RAII.
Those statements are mostly true and also worth talking about, but they're not pertinent to that error (remotely provided JS not behaving correctly), or the eventual crash (which you'd cause exactly the same way for the same reason in Rust with a .unwrap() call).
Not exactly the same. `.unwrap()` will never lead to UB, but this can in Zig in release mode.
Also `unwrap()`s are a lot more obvious than just a ?. Dangerous operations should require more ceremony than safe ones. Surprising to see Zig make such a mistake.
Re: Show HN: Lightpanda, an open-source headless browser in Zig
#110Earlier quoted context omitted.
Playwright can run webkit very easily and it's dramatically less resource-intensive than Chrome.
Yes but WebKit is not a browser per se, it's a rendering engine. It's less resource-intensive than Chrome, but here we are talking orders of magnitude between Lightpanda and Chrome. If you are ~10x faster while using ~10x less RAM you are using ~100x less resources.
Careful, as you implement misssing features your RAM usage might grow too. Happened to many projects, lean at the beggining, get's just as slow when dealing with real world mesiness.