Wow. Lightpanda is absolutely bonkers of a project. I'd pay dearly for such an option a few years back.
Lightpanda migrate DOM implementation to Zig
11–20 of 144 posts
Re: Lightpanda migrate DOM implementation to Zig
#12Earlier quoted context omitted.
Hi, I am Francis, founder of Lightpanda. We wrote a full article explaining why we choose Zig over Rust or C++, if you are interested: https://lightpanda.io/blog/posts/why-we-built-lightpanda-in-... Our goal is to build a headless browser, rather than a general purpose browser like Servo or Chrome. It's already available if you would like to try it: https://lightpanda.io/docs/open-source/installation
Off topic note: I read the website and a few pages of the docs and it's unclear to me for what I can use LightPanda safely. Like say I wanted to swap my it as my engine on playwright, what are the tradeoffs? What things are implemented, what isnt?
Thanks for the feedback, we will try to make this clearer on the website. Lightpanda works with Playwright, and we have some docs[1] and examples[2] available.
Web APIs and CDP specifications are huge, so this is still a work in progess. Many websites and scripts already work, while others do not, it really depends on the case. For example, on the CDP side, we are currently working on adding an Accessibility tree implentation.
[1] https://lightpanda.io/docs/quickstart/build-your-first-extra...
[2] https://github.com/lightpanda-io/demo/tree/main/playwright
Re: Lightpanda migrate DOM implementation to Zig
#13This reminds me of the Servo project's journey. Always impressed to see another implementation of the WHATWG specs. It's interesting to see Zig being chosen here over Rust for a browser engine component. Rust has kind of become the default answer for "safe browser components" (e.g., Servo, Firefox's oxidation), primarily because the borrow checker maps so well to the ownership model of a DOM tree in theory. But in pr…
Hi, I am Francis, founder of Lightpanda. We wrote a full article explaining why we choose Zig over Rust or C++, if you are interested: https://lightpanda.io/blog/posts/why-we-built-lightpanda-in-... Our goal is to build a headless browser, rather than a general purpose browser like Servo or Chrome. It's already available if you would like to try it: https://lightpanda.io/docs/open-source/installation
I see you're using html5ever for HTML parsing, and like it's trait/callback based API (me too). It looks like style/layout is not in scope at the moment, but if you're ever looking at adding style/layout capabilities to lightpanda, then you may find it useful to know that Stylo [0] (CSS / style system) and Taffy [1] (box-level layout) are both avaiable with a similar style of API (also Parley [2] which has a slightly different API style but can be combined with Taffy to implement inline/text layout).
[0]: https://github.com/servo/stylo
[1]: https://github.com/DioxusLabs/taffy
[2]: https://github.com/linebender/parley
---
Also, if you're interested in contributing C bindings for html5ever upstream then let me know / maybe open a github issue.
Re: Lightpanda migrate DOM implementation to Zig
#14This reminds me of the Servo project's journey. Always impressed to see another implementation of the WHATWG specs. It's interesting to see Zig being chosen here over Rust for a browser engine component. Rust has kind of become the default answer for "safe browser components" (e.g., Servo, Firefox's oxidation), primarily because the borrow checker maps so well to the ownership model of a DOM tree in theory. But in pr…
Hi, I am Francis, founder of Lightpanda. We wrote a full article explaining why we choose Zig over Rust or C++, if you are interested: https://lightpanda.io/blog/posts/why-we-built-lightpanda-in-... Our goal is to build a headless browser, rather than a general purpose browser like Servo or Chrome. It's already available if you would like to try it: https://lightpanda.io/docs/open-source/installation
Choosing something like Zig over C++ on simplicity grounds is going to be a false economy. C++ features exist for a reason. The complexity is in the domain. You can't make a project simpler by using a simplistic language: the complexity asserts itself somehow, somewhere, and if a language can't express the concept you want, you'll end up with circumlocution "patterns" instead.
Build system complexity disappears when you set it up too. Meson and such can be as terse as your Curl example.
I mean, it's your project, so whatever. Do what you want. But choosing Zig for the stated reasons is like choosing a car for the shape of the cupholders.
Re: Lightpanda migrate DOM implementation to Zig
#15This reminds me of the Servo project's journey. Always impressed to see another implementation of the WHATWG specs. It's interesting to see Zig being chosen here over Rust for a browser engine component. Rust has kind of become the default answer for "safe browser components" (e.g., Servo, Firefox's oxidation), primarily because the borrow checker maps so well to the ownership model of a DOM tree in theory. But in pr…
I don't think it's really that bad in Rust. If you're happy with an arena in Zig you can do exactly the same thing in Rust. There are a ton of options listed here: https://donsz.nl/blog/arenas/ Some of them even prevent use after free (the "ABA mitigation" column).
I'm not super experienced with zig, but I always think that in the same way that rust forces you to think about ownership (by having the borrow checker - note: I think of this as a good thing personally) zig makes you think upfront about your allocation (by making everything that can allocate take an allocator argument.).
It makes everything very explicit, and you can always _see_ where your allocations are happening in a way that you can't (as easily, or as obviously - imo) in rust.
It seems like something I quite like. I'm looking forward to rust getting an effects system/allocator api to help a little more with that side of things.
Re: Lightpanda migrate DOM implementation to Zig
#16This table is informative as to exactly what lightpanda is: https://lightpanda.io/blog/posts/what-is-a-true-headless-bro... TL;DR: It does the following: - Fetch HTML over the network - Parse HTML into a DOM tree - Fetch and execute JavaScript that manipulates the DOM But not the following: - Fetch and parse CSS to apply styling rules - Calculate layout - Fetch images and fonts for display - Paint pixels to render th…
When I was working before on something that used headless browser agents, the ability to do a screenshot (or even a recording) was really great for debugging... so I am not sure about the "no paint". But hey everything in life is a trade-off.
yeah I feel the same, I think even having a screenshot of part of rendered page or full page can be useful even for machines considering how heavy those HTML can be to parse and expensive for LLM context. Sometimes (sub)screenshot is just a better kind of compression
Re: Lightpanda migrate DOM implementation to Zig
#17Earlier quoted context omitted.
When I was working before on something that used headless browser agents, the ability to do a screenshot (or even a recording) was really great for debugging... so I am not sure about the "no paint". But hey everything in life is a trade-off.
yeah I feel the same, I think even having a screenshot of part of rendered page or full page can be useful even for machines considering how heavy those HTML can be to parse and expensive for LLM context. Sometimes (sub)screenshot is just a better kind of compression
Yes HTML is too heavy and too expensive for LLM. We are working on a text-based format more suitable for AI.
Re: Lightpanda migrate DOM implementation to Zig
#18I've been using it for months now ever since I saw their presentation at GitHub
This is a common flow for me
lightpanda url | markitdown (microsoft) | sd (day50 streamdown)
It's solid. Definitely worth a check
Re: Lightpanda migrate DOM implementation to Zig
#19This reminds me of the Servo project's journey. Always impressed to see another implementation of the WHATWG specs. It's interesting to see Zig being chosen here over Rust for a browser engine component. Rust has kind of become the default answer for "safe browser components" (e.g., Servo, Firefox's oxidation), primarily because the borrow checker maps so well to the ownership model of a DOM tree in theory. But in pr…
Too late now, but is the requirement for shared mutable state inherent in the problem space? Or is it just because we still thought OOP was cool when we started on the DOM design?
Re: Lightpanda migrate DOM implementation to Zig
#20This reminds me of the Servo project's journey. Always impressed to see another implementation of the WHATWG specs. It's interesting to see Zig being chosen here over Rust for a browser engine component. Rust has kind of become the default answer for "safe browser components" (e.g., Servo, Firefox's oxidation), primarily because the borrow checker maps so well to the ownership model of a DOM tree in theory. But in pr…
I don't think it's really that bad in Rust. If you're happy with an arena in Zig you can do exactly the same thing in Rust. There are a ton of options listed here: https://donsz.nl/blog/arenas/ Some of them even prevent use after free (the "ABA mitigation" column).
Which is hardly any different from me using PurifyPlus back in 2000.