Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Lightpanda, an open-source headless browser in Zig

github.com

Post reply on HN
Print view View original

51–60 of 141 posts

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#51

Post by sangnoir »

slt2021 wrote: post #50
jjcoffman wrote: post #42

Earlier quoted context omitted.

If your browser doesn't play nicely and obey robots.txt when its headless I don't think it's that crazy to block the browser and not the user.

it is trivial to spoof user-agent, if you want to stop a motivated scraper, you need a different solution that exploits the fact that robots use headless browser

> it is trivial to spoof user-agent

It's also trivial to detect spoofed user agents via fingerprinting. The best defense against scrapers is done in layers, with user-agent name block as the bare minimum.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#52

Post by cush »

fbouvier wrote: post #18
JoelEinbinder wrote: post #7

Earlier quoted context omitted.

When I've talked to people running this kind of ai scraping/agent workflow, the costs of the AI parts dwarf that of the web browser parts. This causes computational cost of the browser to become irrelevant. I'm curious what situation you got yourself in where optimizing the browser results in meaningful savings. I'd also like to be in that place! I think your ram usage benchmark is deceptive. I'd expect a minimal bro…

The cost of the browser part is still a problem. In our previous startup, we were scraping >20 millions of webpages per day, with thousands of instances of Chrome headless in parallel. Regarding the RAM usage, it's still ~10x better than Chrome :) It seems to be coming mostly from v8, I guess that we could do better with a lightweight JS engine alternative.

> there are hundreds of Web APIs, and for now we just support some of them (DOM, XHR, Fetch)

> it's still ~10x better than Chrome

Do you expect it to stay that way once you've reached parity?

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#55

Post by 10000truths »

katiehallett wrote: post #6
m3kw9 wrote: post #4

How does this work because the browser needs to render a page and the vision model needs to know where a button is, so it still needs to see an image. How does headless make it easier?

Headless mode skips the visual rendering meant for humans, but the DOM structure and layout still exist, allowing the model to parse elements programmatically (e.g. button locations). Instead of 'seeing' an image, the model interacts with the page's underlying structure, which is faster and more efficient. Our browser removes the rendering engine as well, so it won't handle 100% of automation use cases, but it's also…

The issue is that DOM structure does not correspond one-to-one with perceived structure. I could render things in the DOM that aren't visible to people (e.g. a transparent 5px x 5px button), or render things to people that aren't visible in the DOM (e.g. Facebook's DOM obfuscation shenanigans to evade ad-blocking, or rendering custom text to a WebGL canvas). Sure, most websites don't go that far, but most websites also aren't valuable targets for automated crawling/scraping. These kinds of disparities will be exploited to detect and block automated agents if browser automation becomes sufficiently popular, and then we're back to needing to render the whole browser and operate on the rendered image to keep ahead of the arms race.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#56

Post by frankgrecojr »

The hello world example does not work. In fact, no website I've tried works. It's usually always panics. For the example in the readme, the errors are:

```

./lightpanda-aarch64-macos --host 127.0.0.1 --port 9222

info(websocket): starting blocking worker to listen on 127.0.0.1:9222

info(server): accepting new conn...

info(server): client connected

info(browser): GET https://wikipedia.com/ 200

info(browser): fetch https://wikipedia.com/portal/wikipedia.org/assets/js/index-2...: http.Status.ok

info(browser): eval script portal/wikipedia.org/assets/js/index-24c3e2ca18.js: ReferenceError: location is not defined

info(browser): fetch https://wikipedia.com/portal/wikipedia.org/assets/js/gt-ie9-...: http.Status.ok

error(events): event handler error: error.JSExecCallback

info(events): event handler error try catch: TypeError: Cannot read properties of undefined (reading 'length')

info(server): close cmd, closing conn...

info(server): accepting new conn...

thread 5274880 panic: attempt to use null value

zsh: abort ./lightpanda-aarch64-macos --host 127.0.0.1 --port 9222

```

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#57

Post by JambalayaJimbo »

gkbrk wrote: post #45
tossandthrow wrote: post #44

Earlier quoted context omitted.

Where do you read DRM? Parent commenter merely and humbly asks the author of the library to make sure that it has sane defaults and support for ethical crawling. I find it disturbing that you would recommend against that.

Here's what the parent comment wrote. > And there should not be an option to override that. This is not just a sane default. This is software telling you what you are allowed to do based on what the rights owner wants, literally DRM. This is exactly like Android not allowing screenshots to be taken in certain apps because the rights owner didn't allow it.

This is software telling you what you are allowed to do based on what the software developer wants* (assuming the developers cares of course...). Which is how all software works. I would not want my users of my software doing anything malicious with it, so I would not give them the option.

If I create an open-source messaging app I am also not going to give users the option of clicking a button to spam recipients with dick pics. Even if it was dead-simple for a determined user to add code for this dick pic button themselves.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#58

Post by nwienert »

fbouvier wrote: post #18
JoelEinbinder wrote: post #7

Earlier quoted context omitted.

When I've talked to people running this kind of ai scraping/agent workflow, the costs of the AI parts dwarf that of the web browser parts. This causes computational cost of the browser to become irrelevant. I'm curious what situation you got yourself in where optimizing the browser results in meaningful savings. I'd also like to be in that place! I think your ram usage benchmark is deceptive. I'd expect a minimal bro…

The cost of the browser part is still a problem. In our previous startup, we were scraping >20 millions of webpages per day, with thousands of instances of Chrome headless in parallel. Regarding the RAM usage, it's still ~10x better than Chrome :) It seems to be coming mostly from v8, I guess that we could do better with a lightweight JS engine alternative.

Playwright can run webkit very easily and it's dramatically less resource-intensive than Chrome.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#59

Post by danielsht »

fbouvier wrote: post #2

Author here. The browser is made from scratch (not based on Chromium/Webkit), in Zig, using v8 as a JS engine. Our idea is to build a lightweight browser optimized for AI use cases like LLM training and agent workflows. And more generally any type of web automation. It's a work in progress, there are hundreds of Web APIs, and for now we just support some of them (DOM, XHR, Fetch). So expect most websites to fail or c…

Very impressive! At Airtop.ai we looked into lightweight browsers like this one since we run a huge fleet of cloud browsers but found that anything other than a non-headless Chromium based browser would trigger bot detection pretty quickly. Even spoofing user agents triggers bot detection because fingerprinting tools like FingerprintJS will use things like JS features, canvas fingerprinting, WebGL fingerprinting, font enumeration, etc.

Can you share if you've looked into how your browser fares against bot detection tools like these?

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#60

Post by randunel »

bityard wrote: post #47
gkbrk wrote: post #43

Earlier quoted context omitted.

Please don't. Software I installed on my computer needs to the what I want as the user. I don't want every random thing I install to come with DRM. The project looks useful, and if it ends up getting popular I imagine someone would make a DRM-free version anyway.

I feel like you may have a misunderstanding of what DRM is. Talking about DRM outside the context of media distribution doesn't really make any sense. Yes, someone can fork this and modify it however they want. They can already do the same with curl, Firefox, Chromium, etc. The point is that this is project is deliberately advertising itself as an AI-friendly web scraper. If successful, lots of people who don't know…

It's exactly DRM, management of legal access to digital content. The "media" part has been optional for decades.

The comment they replied to didn't suggest sane defaults, but DRM. Here's the quote, no defaults work that way (inability to override):

> At a _bare_ minimum, that means obeying robot.txt and NOT crawling a site that doesn't want to be crawled. And there should not be an option to override that.

Post reply on HN