Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Lightpanda, an open-source headless browser in Zig

github.com

Post reply on HN
Print view View original

81–90 of 141 posts

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#81

Post by AndyKelley »

the__alchemist wrote: post #79

I have a meta question from browsing the repo: Why do C, C++, and Zig code bases, by convention, include a license at the top of every module" IMO it makes more sense to insetead include of an overview of the module's purpose, and how it fits in with the rest of the program, and one license at the top-level, as the project already has.

100% of my projects, including the Zig compiler itself, have only the license file at the root of the project tree, except of course for files that were copy pasted from other projects.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#82

Post by zlagen »

fbouvier wrote: post #70
zlagen wrote: post #63

what do you think would be the use cases for this project? being lightweight is awesome but usually you need a real browser for most use cases. Testing sites and scraping for example. It may work for some scraping use cases but I think that if the site uses any kind of bot blocking this is not going to cut it.

There are a lot of uses cases: - LLM training (RAG, fine tuning) - AI agents - scraping - SERP - testing - any kind of web automation basically Bot protection of course might be a problem but it depends also on the volume of requests, IP, and other parameters. AI agents will do more and more actions on behalf of humans in the future and I believe the bot protection mechanism will evolve to include them as legit.

thanks, it doesn't seem like it's the direction it's going at the moment. If you look at the robots.txt of many websites, they are actually banning AI bots from crawling the site. To me it seems more likely that each site will have its own AI agent to perform operations but controlled by the site.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#83

Post by zlagen »

corford wrote: post #67
katiehallett wrote: post #15

Earlier quoted context omitted.

Thanks! The three of us worked together at our former company - ecomm saas start up where we spent a ton of $ on scraping infrastructure spinning up headless Chrome instances. It started out as more of an R&D thesis - is it possible to strip out graphical rendering from Chrome headless? Turns out no - so we tried to build it from scratch. And the beta results validated the thesis. I wrote a whole thing about it here…

Not sure what category of ecomm sites you were scraping but I scrape >10million ecomm URLs daily and, honestly, in my experience the compute is not a major issue (8 times out of 10 you can either use API endpoints and/or session stuffing to avoid needing a browser for every request; and in the 2 out of 10 sites where you really need a browser for all requests it's usually to circumvent aggressive anti-bot which means…

fully agree here, using a browser for everything is the dumb way. You just usually use it to circumvent the blocking and then reuse the cookies to call the endpoints directly.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#84

Post by ksec »

fbouvier wrote: post #11
sesm wrote: post #9

Earlier quoted context omitted.

Great job! And good luck on your journey! One question: which JS engines did you consider and why you chose V8 in the end?

We have also considered JavaScriptCore (used by Bun) and QuickJS. We did choose v8 because it's state of the art, quite well documented and easy to embed. The code is made to support others JS engine in the future. We do want to add a lightweight alternative like QuickJS or Kiesel https://kiesel.dev/

Thank You I was thinking of JSC and Bun as well. Was half expecting JSC since that combination seems to work well.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#86

Post by zelcon »

frankgrecojr wrote: post #56

The hello world example does not work. In fact, no website I've tried works. It's usually always panics. For the example in the readme, the errors are: ``` ./lightpanda-aarch64-macos --host 127.0.0.1 --port 9222 info(websocket): starting blocking worker to listen on 127.0.0.1:9222 info(server): accepting new conn... info(server): client connected info(browser): GET https://wikipedia.com/ 200 info(browser): fetch http…

That's Zig for you. A ``modern'' systems programming language with no borrow checker or even RAII.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#87

Post by xena »

cchance wrote: post #80
bityard wrote: post #30

Earlier quoted context omitted.

Please put a priority on making it hard to abuse the web with your tool. At a _bare_ minimum, that means obeying robot.txt and NOT crawling a site that doesn't want to be crawled. And there should not be an option to override that. It goes without saying that you should not allow users to make hundreds or thousands of "blind" parallel requests as these tend to have the effect of DoSing sites that are being hosted on…

Its literally open source, any effort put into hamstringing it would just be forked and removed lol

Any barrier to abuse makes abuse harder.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#88

Post by tetris11 »

zelcon wrote: post #85

Why didn't you just fork Chromium and strip out the renderer? This is guaranteed to bitrot when the web standards change unless you keep up with it forever and have perpetual funding. Yes, modifying Chromium is hard, but this seems harder.

Why do anything: because it shows what's possible, and makes the next effort that much more easier.

I call this process of frontier effort and discovery: "science"

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#89

Post by cxr »

Tostino wrote: post #27
fbouvier wrote: post #18

Earlier quoted context omitted.

The cost of the browser part is still a problem. In our previous startup, we were scraping >20 millions of webpages per day, with thousands of instances of Chrome headless in parallel. Regarding the RAM usage, it's still ~10x better than Chrome :) It seems to be coming mostly from v8, I guess that we could do better with a lightweight JS engine alternative.

You may reduce ram, but also performance. A good JIT costs ram.

If your target is consistent enough (perhaps even stationary), then at some point "JIT" means wasting CPU cycles.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#90

Post by zelcon »

tetris11 wrote: post #88
zelcon wrote: post #85

Why didn't you just fork Chromium and strip out the renderer? This is guaranteed to bitrot when the web standards change unless you keep up with it forever and have perpetual funding. Yes, modifying Chromium is hard, but this seems harder.

Why do anything: because it shows what's possible, and makes the next effort that much more easier. I call this process of frontier effort and discovery: "science"

Redoing what others have already done is not what I think of when I hear "frontier effort"
Post reply on HN