Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Lightpanda, an open-source headless browser in Zig

github.com

Post reply on HN
Print view View original

131–140 of 141 posts

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#131

Post by andrethegiant »

internet_points wrote: post #119
bityard wrote: post #30

Earlier quoted context omitted.

Please put a priority on making it hard to abuse the web with your tool. At a _bare_ minimum, that means obeying robot.txt and NOT crawling a site that doesn't want to be crawled. And there should not be an option to override that. It goes without saying that you should not allow users to make hundreds or thousands of "blind" parallel requests as these tend to have the effect of DoSing sites that are being hosted on…

Yes! Having long time ago done some minor web scraping, I did not put any work at all into following robots.txt, simply because it seemed like a hassle and I thought "meh it's not that much traffic is it and boss wants this done yesterday". But if the tool defaulted to following robots.txt I certainly wouldn't have minded, it would have caused me to get less noise and my tool to behave better. Also, throttling reques…

This is why I’m making crawlspace.dev, a crawling PaaS that respects robots.txt, implements proper caching, etc by default.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#132

Post by emporas »

aniviacat wrote: post #124
emporas wrote: post #104

Earlier quoted context omitted.

Servers operate on top of tcp/ip not to serve information, rather to serve information plus something else, usually ads. This is usually implemented with websites and captchas n stuff. That's a problem of misaligned economic incentives. If there is a blockchain which enables micro-transactions of 0.000001 cent per request, and in the order of a million tps or a billion tps, then servers have no reason not to accept m…

No need for blockchain, microtransaction functionality should be integrated into our existing payment methods.

Existing payment methods, paypal, google pay etc, have been absolutely crucial for internet payments, but the micro in the word never ends.

If there are internet payments with a minimum payment of 1 cent, then we need payments of 0.1 cents. If that's achieved, then we need 0.01 cents minimum transaction. The micro in the transaction always needs to be smaller (and faster).

Free competition (or perfect competition) over a well defined landscape, internet protocols that is, has proven to always deliver better quality goods and lower price. Money derived from governments is far, far from free competition, let alone well defined internet protocols, and there is a point in which existing payment methods get stuck and cannot deliver smaller transactions.

I don't personally know where and when that point is, but if i have to guess, existing payment methods have reached that minimum point for at least a decade. In other words, their transactions minimums have to be high enough for them to make a profit. Yes, they can implement microtransactions, but they will not be profitable.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#133

Post by oever »

fbouvier wrote: post #103
zelcon wrote: post #85

Why didn't you just fork Chromium and strip out the renderer? This is guaranteed to bitrot when the web standards change unless you keep up with it forever and have perpetual funding. Yes, modifying Chromium is hard, but this seems harder.

It was my first idea. Forking Chromium has obvious advantages (compatibility). But it's not architectured for that. The renderer is everywhere. I'm not saying it's impossible, just that it did look more difficult to me than starting over. And starting from scratch has other benefits. We own the codebase and thus it's easier for us to add new features like LLM integrations. Plus reducing binary size and startup time,…

The Chromium/Webkit renderer used to have multiple rendering backends. You might use or add a no-op backend.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#134

Post by mpalmer »

bityard wrote: post #30
fbouvier wrote: post #2

Author here. The browser is made from scratch (not based on Chromium/Webkit), in Zig, using v8 as a JS engine. Our idea is to build a lightweight browser optimized for AI use cases like LLM training and agent workflows. And more generally any type of web automation. It's a work in progress, there are hundreds of Web APIs, and for now we just support some of them (DOM, XHR, Fetch). So expect most websites to fail or c…

Please put a priority on making it hard to abuse the web with your tool. At a _bare_ minimum, that means obeying robot.txt and NOT crawling a site that doesn't want to be crawled. And there should not be an option to override that. It goes without saying that you should not allow users to make hundreds or thousands of "blind" parallel requests as these tend to have the effect of DoSing sites that are being hosted on…

If it's already a problem, nothing this developer does will improve it, including crippling their software and removing arguably legitimate use cases.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#136

Post by hansvm »

jbggs wrote: post #99
hansvm wrote: post #94

Earlier quoted context omitted.

Those statements are mostly true and also worth talking about, but they're not pertinent to that error (remotely provided JS not behaving correctly), or the eventual crash (which you'd cause exactly the same way for the same reason in Rust with a .unwrap() call).

you shouldn't be unwrapping, error cases should be properly handled. users shouldn't see null dereference errors without any context, even in cli tools...

That too, as a general coding pattern. I was commenting on the criticism of Zig as a sub-par system's language though, contrasting with a language most people with that opinion seem to like.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#137

Post by hansvm »

IshKebab wrote: post #109
hansvm wrote: post #94

Earlier quoted context omitted.

Those statements are mostly true and also worth talking about, but they're not pertinent to that error (remotely provided JS not behaving correctly), or the eventual crash (which you'd cause exactly the same way for the same reason in Rust with a .unwrap() call).

Not exactly the same. `.unwrap()` will never lead to UB, but this can in Zig in release mode. Also `unwrap()`s are a lot more obvious than just a ?. Dangerous operations should require more ceremony than safe ones. Surprising to see Zig make such a mistake.

> UB vs "safe" panic

Yes, it's not exactly the same if you compile in ReleaseFast instead of ReleaseSafe. Both are bad though, and I'd tend to blame the coding pattern for the observed behavior rather than quibble about which unacceptable scenario is worse.

I see people adopting forced null unwrapping for dumb reasons all the time. For the remaining reasons, do you have a sense of what the language deficiencies are which make that feature helpful? I see it for somewhat sane reasons when crossing language boundaries. Does anything else stand out?

> ceremony

Yes. Thankfully ".?" is greppable, but I wouldn't mind more ceremony there, and less for `try` coding patterns.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#138

Post by cchance »

xena wrote: post #87
cchance wrote: post #80

Earlier quoted context omitted.

Its literally open source, any effort put into hamstringing it would just be forked and removed lol

Any barrier to abuse makes abuse harder.

Not really lol, literally if you add a robots.txt check someone can just create a fork repo with a git action that escapes that routine every time the original is pushed... Adding options for filter and respecting things is great even as defaults, but trying to force "good behavior" tends to just lead to people setting up a workaround that everyone eventually uses because why use the hamstrung version instead of the open version and make your own choices.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#139

Post by dolmen »

xena wrote: post #39
fbouvier wrote: post #2

Author here. The browser is made from scratch (not based on Chromium/Webkit), in Zig, using v8 as a JS engine. Our idea is to build a lightweight browser optimized for AI use cases like LLM training and agent workflows. And more generally any type of web automation. It's a work in progress, there are hundreds of Web APIs, and for now we just support some of them (DOM, XHR, Fetch). So expect most websites to fail or c…

How do I make sure that people can't use lightpanda to bypass bot protection tools?

One of Lightpanda's goals is to ease building bots.

Re: Show HN: Lightpanda, an open-source headless browser in Zig

#140

Post by niutech »

fbouvier wrote: post #2

Author here. The browser is made from scratch (not based on Chromium/Webkit), in Zig, using v8 as a JS engine. Our idea is to build a lightweight browser optimized for AI use cases like LLM training and agent workflows. And more generally any type of web automation. It's a work in progress, there are hundreds of Web APIs, and for now we just support some of them (DOM, XHR, Fetch). So expect most websites to fail or c…

Congratulations! But does it support Google Account login? And ReCAPTCHA?
Post reply on HN