Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dav2d

code.videolan.org

Post reply on HN
Print view View original

71–80 of 199 posts

Re: Dav2d

#71

Post by pixelpoet »

tensor wrote: post #4

Not on topic, but wow the internet has very quickly devolved into: click -> "making sure you're not a bot", click -> "making sure you're a human", click -> "COOKIES COOKIES COOKIES", click -> "cloudflare something something"

No one's even clicking anymore, everything implores me to tap or swipe these days, and everything is optimised for humans with one eye above the other.

Then I press the X to close the all-caps banner commanding me to install the app, upon which I get sent to the app store. Users of the website refer to it as an app.

Re: Dav2d

#73

Post by Y-bar »

hectormalot wrote: post #65
thresh wrote: post #16

Earlier quoted context omitted.

We had to set it up on the parts of VideoLAN infra so the service would remain usable. Otherwise it was under a constant DDoS by the AI bots.

Maybe I’m naive about this, but I didn’t expect AI scrapers to be that big of a load? I mean, it’s not that they need to scrape the same at 1000+ QPS, and even then I wouldn’t expect them to download all media and images either? What am I missing that explains the gap between this and “constant DDoS” of the site?

They are a scourge, they never rate-limit themselves, there are a hundred of them, and a significant number don’t respect robots.txt. Many of them also end up our meta:no-index,no-follow search pages leading to cost overruns on our Algolia usage. We spend way too much time adjusting WAF and other bot-controls than we should have.

Re: Dav2d

#74

Post by nijave »

hectormalot wrote: post #65
thresh wrote: post #16

Earlier quoted context omitted.

We had to set it up on the parts of VideoLAN infra so the service would remain usable. Otherwise it was under a constant DDoS by the AI bots.

Maybe I’m naive about this, but I didn’t expect AI scrapers to be that big of a load? I mean, it’s not that they need to scrape the same at 1000+ QPS, and even then I wouldn’t expect them to download all media and images either? What am I missing that explains the gap between this and “constant DDoS” of the site?

I think there's a few things at play here

- AI scrapers will pull a bunch of docs from many sites in parallel (so instead of a human request where someone picks a single Google result, it hits a bunch of sites)

- AI will crawl the site looking for the correct answer which may hit a handful of pages

- AI sends requests in quick succession (big bursts instead of small trickle over longer time)

- Personal assistants may crawl the site repeatedly scraping everything (we saw a fair bit of this at work, they announced themselves with user agents)

- At work (b2b SaaS webapp) we also found that the personal assistant variety tended to hammer really computationally expensive data export and reporting endpoints generally without filters. While our app technically supported it, it was very inorganic traffic

That said, I don't think the solution is blanket blocks. Really it's exposing sites are poorly optimized for emerging technology.

Re: Dav2d

#75

Post by [deleted] »

hectormalot wrote: post #65
thresh wrote: post #16

Earlier quoted context omitted.

We had to set it up on the parts of VideoLAN infra so the service would remain usable. Otherwise it was under a constant DDoS by the AI bots.

Maybe I’m naive about this, but I didn’t expect AI scrapers to be that big of a load? I mean, it’s not that they need to scrape the same at 1000+ QPS, and even then I wouldn’t expect them to download all media and images either? What am I missing that explains the gap between this and “constant DDoS” of the site?

[deleted]

Re: Dav2d

#76

Post by kllrnohj »

dcsommer wrote: post #18

We must not continue to develop media codecs in memory unsafe languages. Small, auditable sections can opt-out perhaps, but choosing default-unsafe for this type of software is close to professional negligence.

For the codec itself, the majority of it is performance sensitive and often has a significant amount of assembly even, so a memory safe language doesn't change much.

However for the container/extractor... those should absolutely be in a memory safe language, and those are were a lot of the exploits/crashes are, too, as metadata is more fuzzy.

As a practical example of this see something like CrabbyAVIF. All the parser code is rust, but it delegates to dav1d for the actual codec portion

Re: Dav2d

#78

Post by computerbuster »

esseph wrote: post #41
fishgoesblub wrote: post #37

Earlier quoted context omitted.

Of the 3 software AV1 encoders, the only one that is fully dead is the Rust encoder (rav1e). If people truly wanted memory safe encoders/decoders, they would fund and develop them.

> If people truly wanted memory safe encoders/decoders Really? How many codecs have your neighbors contributed money for the development of, just curious.

I think these conversations are directed by the parties funding the efforts. Example: "we (large company) want a fast AV2 decoder" -> they pay a specialized team to do it -> this team works in C for the most part, so it is done in C. If there were financial incentives to do it in Rust, they'd pay more for a Rust decoder.

Re: Dav2d

#79

Post by tomwheeler »

tensor wrote: post #4

Not on topic, but wow the internet has very quickly devolved into: click -> "making sure you're not a bot", click -> "making sure you're a human", click -> "COOKIES COOKIES COOKIES", click -> "cloudflare something something"

At least this one was significantly faster than Cloudflare and required no action on my part.

Re: Dav2d

#80

Post by maxloh »

dcsommer wrote: post #18

We must not continue to develop media codecs in memory unsafe languages. Small, auditable sections can opt-out perhaps, but choosing default-unsafe for this type of software is close to professional negligence.

Decoders written in Rust will be a lot slower than the equivalents in assembly.
Post reply on HN