Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let us serve you, but don't bring us down

blog.archive.org

Post reply on HN
Print view View original

231–240 of 255 posts

Re: Let us serve you, but don't bring us down

#231

Post by schoen »

remram wrote: post #219
schoen wrote: post #93

Earlier quoted context omitted.

You could also do an in-between step of host -t soa example.com which should give you domains that have any DNS record at all, not just an A record.

Is there still a point talking to the registrar then? How common is it for a domain to exist with the registrar but have no SOA record?

Hmmm, I'd have to think about this more to be sure. I'm not immediately positive of the answer.

If the registrant creates any DNS records at all, then the SOA will need to exist for the zone to be valid; but I don't recall whether the registrar is or isn't required to publish a zone for a registered domain that otherwise contains no records. (Also whether the registrar is required to inform the registry of authoritative nameservers for the domain at all times from the moment that the domain is registered, and then whether that information would lead to the synthesis of a SOA record.)

I guess I can either try this (if I could find a no-frills-enough registrar that doesn't create any records at all for add-on hosting services or domain parking) or try to take some more ICANN coursework to find out the answer.

Or maybe someone else reading this thread knows whether we can have a domain in practice that is registered but has no published SOA.

Re: Let us serve you, but don't bring us down

#232

Post by rodgerd »

crazydoggers wrote: post #91
crazydoggers wrote: post #18

Earlier quoted context omitted.

Same way a library does. The information is made available to anyone who wants it, the copyrights are maintained intact, and they don’t attempt to profit from the material. Copyright law specifically allows for libraries and archives to make copies of copyrighted material. Without such laws, without libraries, knowledge could not be guaranteed to be shared freely among the public, resulting in ever growing knowledge…

I just wanted to add, while I don’t agree with the assessment that copyright holders need to be protected from something like archive.org, i don’t think the parent comment deserved to be flagged, so I vouched for it. I think the question was raised in sincerity, and I think it offers a point of discussion for those not familiar with the issues. I don’t think it’s helpful to flag things people disagree with, as long a…

> i don’t think the parent comment deserved to be flagged, so I vouched for it.

Since archive.org have now updated that the problem scraper is now evading countermeasures and bringing them down repeatedly, and has been identified as an "AI" company, it could end up being an existential risk: if companies start using archive.org as a large-scale commercial IP theft proxy, they will likely face even more legal challenges than they already do.

Re: Let us serve you, but don't bring us down

#233

Post by marginalia_nu »

RugnirViking wrote: post #213
marginalia_nu wrote: post #130

Earlier quoted context omitted.

In practice they're often using botnets. It's what makes this such a difficult problem to deal with. If they weren't it would be trivial to rate-limit or block problematic IPs or subnets.

as I mentioned in the post: why would a legitimate scraper be using a botnet?

In most cases, they aren't legitimate scrapers. They have a complete disregard for the resources they use from the websites they scrape and use a botnet to hide their point of origin.

Re: Let us serve you, but don't bring us down

#234

Post by mellosouls »

hgsgm wrote: post #188
mellosouls wrote: post #109

Earlier quoted context omitted.

There are various underfunded digital archives in the EU and elsewhere that have legal duties to preserve online content relating to their specific countries. They may be "of very questionable value" to you but your solution to remove funding from them to channel it to a much wealthier organisation in a wealthier country is neither ethical, legal or practical.

This is almost completely unresponsive to what parent actually wrote.

Not at all. The suggestion was to shift countries government funding from their own projects to a foreign, better funded initiative on the implied assumption they don't know what they are doing.

The suggestion was well-motivated but presumably guided by a lack of understanding of the existing digital landscape of international publicly funded projects and their obligations and constraints.

Re: Let us serve you, but don't bring us down

#235

Post by otikik »

coldpie wrote: post #204
Lalabadie wrote: post #163

Earlier quoted context omitted.

Some zealous systems will infer a very regular request rate as coming from automated services and block them, no matter how gentle the rate.

Is this speculation, or something that has actually been seen?

I would block it, if I was the maintainer. As the linked post mentions, automated requests can "ramp up" at any moment, risking server stability. By preemptively blocking automated parsing (on a resource which primary usage is individual requests, not mass ones) I would avoid future problems for myself. Let them contact us via support if they really need an exception.

In general I would rate limit by IP anything connected to the internet.

Re: Let us serve you, but don't bring us down

#236

Post by npteljes »

maksimur wrote: post #75
x-complexity wrote: post #26

Unpopular opinion: Severely rate limit retrieving the files from the website / HTTP endpoint, and loudly point towards downloading the files via torrents. The torrent protocol was meant to relieve this level of server load in mind.

Torrents have the habit of disappearing when no users keep them alive. It happened to me enough times to be wary of such solution. If there's a way to keep them alive regardless of interest I'm all for it.

Torrent needs seeders the same as URLs need their original servers. There's a way to upgrade http to torrent though, like this: https://en.wikipedia.org/wiki/BitTorrent#Web_seeding

Re: Let us serve you, but don't bring us down

#237

Post by coldpie »

otikik wrote: post #235
coldpie wrote: post #204

Earlier quoted context omitted.

Is this speculation, or something that has actually been seen?

I would block it, if I was the maintainer. As the linked post mentions, automated requests can "ramp up" at any moment, risking server stability. By preemptively blocking automated parsing (on a resource which primary usage is individual requests, not mass ones) I would avoid future problems for myself. Let them contact us via support if they really need an exception. In general I would rate limit by IP anything conn…

So the answer is it's just speculation and has never been seen in the wild :)

Re: Let us serve you, but don't bring us down

#238

Post by coldpie »

tastysandwich wrote: post #227
coldpie wrote: post #204

Earlier quoted context omitted.

Is this speculation, or something that has actually been seen?

I can't say how the website I'm scraping would respond if I just went full throttle. But it's also just a matter of courtesy anyway not to make ten thousands requests per second. Funny story - at work we once had a huge spike in requests from a single IP. We all crowded around, thinking it was some malicious hacker from France. How exciting - we're now interesting enough to warrant a DoS! Turns out another team in th…

I was asking about randomizing the interval between requests.

Re: Let us serve you, but don't bring us down

#239

Post by hombre_fatal »

scotty79 wrote: post #225
hombre_fatal wrote: post #224

Earlier quoted context omitted.

Because it doesn’t accomplish anything. Things take longer for honest users while botnet abusers don’t even notice that the rented hardware is burning more CPU. Nor does it matter because each request still goes through.

For honest users it's not noticeable in normal conditions. But the DDOSer will DDOS themselves. They might make your service slow down for the users when they attack but not because your server gets swamped just because you choose to increase the difficulty of the challenge while you are attacked. DDOSers will notice because they won't be able to make requests as fast. At least not the ones that are costly for you. I…

I think the part you're not addressing is the fact that DDOSers aren't using their own hardware and more importantly: their request still goes through at the end so what do they care if it took 2000ms longer?

I actually do like your puzzle challenge idea from an obfuscation standpoint of making an API less attractive for clients that aren't your own, though. A challenge system + an annoying format like Protobufs instead of JSON is too much work for a lot of abusers.

Biggest problem with DDOS though is that if the volume is even reaching your application server, you're probably hosed.

Re: Let us serve you, but don't bring us down

#240

Post by jlokier »

remram wrote: post #222
jlokier wrote: post #169

Earlier quoted context omitted.

This is what "proof of storage" or "proof of data availability" blockchain networks are for. They use economic incentives to continuously pay nodes a small amount to store some data and keep it available, and the cryptographic sampling mechanism ensures that less popular data must remain in the available dataset for nodes to be paid, even if it is rarely requested in full.

If you are willing to spend money, there is no problem keeping content up with existing technology. Like always, blockchain solutions miss the problem entirely.

It would be nice not to shoot the messenger. The problem described by maksumur is a real problem people encounter often - torrents for less popular large files that people only occasionally want, but do want or need, have a habit of disappearing, or being non-functional when you do find a seed. Popularity plays too strong a role.

As far as I know the proof-of-data-availability folks (whether using a blockchain or not) are the only folks trying to solve this problem with a serious technical solution at a cost scale lower than "rent your own server and have deep pockets for bandwidth" at the moment.

It's not about "willing to spend money", as if there is only one threshold to pass.

Torrents are used largely because they diffuse the cost of hosting, so that people (and organisations) don't need deep pockets to distribute large files to many others. There is an inevitable infrastructure cost but it's spread out more fairly among users.

The proof-of-data-availability stuff is similar, but takes it further so that a wider range of data stays publically available to whoever wants to download it, whenever they want, than it otherwise would be. It is really just a more fancy version of torrenting that is less prone to the excessive per-file popularity fluctuations that torrents suffer from. The objective is to lower costs compared with the current state of the internet, not add more. And it does not specifically require a blockchain.

If you know of something else tackling the problem I'd be love to hear about it. Torrent seed sites don't qualify, as they don't solve the problem: Most files aren't available on them either, and you have to pay for the more obscure content they do have.

Post reply on HN