I use the Internet Archive a lot, and I get rate limited after opening (manually!) a few dozen pages now. The message is clear: archive.org is meant for one-off lookups, if you're doing real historical research, you're not welcome. I thought creating an account and even donating money might help me, but nope.
Let us serve you, but don't bring us down
211–220 of 255 posts
Re: Let us serve you, but don't bring us down
#212reminds me of a general pattern I've observed in life: 1. whatever interface you expose to the public, some will abuse it 2. rules/laws/countermeasures are then added and enforced, often universally (though sometimes inconsistently) 3. which then bites, annoys, insults or otherise burdens or adds to the prices paid by ALL the OTHER non-abusing users pf that same interface always. eventually. every time
It's a digital Tragedy of the Commons: https://en.m.wikipedia.org/wiki/Tragedy_of_the_commons
agreed
Re: Let us serve you, but don't bring us down
#213Earlier quoted context omitted.
I hate to be a pedant but I see it used wrong a lot. "DDOS" stands for Distributed denial of service, specifically indicating the traffic is coming from many sources (wide range of ip), which is what makes it so hard to defend against. Someone scraping too fast would be performing an unintentional DOS, because they probably arent scraping using a botnet (and if they were they probably do in fact intend to attack)
In practice they're often using botnets. It's what makes this such a difficult problem to deal with. If they weren't it would be trivial to rate-limit or block problematic IPs or subnets.
as I mentioned in the post: why would a legitimate scraper be using a botnet?
Re: Let us serve you, but don't bring us down
#214I have a side project that scrapes thousands and thousands of pages of a single website. So as not to piss them off (and so they don't try to block me), my script will take about 6 hours. Between each page fetch it sleeps for a small, random amount of time. It's been working like that for years.
I build a system that scraped GitHub. Even though GitHub can clearly handle lots of traffic I still rate limited the hell out of it. The only time I've ever seen scrapers get banned is when go super fast. Unless it's LinkedIn and Instagram who guard their product aka data as much as possible.
> I still rate limited the hell out of it
Merely for your consideration, they actually do a great job of indicating in the response how many more requests per "window" the current authentication is allowed, and a header containing the epoch at which time the window will reset: https://docs.github.com/en/rest/overview/resources-in-the-re...
I would suspect, all things being equal, that politely spaced requests are better than "as fast as computers can go" but I was just trying to point out that one need not bend oppressively over the other direction when the site is kind enough to tell the caller about the boundaries in a machine-readable format
Re: Let us serve you, but don't bring us down
#215We have a service that was often crawled by bots on cloud providers, luckily we can block AWS IPs without to much hassle.
I also recently learned they publish changes to their IP netblocks almost daily as part of their new "please use more SNS" blog post: https://news.ycombinator.com/item?id=35056973
For example: https://aws-new-features.s3.us-east-1.amazonaws.com/update/2... (see the `NEW_ip_prefixes` and `ip_prefixes_REMOVED` keys)
Re: Let us serve you, but don't bring us down
#216Earlier quoted context omitted.
> cheap for people who own mining ASICs and expensive for occasional users. Seems like it should be the same cost (barring the friction of having a wallet, etc.) for both sets of people since Bitcoin is just a commodity and the value is the same to everyone, miner or not. For example, a miner should value some fraction of a BTC the same way anyone else does, since they can sell or buy it at the same price a normal us…
Miners have ASICs - specialized silicon that ONLY mines Bitcoin, but does so many times faster than a CPU or GPU can. The average low-demand user does not have such hardware. Furthermore, because Bitcoin increases difficulty to maintain a fixed transaction rate, ASICs have to be replaced with newer models every few years. So older ASICs are going to be cheaper, but they might still be worth buying if you're a spammer…
Well said, that's exactly my point.
Re: Let us serve you, but don't bring us down
#217Earlier quoted context omitted.
I build a system that scraped GitHub. Even though GitHub can clearly handle lots of traffic I still rate limited the hell out of it. The only time I've ever seen scrapers get banned is when go super fast. Unless it's LinkedIn and Instagram who guard their product aka data as much as possible.
> I still rate limited the hell out of it Merely for your consideration, they actually do a great job of indicating in the response how many more requests per "window" the current authentication is allowed, and a header containing the epoch at which time the window will reset: https://docs.github.com/en/rest/overview/resources-in-the-re... I would suspect, all things being equal, that politely spaced requests are bet…
That's on the API. For what I wanted it was do various HTTP requests to the website and do git clones.
Re: Let us serve you, but don't bring us down
#218Earlier quoted context omitted.
> cheap for people who own mining ASICs and expensive for occasional users. Seems like it should be the same cost (barring the friction of having a wallet, etc.) for both sets of people since Bitcoin is just a commodity and the value is the same to everyone, miner or not. For example, a miner should value some fraction of a BTC the same way anyone else does, since they can sell or buy it at the same price a normal us…
Miners have ASICs - specialized silicon that ONLY mines Bitcoin, but does so many times faster than a CPU or GPU can. The average low-demand user does not have such hardware. Furthermore, because Bitcoin increases difficulty to maintain a fixed transaction rate, ASICs have to be replaced with newer models every few years. So older ASICs are going to be cheaper, but they might still be worth buying if you're a spammer…
Yes, sorry I wasn't expressing myself very clearly. I agree if the goal is to impose a uniform (per-unit) cost to users, you should avoid widely-optimized hash functions.
But even if you pick a novel function that doesn't have special-purpose hardware, spammers can still optimize their setups to lower the effective unit cost below what a legitimate user faces, by doing normal miner activities (picking hardware, scaling up, moving to where electricity is cheap, etc.).
Since your goal is to maximally discriminate between legitimate and spam use cases, you'd want spammers to face at least the same per-unit cost as legitimate users.
What's one way you can do that? Well, how about charging actual currency, whether Bitcoin or fiat? Money has the useful property of having the same nominal value for everyone, and not being amenable to further optimization.
In short, forcing users to actually run PoW themselves doesn't really make economic sense. Even if you're avoiding existing hash functions, it's mostly worse than just charging money because spammers have better ability to optimize against it.
And if charging money doesn't work, switching to local PoW is unlikely to be better.
Re: Let us serve you, but don't bring us down
#219Earlier quoted context omitted.
I usually make it a two step process: host example.com If this returns with an IP address, no need to talk to a registrar. Only if there is no IP address, I go with whois example.com
You could also do an in-between step of host -t soa example.com which should give you domains that have any DNS record at all, not just an A record.
Is there still a point talking to the registrar then? How common is it for a domain to exist with the registrar but have no SOA record?
Re: Let us serve you, but don't bring us down
#220Earlier quoted context omitted.
I answered on the other comment, but at that point, in case there is a blockchain, which supports a millionth of a cent transaction, then exchanges (Binance, Coinbase etc) are not so useful. If every person just needs one cent for a million http requests, then one guy in your neighbourhood or your town, or your city might have some of it, you message him and he will send you a cent for free. You buy him a coffee, and…
Yikes, that sounds terrible. The current best alternative is a visual captcha. I'm not driving around town to buy coffee for some random person just to be able to use the internet. That's not the sort of internet I want.
I am not suggesting that you should do that. As soon as we reach the sweet spot of transaction fees, between a thousandth of a cent, and a millionth of a cent, then every person will have a wallet with one dollar inside it, and send it to a site he visits and desires to not be considered a spammer.
In case you run out of proof_of_burn a friend of yours might send you just a cent, and you are good to go, for one hundred thousand http requests more. There is no need for a normal person, to hold more than a handful of dollars for every year's internet use.
That renders exchanges almost useless. Not totally useless, but much less relevant than they are today.
I personally wouldn't care less, if there is one bitcoin/blockchain which reaches that sweet spot, or if they are a hundred, including litecoin, ripple etc. PoW is meant to be used for practical reasons. Blockchain however constitutes an economic system, of suppliers-miners and consumers-users, it is more than just a software program. It will evolve in the future, and it requires some crucial time.
For the moment there no API which provides the kind of PoW service to be very useful, and some hacks might be required to mitigate side effects of relentless scraping. These are just hacks, useful today, but the real solution is coming soon. Web3 some people call it, or Cryptocosm is another name of it.