Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ban me at the IP level if you don't like me

boston.conman.org

Post reply on HN
Print view View original

471–480 of 516 posts

Re: Ban me at the IP level if you don't like me

#471

Post by dilyevsky »

hinkley wrote: post #469
dilyevsky wrote: post #427

Earlier quoted context omitted.

> And if you 429 Google’s bots they will reduce your pagerank. That’s straight up extortion from a company that also sells cloud services. Googlebot uses different IP space from gcp

They use the same bank accounts and stock ticker. This is basically a non sequitur. The point is they’re getting paid to run cloud servers to keep their bots happy and not dropping your website to page six.

I thought the argument was that if you run on gcp you can masquerade as googlebot and not get a 429 which is obviously false. Instead it looks like the argument is more of a tinfoil hat variety.

btw you don't get dropped if you issue temporary 429s only when it's consistent and/or the site is broken. that is well documented. and wtf else are they supposed to do if you don't allow to crawl it and it goes stale?

Re: Ban me at the IP level if you don't like me

#472

Post by genewitch »

imoverclocked wrote: post #378
reincoder wrote: post #374

I work for IPinfo. If you need IP Address CIDR blocks for any country or ASN, let me know. I have our data in front of me and can send it over via Github Gist. Thank you.

I’ve been having a heck of a time figuring out where some malicious traffic is coming from. Nobody has been able to give me a straight answer when I give them the ip: 127.1.5.12 Maybe you can help trace-a-route to them? I’d just love to know whois behind that IP. If nothing else, I could let them know to be standards compliant and implement rfc 3514.

the malicious traffic is coming from inside the house

Re: Ban me at the IP level if you don't like me

#473

Post by xp84 »

socalgal2 wrote: post #414
xp84 wrote: post #398

Earlier quoted context omitted.

Really makes you think about the calibre of minds being applied to buzzy problem spaces these days, doesn't it?

do we know they didn't download the DB? Maybe the new traffic is the LLM reading the site? (not the training) I don't know that LLMs read sites. I only know when I use one it tells me it's checking site X, Y, Z, thinking about the results, checking sites A, B, C etc.... I assumed it was actually reading the site on my behalf and not just referring to its internal training knowledge. Like how people are training LLMs,…

I would guess site operators can tell the difference between an exhaustive crawl and the targeted specific traffic I'd expect to see from an LLM checking sources on-demand. For one thing, the latter would have time-based patterns attributable to waking hours in the relevant parts of the world, whereas the exhaustive crawl traffic would probably be pretty constant all day and night.

Also to be clear I doubt those big guys are doing these crawls. I assume it's small startups who think they're gonna build a big dataset to sell or to train their own model.

Re: Ban me at the IP level if you don't like me

#474

Post by renewiltord »

johneth wrote: post #447
renewiltord wrote: post #372

I would never have considered this, but someone on HN pointed out that web user agents work like this. Servers send ads and there is no way for them to enforce that browsers render the ads and hide the content or whatever. The user agent is supposed to act for the user. "Your business model is not my problem", etc. Well, my user agents work for me, not for you - the server guy who is complaining about this and that.…

Well done on pointing out exactly what everyone here is saying in the most arrogant way possible. Also, well done on linking to your own comment where people explain this to you. The problem is that there is no way to "block me if you don't want me". That's the entire issue. The methods these scrapers use mean it's nigh on impossible to block them.

"Your inability to engineer is not my problem".

See: https://news.ycombinator.com/item?id=45018660

Re: Ban me at the IP level if you don't like me

#475

Post by 0x457 »

nitwit005 wrote: post #467
0x457 wrote: post #382

Earlier quoted context omitted.

So weird to scrape wikipedia when you can just download db dumb from them.

When you have a pile of funding, and you get told to do things quickly.

But the correct way (getting a sql dump) is faster?

Re: Ban me at the IP level if you don't like me

#476

Post by lxgr »

closewith wrote: post #364
lxgr wrote: post #353

Earlier quoted context omitted.

> Chargebacks outside the US are a true arbitration process, not the rubberstamped refunds they are there. What's true is that in the US, the cardholder can often just say "I've never heard of that merchant", since 3DS is not really a thing, and generally merchants are relatively unlikely to have compelling evidence to the contrary. But for all non-fraud disputes, they follow the same process.

As commented elsewhere, you're just wrong. It's a significant burden of proof for a cardholder to win a dispute for non-compliance with card network rules and it very rarely happens (outside of actual merchant fraud, which is much rarer in the EU). Again, you're not aware of the reality outside the US.

> It's a significant burden of proof for a cardholder to win a dispute for non-compliance with card network rules

That's true, but "fraud" and "compliance" aren't the only dispute categories, not by far.

In this case, using Mastercard as an example (as their dispute rules are public [1]), the dispute category would be "Refund not processed".

The corresponding section explicitly lists this as a valid reason: "The merchant has not responded to the return or the cancellation of goods or services."

> Again, you're not aware of the reality outside the US.

Repeating your incorrect assumption doesn't make it true.

[1] https://www.mastercard.us/content/dam/public/mastercardcom/n...

Re: Ban me at the IP level if you don't like me

#478

Post by leviathant »

ygritte wrote: post #6
Etheryte wrote: post #3

One starts to wonder, at what point might it be actually feasible to do it the other way around, by whitelisting IP ranges. I could see this happening as a community effort, similar to adblocker list curation etc.

Came here to say something similar. The sheer amount of IP addresses one has to block to keep malware and bots at bay is becoming unmanageable.

Knowing my audience, I've blocked entire countries to stop the pain. Even that was a bit of whack-a-mole. Blocking China cooled off the traffic for a few days, then it came roaring back via Singapore. Blocked Singapore, had a reprieve for a while, and then it was India, with a vengeance.

Cloudflare has been a godsend for protecting my crusty old forum from this malicious, wasteful behavior.

Re: Ban me at the IP level if you don't like me

#479

Post by pabs3 »

johnisgood wrote: post #448
pabs3 wrote: post #428

Earlier quoted context omitted.

Personal sites are definitely interesting, way more interesting than most of the rest of the web. I was thinking I would put your site into archive.org, using ArchiveBot, with reasonable crawl delay, so that it is preserved if your hardware dies. Ask on the ArchiveTeam IRC if you want that to happen. https://chat.hackint.org/?join=%23archiveteam-bs

It is a public git repository for the most part, that is the essence of my website, not really much writings besides READMEs, comments in code and commits.

A public git repository is even more interesting, for both ArchiveTeam Codearchiver, and Software Heritage. The latter offers an interface for saving code automatically.

https://wiki.archiveteam.org/index.php/Codearchiver https://wiki.archiveteam.org/index.php/Software_Heritage https://archive.softwareheritage.org/save/

Re: Ban me at the IP level if you don't like me

#480

Post by SoftTalker »

Dylan16807 wrote: post #369
SoftTalker wrote: post #322

Earlier quoted context omitted.

You probably shouldn't log usernames then, or really any form fields, as users might accidentally enter a password into one of them. Kind of defeats the point of web forms, but safety is important!

Are you using a very weird definition of "logging" to make a joke? Web forms don't need any logging to work.

You save them in a database. Probably in clear text. Six of one, half-dozen of the other.
Post reply on HN