Live data from Hacker News

A year of fighting scrapers on my 1.5 million-page website

patronview.com

291–300 of 451 posts

Re: A year of fighting scrapers on my 1.5 million-page website

#291
post #107

Earlier quoted context omitted.

It doesn't detect "real browsers". It simply adds some friction and is niche enough that AI scrapers have not bothered to bypass it yet.

Yeah this is one of those things that hurts some real users and AI scrapers have figured out how to bypass. Not worth it IMO.

How do you figure it hurts real users? The amount of compute/energy used on the proof of work is pretty minimal. You're using more when you watch a YouTube video or browse a JS-heavy web app.

Of course a sophisticated scraper can "figure out" how to bypass. It isn't trying to be foolproof, it's adding an extra cost to deter massive amounts of bot traffic.

I put it in front of my hobby project because I can't afford to serve hundreds of thousands of bot requests from residential proxies all across the world, and I didn't want to route all of my traffic through a third party company like Cloudflare.

I've been happy with Anubis.

Re: A year of fighting scrapers on my 1.5 million-page website

#292

Earlier quoted context omitted.

I disagree. I personally consider these my biggest problems with the Web: - Bias, specifically commercial bias - Webpage formatting: every website looks different, hides the information I want in different places. Sure, it "looks pretty", but I don't want pretty; I want info - Scams/SEO/etc... The LLMs are very good at reading from multiple sources, parsing, and presenting only the data in a consistent format. There…

And if you're lucky, it won't include rocks as an ingredient.

They're an important part of your diet if you happen to have a gizzard.

I've also been known to include some ground up rocks in my meals. I'm pretty picky though about them, I mostly just want a couple specific varieties.

Re: A year of fighting scrapers on my 1.5 million-page website

#293
post #206

Earlier quoted context omitted.

I don’t see the sliding scale - if the website didn’t want a bot they could have blocked you. They clearly don’t mind so what’s the issue? You still used a bot.

> I don’t see the sliding scale - if the website didn’t want a bot they could have blocked you. The point is that not all bots are bad. Assuming as much / implementing policies to that effect won't block all bots but it _will_ block the portion of bots that represent users considering giving you money. I have a series of bots that monitor various eCom sites to monitor prices over time for big-ticket items I am consid…

The tradeoff here is a classic false-positive vs false-negative issue. If the cost of the bad-bot false positive (which blocks your bot out) is lower than the cost of the bad-bot false negatives (which allow bad bots in), then it is still a good tradeoff, if a suboptimal situation.

Re: A year of fighting scrapers on my 1.5 million-page website

#294
post #237

Earlier quoted context omitted.

Because AI cannot be trusted as a reliable source of information. Information is more trustworthy when it comes from from primary sources on the open web.

Why not ? Can google be trusted ? Or facebook ? How do people have been accessing the internet for the last decade you reckon and how's this any different ?

Google (as it originally) existed wasn't a source of information, it was an *index* of it. You were trusting google as a source for {search_query}, you were trusting the links it gave as a source (based on your own evaluation). LLMs are fundamentally different, because you are trusting the software to actually generate the information in a truthful way.

If you use google (sans AI), you're putting some trust in their page ranking algorithm. If you use it with AI, you're trusting the same algorithm (since that's how the model gets it's sources), but then you're trusting the model to evaluate the sources for credibility and extract the information you actually want.

Re: A year of fighting scrapers on my 1.5 million-page website

#295

Earlier quoted context omitted.

you know what a browser is called by the web site? check the header that tells the version. USER AGENT not user, an agent on behalf of the user. the web was intended to be usable by agents belonging to users. in our path of the multiverse, the main agent we think of happens to be live interactive browser. but that need not have been, nor will it be in the future, the dominant USER AGENT. for a glimpse at one possible…

I fancy the approach myself... but, much like the grand ideas of OSI in networking, Tim Burners Lee's idea of the Web was never what was going to be used and a lot of the original terms/concepts just plain don't make sense for the Web that exists. It was, originally, a true document system, nothing more. It wasn't until many changes from the original idea that it started to be what people wanted & actually use today.…

I was on the internet before HTML and long before Mosaic...

I agree with one aspect of the narrative you're saying, his idea was held as being for «documents». To be sure, document was more abstract than document on a Mac.

Either way, I'm very sad it mostly turned into animated brochures instead.

But the reason styles were separate, the reason everyone talked about semantics, was always about the universal knowledge DAG and enabling machines to help us navigate and retrieve it. It's why so many of the early agents referenced navigation (Spyglass, Navigator) or the interconnectedness of things (Mosaic).

So, ok, big document “store”, but that's a lifeless noun. The information you are looking for, the dots connected for you surfacing what you didn't know, that's a living growing thing, an agentic retrieval savvy knowledge engine, distributed for curation and resilience.

The librarian was always to be a machine or many, on our behalf.

Re: A year of fighting scrapers on my 1.5 million-page website

#296
post #101

The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see. A second side effect of a knee-jerk reaction to bots…

Agreed! Cloudflare absolutely destroys user experience and honestly doesn't seem that effective in practice.

What's worked for me is I block any client that don't support brotli compression and http2. Seems to work well enough for stopping scrapers.

Re: A year of fighting scrapers on my 1.5 million-page website

#297

Earlier quoted context omitted.

I disagree. I personally consider these my biggest problems with the Web: - Bias, specifically commercial bias - Webpage formatting: every website looks different, hides the information I want in different places. Sure, it "looks pretty", but I don't want pretty; I want info - Scams/SEO/etc... The LLMs are very good at reading from multiple sources, parsing, and presenting only the data in a consistent format. There…

LLMs are not unbiased. Nothing is unbiased.

Yeah I had to laugh at that. LLM's solve commercial bias? Coming from seemingly multiple the most valuable companies of all time?

Re: A year of fighting scrapers on my 1.5 million-page website

#298

Earlier quoted context omitted.

> for a glimpse at one possible futureverse, check out what the home assistant community is up to, how they assemble then update the ambient information displays on their walls. their USER AGENTS are doing what most HN-style "hackers" dreamed of reading scifi as kids. I would love some pointers to read up on this, what are some examples of this? Genuinely, not being a troll or ironic

> I would love some pointers to read up on this, what are some examples of this? Genuinely, not being a troll or ironic There's a "look what I built!" thread every few days on the /r/homeassistant sub and the home assistant forums, too. The data sources vary but it's usually a mix of devices that the user has locally and some remote data sources like weather/public transit schedules APIs or RSS and in some cases also…

It's an interesting example. People did very similar things during the "desktop shell modification" scene in the early 2000s, by far the most common thing people modified their desktop shell to do was display a bunch of charts and graphs of semi-random information, usually about their local computer.

It was useless in 99.9% of the cases. You don't need or want to have constant access to how much free ram your computer currently has, as if that even meant anything.

So, after a few steps, that leads to the question "what is useful information? How do you define that?"

A basic attempt at it might be: useful information is something that causes you to make or change a previous decision. Take an action you wouldn't have otherwise done.

If you're living in the american southwest, and it's between the months of like, march and, dunno, septemeber, you probably don't need to know the current outside temperature to decide if you're wearing shorts or a jacket when you get up in the morning (you're wearing shorts because it's always hot)

Theoretically there could be some kind of "extreme" weather, aka rain, that causes you to do something different, bring an umbrella, but it's pretty difficult to not notice it currently raining, even if you're inside a house.

So what other information do you want? There's all sorts of stuff that's considered "news", but I'd argue that at least 98% of that is basically just entertainment. We'd need some kind of article or perhaps even a book to hammer out all the nuances involved, but seeing a headline in the morning that trump has caved in to iran again or killed some more random people on a boat, while sort of important to know, doesn't really change what you're going to be doing that day.

Knowing ICE is trying to do some domestic terrorism near where you might be could be useful information, but there's not a real good way to track that.

So yeah, what kind of information is actually useful to have in the morning? Stuff like calendar appointments would count, knowing you have things scheduled at specific moments or tasks to do, go grocery shopping, etc, todo lists in otherwords, but what else? That's basically all self-generated stuff by you about you, you don't really need any outside data sources.

Re: A year of fighting scrapers on my 1.5 million-page website

#299
post #117
post #101

The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see. A second side effect of a knee-jerk reaction to bots…

> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. No; in this case you are not a user, you are a bot user.

I think it's time for people to build a local database of every site they've ever visited, and then they can give their LLMs access to that. I might be willing to pass this data off to a company to store for me. Companies already store all my emails and money--why not trust them with this too?

Like, all the comments of this post would go into my personal database simply because I loaded the page, and it would help me find old information I've read, and could also inform LLMs I use.

This should be built into browsers.

Re: A year of fighting scrapers on my 1.5 million-page website

#300
post #206

Earlier quoted context omitted.

I don’t see the sliding scale - if the website didn’t want a bot they could have blocked you. They clearly don’t mind so what’s the issue? You still used a bot.

> I don’t see the sliding scale - if the website didn’t want a bot they could have blocked you. The point is that not all bots are bad. Assuming as much / implementing policies to that effect won't block all bots but it _will_ block the portion of bots that represent users considering giving you money. I have a series of bots that monitor various eCom sites to monitor prices over time for big-ticket items I am consid…

The fundamental issue here is the way we use the internet is changing… and what we are seeing is some friction as that works its way through.

I have a business where I clearly don’t want any bots. A retailer, maybe they do.

Post reply on HN