Accessing Publicly Available Information on the Internet Is Not a Crime

However: That’s not what this article is about. That we don’t have a perfect solution for whatever weird corner cases (accidentally clicking on child porn?), should not change this very honest, serious and real issue the eff is addressing here. It is a distraction. We can hypothesise about edge cases until the cows come home, but to what end?

I get how a life of working in binary makes us immediately jump to the corner cases. It’s a curse on any legal discussion on HN. But it’s not relevant, and, imo, it dilutes the energy.

Edit : that came out harsh so I’d like to clarify: I get, 100%, where this “looking for the flaws” mentality comes from. It’s what makes a good programmer. A function that only follows the spec for 75% of its possible inputs is wrong. A law, not necessarily. We need to be careful not to keep our engineering hats on when switching to discussing law.

After doing something that pisses a lot of people off, they start getting 1000X calls per day on the same number, almost all complaints.

This cases actual damages (no "normal" customers can get through) and is also clearly outside the scope of "normal" usage.

Do you think the same rules apply?

The only people a robots.txt law would affect are private users who set up a Python script to scrape a single page for themselves to check for something, things like archive.org, researchers, automated website testers, etc. while anyone nefarious can just rent a shady VPN or use a server in Russia, China, Middle East, etc.

Requiring a human barely increases the cost if that data is so valuable in the first place and would be last resort anyway, far after just running the bots from a shady country, for captcha it's done because it's technically easier/cheaper (although supposedly automated solvers exist too).

But laws that punish outright gross negligence would help protect everyone who uses these American websites (and most of the world does) from data leaks of data that is arguably way more sensitive (emails, unhashed passwords, SS and CC numbers, real names even like in Ashley Madison case, etc.).

LinkedIn used sha1 with no salt as recently as 2012 (when they were hacked) for passwords and over 100 million such username + password combinations got stolen. Not only is sha1 not good enough for passwords but for many common and simple words (yes, yes, they are bad passwords, but people do use them) just googling can "crack" them due to lack of salt. The law should either go both ways or neither.

To suggest such heavy handed laws like considering robots.txt ignorance hacking while multi million corporations with millions of users get away with stuff like that (and I mean true negligence of most basic practices, not some obscure bug in the underlying software or something else that isn't absolutely obvious) over and over and over again that every random free my-first-login-page and my-first-SQL-injection-prevention tutorials advise against is absolutely ridiculous and anti-consumer.

Okay. Start with something simple then - how would you define a "bot" and thus subject to your robots.txt rule?

Is my web-browser a bot? What about a proxy? What about a deaf persons screen reader?

If my web-browser pre-fetches links near my mouse pointer, is that a bot? What if it downloads the whole of an article split over, say, ten pages?

I think of robots.txt similar to posting a "No Trespassing" sign. For a private residence, it's almost not even required, yet for something like a shopping mall during opening hours, the default assumption is that anyone is allowed to be there without a specific invitation, until they are expressly asked to leave and not come back.

Trying to nail down the exact line is a tough issue.

IANAL but, uh, seems legit... ¯\_(ツ)_/¯