Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Accessing Publicly Available Information on the Internet Is Not a Crime

eff.org

Post reply on HN
Print view View original

271–280 of 299 posts

Re: Accessing Publicly Available Information on the Internet Is Not a Crime

#271

Post by paulus_magnus2 »

_mhr_ wrote: post #246
paulus_magnus2 wrote: post #216

How about: if you want me not to scrape it, keep it off my internet?? Actually I'm considering building "API-fication" of websites with bindings for major languages (Java, Python, JS). With luck websites could participate by providing & maintaining a parseable API-sitemap. This would open door to my 2nd project: orchestration a-la BPEL on top of websites. visual editor, macros, scripting. Call this PIPES 2.0

Can you provide some use-cases for why this would be useful in a way that wouldn't violate most sites' ToS?

- a lot of online stores, hotels need to constantly update prices based on what competitors do.

- cleaning (big) data. Automatically reconcile data to canonical format / names using authoritative source (say wikipedia)

Can you understand even the simplest TOS? I'd argue most (all?) are too restrictive to be enforceable. https://tosdr.org/

Re: Accessing Publicly Available Information on the Internet Is Not a Crime

#272

Post by ben_w »

alexdoma wrote: post #210
feelin_googley wrote: post #203

Here is an example of the "good bot"/"bad bot" nonsense in action. This is an article about the LinkedIn v hiQ case at AdWeek. curl --user-agent INSERT_ANYTHING_HERE http://www.adweek.com/digital/rami-essaid-distil-networks-guest-post-linkedin-hiq-labs/ It seems AdWeek can distinguish a "good bot" from a "bad bot" irrespective of the behavior of the user^W bot, i.e., whether it is one single HTTP request or 10,000 co…

What would be a better solution, IP address check to allow only known google crawlers perhaps?

That practically invites them to present a different page to google as to a normal user, the former pure SEO, the latter perhaps pure advertising.

Re: Accessing Publicly Available Information on the Internet Is Not a Crime

#273

Post by tomarr »

odorousrex wrote: post #181
gwbas1c wrote: post #168

Earlier quoted context omitted.

> Access that does not require authentication should never be a crime. Careful, this could legitimize things like accidental denial of service. Depending on circumstances, even basic scraping could cause problems. (I need to be vague to avoid violating an NDA.) A major internet site had a URL that went something like somedomain/group?id=xxxxx. It turns out that a simple scraper, that called id=1, id=2, id=3, ect, ect…

>A major internet site had a URL that went something like somedomain/group?id=xxxxx. It turns out that a simple scraper, that called id=1, id=2, id=3, ect, ect, caused a major problem! This is a failure on the part of the developers at that "major internet site". Using a guid instead of consecutive IDs, a rate limiter, hell even just a cache...or all of the above. There are lots of solutions here. You have to take ro…

>You have to take robot scraping and indexing into consideration, and assume people will ignore robots.txt. (Certain bots, i.e. msnbot/bingbot are quite aggressive!)

Who owns LinkedIn again?

Re: Accessing Publicly Available Information on the Internet Is Not a Crime

#274

Post by blowski »

nostrademons wrote: post #247
bottled_poe wrote: post #242

Earlier quoted context omitted.

Scraping is not an intended purpose for most websites. Unless the website specifically states that this is an intended function, it is not reasonable to assume so. In fact it may be in violation of the terms and conditions of the given website.

If the law assumed that only intended functions are permissible, innovation would be a crime. By definition, innovation is finding new and unforeseen uses for resources.

You both make good points. If you make the law too strict you punish reasonable uses of the website, like scraping a few publicly available pages to help users. If you make it too lenient you permit DOS attacks.

It’s not easy to craft a law that will punish bad behaviour without blocking innovation.

Re: Accessing Publicly Available Information on the Internet Is Not a Crime

#275

Post by throwanem »

JepZ wrote: post #241
gwbas1c wrote: post #168

Earlier quoted context omitted.

> Access that does not require authentication should never be a crime. Careful, this could legitimize things like accidental denial of service. Depending on circumstances, even basic scraping could cause problems. (I need to be vague to avoid violating an NDA.) A major internet site had a URL that went something like somedomain/group?id=xxxxx. It turns out that a simple scraper, that called id=1, id=2, id=3, ect, ect…

Honestly, we all know the wild west mentality of the internet (yes, it is post national as in 'above the law') and therefore, everyone should assume attacks like that and build defenses against them. Building a service which could be brought to a 'major problem' with simple requests leading to high server loads is just negligent . What would that site do if someone actually wanted to attack it?!? I am not saying that…

> the wild west mentality of the internet (yes, it is post national as in 'above the law')

No. It's only been ahead of most laws for a while, as all frontiers are while they remain frontiers. But all frontiers eventually close, and laws catch up with them as they do so. That is what we're seeing now, and have been for a decade or more.

Re: Accessing Publicly Available Information on the Internet Is Not a Crime

#276

Post by da_chicken »

alasdair_ wrote: post #223
da_chicken wrote: post #198

Earlier quoted context omitted.

If the activity caused actual damages and was outside the scope of normal usage? Yes. You're still culpable if your actions break your neighbor's window, even if it was accidentally while you were opening it.

Say a business publishes a phone number and they typically get X calls per day. After doing something that pisses a lot of people off, they start getting 1000X calls per day on the same number, almost all complaints. This cases actual damages (no "normal" customers can get through) and is also clearly outside the scope of "normal" usage. Do you think the same rules apply?

Yeah, that's not how the comment read when I responded. It said "be held liable," not "do time."

I must've responded while he was editing it, and I didn't catch the change.

Re: Accessing Publicly Available Information on the Internet Is Not a Crime

#277

Post by da_chicken »

__jal wrote: post #201
da_chicken wrote: post #198

Earlier quoted context omitted.

If the activity caused actual damages and was outside the scope of normal usage? Yes. You're still culpable if your actions break your neighbor's window, even if it was accidentally while you were opening it.

Unless I'm missing something, you're proposing criminal penalties for tort liabilities. Yes, if my crappy software costs you money by knocking your site offline by accident, I should make you whole. I think it has to be something substantially more impactful, clearly intentionally malicious, or in some other way much worse than aggressive timeouts before we start thinking criminal penalties.

No, when I responded it read "be held liable." I must've commented while it was being edited.

Re: Accessing Publicly Available Information on the Internet Is Not a Crime

#278

Post by ams6110 »

rayiner wrote: post #85
xoa wrote: post #52

Earlier quoted context omitted.

> Just because the cafe on the corner has its door open and lets you stroll in off the street doesn't mean that the property owner doesn't retain the right to exclude people. While I don't know about the EFF's overall argument, as an absolute statement I don't think you are correct here. In the USA at least, "Public Accommodations" (which your cafe example would be) are in fact subject to regulations that limit their…

Laws against discrimination don't turn private property into quasi public property. They are narrow exceptions to the way in which property owners exercise their right to exclude. Neither the corner cafe nor Linked In can refuse to serve a request by someone because the person is black. But both the corner cafe and Linked In can refuse to service someone for any non-discriminatory reason, such as say because they're…

Or, more reasonably, because they are refusing to comply with some expectations for behavior that apply to everyone there.

Re: Accessing Publicly Available Information on the Internet Is Not a Crime

#280

Post by davvolun »

jstarfish wrote: post #218
dec0dedab0de wrote: post #206

Earlier quoted context omitted.

No, that is a failure of the developer of the scraper. I am definitely pro scraping, but you have to be a good neighbor.

How the hell is the scraper dev supposed to anticipate how poorly-written these particular views are with no backend knowledge? If not an automated scraper, a thundering herd from content gone viral would trigger the same result.

I've done some scraping work -- one of my rules of thumb is to always assume the worst of their site and try to be as gentle as possible.
Post reply on HN