Live data from Hacker News

Web Scraping Is Vital to Democracy

themarkup.org

111–120 of 132 posts

Re: Web Scraping Is Vital to Democracy

#111

Clearly its time to come up with rules for this activity. Picking an apple from a tree in a public park is clearly harmless. Bringing in a combine harvester to take every scrap is not. How do we set a boundary on scraping?

They should sell scraping licenses, right? /s

Re: Web Scraping Is Vital to Democracy

#112

Earlier quoted context omitted.

I totally agree with the first part where you say it's fine to scrape. It is just an automation. But you don't mention what you do with this data after. 1. You store it somewhere (not in your brain) 2. You extract value out of it (directly or indirectly) That's why I understand why it is a problem for those who publish this data. One thing I always try to say to everyone who argues about web scraping: web scraping is…

> That's why I understand why it is a problem for those who publish this data. I still don't understand why it's a problem for those who publish the data. If the "data" is facts (e.g. lists of values associated with objects, like the colors available on automobiles), this data is not protected by copyright, there is no remedy if it's republished, and if a business relies on limiting access to this data, that business…

Are news just facts? Or do they also incorporate the bias of the author?

Sure, you could argue that news are facts, and that they should be free from copyright. But that would only work if your news is just bullet points of the factual stuff. But if any element of bias is present, it become a distinct property of the author, hence it should be copyrightable.

Of course, with today's tech, it's easier to remove the bias element from the factual data.

Re: Web Scraping Is Vital to Democracy

#113
post #61
post #58

Earlier quoted context omitted.

I think of placing a zillion cameras in public places as nothing less than automation of human work. Instead of hiring a million agents to record what the population is doing, you hire one programmer to automate that data collection process. Totally different? Yes, but it shows that scale can affect whether something is OK to do or not, at least for some (I accept that people watching the streets can be useful, but a…

If, for example, I start scraping all the articles on a newspaper’s web site, publish them on a web site, adding my own ads, most people would think that shouldn’t be legal. Publishing? Ads?

Republishing, duh.

Re: Web Scraping Is Vital to Democracy

#114
post #20

I think of web scraping as nothing less than automation of human work. There really is no hacking or unauthorized access involved. It's either me, using a browser, to read something publicly accessible on the internet, or it's my computer, which I've programmed to read those things for me, so I can focus on creative work or spend time with friends and family. This applies not only to personal life, but businesses as…

> There really is no hacking or unauthorized access involved. I'm not sure about that. If I had a phone book full of phone numbers (those heavy ones from 80s), would calling every number in that book to find the one I'm looking for be legal/ethical? P.S: I agree that "Web Scraping Is Vital to Democracy".

I can tell you haven't cold called ever in your life.

Re: Web Scraping Is Vital to Democracy

#115

Clearly its time to come up with rules for this activity. Picking an apple from a tree in a public park is clearly harmless. Bringing in a combine harvester to take every scrap is not. How do we set a boundary on scraping?

a website would be like an apple tree if there was an infinite number of apples in it. so you could bring a harvester and still leave enough apples for everyone.

Re: Web Scraping Is Vital to Democracy

#116

Earlier quoted context omitted.

Can the case be made that recording in public is a right (as if always should be) but trying to track where everyone is at every point of time is stalking at a mass scale which should be illegal as stalking on a one to one scale is illegal? For reference this is what one site (findlaw) has given as what constitutes the crime of stalking: The crime of stalking can be simply described as the unwanted pursuit of another…

> The crime of stalking can be simply described as the unwanted pursuit of another person. Examples of this type of behavior includes following a person, appearing at a person's home or place of business, making harassing phone calls, leaving written messages or objects, or vandalizing a person's property. Except for the last point, that sounds like ads :)

ads vandalize property by causing unwanted images to appear on and unwanted sounds to emit from personal property

Re: Web Scraping Is Vital to Democracy

#117

I use web scraping on my employer's Oracle EBS instance because coordinating with the battle-axe personalities that run ERP's is such a pain in the ass I would rather have a root-canal. With Web Scraping, I don't even have to talk to them.

For others: it looks like EBS in this context means "oracle e-business suite" and ERP means "enterprise resource planning" (maybe everyone else knows these, but just in case) https://en.wikipedia.org/wiki/Oracle_Applications#Oracle_E-B... https://en.wikipedia.org/wiki/Enterprise_resource_planning

Yes, that's right, and those of us that don't have to deal with these dreadful systems are better off :-)

Re: Web Scraping Is Vital to Democracy

#118

Earlier quoted context omitted.

> That's why I understand why it is a problem for those who publish this data. I still don't understand why it's a problem for those who publish the data. If the "data" is facts (e.g. lists of values associated with objects, like the colors available on automobiles), this data is not protected by copyright, there is no remedy if it's republished, and if a business relies on limiting access to this data, that business…

So if somebody compiles the data, charges an N amount for access, it's fine that some entity buys a one time access and then releases it for free?

This is usually explicitly prohibited.

If it's not prohibited for some reason, e.g. because the term of the prohibition, such as copyright, has expired, why not.

Re: Web Scraping Is Vital to Democracy

#119
post #82
post #8

This article focuses a lot on _good_ of web-scraping but I don't think we even need to go that far. Fight against web-scrapers just seems like a complete logical oxymoron: they want data to be public but also select who gets to see it. Our whole web infrastructure are based around clearly distinct public/private exchanges - there's no middleground this and yet people create these absurd hacks like captchas and finger…

I think web-scraping is ok but it should be treated as copyrighted data. Let's take a real example: We've built a company that creates ratings for doctors via a private process. We'd like the doctors to be able to show their good rating on their website via a linked widget. How do we stop a competitor from not only scraping our data but using it as their own and/or selling it to someone else?

By network effects and brand loyalty.

Re: Web Scraping Is Vital to Democracy

#120

Earlier quoted context omitted.

> The crime of stalking can be simply described as the unwanted pursuit of another person. Examples of this type of behavior includes following a person, appearing at a person's home or place of business, making harassing phone calls, leaving written messages or objects, or vandalizing a person's property. Except for the last point, that sounds like ads :)

ads vandalize property by causing unwanted images to appear on and unwanted sounds to emit from personal property

Vandalism usually means permanent damage. You can call the police on your neighbor for playing loud music at 3 am, but I don't think it would be correct to accuse them of vandalizing your property.
Post reply on HN