Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping Is Vital to Democracy

themarkup.org

Post reply on HN
Print view View original

31–40 of 132 posts

Re: Web Scraping Is Vital to Democracy

#31

Post by krageon »

sokoloff wrote: post #26
mnmkng wrote: post #20

I think of web scraping as nothing less than automation of human work. There really is no hacking or unauthorized access involved. It's either me, using a browser, to read something publicly accessible on the internet, or it's my computer, which I've programmed to read those things for me, so I can focus on creative work or spend time with friends and family. This applies not only to personal life, but businesses as…

While I have substantial agreement with your point of view, when web content is substantially ad-funded, automated scraping effectively bypasses the “payment”. I say this as someone who runs an adblocker, installs them for family, and doesn’t derive income from hosting ads, so I’m not pro-ad; I just realize that while it’s not a crime for me to dump the whole bowl of waiting room candies into my backpack, that’s goin…

Advertisements are harmful to every layer of society, mostly because they prey on you to instill desires that you probably wouldn't have had on your own (it being the case that this is their entire added value proposition). They should not be tolerated, and the fact that they ever were is a travesty.

That being the case, any argument founded on "but advertisements" does not hold water.

Re: Web Scraping Is Vital to Democracy

#32

Post by krageon »

amelius wrote: post #7
1vuio0pswjnm7 wrote: post #6

"In both of those instances, the pages and data scraped are publicly available on the internet-no hacking necessary-but sites involved could easily change the fine print on their terms of service to label the aggregation of that information "unauthorized."" They could. But if they filed a claim under the CFAA without ever sending a cease and desist letter to the alleged intruder, I think the claim would be dismissed.…

So robots.txt has no meaning?

No, because robots.txt is a friendly request and not a demand. It can never be a demand.

Re: Web Scraping Is Vital to Democracy

#33

Post by norwegianwhale »

bzb6 wrote: post #23

You should also have called it a “human right” for improved virtue signalling.

"Human right" is one of those sacred terms that are supposed to mean so much but mean nothing. "Human rights" are just superstitious conventions about what's good/bad. "No one can deprive you of housing" oh look I am gonna set up my tent here in this warm shop and claim human rights protection.

Re: Web Scraping Is Vital to Democracy

#34

Post by agustif »

Last weekend I made a little playwright script to checkout stock item status of decathlon products. The bike -Rodillo- Has been out of stock for several weeks.

While coding it it was briefly on stock but I was too slow with my card input hahaha and so I neeed to now automate the buy part as soon as it's available, or make that run every 5min and mail/notify me if available...

Btw this was super easy, playwright as a puppeteer successor really rocks, another thing hard to hate from MS like TS or VS Code

PS: I brought down script execcution from 7 secocnds to 2 seocnds, by not loading any unneeded stuff css images external js fonts.

I felt like a good scraper citizen doing so, it was just two linnes to block by request type on the network level, and it worked like a charm to speed up the process.

Re: Web Scraping Is Vital to Democracy

#35

Post by artembugara »

mnmkng wrote: post #20

I think of web scraping as nothing less than automation of human work. There really is no hacking or unauthorized access involved. It's either me, using a browser, to read something publicly accessible on the internet, or it's my computer, which I've programmed to read those things for me, so I can focus on creative work or spend time with friends and family. This applies not only to personal life, but businesses as…

I totally agree with the first part where you say it's fine to scrape. It is just an automation.

But you don't mention what you do with this data after. 1. You store it somewhere (not in your brain) 2. You extract value out of it (directly or indirectly)

That's why I understand why it is a problem for those who publish this data.

One thing I always try to say to everyone who argues about web scraping: web scraping is not a problem, problem is what you do with the information you scraped.

Disclaimer: We crawl the web for news (https://newscatcherapi.com/)

Re: Web Scraping Is Vital to Democracy

#36

Post by LunaSea »

krageon wrote: post #31
sokoloff wrote: post #26

Earlier quoted context omitted.

While I have substantial agreement with your point of view, when web content is substantially ad-funded, automated scraping effectively bypasses the “payment”. I say this as someone who runs an adblocker, installs them for family, and doesn’t derive income from hosting ads, so I’m not pro-ad; I just realize that while it’s not a crime for me to dump the whole bowl of waiting room candies into my backpack, that’s goin…

Advertisements are harmful to every layer of society, mostly because they prey on you to instill desires that you probably wouldn't have had on your own (it being the case that this is their entire added value proposition). They should not be tolerated, and the fact that they ever were is a travesty. That being the case, any argument founded on "but advertisements" does not hold water.

It's not founded on "but advertisements" it's founded on "but paying for content". The fact that websites are developed by people (wages), hosted on infrastructure (hardware renting) and require countless other jobs is somehow magically forgotten in these discussions.

Re: Web Scraping Is Vital to Democracy

#37

Post by brandonr49 »

lopmotr wrote: post #28
mschuster91 wrote: post #10

Earlier quoted context omitted.

Not OP, but it's very hard to intercept and monitor the traffic of an app that uses code signing, obfuscation and certificate pinning properly. What used to be a five minute "put tcpdump on the access point / router to work" job pre-HTTPS everywhere now is a many days worth job of messing around - out of reach for all but the really dedicated, and as a result it is very hard for an user to have actual visibility over…

Does that really matter though? Isn't the proper way to evaluate what data is being shared to assume each app exploits its permissions up to the level of trust you have in the company that provided it? Eg. if you worry Facebook is stealing your contact list, then if their app has that permission, you should assume they are stealing it. No need to bother checking if they actually are. They might do it when you're not…

There's a bit of circular logic here because I can't evaluate who to trust if I can't effectively monitor what the developer is doing on my machine.

Re: Web Scraping Is Vital to Democracy

#38

Post by jagged-chisel »

artembugara wrote: post #35
mnmkng wrote: post #20

I think of web scraping as nothing less than automation of human work. There really is no hacking or unauthorized access involved. It's either me, using a browser, to read something publicly accessible on the internet, or it's my computer, which I've programmed to read those things for me, so I can focus on creative work or spend time with friends and family. This applies not only to personal life, but businesses as…

I totally agree with the first part where you say it's fine to scrape. It is just an automation. But you don't mention what you do with this data after. 1. You store it somewhere (not in your brain) 2. You extract value out of it (directly or indirectly) That's why I understand why it is a problem for those who publish this data. One thing I always try to say to everyone who argues about web scraping: web scraping is…

> That's why I understand why it is a problem for those who publish this data.

I still don't understand why it's a problem for those who publish the data.

If the "data" is facts (e.g. lists of values associated with objects, like the colors available on automobiles), this data is not protected by copyright, there is no remedy if it's republished, and if a business relies on limiting access to this data, that business needs a new business plan. Charge for access to cover the costs of obtaining and organizing the data, but understand that clients are legally allowed to make copies and use them however they see fit, even if that impacts the supplier's ability to charge for access.

If the "data" is prose, it's under copyright and republishing without a license has remedies under law. Maintaining copies of articles for the purpose of processing them to obtain other data (e.g. how many nouns? what adjectives are near those nouns? etc...) isn't protected.

Re: Web Scraping Is Vital to Democracy

#39

Post by hhjj »

IfOnlyYouKnew wrote: post #11
wraptile wrote: post #8

This article focuses a lot on _good_ of web-scraping but I don't think we even need to go that far. Fight against web-scrapers just seems like a complete logical oxymoron: they want data to be public but also select who gets to see it. Our whole web infrastructure are based around clearly distinct public/private exchanges - there's no middleground this and yet people create these absurd hacks like captchas and finger…

You just state that "there is no middle ground", then go on describing instances of what arguably is "middle ground" as "absurd hacks". What's it gonna be? There's a POV that, on the internet, "anything goes", i. e. whatever you can do, you're allowed to do. Then, there's a perspective that works a lot like the offline world, where any clear communication that a reasonable person would understand as denying them acce…

The offline world is the world you are by default, you can't choose to put your house elsewhere. On contrary, when you put information on internet you choose to do it there (at least for website owners...).

If you want to put restrictions on the usage of your data, make people sign a contract before accessing it.

Also a contract should not have "the public" as a party or a subset of, you should be able to identify the parties you have contracted with. Else you may end up with warrants targeting everyone or a subset of...

Re: Web Scraping Is Vital to Democracy

#40

Post by petr25102018 »

jagged-chisel wrote: post #38
artembugara wrote: post #35

Earlier quoted context omitted.

I totally agree with the first part where you say it's fine to scrape. It is just an automation. But you don't mention what you do with this data after. 1. You store it somewhere (not in your brain) 2. You extract value out of it (directly or indirectly) That's why I understand why it is a problem for those who publish this data. One thing I always try to say to everyone who argues about web scraping: web scraping is…

> That's why I understand why it is a problem for those who publish this data. I still don't understand why it's a problem for those who publish the data. If the "data" is facts (e.g. lists of values associated with objects, like the colors available on automobiles), this data is not protected by copyright, there is no remedy if it's republished, and if a business relies on limiting access to this data, that business…

Databases are actually protected under some copyright laws in some countries.
Post reply on HN