181–190 of 196 posts
Its easy to detect chrome headless so scraping with it is not really how "big" boys do it :D the only scrapers/bots that are really hard to detect are the ones running and controlling real browser and not chromium. I do a lot od research aggainst abitbot systems, some times is friday night. If you spend each one in pub it doesnt mean your normal.
Earlier quoted context omitted.
No? If you place information publicly on a website it's pretty much free game, no copyright violation, especially regarding user generated information. That's my take, but legally it's a gray area and it's still going back and forth in the courts (at least in the US) but for a while before a decision was vacated by the supreme court scraping publicly available information on a site was legally protected and seemingly…
If we are to live in a mutually prosperous society, how is the labor and therefore well-being of the content creator improved by a web scraper? Does this precedent not injury future opportunities for exercising one’s life to making website data available for others to scrape? By my understanding any website with a copyright disclaimer warrants their data as exclusively their own and are granting permission for other…
If you can't make that trade then you've weighed the value provided by an organization like google to be more valuable than the copyright of these content creators and I want other players who may want to be able to challenge google to have the same protections and access google does to have a chance at providing the same value.
Earlier quoted context omitted.
If we are to live in a mutually prosperous society, how is the labor and therefore well-being of the content creator improved by a web scraper? Does this precedent not injury future opportunities for exercising one’s life to making website data available for others to scrape? By my understanding any website with a copyright disclaimer warrants their data as exclusively their own and are granting permission for other…
I'm actually cool assuming there is implicit harm and no benefit, but by that logic we need to tear google down too. I'm cool making that trade but it has to be done equally. If you can't make that trade then you've weighed the value provided by an organization like google to be more valuable than the copyright of these content creators and I want other players who may want to be able to challenge google to have the…
Earlier quoted context omitted.
I'm actually cool assuming there is implicit harm and no benefit, but by that logic we need to tear google down too. I'm cool making that trade but it has to be done equally. If you can't make that trade then you've weighed the value provided by an organization like google to be more valuable than the copyright of these content creators and I want other players who may want to be able to challenge google to have the…
Well actually, isn’t Google improving the value of the content ergo property itself by improving its accessibility? I was inferring a one-way street with the accumulated data that can lead to server crashes - which I don’t believe Google’s web crawl does at all (in fact that would be counter-productive).
Earlier quoted context omitted.
You've been in the SEO industry for some time and never heard SERP?
The web programming industry, not the SEO industry
Its easy to detect chrome headless so scraping with it is not really how "big" boys do it :D the only scrapers/bots that are really hard to detect are the ones running and controlling real browser and not chromium. I do a lot od research aggainst abitbot systems, some times is friday night. If you spend each one in pub it doesnt mean your normal.
puppeteer-extra and undetected-chromedriver beg to differ :)
Earlier quoted context omitted.
Why not create api endpoint and charge mild cost for that data? You’ll make money instead of spending it.
Do you honestly believe all site scraper people/companies are ethical enough to go to whoever pays /them/ to scrape data from a competitor's site and say "oh they offer an API to access this data let's pay for that", instead of "why pay for that data when we can scrape it right off their site"? Also, not all types of company will provide API endpoints. It all depends on the type of site - for example, an online shop…
Valuable public data is going to be scraped - this is inevitable. Even paywalled or signup protected valuable data is going to be scraped.
Why not sell valuable data for reasonable price then.
Basic question, how does one profit from scraping data and what kinda data? Taking a stab at answering it: you scrape the data and build a business around selling it. Stock prices? But that's boring, plus how many others are doing it? I bet a lot.