Ask HN: Can web scraping be the basis of a viable business model? • Hacker News Board

Re: Ask HN: Can web scraping be the basis of a viable business model?

#111

Post by asdfasgasdgasdg » Sun, Dec 19, 2021, 10:19 PM UTC

bagacrap wrote: post #110

asdfasgasdgasdg wrote: post #88

Earlier quoted context omitted.

> Google, a trillion dollar company, is essentially the world's largest web scraper. Even just considering the parts of Google that it takes to bring you the N blue links part of the Google SERP, the web scraper is probably the least interesting and significant piece of technology in the stack. It's beyond reductive to say that Google is in essence a large web scraper, or a web scraper of any kind. It is like saying…

why is Google's web scraping boring/easy? I'm sure it must be very sophisticated/optimized since they want it to be done often and cheaply

It is not boring, it is just not as interesting/difficult as the remainder of the stack.

Re: Ask HN: Can web scraping be the basis of a viable business model?

#112

Post by ozfive » Mon, Dec 20, 2021, 4:20 PM UTC

preinheimer wrote: post #94

ozfive wrote: post #91

Earlier quoted context omitted.

Wasn't there a ruling that web scraping was legal now?

The LinkedIn case, it's still up in the air i think - https://news.bloomberglaw.com/us-law-week/supreme-court-scra...

Thanks for this!

Re: Ask HN: Can web scraping be the basis of a viable business model?

#113

Post by edmundsauto » Tue, Dec 21, 2021, 1:17 AM UTC

stef25 wrote: post #67

edmundsauto wrote: post #20

Yes, although I would encourage you to think about something higher on the value chain than raw data feeds. Those exist and have become an increasingly difficult market to compete in. You can buy a custom feed for like $250/mo. Instead, think about what people want to do with the data. For example, if you are going to scrape diamond prices, don’t try to sell that feed. Set up a website with a UI so people can researc…

Like camelcamelcamel.com I guess?

Exactly. The data is interesting, but a commodity. If the market is big enough, figure out what they’re using it for and help them use it. Sell access to processed data.

Re: Ask HN: Can web scraping be the basis of a viable business model?

#114

Post by itake » Sat, Dec 25, 2021, 3:35 PM UTC

exdsq wrote: post #23

itake wrote: post #7

You might want to look at web scraping for data scientists. I am trying to build a ML Model for NSFW text detection in multiple languages and I am not looking forward to scraping p*rn and youtube websites for comments.

Could you expand on this a little? What’s the problem and what are you looking for as a solution?

Humans submit NSFW text content to platforms that are no intended to host such content (like chats in video games or reviews for products). It is typically too expensive for a company to hire humans to review all text content, so they may want an ML model that can help them identify the unwanted text.