Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Can web scraping be the basis of a viable business model?

news.ycombinator.com

Post reply on HN
Print view View original

101–110 of 114 posts

Re: Ask HN: Can web scraping be the basis of a viable business model?

#101

Post by [deleted] »

elliottinvent wrote: post #70
apienx wrote: post #9

Google, a trillion dollar company, is essentially the world's largest web scraper. So...yes! You'll almost certainly find a way to monetize that. Monopolies, lobbying and protectionism got in the way of keeping the web truly machine readable. There's tremendous value in restoring some of it.

> Monopolies, lobbying and protectionism got in the way of keeping the web truly machine readable. Exactly and that ship has long since sailed. The good ship Web 3.0 (semantic web) launched in ‘99 and was a ghost ship until recently when it was boarded by crypto pirates now flying the web 3.0 flag. > There's tremendous value in restoring some of it. To this comment and OP, my startup is using web scraping to pre-popu…

[deleted]

Re: Ask HN: Can web scraping be the basis of a viable business model?

#102

Post by mattzito »

There is always value in having a store of data that is unique or differentiated in some way. The trick is figuring out what that is and who might be interested.

For a while I ran product for a social monitoring company and our traditional user base was brands, agencies, etc. who would use our giant database of public content to do market research, etc. At various points we would get inbound requests from someone with a unique ask - I recall:

- a military historian working on a government grant who wanted to analyze the social media activities of various militias in a particular part of the world

- several pharma companies looking for adverse drug reaction reports online

- hedge funds looking for deep sentiment trends in particular areas for perception of certain businesses

- some company looking to find properties where women made announcements that they were pregnant.

And then there’s always the requests for X but in Y language/country. “There’s a Twitter like service in Bangladesh, can you get that data?”.

All of these people had money to spend and specific interests - we couldn’t help most of them as the economics didn’t work out in terms of building a scalable business, but if you can find a niche and run things lean, there’s a real long tail of opportunity there.

Re: Ask HN: Can web scraping be the basis of a viable business model?

#103

Post by hartator »

yosito wrote: post #100
hartator wrote: post #97

I am the CEO of https://serpapi.com . Please consider helping us support the EFF actions. Outside the obvious vested interests of our business, I truly believe scraping the web is a force of good and progress. And the EFF work in ensuring web scraping stays a legal practice in the Unite States has been outstanding. [1] [1] https://www.eff.org/deeplinks/2021/07/eff-ninth-circuit-rece...

I guess that's SERP API. In Spanish, ser papi means "to be a daddy".

Yes, SerpApi. Not the Spanish one!

Re: Ask HN: Can web scraping be the basis of a viable business model?

#104

Post by 14 »

I saw a project here on HN that was basically a webscraper for non developers that aimed to make it easy to scrape data from various sites. What some warned the project did not do is warn you that you could potentially be banned from some services by using bots and scrapers. I forget the project name but once someone warned it could potentially have your facebook or whatever banned I decided not to try it out. I would warn users be very careful about the TOS for each site you decide to scrape if you are logged in as a user.

Re: Ask HN: Can web scraping be the basis of a viable business model?

#105

Post by mateo1 »

A good think to ask yourself is whether there's anything left out there that's both accessible (legally and technically) and worth scraping at the same time.

Search engines deliberately wiped out personal websites, blogs, small news organisations etc, and spammers drowned out the remaining real user generated content from the www. Social media ate the forums and closed the doors.

Websites now aren't websites but businesses and they don't like people snooping around

Re: Ask HN: Can web scraping be the basis of a viable business model?

#108

Post by arinze11 »

Hi , I am the ceo of http://webautomation.io the largest marketplace for no-code webscrapers. I can tell you first hand the Market you are going after is very big and getting bigger everday. We get thousands of business's/people sign up everymonth . So my advice is not to worry as I am sure you will find a viable business with your proposed product

Re: Ask HN: Can web scraping be the basis of a viable business model?

#109

Post by ryuta »

arinze11 wrote: post #108

Hi , I am the ceo of http://webautomation.io the largest marketplace for no-code webscrapers. I can tell you first hand the Market you are going after is very big and getting bigger everday. We get thousands of business's/people sign up everymonth . So my advice is not to worry as I am sure you will find a viable business with your proposed product

Do you guys have an in-house team to deal with legal troubles?

Re: Ask HN: Can web scraping be the basis of a viable business model?

#110

Post by bagacrap »

asdfasgasdgasdg wrote: post #88
apienx wrote: post #9

Google, a trillion dollar company, is essentially the world's largest web scraper. So...yes! You'll almost certainly find a way to monetize that. Monopolies, lobbying and protectionism got in the way of keeping the web truly machine readable. There's tremendous value in restoring some of it.

> Google, a trillion dollar company, is essentially the world's largest web scraper. Even just considering the parts of Google that it takes to bring you the N blue links part of the Google SERP, the web scraper is probably the least interesting and significant piece of technology in the stack. It's beyond reductive to say that Google is in essence a large web scraper, or a web scraper of any kind. It is like saying…

why is Google's web scraping boring/easy? I'm sure it must be very sophisticated/optimized since they want it to be done often and cheaply
Post reply on HN