Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Can web scraping be the basis of a viable business model?

news.ycombinator.com

Post reply on HN
Print view View original

71–80 of 114 posts

Re: Ask HN: Can web scraping be the basis of a viable business model?

#71

Post by eatonphil »

sonofhans wrote: post #25

I worked at AboutUs.org for a while, and that’s what we did. Good news: it was fun and rewarding. In many ways it felt like a satisfying old-skool problem: scrape, find edge case, patch it, scrape again. We were scraping 100 million domains once a week with a team of six engineers, one UX (me), and Ward Fucking Cunningham as wiki expert. Ward in particular was great at prototyping solutions. It is an arms race, since…

People make their own blogs and sites with React and other JavaScript frameworks all the time now... It won't even be actually dynamic content before you need to be able to execute JavaScript.

Re: Ask HN: Can web scraping be the basis of a viable business model?

#72

Post by that_guy_iain »

stef25 wrote: post #69
bjourne wrote: post #53

Web scraping is a legal gray area in many or most jurisdictions. In some jurisdictions, depending on the tos of the web site itself, scraping it might be illegal. In others republishing the scraped information in any form might be illegal. In others still you might not be allowed to use the scraped data for any commercial purpose. "But what about Google?" Google is worth 100 billion dollars and can play by completely…

Google also plays nice. I manage a few sites that get hammered by scrapers to the extent that it causes big spikes in CPU, something that doesn't happen with G.

So many people don't even think about number of requests and often want the job to end as quickly as possible. For me, that is thinking short term, I've seen requests per second as low as 4 cause performance issues for a company. If you want to scrape long term you need to think requests per minute. That is unless you're dealing with a mega site then it's keeping a low request per second.

Re: Ask HN: Can web scraping be the basis of a viable business model?

#73

Post by that_guy_iain »

DonHopkins wrote: post #39
bdcravens wrote: post #36

Yes, but scraping is a small part of the overall puzzle. As developers, we overestimate how valuable tools are (as opposed to solutions). I think the better opportunity is not to be another scraping-as-a-service provider, but to niche down to a solution that uses your scraping technology.

If you start yet another scraping-as-a-service provider, you're attempting to provide a paid service for people who just want to steal content for free. Not gonna work.

Scraping as a service can be profitable if you target people who are looking for leads to cold email/spam. Lead gen is one of the few areas you can easily charge $100 as your entry level package and at that price it's very easy to make money if you have half decent marketing.

Re: Ask HN: Can web scraping be the basis of a viable business model?

#74

Post by rank0 »

Some major issues from my experience web scraping:

1. Changes in data structures. If some site randomly decides to alter the format of their json/xml objects for their frontend api it may brake your scraper and anything that relies on that scraper’s output.

2. Security controls like rate limiting, captcha, ip blacklisting, auth systems.

3. Html which is rendered via complicated client side JavaScript blobs or web sockets. You’ll need a Headless browser engine like selenium and some site-specific parsing logic.

4. Legal issues.

Re: Ask HN: Can web scraping be the basis of a viable business model?

#78

Post by adinosaur123 »

I don't wish to hijack this thread, but I've been pondering a similar question. I've been working on a product that requires a very large amount of data that, as far as I can tell, can only be gathered by scraping (real estate data - even data vendors like estated.com don't have stuff like sales data).

Many, many websites contain legal language that forbids automatic data collection/scraping. How can a business be built in such a case?

Perhaps OPs tool only scrapes a select few sites that don't prohibit scraping, but that seems like the exception, not the norm.

Re: Ask HN: Can web scraping be the basis of a viable business model?

#79

Post by melony »

elliottinvent wrote: post #70
apienx wrote: post #9

Google, a trillion dollar company, is essentially the world's largest web scraper. So...yes! You'll almost certainly find a way to monetize that. Monopolies, lobbying and protectionism got in the way of keeping the web truly machine readable. There's tremendous value in restoring some of it.

> Monopolies, lobbying and protectionism got in the way of keeping the web truly machine readable. Exactly and that ship has long since sailed. The good ship Web 3.0 (semantic web) launched in ‘99 and was a ghost ship until recently when it was boarded by crypto pirates now flying the web 3.0 flag. > There's tremendous value in restoring some of it. To this comment and OP, my startup is using web scraping to pre-popu…

Web3 (without the decimal point) these days usually refer to internet Ponzi schemes.
Post reply on HN