Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Can web scraping be the basis of a viable business model?

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 114 posts

Re: Ask HN: Can web scraping be the basis of a viable business model?

#61

Post by deanebarker »

There was a company in my city that did something like this. They didn't survive.

They crawled the data, but also had a services component to do something with the data. Like, they had contracts with pharma companies to search for indications that a page was selling counterfeit drugs.

I'm not sure of the exact details of why they didn't make it.

Also, I'm thinking about Recorded Future (https://www.recordedfuture.com). They do something like this -- again, the mechanics of scraping, and a services component for analysis.

Re: Ask HN: Can web scraping be the basis of a viable business model?

#63

Post by hooande »

Web scraping can be a viable business. It depends on what you're scraping and who your customers are.

Are there a couple thousand people who would pay for a SaaS offering? Then it's a business. The real goal would be identifying a hair on fire problem that you are in a unique position to solve. That's always the problem, and it has nothing to do with web scraping in particular.

Re: Ask HN: Can web scraping be the basis of a viable business model?

#64

Post by tempestn »

bjourne wrote: post #53

Web scraping is a legal gray area in many or most jurisdictions. In some jurisdictions, depending on the tos of the web site itself, scraping it might be illegal. In others republishing the scraped information in any form might be illegal. In others still you might not be allowed to use the scraped data for any commercial purpose. "But what about Google?" Google is worth 100 billion dollars and can play by completely…

Yeah, I was going to say something like this. And one step better than Google, but still challenging, there are also a lot of sites out there that prohibit scraping but make an exception for "general purpose search engines". So if you're providing a specialized scraper that's used for specific purposes, you'd likely run into problems there too.

As well as the actual legal challenges, you'll also have the perception thereof, which could make businesses wary of relying on your services.

Re: Ask HN: Can web scraping be the basis of a viable business model?

#65

Post by dbtc »

apienx wrote: post #9

Google, a trillion dollar company, is essentially the world's largest web scraper. So...yes! You'll almost certainly find a way to monetize that. Monopolies, lobbying and protectionism got in the way of keeping the web truly machine readable. There's tremendous value in restoring some of it.

Web scraping is just one way to get the data to facilitate the sale and service of ads.

Re: Ask HN: Can web scraping be the basis of a viable business model?

#66

Post by stef25 »

itake wrote: post #7

You might want to look at web scraping for data scientists. I am trying to build a ML Model for NSFW text detection in multiple languages and I am not looking forward to scraping p*rn and youtube websites for comments.

You probably need NSFW text which is its own niche and should be straight forward to collect?

Re: Ask HN: Can web scraping be the basis of a viable business model?

#67

Post by stef25 »

edmundsauto wrote: post #20

Yes, although I would encourage you to think about something higher on the value chain than raw data feeds. Those exist and have become an increasingly difficult market to compete in. You can buy a custom feed for like $250/mo. Instead, think about what people want to do with the data. For example, if you are going to scrape diamond prices, don’t try to sell that feed. Set up a website with a UI so people can researc…

Like camelcamelcamel.com I guess?

Re: Ask HN: Can web scraping be the basis of a viable business model?

#68

Post by ChikkaChiChi »

Scraping services on their own are a viable business product, but the power to assign metadata and contextualization is where the unicorn lies.

The service itself will always be in flux because of how freeform hypertext is as a schema. So many other comments here reflect that better than I could.

The fact is that any chunk of data you're handing over to clients still needs to be handled by their team and in my experience reality often falls short of expectations. If you can somehow deliver them something cleaner (or even something that can help them reach conclusions faster), then you have a product with a high value prop.

Re: Ask HN: Can web scraping be the basis of a viable business model?

#69

Post by stef25 »

bjourne wrote: post #53

Web scraping is a legal gray area in many or most jurisdictions. In some jurisdictions, depending on the tos of the web site itself, scraping it might be illegal. In others republishing the scraped information in any form might be illegal. In others still you might not be allowed to use the scraped data for any commercial purpose. "But what about Google?" Google is worth 100 billion dollars and can play by completely…

Google also plays nice. I manage a few sites that get hammered by scrapers to the extent that it causes big spikes in CPU, something that doesn't happen with G.

Re: Ask HN: Can web scraping be the basis of a viable business model?

#70

Post by elliottinvent »

apienx wrote: post #9

Google, a trillion dollar company, is essentially the world's largest web scraper. So...yes! You'll almost certainly find a way to monetize that. Monopolies, lobbying and protectionism got in the way of keeping the web truly machine readable. There's tremendous value in restoring some of it.

> Monopolies, lobbying and protectionism got in the way of keeping the web truly machine readable.

Exactly and that ship has long since sailed. The good ship Web 3.0 (semantic web) launched in ‘99 and was a ghost ship until recently when it was boarded by crypto pirates now flying the web 3.0 flag.

> There's tremendous value in restoring some of it.

To this comment and OP, my startup is using web scraping to pre-populate machine-readable data for a DNS-based protocol called NUM [0]. So as others have said, whilst web scraping itself may be difficult to build into a viable business, it can be a key component of a viable business. Email in my profile if you want to discuss.

0. https://num.uk/blog/we-crawled-5m-uk-websites-and-published-...

Post reply on HN