Ask HN: Can web scraping be the basis of a viable business model? • Hacker News Board

Re: Ask HN: Can web scraping be the basis of a viable business model?

#51

Post by bdcravens » Sat, Dec 18, 2021, 7:19 PM UTC

DonHopkins wrote: post #39

bdcravens wrote: post #36

Yes, but scraping is a small part of the overall puzzle. As developers, we overestimate how valuable tools are (as opposed to solutions). I think the better opportunity is not to be another scraping-as-a-service provider, but to niche down to a solution that uses your scraping technology.

If you start yet another scraping-as-a-service provider, you're attempting to provide a paid service for people who just want to steal content for free. Not gonna work.

There are legit use cases for browser automation, but a ton of bad actors too. I think your point illuminates something that many fail to fully consider: What are all the ways my service/software can be abused?

Re: Ask HN: Can web scraping be the basis of a viable business model?

#52

Post by lubujackson » Sat, Dec 18, 2021, 7:19 PM UTC

xedarius wrote: post #17

I went for an interview once at a hedge fund. There were a surprising amount of questions about web scraping. I very much got the feeling it was an active and ongoing problem. So yes I do think there’s a business in there.

AFAIK Plaid does a fair bit of old school scraping behind the scenes - a lot of the "nice new web" is built on the backs of old kludgy websites doing the same scraping things people were doing 20 years ago.

Finance/banks are especially... inconsistent, to say the least.

Re: Ask HN: Can web scraping be the basis of a viable business model?

#53

Post by bjourne » Sat, Dec 18, 2021, 7:34 PM UTC

Web scraping is a legal gray area in many or most jurisdictions. In some jurisdictions, depending on the tos of the web site itself, scraping it might be illegal. In others republishing the scraped information in any form might be illegal. In others still you might not be allowed to use the scraped data for any commercial purpose.

"But what about Google?" Google is worth 100 billion dollars and can play by completely different rules than scraping startups.

Re: Ask HN: Can web scraping be the basis of a viable business model?

#54

Post by fifilura » Sat, Dec 18, 2021, 7:42 PM UTC

shtopointo wrote: post #43

Isn't one big issue that any website that has data worth scraping has ToS that disallow scraping? e.g. craigslist? Would think this is a legal concern...

What about government/municipality data?

We paid for that once, surely it shouldn't be necessary to charge for it again?

Re: Ask HN: Can web scraping be the basis of a viable business model?

#55

Post by tommoor » Sat, Dec 18, 2021, 7:43 PM UTC

ClearBit and Plaid haven't been mentioned yet – both examples of multi-million/billion dollar business built on the back of scraping.

The more specialized you can get the better chances of success imo. Generic web scrapers are dime-a-dozen

Re: Ask HN: Can web scraping be the basis of a viable business model?

#56

Post by dradicchi » Sat, Dec 18, 2021, 7:49 PM UTC

Please, see https://www.datallog.com/

They are a profitable startup and have a SDK solution for scale web scraping bots on many business fields.

I'm a advisor/investor in Datallog. You can connect with the founder in https://www.linkedin.com/in/joelder-maragno-arcaro/

Re: Ask HN: Can web scraping be the basis of a viable business model?

#57

Post by nomdep » Sat, Dec 18, 2021, 7:52 PM UTC

Instead of competing head-on with ScrapingHub/etc, you could ask yourself “why companies doing X pay them” and sell a specific product for that niche.

Re: Ask HN: Can web scraping be the basis of a viable business model?

#58

Post by kccqzy » Sat, Dec 18, 2021, 7:53 PM UTC

Of course. See Yodlee and Plaid.

Re: Ask HN: Can web scraping be the basis of a viable business model?

#59

Post by tixocloud » Sat, Dec 18, 2021, 7:54 PM UTC

There are many data providers serving the financial services/capital markets sector that provide everything from raw data feeds to insights based off of web-scraped data. What will be important for you is to identify a good niche, understand which part of the data value chain your customer needs, and deliver the data in a way that fits their workflow.

The value is in the decisions that can be made based on the data being sold rather than the method at which you extract it from. If you focus on the value of the data and who needs it, you’ll likely find a viable business model.

Re: Ask HN: Can web scraping be the basis of a viable business model?

#60

Post by nojs » Sat, Dec 18, 2021, 8:04 PM UTC

I cofounded a company based on scraping academic publications. We ended up getting a lot of traffic (millions of pageviews per month) because we had good SEO, but it’s not necessarily a defensible business model by itself.

You’ll likely have to do more manual data cleaning than you expect, and get some amount of pushback from the sources you’re crawling (depending how commercially valuable the data is).