Ask HN: Can web scraping be the basis of a viable business model?

Ultimately people want information from scraping, because to a lot of businesses, the information is what's valuable.

Consider, apart from a tool for general purpose scraping, what information a specialized scraper might obtain for a valuable but underserved industry that can profit from the data.

I have a buddy that scrapes data specifically for the tanker/shipping industry, for example.

General purpose scraping will involve a lot of competition and a bit of an arms race. Niche scraping lets you fly a little below the radar.

Are you falling in love with the code?

I think you'll have a lot more luck finding 2-3 initial customers before you try to raise money. It's always easier to explain what your product does, and who the target market is, in terms of actual customers; instead of hypotheticals.

Remember, the goal is to build a business. If you fall in love with the code, it's too easy to build something you enjoy working on, but has limited commercial value.

Scraping isn't the problem, it's getting access to the pages to scrape at high-enough volume, at low-enough cost.

Hey! Hopefully this comment comes across as helpful rather than hurtful. I'm also the founder of a developer tool, and it's hard to raise money for! (I know you said yours isn't strictly a developer tool, but you mentioned open source so I think it's fair to assume it falls in this category.)

I think you're missing a step, which is where _you_ answer if web scraping can be a viable business model. You're attempting to convince VCs with logic (and a few assumptions), but there's an easier (or harder) way to do it... convince them by making money.

Most VCs aren't ideologues, and don't have an opinion about business models. They will be convinced if you simply show them you're making money. It's not their job to decide if an idea can make money or not; that's your job as a founder.

I applied to YC twice. The first time we spent the 10 minutes talking about if it could make money or not, and never got anywhere good. We got rejected. The second time we were making money, the conversation was smoother, and we got in. It's so much easier to be able to replace "I believe" with "our customers believe". It changes the conversation completely. You don't need to be making billions of dollars; just enough to show that people want what you're making!

tl;dr You're trying to convince VCs when you need to be convincing customers!

(For the record, a lot of what I said here is very money-driven and that's not how I build my company. However, in the context of VCs, which are purely financially driven, it's how you should be thinking about it.)

Good luck, and let me know if I can help! My email is in my profile if you want to talk!

If you start yet another scraping-as-a-service provider, you're attempting to provide a paid service for people who just want to steal content for free. Not gonna work.