Ask HN: Can web scraping be the basis of a viable business model?

#1

Post by fforflo » Sat, Dec 18, 2021, 3:25 PM UTC

I'm a data engineer at heart, and I never did or enjoyed front-end work. Having said that I always was happy to code and evolve crawlers and web scrapers. Now I've taken some time off from work and gigs and I'm working on a side-project I've been hacking for some time.

Without getting into the details yet: it aims to make web data collection a little bit easier for non-devs. I'll soon have an MVP and will start pitching to investors: aiming for an open-source business model (after a few months of stealth development) and eventually a typical SaaS offering for extra functionality.

At this point I'm trying to consolidate and counter the steel-man counter-arguments I should expect from investors. The most obvious one: as one can imagine, the product it's not magic and, after a certain point it does require some manual work from the customer, hence this is an aspect I should prepare for.

I have done some preliminary analysis of the space of potential competitors (think import.io, Apify, Zyte/ScarpingHub, etc.) and described opportunities for differentiation. What I'm afraid of is getting sidetracked in a discussion of "um, this is web scraping and it's hard to make a business on top of it".

I understand that there's not much context now and one could easily say "well yeah, anything could be possible with a good team, product...", but I'm reaching out to the HN community to gather some considerations, mental models and pointers, I may not think of myself at this point.

Re: Ask HN: Can web scraping be the basis of a viable business model?

#2

Post by wcerfgba » Sat, Dec 18, 2021, 3:46 PM UTC

You've already identified some of your competitors, you should go in to more detail and try to answer:

* What features are common among my competitors?

* What features are unique?

* Who are target customers and users? Is there any overlap, or do some competitors target unique market segments?

This last question ties in to a discussion I was having with a friend recently. In B2B sales, your customers are businesses, but your users are people in those businesses with certain roles and responsibilities. Understanding the difference is key, because you will often need to develop your sales and marketing strategies based on the business/customer profile, but your UX will depend on the needs of the users within those businesses.

In my opinion you are more likely to be successful if you can get an initial foothold in a market by identifying a specific target of customers and users, solving their use case very well, developing a moat, and then growing out from that foothold to provide a wider set of options. Web scraping is just a tool. You need to find businesses who can gain value from scraping or from scraped data. Are there businesses who, for whatever reason, would not be able to adopt one of your competitors' products, or would find that adoption difficult? Maybe you could specialise in scraping a particular kind of data, or providing a full-stack solution for companies with limited in-house technical expertise (like some kind of consulting, you hop on a call with the client, they tell you what they want to scrape, and you set up a hosted solution which provides a SQL or Excel interface to the data).

In short, successful product development is all about understanding customer and user pain and needs. If you can find pains or needs which are a common theme for a particular demographic of companies and roles, you can work with those people to understand their problems and make a product which is very valuable to them.

Re: Ask HN: Can web scraping be the basis of a viable business model?

#3

Post by ethor » Sat, Dec 18, 2021, 4:47 PM UTC

In my opinion you should start with defining your value proposition and target market, i.e. how do I create value for customers, and who are my customers?

Re: Ask HN: Can web scraping be the basis of a viable business model?

#4

Post by test0account » Sat, Dec 18, 2021, 4:47 PM UTC

Is Google not just one massive web scraper?

Re: Ask HN: Can web scraping be the basis of a viable business model?

#5

Post by fforflo » Sat, Dec 18, 2021, 4:49 PM UTC

wcerfgba wrote: post #2

You've already identified some of your competitors, you should go in to more detail and try to answer: * What features are common among my competitors? * What features are unique? * Who are target customers and users? Is there any overlap, or do some competitors target unique market segments? This last question ties in to a discussion I was having with a friend recently. In B2B sales, your customers are businesses, b…

Thanks a lot for your comment.

You're exactly right on the tricky relationship between developers as internal ambassadors to businesses - customers. I think it somehow applies to almost any developer-facing tool.

Your recommendation to focus on a specific vertical at first makes sense too. Helps prioritising the backlog as well.

Re: Ask HN: Can web scraping be the basis of a viable business model?

#6

Post by sjg007 » Sat, Dec 18, 2021, 4:50 PM UTC

It is valuable. There's a lot of Robotic Process Automation, competitive analysis etc...

>the product it's not magic and, after a certain point it does require some manual work from the customer, hence this is an aspect I should prepare for.

Can you make it magic or maybe develop end to end solutions for your first "customers" using your product? That sounds like the schlep you need to do.

Sounds promising! Find yourself a customer or two!

If you really want to go the open source route, just focus on that and then see if people pick it up and use it. Then you'd offer the SaaS.

Re: Ask HN: Can web scraping be the basis of a viable business model?

#7

Post by itake » Sat, Dec 18, 2021, 4:50 PM UTC

You might want to look at web scraping for data scientists. I am trying to build a ML Model for NSFW text detection in multiple languages and I am not looking forward to scraping p*rn and youtube websites for comments.

Re: Ask HN: Can web scraping be the basis of a viable business model?

#8

Post by wombatpm » Sat, Dec 18, 2021, 4:52 PM UTC

If your potential customers are willing to pay to scrape data, why aren't they will to pay for the data from the source directly? Is it not available or is it considered exclusive or proprietary. I'm thinking about the lawsuits around deep linking and TicketMaster. Web scraping at scale is a never ending arms race because designs evolve or the host is actively trying to thwart you.

Re: Ask HN: Can web scraping be the basis of a viable business model?

#9

Post by apienx » Sat, Dec 18, 2021, 4:53 PM UTC

Google, a trillion dollar company, is essentially the world's largest web scraper. So...yes! You'll almost certainly find a way to monetize that.

Monopolies, lobbying and protectionism got in the way of keeping the web truly machine readable. There's tremendous value in restoring some of it.

Re: Ask HN: Can web scraping be the basis of a viable business model?

#10

Post by fforflo » Sat, Dec 18, 2021, 4:57 PM UTC

apienx wrote: post #9

Google, a trillion dollar company, is essentially the world's largest web scraper. So...yes! You'll almost certainly find a way to monetize that. Monopolies, lobbying and protectionism got in the way of keeping the web truly machine readable. There's tremendous value in restoring some of it.

> keeping the web truly machine readable. There's tremendous value in restoring some of it.

:-)