If you can apply the scraping to a use case and sell that use case you have a better chance at a viable business model. Examples come to my mind is; builtwith (scrape sites and publish the list of technologies they use), ahrefs (scrape sites and find outgoing/incoming links) etc.
Ask HN: Can web scraping be the basis of a viable business model?
11–20 of 114 posts
Re: Ask HN: Can web scraping be the basis of a viable business model?
#12If you’re scraping someone else’s data, do you know what they copyright status is? Have you made deals with the original sources that permits you to use their data? How will you deal with lawsuits and the constant blocking of your scrapers?
Re: Ask HN: Can web scraping be the basis of a viable business model?
#13If your potential customers are willing to pay to scrape data, why aren't they will to pay for the data from the source directly? Is it not available or is it considered exclusive or proprietary. I'm thinking about the lawsuits around deep linking and TicketMaster. Web scraping at scale is a never ending arms race because designs evolve or the host is actively trying to thwart you.
I like these two questions! They can crystallise things.
Wasn't aware of the Ticketmaster v. Tickets.com case. Will have to print it along the SCOTUS LinkedIn v. hiQ ruling.
Re: Ask HN: Can web scraping be the basis of a viable business model?
#14From what i know thats how skyscanner started, not sure if ita very popular in US, but it is in Europe. Now they're paying/having deqls for data, but they started with scraping the hell out of airline sites.
Re: Ask HN: Can web scraping be the basis of a viable business model?
#15Yes.
Re: Ask HN: Can web scraping be the basis of a viable business model?
#16It is for openexchangerates.org
Makes the founder a hefty sum.
Re: Ask HN: Can web scraping be the basis of a viable business model?
#17I went for an interview once at a hedge fund. There were a surprising amount of questions about web scraping. I very much got the feeling it was an active and ongoing problem. So yes I do think there’s a business in there.
Re: Ask HN: Can web scraping be the basis of a viable business model?
#18An online law editorial I used to work at around 10 years ago basically scraped free available content from different sources around the web, repackaged it, and sold it online via subscription model. They're still around.
Re: Ask HN: Can web scraping be the basis of a viable business model?
#19I co-founded https://packetstream.io
There are a handful of companies doing very well with models similar to what you’re describing. I can’t mention specific customers, but I see some of them doing very large scraping volume through our network.
It’s an industry where having a good product is more important than the amount you’re spending on marketing. If developers are happy with your product they’ll take it with them to future companies/projects and share it with colleagues.
It can be a cat-and-mouse development cycle where the sites you target break your functionality and you’ll have customers that will want fixes to be implement ASAP because they rely on your tools to make money.
I don’t know what you’re building exactly, but keep in mind there’s a good chance that you’ll need to commit to long-term, continuous, rapid development cycles if you want to retain customers.
Best of luck!
Re: Ask HN: Can web scraping be the basis of a viable business model?
#20Yes, although I would encourage you to think about something higher on the value chain than raw data feeds. Those exist and have become an increasingly difficult market to compete in. You can buy a custom feed for like $250/mo.
Instead, think about what people want to do with the data. For example, if you are going to scrape diamond prices, don’t try to sell that feed. Set up a website with a UI so people can research diamond prices, and get alerts when specific thresholds are met or items come in stock. Monetize with ads.