Yes if you point it at a well defined target market and create a solution.
Ask HN: Can web scraping be the basis of a viable business model?
41–50 of 114 posts
Re: Ask HN: Can web scraping be the basis of a viable business model?
#42Hey! Hopefully this comment comes across as helpful rather than hurtful. I'm also the founder of a developer tool, and it's hard to raise money for! (I know you said yours isn't strictly a developer tool, but you mentioned open source so I think it's fair to assume it falls in this category.) I think you're missing a step, which is where _you_ answer if web scraping can be a viable business model. You're attempting t…
You're not being hurtful. :)
I'll send you an email too, but because others are raising this:
Good point and it's a potential trap of tools built from developers for developers.
There's this fine line between finding it useful and being willing to pay for it. Being honest to my self: Do I find this useful ? Yes Makes my web scraping easier? Yes Would I pay for it? Maybe... but the problem is I've never paid for a scraping tool either.
But other people may pay for it. To get to that point though I'll probably need a few more hands on deck, who need salaries and for that you need external funds.
Re: Ask HN: Can web scraping be the basis of a viable business model?
#43Isn't one big issue that any website that has data worth scraping has ToS that disallow scraping? e.g. craigslist?
Would think this is a legal concern...
Re: Ask HN: Can web scraping be the basis of a viable business model?
#44I did so for years, scraping university press releases, obscure government data (like the Federal Register), jail/prison rosters, Reddit posts/comments/users etc. Companies like LexusNexus have been doing it for far longer than the digital world, offering it as a clipping service. If you can find a niche, it can be a regular subscription income stream.
Re: Ask HN: Can web scraping be the basis of a viable business model?
#45And to answer your question, because my other post was a question itself:
> after a certain point it does require some manual work from the customer
Once you gain traction, you can become a platform, intermediary between customers and engineers that will fine tune scraping to what the customer needs. This could either be some sort of "Solution Engineer" that the company hires, or open it up to outside developers that get paid per integration (either by you or by the customer, or both). There's a solution to every problem.
As far as the business itself, I think you could be on to something. Of course, ideas are cheap and it's the execution that counts, but here's how I'd think about it: with scraping, every website on the web has an API. Before, only 0.1% of websites had an API.
And certainly wouldn't hurt to change the "scraping" word – such an ugly word.
Re: Ask HN: Can web scraping be the basis of a viable business model?
#46Never mind business, a web-scraping command-line utility as comprehensive and easy to use as say curl would be something. I would even pay for that.
Re: Ask HN: Can web scraping be the basis of a viable business model?
#47Isn't one big issue that any website that has data worth scraping has ToS that disallow scraping? e.g. craigslist? Would think this is a legal concern...
Can breaking terms of service result in anything more than being banned from the site you are scraping?
Re: Ask HN: Can web scraping be the basis of a viable business model?
#48Isn't one big issue that any website that has data worth scraping has ToS that disallow scraping? e.g. craigslist? Would think this is a legal concern...
Also it's very easy to get blacklisted if you scrape too often or too quickly. You definitely have to be "polite" about it.
Re: Ask HN: Can web scraping be the basis of a viable business model?
#49Maybe only kinda sorta relevant, but I interned at RefME for a summer - basically a Zotero competitor. There was a lot of value (for users) in scraping web pages to autofill to author, title, date etc so it could generate the references, something I started to work on (but then they got bought out so not sure what ended up happening)
Any I could've imagining this being a paid pro feature down the line
Re: Ask HN: Can web scraping be the basis of a viable business model?
#50Hey! Hopefully this comment comes across as helpful rather than hurtful. I'm also the founder of a developer tool, and it's hard to raise money for! (I know you said yours isn't strictly a developer tool, but you mentioned open source so I think it's fair to assume it falls in this category.) I think you're missing a step, which is where _you_ answer if web scraping can be a viable business model. You're attempting t…
You're not being hurtful. :) I'll send you an email too, but because others are raising this: Good point and it's a potential trap of tools built from developers for developers. There's this fine line between finding it useful and being willing to pay for it. Being honest to my self: Do I find this useful ? Yes Makes my web scraping easier? Yes Would I pay for it? Maybe... but the problem is I've never paid for a scr…
Great! Looking forward to talking :)
If I were you, I'd do everything possible to answer your initial question (is web scraping a viable business) before bringing other people and their money into it. Everything gets much, much harder once other people's time and money is involved (both the process of convincing them, AND the potential process of realizing you were wrong).
There's lots of ways to build a business, so take my advice with a grain of salt. But especially if you're worried about it, I'd highly recommend focusing on getting to a few paying customers before involving VCs or employees. If you're here asking that question, you can be sure they will be too (minus the initial conviction you have).
(The one exception is a co-founder! It's lonely starting a company, so you should definitely consider trying to find a co-founder that you love working with.)