Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Can web scraping be the basis of a viable business model?

news.ycombinator.com

Post reply on HN
Print view View original

81–90 of 114 posts

Re: Ask HN: Can web scraping be the basis of a viable business model?

#81

Post by nlh »

adinosaur123 wrote: post #78

I don't wish to hijack this thread, but I've been pondering a similar question. I've been working on a product that requires a very large amount of data that, as far as I can tell, can only be gathered by scraping (real estate data - even data vendors like estated.com don't have stuff like sales data). Many, many websites contain legal language that forbids automatic data collection/scraping. How can a business be bu…

Read up on LinkedIn vs. HiQ. As long as that ruling holds (and it might not), the tl;dr is: If it's on the open web, you can scrape it. You might be violating some Terms of Service (that you never agreed to), but you're not violating (US) law.

If it's NOT on the public web - e.g. it's behind a login, then you can be sued, as you'll have had to explicitly agree to Terms of Service during your account creation and you'll then be in explicit violation of that ToS.

Re: Ask HN: Can web scraping be the basis of a viable business model?

#82

Post by GhettoComputers »

adinosaur123 wrote: post #78

I don't wish to hijack this thread, but I've been pondering a similar question. I've been working on a product that requires a very large amount of data that, as far as I can tell, can only be gathered by scraping (real estate data - even data vendors like estated.com don't have stuff like sales data). Many, many websites contain legal language that forbids automatic data collection/scraping. How can a business be bu…

Do it manually. I wonder what automatically means legally or scraping. It’s pretty hard to enforce those requirements, because I assume it’s being broken by search providers.

If it’s a derivative work like copilot, I wonder if there’s a legal case to say you can’t do it. I assume you’re doing something like an RSS feed for pricing suggestions with commissions? I just looked this up and it seems like it’s legal to do so but their information is copyrighted. https://law.stackexchange.com/questions/15556/is-scraping-re...

Re: Ask HN: Can web scraping be the basis of a viable business model?

#84

Post by pb7 »

bjourne wrote: post #53

Web scraping is a legal gray area in many or most jurisdictions. In some jurisdictions, depending on the tos of the web site itself, scraping it might be illegal. In others republishing the scraped information in any form might be illegal. In others still you might not be allowed to use the scraped data for any commercial purpose. "But what about Google?" Google is worth 100 billion dollars and can play by completely…

>Google is worth 100 billion dollars

$100B from $2T*. :-)

Re: Ask HN: Can web scraping be the basis of a viable business model?

#85

Post by NavinF »

thruway516 wrote: post #46

Never mind business, a web-scraping command-line utility as comprehensive and easy to use as say curl would be something. I would even pay for that.

Are you looking for output that’s more structured than curl’s or for a way to run curl on silly sites that block you?

Re: Ask HN: Can web scraping be the basis of a viable business model?

#86

Post by NavinF »

suifbwish wrote: post #24

I’m curious how you deal with JavaScript that will load other pages including other JavaScript documents that cannot be loaded until the first set of JavaScript is executed. I’ve played with the chromium web driver a few times but it seems to be tricky to implement in a completely headless environment.

For most use cases, headless chromium will work out of the box. For the rest, set $DISPLAY to a virtual framebuffer like Xvfb

Re: Ask HN: Can web scraping be the basis of a viable business model?

#87

Post by notduncansmith »

It will help to recognize the key use-cases and provide lots of support out of the box like pre-built scrapers for price comparison, social media mentions (or other analysis), whatever you find that people will pay for.

Make sure your pricing is clear so the profit calculation for the customer is transparent.

You then have a tangible product line you can pitch to investors regardless of whether they can appreciate the more abstract solution/platform.

Re: Ask HN: Can web scraping be the basis of a viable business model?

#88

Post by asdfasgasdgasdg »

apienx wrote: post #9

Google, a trillion dollar company, is essentially the world's largest web scraper. So...yes! You'll almost certainly find a way to monetize that. Monopolies, lobbying and protectionism got in the way of keeping the web truly machine readable. There's tremendous value in restoring some of it.

> Google, a trillion dollar company, is essentially the world's largest web scraper.

Even just considering the parts of Google that it takes to bring you the N blue links part of the Google SERP, the web scraper is probably the least interesting and significant piece of technology in the stack. It's beyond reductive to say that Google is in essence a large web scraper, or a web scraper of any kind. It is like saying that a person is, in essence, the world's largest mouth.

I think it's incredibly difficult to build a profitable business in this space. The number of customers who a) need to scrape the web b) aren't sophisticated enough to do it themselves and c) are big enough to make $$$ from are small. The important bit is always processing the web pages for whatever content is salient to the given customer. Which means that you need to deliver the web pages to them. So effectively your business is providing nothing more than URL lists and potentially some additional metadata compared to what the customer would get if they fetched the pages themselves. There are definitely some other complexities you could resolve, but it's hard to imagine that those benefits would be enough to build a business on.

Re: Ask HN: Can web scraping be the basis of a viable business model?

#89

Post by lifefeed »

apienx wrote: post #9

Google, a trillion dollar company, is essentially the world's largest web scraper. So...yes! You'll almost certainly find a way to monetize that. Monopolies, lobbying and protectionism got in the way of keeping the web truly machine readable. There's tremendous value in restoring some of it.

And there's another entire ecosystem around scraping Google's results and wrapping it in an API.

Just scraping upon scraping.

Re: Ask HN: Can web scraping be the basis of a viable business model?

#90

Post by proszkinasenne2 »

Sure, it can be! Also, as some people have already pointed out, this is often a gray area where people go beyond violating ToS. Some good examples are privacy violations (scraping personal data), credentials stuffing etc.

Recently, there is a boom of "anti-bot" services. These are essentially SaaS businesses that "protect" websites from being scraped by automated software. As you onboard the first customer who wants to extract data from a bot-protected website, you are going to run into an unlimited waterfall of stupid troubles. Your bots will be blocked, will consume excessive amount of data, kill your CPU/GPU performance.

I have shared some highlights on how to bypass these recently on HN [1], but it is sadly only the tip of the iceberg. On the other hand, since the post has been featured on HN I have been reached by more than 50 companies and individuals whose business operating model is based solely on data extraction/automated scraping. These are (in my opinion) successful companies, and two out of these are part of YC.

[1] https://news.ycombinator.com/item?id=29060272

Post reply on HN