Ask HN: How do you earn a living from scraping in 2018?

#1

Post by hoerzu » Tue, Jul 24, 2018, 8:14 PM UTC

I have been scraping housing markets. LinkedIn paired with hunter.io for creating mailing lists of companies. Recently I started MITM and Burp to intercept Apps to get direct api access. I created a docker container using different tor endpoints for different ips. I even used ssh proxying to create my own ip network.

To bypass recaptcha I build a container with chrome + puppeteer with anticaptcha preinstalled and pre logged in.

Yet I wonder if it is still worth it. Kinda looking for a mentor or inspiration.

Been a long time HN reader and I wonder what you think? Why scraping is still worth it in 2018?

Re: Ask HN: How do you earn a living from scraping in 2018?

#2

Post by _8usx » Tue, Jul 24, 2018, 10:33 PM UTC

> Recently I started MITM and Burp to intercept Apps to get direct api access.

Kind of a party foul in my book. .

Re: Ask HN: How do you earn a living from scraping in 2018?

#3

Post by paulcole » Tue, Jul 24, 2018, 10:35 PM UTC

You don't earn a living from scraping, you earn a living by solving a problem and providing value.

What problems are your projects solving? Are people were paying (in either effort or money) to solve these problems currently?

Re: Ask HN: How do you earn a living from scraping in 2018?

#4

Post by shurelock » Tue, Jul 24, 2018, 11:13 PM UTC

I think his question really is where can I make money from sccrapping in 2018.

Re: Ask HN: How do you earn a living from scraping in 2018?

#5

Post by gesman » Tue, Jul 24, 2018, 11:59 PM UTC

Build something for others worth scraping

Re: Ask HN: How do you earn a living from scraping in 2018?

#6

Post by verismal » Wed, Jul 25, 2018, 12:11 AM UTC

Microsoft rewards. Live off Amazon gift cards

Re: Ask HN: How do you earn a living from scraping in 2018?

#7

Post by hoerzu » Wed, Jul 25, 2018, 12:31 AM UTC

_8usx wrote: post #2

> Recently I started MITM and Burp to intercept Apps to get direct api access. Kind of a party foul in my book. .

I understand what you mean. Still causing less traffic ;) and I'm intercepting my own devices.

Re: Ask HN: How do you earn a living from scraping in 2018?

#8

Post by dchuk » Wed, Jul 25, 2018, 12:31 AM UTC

Are you just tinkering with all of those ideas above? Or where are you finding these ideas?

Like others have said, start with the problem, rather than coming up with solutions first.

Do you have any of the things you listed on github or anything? Curious to checkout a few of them.

Re: Ask HN: How do you earn a living from scraping in 2018?

#9

Post by hoerzu » Wed, Jul 25, 2018, 12:32 AM UTC

paulcole wrote: post #3

You don't earn a living from scraping, you earn a living by solving a problem and providing value. What problems are your projects solving? Are people were paying (in either effort or money) to solve these problems currently?

I give a price prediction for the rent of a house. So the buyer can immediately calculate ROI.

Re: Ask HN: How do you earn a living from scraping in 2018?

#10

Post by hoerzu » Wed, Jul 25, 2018, 12:33 AM UTC

shurelock wrote: post #4

I think his question really is where can I make money from sccrapping in 2018.

Upwork looks very cheapy imho