How to Scrape Zillow with Python and Scrapy

#1

Post by rl1987 » Sat, Jul 02, 2022, 4:25 PM UTC

How to Scrape Zillow with Python and Scrapy
trickster.dev

Re: How to Scrape Zillow with Python and Scrapy

#2

Post by gravypod » Sun, Jul 03, 2022, 4:01 AM UTC

Is there any cleaner data source for home listings? I wish I could just setup a system to automatically filter listings by criteria that are important to me.

Re: How to Scrape Zillow with Python and Scrapy

#3

Post by xfour » Sun, Jul 03, 2022, 4:31 AM UTC

gravypod wrote: post #2

Is there any cleaner data source for home listings? I wish I could just setup a system to automatically filter listings by criteria that are important to me.

Unfortunately no, the underlying systems companies such as Zillow use require basically joining to access to the local Multiple Listing Services (MLS) of a certain market. All of these have different rules of how data can be used.

Re: How to Scrape Zillow with Python and Scrapy

#4

Post by cute_boi » Sun, Jul 03, 2022, 12:46 PM UTC

I tried to scrape zillow in past (2019) and it was giving me captcha headache. Didn't author faced similar issues?

Re: How to Scrape Zillow with Python and Scrapy

#5

Post by slezyr » Sun, Jul 03, 2022, 3:47 PM UTC

cute_boi wrote: post #4

I tried to scrape zillow in past (2019) and it was giving me captcha headache. Didn't author faced similar issues?

> As of 2022, Zillow uses PerimeterX for scraping countermeasures. This can be bypassed by using ISP proxies from BrightData.

Why ask without reading the article?

Re: How to Scrape Zillow with Python and Scrapy

#6

Post by throwaway193948 » Mon, Jul 04, 2022, 6:57 AM UTC

what’s the point of scraping a scraper?

Re: How to Scrape Zillow with Python and Scrapy

#7

Post by cute_boi » Mon, Jul 04, 2022, 11:29 AM UTC

slezyr wrote: post #5

cute_boi wrote: post #4

I tried to scrape zillow in past (2019) and it was giving me captcha headache. Didn't author faced similar issues?

> As of 2022, Zillow uses PerimeterX for scraping countermeasures. This can be bypassed by using ISP proxies from BrightData. Why ask without reading the article?

Well, I used luminati( Currently BrightData). And, it was still giving me captcha. In fact, it was giving me more captcha in BrightData. So, I am not sure.

Re: How to Scrape Zillow with Python and Scrapy

#8

Post by Tarq0n » Mon, Jul 04, 2022, 3:47 PM UTC

throwaway193948 wrote: post #6

what’s the point of scraping a scraper?

You want a machine-readable dataset rather than a website, and the aggregator has already done some of the work for you.

Scraping all of Zillow's sources instead would explode the complexity of the project.

Re: How to Scrape Zillow with Python and Scrapy

#9

Post by Kimcha » Mon, Jul 04, 2022, 4:56 PM UTC

cute_boi wrote: post #4

I tried to scrape zillow in past (2019) and it was giving me captcha headache. Didn't author faced similar issues?

There are services that use humans to solve captchas for a very reasonable price, such as 2Captcha.

Some of the protections are better than others. For example, you may need to use a real browser with Java script support instead of just HTTP requests.

Browserless works well if you enable the stealth functions.