Live data from Hacker News

Finding the perfect house using open data

dealloc.me

41–50 of 94 posts

Re: Finding the perfect house using open data

#41

Earlier quoted context omitted.

I know it's anecdotal, but most realtors I have met actively dislike Trulia/Zillow/etc

That's right, because they generally have their own proprietary source of data ( http://en.wikipedia.org/wiki/Multiple_listing_service ), which is a big part of the appeal of working with a realtor -- they learn of new listings faster than you can by relying on public, "open source" data. And realtors aren't stupid; they noticed what happened when other professions that were built on closed data opened their database…

I'm married to a Realtor, so I've had plenty of direct access to the local MLS system, and also I've overheard many conversations between my wife and her clients about Zillow.

From what I've seen and heard, Zillow's data isn't just delayed by a week; it can be years out of date, and grossly inaccurate as well. Both buyers and sellers are very mislead about property values based on Zillow because they see the last sale price and Zillow's estimate of the current value. Zillow has no idea about the local market conditions though, so it can over or under estimate the current value by a huge percentage.

Zillow also uses property tax valuations. The towns in and around my area have gone through a couple of re-evaluations in the past five years, and one thing I've learned is that the local governments value houses quite a bit below market value, for political reasons: owners can appeal the valuation, but only if the valuation is some percentage over market value. So instead they consistently value the houses below market value, and charge a higher tax rate to keep the revenue the same. Most people think they're getting away with something because the valuation is lower than they believe their house is worth, and the handful of people who realize what's going on can't do anything because they don't meet the qualifications for appealing.

Anyway, Zillow uses those low valuations as part of their estimate, so buyers coming into the area who are using Zillow instead of a Realtor have no idea how much houses are actually worth around here, and they waste everyone's time arguing about prices and by making low-ball offers that have no chance of being accepted. Realtors hate having their time wasted, because they work on pure commission. So they hate Zillow.

As far as Realtors wanting to keep MLS data to themselves so they don't go the way of travel agents, I think it's more complex than that. For one thing, it's the MLS people that are keeping the data secret, not the Realtors. The Realtors have to pay for access to MLS, which would stop if the MLS data were public. That could put a lot of money back into Realtor's pockets. This is significant, because it's up-front money: the Realtor has to pay for MLS whether they have any closings and earn any commissions or not. It's a fixed cost of business, which hurts when you don't have any guaranteed income.

Also, Realtors bring a lot more value to the game than travel agents did, at least since the early days of the travel industry. Realtors aren't just facilitating searching the MLS; they're managing the entire multi-month process of buying or selling a home. On the buying side that includes locating suitable areas, managing access to properties for viewing, advising on property value and market conditions, negotiating the offer, helping with inspections, helping to navigate the pitfalls between attorney review and closing (often including helping with a mortgage), and being there with you at the closing. On the selling side there's most of that, plus getting the house prepared for sale, taking good photographs and writing the advertising, holding open houses including the critical Realtor-only open house that shows all of the other Realtors (and, by proxy, all of their clients) your home, and helping you to deal with inspection issues that come up and your final move-out.

It's a harsh business to be in, with a huge amount of work for a paycheck that may or may not ever come, and opportunity costs from clients you can't work with when you're tied up managing the deals that your existing clients are working through. Making the MLS data public wouldn't change any of that; instead it would mislead buyers and sellers into thinking they can do it on their own, with no idea about the quagmire they're walking into.

Let me put it another way: if you're sued, or put on trial for a crime, you have every right to represent yourself in court. But that's almost always a really bad idea because you're not a professional lawyer, and when you're in court you want someone who does this kind of thing for a living, instead of someone (you) who's never/rarely done it before.

Re: Finding the perfect house using open data

#42
post #37
post #6

Great, way to go! Reminds me of Bill Gates book "The Road Ahead" (1st ed, 1995, p. 158) - it's about his vision of a next-gen network that would replace the middle-man beside other innovative things: [...] most market places are very inefficient. For instance, if you are trying to find a doctor, lawyer, accountant, or similar professional, or are buying a house, information is incomplete and comparisons are difficult…

Amazing that the book was written by a man who failed to communicate that (rather obvious) idea to the employees of the multi-billion dollar company he owned and directly managed.

He had the financial incentive to not make that happen, namely to lock people onto IE/Windows and direct them out of AOL and into a 1990's MSN walled garden.

If anything, Gates the futurist vs Gates the businessperson shows us that, regardless of what the pro-lassiez faire types say, most often corporations left unfettered don't product the best results, but only results that directly benefit themselves and that often translates into things like vendor lock-in, poor software, poor support, lack of progress.

I mean, does Gates even question why the market for the things he complains about are so inefficient? Its because its full of guys like himself.

Re: Finding the perfect house using open data

#43

Earlier quoted context omitted.

> The current state of affairs appears to be that it does eventually flow out to them, but with a significant (1 week+) delay I don't think that's true, at least not for Redfin. According to them: "Homes on the MLS show up on Redfin in 30 minutes or less." - http://forums.redfin.com/t5/Inland-Empire/How-quickly-does-M... "Local brokerage sites like Redfin show homes for sale immediately, as opposed to the seven to ni…

As I described in my other comment, Redfin is a completely different beast than Zillow and Trulia, because they are an actual real estate brokerage. That gives them direct MLS access rather than having to scrape as the other sites do. [I have no affiliation either, but I used Redfin to find my current house and I think they're great.]

Ok, I guess, but they're different from a normal brokerage in that you can access their database without signing an exclusivity agreement, meeting anyone in person, or talking to anyone at all. They're as publicly accessible as Zillow or Trulia, and from a user experience standpoint they're more like Zillow than they are like your local Coldwell Banker.

I get what you're saying about data accessibility, it seems like Zillow just found a way to hack the restrictions (by becoming a broker).

Re: Finding the perfect house using open data

#44
post #38

Earlier quoted context omitted.

So basically, you're saying the real estate agents are just the next casualty from the collateral damage that is the elimination of friction in society, and they are doing what they can to remain as middlemen.

Zillow has two sources of data: quality data volunteered by real estate agents , and crap wrong data scraped from everywhere else. MLSes, for all their faults, have rather strict rules about accuracy of their data feed. The best listings on Zillow already sold, months ago.

Zillow and Trulia aren't scraping.

The big hole in Zillow and Trulia is listings under contract. A listing goes "pending" (under contract), the MLS knows about it right away. Which is why people see Redfin updated quickly -- because as said elsewhere, Redfin displays data from the MLS.

Sites like Trulia and Zillow also get data from MLS, sometimes directly, usually indirectly thru other data access deals. And they also purchase access to a lot of public record databases, things like that.

So when a house goes pending, these sites don't see that until it closes and becomes a public record, unless the agent themselves updates the listing on the site.

Most this is explained elsewhere in this discussion, I just wanted to make the point, though, that scraping isn't part of the process. I'm sure it was early on, but those days are long gone.

Re: Finding the perfect house using open data

#45
When I was looking for a place to live, I made my own program to scrape the MLS, and did similar things to this. My rectangles were hand drawn, not scraped from open data though, and were more based on the neighborhoods that had houses I knew I didn't like.

Re: Finding the perfect house using open data

#46
post #23

Earlier quoted context omitted.

I know it's anecdotal, but most realtors I have met actively dislike Trulia/Zillow/etc

For those not in the states nor in the trade, could you elaborate on why realtors dislike Trulia and the likes ?

Because a realtor thinks they own the data. So they resent that Zillow and Trulia are using their data to generate leads, which they then sell back to the agent.

Whether it's reasonable to act like they own the data is a different question.

Re: Finding the perfect house using open data

#47
This is awesome. I love analysis like this.

Funny enough, something very similar to this was the impetus for me starting Estately. Early on, we had a bunch of wild spatial queries that would let you even filter by distance to specific transit line stops: http://blog.estately.com/wp-content/uploads/2008/06/near_tra...

When we ran stats on what searches people were actually doing, we found that very few consumers actually used these kinds of filters. We found we had a small and passionate audience for specific geo features and a much larger audience for "great site to search for and learn about homes." It took me way to long to come to terms with this. We still have walkscore search and distance from a neighborhood, school or address search even though they are very little used.

Why wasn't it popular? My guess is it's 80% because the market was small and 20% because we didn't present them right.

The thing about real estate - especially in the current market - is there just are few enough homes for sale that meet your price / proximity to work / school / bedrooms criteria that most people want to just glance at all of them and rule them in or out. Additionally, most people start with a bunch of "rules," then they bend one, change their mind on another and finally compromise on the last one.

The typical home search starts so clean, but dissolves over time as you tour houses, learn more about what you do and don't like and find something that works.

Re: Finding the perfect house using open data

#48

Earlier quoted context omitted.

Redfin is an actual real estate brokerage, and are therefore real members of the MLS, with access to the data that provides. The others are simply scraping via publicly available data. This is a major difference.

Close, except Estately gets access to MLS data too. (Founder & CEO here)

Do you guys have an API? I have a couple cool ideas for apps bit don't know where to get data?

Re: Finding the perfect house using open data

#49

Might be worth combining with walkscore data? http://www.walkscore.com/professional/research.php Not sure how freely they distribute this.

walkscore can already map out all "within walking distance of grocery + rail" locations in Portland (and elsewhere). E.g. http://www.walkscore.com/apartments/search/Portland-OR?zoom=...

The actual scores would have to be looked up one at a time, on per-address basis (e.g. http://www.walkscore.com/score/320-pioneer-way-mountain-view... )

Re: Finding the perfect house using open data

#50

Earlier quoted context omitted.

That's right, because they generally have their own proprietary source of data ( http://en.wikipedia.org/wiki/Multiple_listing_service ), which is a big part of the appeal of working with a realtor -- they learn of new listings faster than you can by relying on public, "open source" data. And realtors aren't stupid; they noticed what happened when other professions that were built on closed data opened their database…

I'm married to a Realtor, so I've had plenty of direct access to the local MLS system, and also I've overheard many conversations between my wife and her clients about Zillow. From what I've seen and heard, Zillow's data isn't just delayed by a week; it can be years out of date, and grossly inaccurate as well. Both buyers and sellers are very mislead about property values based on Zillow because they see the last sal…

Folks here should check out http://patrick.net/, run by a SF bay area programmer, for an opposing view on the necessity and desirability of Realtors(tm).

In terms of the MLS, at least in California, which has the largest one, it's run by Realtors(tm) for Realtors(tm). The board members are Realtors(tm); so are the "34 REALTOR® Associations" that are the MLS's customers. You can see crmls.org for more details.

In terms of attorneys, at least you can pay them by the hour. In the SF bay area, where house prices can be 10x-15x the national average, why should a Realtor(tm) be paid 10x-15x as much for doing approximately the same amount of work? Haircuts, which require approximately as much skill on the part of the service provider, don't cost 15x as much here. Also the more you spend, the more the Realtor(tm) gets paid; this precisely misaligns incentives.

Attorneys also have a far higher degree of legal obligation to you than a Realtor(tm) does. Check out the standard real estate contract, which is designed primarily to eliminate any legal responsibility on the part of the Realtor(tm), and only secondarily to conduct the actual house transaction.

Also it's rather silly to compare the amount of knowledge required to represent someone in court vs. show a house. To be a real estate agent in California, you have to spend 18 days taking three simultaneous correspondence courses. One of which can be "Computer Applications in Real Estate," which will teach you such difficult-to-master skills as "setting up e-mails," "using digital cameras," and "designing a power point presentation." Tests are T/F and multiple choice. I am not making this up: http://www.accreditedcareers.com/ca_info.pdf

Here's part of Assignment #1 (representing almost 10% of your final grade!): 1. Search Online for 5 Internet Service Providers and make a list with the URL next to the ISP name and list the price of each. 2. Search Online and find 5 different Online Vendors of computer hardware and make a list with the URL next to the Vendors name. 3. Search Online and find 5 Computer Software Vendors and make a list with the URL next to the Vendors name. 4. Using your URL go to http://www.dre.ca.gov find the latest Real Estate bulletin and just send the URL address to the bulletin on the list. 5. Download Yahoo Messenger and Install program. Using Yahoo Messenger Application set up a username and messenger account and send a request to ca_ares_course@yahoo.com to join your buddies. (this will also allow chat to the instructor)

It's true that Realtors(tm) can provide some value, for instance when they have deep knowledge of the local area. But whenever I've spoken at Realtor(tm) conferences -- a few times at Inman Connect in SF -- I've outraged the audience by telling them they'll be disintermediated as surely as their travel agent brethren, and describing how it will happen. My only regret is that it's taking so long.

If anyone is working on a startup to dislodge the Realtor(tm) effective monopoly[1], let me know.

[1] The DOJ sued the National Association of Realtors(tm) in 2005 for antitrust violations during the antitrust-averse Bush administration. Unfortunately the case was settled in 2008: http://www.nytimes.com/2008/05/28/business/28realty.html?_r=...

Post reply on HN