Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Crawling car dealerships for a real time consumer search engine

demanjo.com

Post reply on HN
Print view View original

41–50 of 66 posts

Re: Show HN: Crawling car dealerships for a real time consumer search engine

#42

Post by MatthewPhillips »

ambirex wrote: post #9

I have built something very similar to this is the past, here are a couple of observations: Scraping: - Dealer web sites are run by a handful of different data brokers, for the most part if you find a good way to scrape one (say dealer who uses http://www.dealer.com/ than you can extend your scraper to get others) - Dealer web sites, in general, are horrible to view. - Learn to love VIN explosion/decoding - http://ww…

How did you do bulk VIN explosion? Pay for it (rates are high in my research) or scrape one of the free sites?

Re: Show HN: Crawling car dealerships for a real time consumer search engine

#43

Post by jbattle »

A few thoughts: - I think CarFax has an affiliate program that might let you link through to their reports, keyed by VIN. These reports are a pretty valuable tool to shoppers as the condition of a used car is driving 50% of the buying decision.

- One data point that is really important to dealers is how long the car has been on the lot. Hypothetically you could just track how long the same car has appeared at the same dealership. The longer the car has been sitting there, the more motivated they'll be to sell it. I remember hearing 30 days is a long time for dealerships to sit on a car. For buyers this could be good information to have.

- Getting the mobile experience right would be a huge win. So often the cars the dealer lists aren't actually the cars they have on the site (I'm not sure how much of this is intentional and how much is a matter of how fast inventory turns over). So when you get to the dealership, you want to quickly and easily be able to comparison shop - get that right and you cover people in that really critical uncomfortable moment.

Re: Show HN: Crawling car dealerships for a real time consumer search engine

#44

Post by negrit »

webtrill wrote: post #38
negrit wrote: post #35

Earlier quoted context omitted.

The only suggestions I have when I type "San Fra" are : San Clara, MB San Josef Bay, BC San Josef, BC San Joseph Bay, BC"

Your IP address resolves to Canada. You can only search for cars in your country. Will have to change this if required at some point

That's even more weird because I'm in France, using a french ISP without any proxy or whatever and I can see cars from Texas, Nevada, ...

Re: Show HN: Crawling car dealerships for a real time consumer search engine

#46

Post by michaelfeathers »

pc86 wrote: post #13
nkozyra wrote: post #11

Servers couldn't handle it.

I really wish people would test their sites before doing this. I can understand if someone else submits your stuff and you had no idea it was going to happen (but even then...), but if you create an account with no history for the explicit purpose of submitting to HN as the OP did, you should at least test it under some semblance of load.

You're assuming the HN submission isn't the load test.

Re: Show HN: Crawling car dealerships for a real time consumer search engine

#49

Post by webtrill »

GuriK wrote: post #48

Great job. Love the idea. Can you tell us which technologies you used ?

Everything is proprietary!! that is our big advantage.

Data store code name "Saycron" sits on the Demanjo's Distributed file system.

Indexing -- > uses a data structure i call "Octo-tree" Outperforms any multidimensional index i came across in any academic papers.

Web server -- > Code name "SlimAPE". simple non blocking.

Web framework -- > Code name "HerikX".

I will write about all of these technologies later on my blog whenever i get to setting up one.

Post reply on HN