Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Crawling car dealerships for a real time consumer search engine

demanjo.com

Post reply on HN
Print view View original

51–60 of 66 posts

Re: Show HN: Crawling car dealerships for a real time consumer search engine

#51

Post by ambirex »

MatthewPhillips wrote: post #42
ambirex wrote: post #9

I have built something very similar to this is the past, here are a couple of observations: Scraping: - Dealer web sites are run by a handful of different data brokers, for the most part if you find a good way to scrape one (say dealer who uses http://www.dealer.com/ than you can extend your scraper to get others) - Dealer web sites, in general, are horrible to view. - Learn to love VIN explosion/decoding - http://ww…

How did you do bulk VIN explosion? Pay for it (rates are high in my research) or scrape one of the free sites?

Built my own scraping some of the free sites, You can do basic make and year off plain VIN, then scraped some of the free sites with break downs of the various VINs to build a a map.

Re: Show HN: Crawling car dealerships for a real time consumer search engine

#52

Post by Kynlyn »

What about trim levels? Trim levels are crucial in the buying decision. For example, if you search for a Ford F-150 on your site, there is no way to see if it's an XLT, King Ranch, Raptor, etc. The price and equipment difference in those vehicles is huge.

What about options and equipment? Does the car have navigation? Sunroof? Most consumers have specific option packages in mind when searching for car online.

This is why VIN explosion is necessary for any serious automotive shopping site. If a consumer can't narrow the vehicles down to a trim and option package level then it won't get wide adoption.

Re: Show HN: Crawling car dealerships for a real time consumer search engine

#53

Post by DucktourDreams »

webtrill wrote: post #49
GuriK wrote: post #48

Great job. Love the idea. Can you tell us which technologies you used ?

Everything is proprietary!! that is our big advantage. Data store code name "Saycron" sits on the Demanjo's Distributed file system. Indexing -- > uses a data structure i call "Octo-tree" Outperforms any multidimensional index i came across in any academic papers. Web server -- > Code name "SlimAPE". simple non blocking. Web framework -- > Code name "HerikX". I will write about all of these technologies later on my b…

>Everything is proprietary!! that is our big advantage.

Do your customers care if it's proprietary or not? What about when there's a security breach and you can't upgrade to fix it? What if you can't even detect it?

> uses a data structure i call "Octo-tree" Outperforms any multidimensional...

premature optimization. There are dozens of bugs in it now. It doesn't help to move fast if you're moving in the wrong direction.

Re: Show HN: Crawling car dealerships for a real time consumer search engine

#54

Post by beachstartup »

mootothemax wrote: post #16
pc86 wrote: post #13

Earlier quoted context omitted.

I really wish people would test their sites before doing this. I can understand if someone else submits your stuff and you had no idea it was going to happen (but even then...), but if you create an account with no history for the explicit purpose of submitting to HN as the OP did, you should at least test it under some semblance of load.

I really wish people would test their sites before doing this. I'm conflicted; whilst ultimately you're right, it's very easy to type those words, and not as easy to test for a realistic load. It's not as simple as throwing ab at your website, you need to use a proper tool (e.g. JMeter), make sure you're testing realistic user behaviour (even basics such as whether images and CSS have an impact), and ensure that you'…

amplified log playback works best in my experience.

of course, this requires at least some public traffic to play back. i've only used proprietary tools for this in my personal experience, but i believe jmeter has this functionality (log sampling).

Re: Show HN: Crawling car dealerships for a real time consumer search engine

#55

Post by webtrill »

Kynlyn wrote: post #52

What about trim levels? Trim levels are crucial in the buying decision. For example, if you search for a Ford F-150 on your site, there is no way to see if it's an XLT, King Ranch, Raptor, etc. The price and equipment difference in those vehicles is huge. What about options and equipment? Does the car have navigation? Sunroof? Most consumers have specific option packages in mind when searching for car online. This is…

We'll be adding the trim levels in the near future. We agree it can be very useful when buying a car. Thanks

Re: Show HN: Crawling car dealerships for a real time consumer search engine

#57

Post by bfung »

Kynlyn wrote: post #52

What about trim levels? Trim levels are crucial in the buying decision. For example, if you search for a Ford F-150 on your site, there is no way to see if it's an XLT, King Ranch, Raptor, etc. The price and equipment difference in those vehicles is huge. What about options and equipment? Does the car have navigation? Sunroof? Most consumers have specific option packages in mind when searching for car online. This is…

VIN explosion doesn't always return truck trims, as many times the actual truck beds are added after the vehicle rolled off the assembly line.

So they will probably need more than VIN explosion for that; there are some companies that do provide that data though.

Re: Show HN: Crawling car dealerships for a real time consumer search engine

#58

Post by Kynlyn »

bfung wrote: post #57
Kynlyn wrote: post #52

What about trim levels? Trim levels are crucial in the buying decision. For example, if you search for a Ford F-150 on your site, there is no way to see if it's an XLT, King Ranch, Raptor, etc. The price and equipment difference in those vehicles is huge. What about options and equipment? Does the car have navigation? Sunroof? Most consumers have specific option packages in mind when searching for car online. This is…

VIN explosion doesn't always return truck trims, as many times the actual truck beds are added after the vehicle rolled off the assembly line. So they will probably need more than VIN explosion for that; there are some companies that do provide that data though.

Trucks are certainly the most difficult to decode. But if you are using something like ChromeData for the VIN data, and combine that with the info from the dealer's site then you can usually narrow the vehicle down to a specific trim level.

Not always however. Dealers frequently have incorrect or missing information on their websites, so garbage-in, garbage out.

This is why scraping dealer websites for data is always going to be problematic. Far better to work with the providers to have them send you the data. It's faster, easier and you get far better data.

Re: Show HN: Crawling car dealerships for a real time consumer search engine

#59

Post by pc86 »

beachstartup wrote: post #54
mootothemax wrote: post #16

Earlier quoted context omitted.

I really wish people would test their sites before doing this. I'm conflicted; whilst ultimately you're right, it's very easy to type those words, and not as easy to test for a realistic load. It's not as simple as throwing ab at your website, you need to use a proper tool (e.g. JMeter), make sure you're testing realistic user behaviour (even basics such as whether images and CSS have an impact), and ensure that you'…

amplified log playback works best in my experience. of course, this requires at least some public traffic to play back. i've only used proprietary tools for this in my personal experience, but i believe jmeter has this functionality (log sampling).

This seems really interesting. When you say proprietary tools, I assume you mean more custom-built solutions and less paid tools available to anyone?

Re: Show HN: Crawling car dealerships for a real time consumer search engine

#60

Post by bfung »

Kynlyn wrote: post #58
bfung wrote: post #57

Earlier quoted context omitted.

VIN explosion doesn't always return truck trims, as many times the actual truck beds are added after the vehicle rolled off the assembly line. So they will probably need more than VIN explosion for that; there are some companies that do provide that data though.

Trucks are certainly the most difficult to decode. But if you are using something like ChromeData for the VIN data, and combine that with the info from the dealer's site then you can usually narrow the vehicle down to a specific trim level. Not always however. Dealers frequently have incorrect or missing information on their websites, so garbage-in, garbage out. This is why scraping dealer websites for data is always…

I used to work for a competitor in the same space as AutoRevo =) Chrome was ok, but there was another provider that had exact vin matches in their catalog. It was a bit more expensive, but made it so the trim field was a non-issue. I don't remember the name, it's been too long.
Post reply on HN