Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Crawling car dealerships for a real time consumer search engine

demanjo.com

Post reply on HN
Print view View original

21–30 of 66 posts

Re: Show HN: Crawling car dealerships for a real time consumer search engine

#21

Post by webtrill »

ambirex wrote: post #9

I have built something very similar to this is the past, here are a couple of observations: Scraping: - Dealer web sites are run by a handful of different data brokers, for the most part if you find a good way to scrape one (say dealer who uses http://www.dealer.com/ than you can extend your scraper to get others) - Dealer web sites, in general, are horrible to view. - Learn to love VIN explosion/decoding - http://ww…

Changing sites is not a problem as the algorithms are generic to target any website!! We do not develop algorithms for each provider out there, that will not be feasible and quiet frankly a waste of time on our part.

For geolocation searches, we auto detect a users location. I cannot guarantee the accuracy of a users location as the geo data source is a free version. You can however easily select the miles to increase the radius of your search.

Re: Show HN: Crawling car dealerships for a real time consumer search engine

#22

Post by ajw0100 »

darkarmani wrote: post #19
walshemj wrote: post #14

Earlier quoted context omitted.

Yep and as I worked on sites for a big Audi dealer in the past they would not be happy giving stuff to scrapers as opposed to getting the lead direct. BTW for information in the UK a lead for our Audi B2C site was worth around £60. Sounds dodgy from a Google perspective republishing other peoples content - though I know that Google looked at doing a niche car product - like they have with hotels etc so might not be a…

> Sounds dodgy from a Google perspective republishing other peoples content Actually, it is just publishing compiled facts. It might violate the ToS, but it probably doesn't violate copyright.

Do you know of any good articles demonstrating the repercussions of violating a ToS vs. violating a copyright?

I'm guessing violating a copyright is more likely to result in aggressive legal action whereas a ToS violation would just get you banned from the service or sent some sort of cease and desist.

Re: Show HN: Crawling car dealerships for a real time consumer search engine

#23

Post by stblack »

Great job! Nicely done.

A suggestion, also a source of competitive advantage: Allow selecting more than one Make per search. Seems nobody does this. I would like to see all SUVs except those by the big North American manufacturers. To do this I need to execute many separate searches.

Re: Show HN: Crawling car dealerships for a real time consumer search engine

#24

Post by brianbreslin »

pc86 wrote: post #13
nkozyra wrote: post #11

Servers couldn't handle it.

I really wish people would test their sites before doing this. I can understand if someone else submits your stuff and you had no idea it was going to happen (but even then...), but if you create an account with no history for the explicit purpose of submitting to HN as the OP did, you should at least test it under some semblance of load.

I would love to see a post on how to test your site for load issues like ones you would face from hn

Re: Show HN: Crawling car dealerships for a real time consumer search engine

#26

Post by webtrill »

stblack wrote: post #23

Great job! Nicely done. A suggestion, also a source of competitive advantage: Allow selecting more than one Make per search. Seems nobody does this. I would like to see all SUVs except those by the big North American manufacturers. To do this I need to execute many separate searches.

Thanks!!

Will deploy multiple selections of filters by weeks end.

Re: Show HN: Crawling car dealerships for a real time consumer search engine

#28

Post by lincolnq »

Samuel_Michon wrote: post #6
carbocation wrote: post #5

Earlier quoted context omitted.

I would guess they did so in the same way that google asks every website operator before crawling and caching. (I.e., I suspect they didn't come to any explicit agreement. If google doesn't, why should they need to?)

Given that Google has been sued over that countless times, as a small startup I would be wary of following their example.

This is probably the wrong attitude towards founding startups. In general, you shouldn't unnecessarily risk the business -- but if people are throwing roadblocks in your way, lots of startups seem to generally do pretty well when they play fast-and-loose with rules. The logic is that nobody's going to bother to sue you until you get big and can defend yourself. Obviously taking a big risk like this isn't ideal, but you shouldn't let it stop you from moving forward with a business.
Post reply on HN