Servers couldn't handle it.
Show HN: Crawling car dealerships for a real time consumer search engine
11–20 of 66 posts
Re: Show HN: Crawling car dealerships for a real time consumer search engine
#12Suggest a map that shows which dealer and where a car is located. There's a subset of car buyers where the car itself is less important than the garage they bought it from, in case they need to bring it back for repairs / tuning etc. This is especially true of used cars. And very important on mobile
Maybe it's something country-specific, but why would you do repairs in the same garage? Or did you mean the dealer's guarantee period?
For folks who lease cars (a poor financial decision long-term, but lower monthly payments) you need to take the car back to the dealership you bought it from at regular intervals (3-10K miles).
Re: Show HN: Crawling car dealerships for a real time consumer search engine
#13Servers couldn't handle it.
I really wish people would test their sites before doing this. I can understand if someone else submits your stuff and you had no idea it was going to happen (but even then...), but if you create an account with no history for the explicit purpose of submitting to HN as the OP did, you should at least test it under some semblance of load.
Re: Show HN: Crawling car dealerships for a real time consumer search engine
#14How did you manage to come into agreement with car dealers to crawl their sites and use their photos in your aggregator? Congratulations!
“Listings are currently sourced from several delearship websites by means of crawling and extracting relevant content available on the host application. If you are a dealer wishing to list and/or promote your inventory on Demanjo, we can help you drive qualified, local shoppers to your dealership.” and “The selection and placement of listings on this page, except featured listings, were determined automatically by a…
Yep and as I worked on sites for a big Audi dealer in the past they would not be happy giving stuff to scrapers as opposed to getting the lead direct.
BTW for information in the UK a lead for our Audi B2C site was worth around £60.
Sounds dodgy from a Google perspective republishing other peoples content - though I know that Google looked at doing a niche car product - like they have with hotels etc so might not be a viable long term business.
Re: Show HN: Crawling car dealerships for a real time consumer search engine
#15How did you manage to come into agreement with car dealers to crawl their sites and use their photos in your aggregator? Congratulations!
As long as they respect the robots.txt, and aren't hitting the site constantly, there shouldn't be a problem technology-wise.
Now, whether this business wants a potential competitor doing this, that's a different story.
Re: Show HN: Crawling car dealerships for a real time consumer search engine
#16Servers couldn't handle it.
I really wish people would test their sites before doing this. I can understand if someone else submits your stuff and you had no idea it was going to happen (but even then...), but if you create an account with no history for the explicit purpose of submitting to HN as the OP did, you should at least test it under some semblance of load.
I really wish people would test their sites before doing this.
I'm conflicted; whilst ultimately you're right, it's very easy to type those words, and not as easy to test for a realistic load.
It's not as simple as throwing ab at your website, you need to use a proper tool (e.g. JMeter), make sure you're testing realistic user behaviour (even basics such as whether images and CSS have an impact), and ensure that you're not getting a false sense of security (e.g. how many connections are actually hitting the server at the same time?).
So yeah, whilst I kinda agree with you, I think it's a lot easier to say than to carry out.
Re: Show HN: Crawling car dealerships for a real time consumer search engine
#17Servers couldn't handle it.
I really wish people would test their sites before doing this. I can understand if someone else submits your stuff and you had no idea it was going to happen (but even then...), but if you create an account with no history for the explicit purpose of submitting to HN as the OP did, you should at least test it under some semblance of load.
I understand what you mean. but we have actually tested the site several times before posting it on HN. We are working hard at this and will be back shortly. Thanks.
Re: Show HN: Crawling car dealerships for a real time consumer search engine
#18Earlier quoted context omitted.
I really wish people would test their sites before doing this. I can understand if someone else submits your stuff and you had no idea it was going to happen (but even then...), but if you create an account with no history for the explicit purpose of submitting to HN as the OP did, you should at least test it under some semblance of load.
I really wish people would test their sites before doing this. I'm conflicted; whilst ultimately you're right, it's very easy to type those words, and not as easy to test for a realistic load. It's not as simple as throwing ab at your website, you need to use a proper tool (e.g. JMeter), make sure you're testing realistic user behaviour (even basics such as whether images and CSS have an impact), and ensure that you'…
I totally agree, and if I tried to do it I'd probably botch it myself. I wouldn't even know where to start as I've ever built anything that needs to be under this kind of load where the deployment isn't handled by people better at it than I.
Re: Show HN: Crawling car dealerships for a real time consumer search engine
#19Earlier quoted context omitted.
“Listings are currently sourced from several delearship websites by means of crawling and extracting relevant content available on the host application. If you are a dealer wishing to list and/or promote your inventory on Demanjo, we can help you drive qualified, local shoppers to your dealership.” and “The selection and placement of listings on this page, except featured listings, were determined automatically by a…
Yep and as I worked on sites for a big Audi dealer in the past they would not be happy giving stuff to scrapers as opposed to getting the lead direct. BTW for information in the UK a lead for our Audi B2C site was worth around £60. Sounds dodgy from a Google perspective republishing other peoples content - though I know that Google looked at doing a niche car product - like they have with hotels etc so might not be a…
> Sounds dodgy from a Google perspective republishing other peoples content
Actually, it is just publishing compiled facts. It might violate the ToS, but it probably doesn't violate copyright.
Re: Show HN: Crawling car dealerships for a real time consumer search engine
#20Servers couldn't handle it.
I really wish people would test their sites before doing this. I can understand if someone else submits your stuff and you had no idea it was going to happen (but even then...), but if you create an account with no history for the explicit purpose of submitting to HN as the OP did, you should at least test it under some semblance of load.
[deleted]