If its being used for competitive price analysis, I wonder if any retail sites will simply block their crawler? I am assuming that they are (correctly) announcing their crawler by its user agent, so could be blocked via robots.txt.
Semantics3 (YC W13) Is A Massive Consumer Products Database To Rule Them All
11–20 of 35 posts
Re: Semantics3 (YC W13) Is A Massive Consumer Products Database To Rule Them All
#12All payments are handled through PayPal? These guys are really putting all of their eggs into one basket with that move.
Why so you say so? PayPal allows payments through their account and payments with credit cards without an account too.
because Paypal has a history of shutting down accounts for no apparent reasons. Then it takes months to get your money out.
Re: Semantics3 (YC W13) Is A Massive Consumer Products Database To Rule Them All
#13I haven't tested their data, but 1 reason why this is hard is due to the SKUs at Walmart and Best Buy. There may be a Samsung 42-inch TV that exists at Amazon, Walmart and Best Buy but have slightly modified specs for each of those mega retailers... and with this, Best Buy has a lot of SKUs that simply die. So once semantics3 has "reconciled" that samsung 42-inch TV across retailers, they'll have to continuously check whether any of the retailers have changed the SKU and/or URL on them.
source: I do this at dealzon.com for a very limited set of data where it's practical
Re: Semantics3 (YC W13) Is A Massive Consumer Products Database To Rule Them All
#14Earlier quoted context omitted.
Why so you say so? PayPal allows payments through their account and payments with credit cards without an account too.
because Paypal has a history of shutting down accounts for no apparent reasons. Then it takes months to get your money out.
We started out in Singapore and the only payment processor we could use there was Paypal. If started now we would have used something more easier and better like Stripe :) That being said, we should be switching over to them soon.
Re: Semantics3 (YC W13) Is A Massive Consumer Products Database To Rule Them All
#15If its being used for competitive price analysis, I wonder if any retail sites will simply block their crawler? I am assuming that they are (correctly) announcing their crawler by its user agent, so could be blocked via robots.txt.
Not to mention, many ecommerce sites explicitly forbid this sort of thing. I'd be interested to know how they got around it.
Re: Semantics3 (YC W13) Is A Massive Consumer Products Database To Rule Them All
#16Some retailers get their own sku's for common products, (to prevent price comparisons?)
Re: Semantics3 (YC W13) Is A Massive Consumer Products Database To Rule Them All
#17If its being used for competitive price analysis, I wonder if any retail sites will simply block their crawler? I am assuming that they are (correctly) announcing their crawler by its user agent, so could be blocked via robots.txt.
Not to mention, many ecommerce sites explicitly forbid this sort of thing. I'd be interested to know how they got around it.
We currently get the pricing data via rss feeds, crawling, data dumps and for some cases also crowdsourcing. In the long run, we also hope to establish merchant relationships and get the data directly.
To the original question on crawling - (I had replied to a similar question previously on HN): "Some great advice here on crawling at scale, which has inspired our crawlers a lot : http://news.ycombinator.com/item?id=4367933 Basically it boils down to three things: 1. If the site is slow,crawl slooowly. 2. If you see non-200 http error codes, stop! 3. Obey robots.txt and speed restrictions."
Re: Semantics3 (YC W13) Is A Massive Consumer Products Database To Rule Them All
#18If its being used for competitive price analysis, I wonder if any retail sites will simply block their crawler? I am assuming that they are (correctly) announcing their crawler by its user agent, so could be blocked via robots.txt.
Some percentage of online retailers will block their crawler. It's a situation where party A wants party B's data and there is really no reason party B would want party A to have it. It occurs even to small and medium-sized online retailers to try to block pricing crawlers. Yahoo Store devs are irked that you still can't upload a custom robots.txt file to the web root. Forbidding unwanted crawlers from poking around is frequently the reason (applies to stores using the store editor).
Maybe in exchange for the privilege of obtaining pricing data, this service could offer to automatically clean up their product data in exchange. Win-win for everyone.
Re: Semantics3 (YC W13) Is A Massive Consumer Products Database To Rule Them All
#19Some retailers get their own sku's for common products, (to prevent price comparisons?)
Some retailers go so far as to invent different names for uncommon brands and use generated SKUs to prevent even the poaching of new manufacturers. Misinformation is used a lot to keep a competitive edge. One of the many challenges this service faces is trying to figure out if mfr A is actually mfr B.
Re: Semantics3 (YC W13) Is A Massive Consumer Products Database To Rule Them All
#20Seems complementary to a service like EDITD (www.editd.com). I'm curious how Semantics3 determines which ecommerce sites to crawl. I remember reading somewhere about how even Amazon only stocks something like 2% of all ecommerce products.
We started off adding most of the major e-commerce stores. As to how we decide which sites to add, well that depends. We usually add sites based on what our users seem to be asking for. More sites means more ability to do interesting queries like comparing the prices between store A and B. Since we have the products disambiguated, it is much easier.
Or if we find that the existing data that we gathered from our existing sources for a certain vertical could be improved, we add new niche sites to augment the data that we already have.