[flagged]
Why? They're cataloguing public data. Do you think the same of search engines - which are equally scraping the web?
If it's not useful to you, nobody is forcing you to use this product.
Show HN: I scraped 25M Shopify products to build a search engine
31–40 of 286 posts
Re: Show HN: I scraped 25M Shopify products to build a search engine
#32Great project. If you continue to crawl the data, be sure to save it so you can detect price changes a la camelcamelcamel.
Great call! I am doing back-ups on Mongo and this is a good use-case for tracking changes. Also trying to figure out how to detect is a product is sold out or not being sold anymore.
Re: Show HN: I scraped 25M Shopify products to build a search engine
#33Basically it’s Amazon
Antizon
Re: Show HN: I scraped 25M Shopify products to build a search engine
#34Cool project but Shopify already has this. https://shop.app https://shop.app/search/results?query=red%20shoes
Yeah absolutely. I hadn't heard of Shop until today but the value proposition is definitely similar. In the next week, I'll add other e-commerce platforms like BigCommerce, WooCommerce, support for custom built sites, etc. to really differentiate the user experience.
Re: Show HN: I scraped 25M Shopify products to build a search engine
#35[flagged]
Bad experience with someone?
Re: Show HN: I scraped 25M Shopify products to build a search engine
#36[flagged]
Why? They're cataloguing public data. Do you think the same of search engines - which are equally scraping the web? If it's not useful to you, nobody is forcing you to use this product.
You really didn't think about this. Search engine crawlers are specifically designed to ignore sites that have the right setups. In other words: they try to be good citizens of the web. Data scrapers on the other hand are known to steal others IP and use it to jump start their own services.
When you do that you're using all the bandwidth of the site while devaluing everything they've built. They will have invested significant time and resources building that IP. But data scrapers think that just because its on a public website it means they can leech it and do what they like. No, there is such a thing as copy right and respecting authors. Fuck anyone who says otherwise.
If you want to be a little kiddiot and steal content you're free to do so. But you're just undermining the work of everyone you steal from. Making it less and less viable for them to make more of it in the future. I stand by what I said. Data miners are fucking parasites and the web would be better off without them.
Re: Show HN: I scraped 25M Shopify products to build a search engine
#37Cool! But how did you get the initial dataset of 643,000+ Shopify stores (data as per your “About” page) in the first place, to then scrape the products from their /products.json feeds? Or did you just try a huge list of domain names at random?
https://www.shopify.com/robots.txt lists a lot of sitemap files, which tend to be a good starting point.
Did this suddenly get changed? Nothing but "# ,: # ,' | # / : # --' / # \/ />/ # /" is shown now.
Re: Show HN: I scraped 25M Shopify products to build a search engine
#38When I search for “op-1”, partial match like “Frontier Co-op Turkey Rub, Organic 1 lb. -- Frontier Co-op” gets ranked higher than “teenage engineering op-1”. I would expect the opposite.
Re: Show HN: I scraped 25M Shopify products to build a search engine
#39Great project. If you continue to crawl the data, be sure to save it so you can detect price changes a la camelcamelcamel.
Great call! I am doing back-ups on Mongo and this is a good use-case for tracking changes. Also trying to figure out how to detect is a product is sold out or not being sold anymore.
I worked on a competitor to CamelCamelCamel years ago. We had this exact issue since people would often click through to a page where the price or availability were different from what we were showing
Ultimately, we ended up adding an interstitial page between the product listing on our site and the page on the seller's site
This interstitial checked to see if we checked the price in the last couple of minutes, and if not, it would run a quick scrape of the page to ensure that we had the most up to date information
I can't remember exactly what the messaging or behavior was when there was a difference. I think there was a message that was displayed if the prices were different. Or if the product was actually out of stock, it would pull the user back into our site with a toast explaining that the product was no longer available
Anything less aggressive than this resulted in more customers experiencing price/availability errors or simply leaving the site, and anything more aggressive resulted in angry site owners who were losing bandwidth to our bots
> Also trying to figure out how to detect is a product is sold out or not being sold anymore
In these cases, either the page will say as much (eg: "Product Unavailable"), have some kind of stock or status code hidden beneath the UI to show that it's not available, or the target page will simply vanish from the web. However, none of these are guarantees. A site could say that a product has been discontinued, but the item could come back later, or under a different SKU, or whatever else
Re: Show HN: I scraped 25M Shopify products to build a search engine
#40[flagged]
Arrested? That’s a hot take.
What is op doing that would warrant such a response? If anything, they’re providing free advertising to their “victims”.