Cool! But how did you get the initial dataset of 643,000+ Shopify stores (data as per your “About” page) in the first place, to then scrape the products from their /products.json feeds? Or did you just try a huge list of domain names at random?
Show HN: I scraped 25M Shopify products to build a search engine
21–30 of 286 posts
Re: Show HN: I scraped 25M Shopify products to build a search engine
#22Cool! But how did you get the initial dataset of 643,000+ Shopify stores (data as per your “About” page) in the first place, to then scrape the products from their /products.json feeds? Or did you just try a huge list of domain names at random?
https://www.shopify.com/robots.txt lists a lot of sitemap files, which tend to be a good starting point.
Re: Show HN: I scraped 25M Shopify products to build a search engine
#23Agora also doesn't return red shoes for the search query "red shoes". Seems like you haven't fully solved the problem yet :) From a technical perspective, crawling 25M products is impressive but the search itself doesn't provide much value to me. I already use large e-commerce sites (amazon, wallmart, ...) and targeted ones (Nordstrom, SSENSE, ...). Sure I may not be searching through all the shopify, wix stores but…
Definitely have not solved the problem yet! The search algorithm prioritizes the brand called "Red Wing Shoe" so still figuring out ways to show real 'red shoes'. Have been thinking about passing the images through a detection tool and tag them to enhance the search experience.
Re: Value Proposition. Absolutely, I think focusing on the SMB-angle and 'local shopping' will help direct users better. I'll definitely take this into account.
Re: Show HN: I scraped 25M Shopify products to build a search engine
#24Cool project but Shopify already has this. https://shop.app https://shop.app/search/results?query=red%20shoes
But seems to have filters (lots of liquor stores use Shopify) - shop.app shows only candy and swag[0], while searchagora shows ~130k results for the actual product [1]
[0]: https://shop.app/search/results?query=Baileys+Irish+Cream+Li... [1]: https://www.searchagora.com/search?query=Baileys%20Irish%20C...
Re: Show HN: I scraped 25M Shopify products to build a search engine
#25[flagged]
Re: Show HN: I scraped 25M Shopify products to build a search engine
#26Cool! But how did you get the initial dataset of 643,000+ Shopify stores (data as per your “About” page) in the first place, to then scrape the products from their /products.json feeds? Or did you just try a huge list of domain names at random?
Bought an initial list of 2m stores for a few hundred dollars from a website called "Built With". Think they are used for building sales outreach lists. Then narrowed down the focus to stores to US only and between $100k - $1m in revenue to keep the initial data set manageable (and the CPU / Storage costs reasonable).
Re: Show HN: I scraped 25M Shopify products to build a search engine
#27Great project. If you continue to crawl the data, be sure to save it so you can detect price changes a la camelcamelcamel.
Re: Show HN: I scraped 25M Shopify products to build a search engine
#28I love your approach; you found a problem and developed a solution for it. And then you got the courage to share with the larger technical community. Good on you.
There's obviously some rough edges (multiple duplicate products, issues with product links linking to empty pages, and no results for broad terms), but don't let that stop you. I'm certain they can all be fixed.
Keep going! At the least, you'll come out of this with an excellent project in your portfolio.
Re: Show HN: I scraped 25M Shopify products to build a search engine
#29This is amazing for finding cute collectibles from my favorite TV show that I would otherwise not noticed among random t-shirt and other "slap the picture and call it co-branded" products! I'm not super sure how long it is going to be around, but I think I'm gonna keep playing with it for a while.
Really happy to hear this. I'll do my best to keep this around :)
Re: Show HN: I scraped 25M Shopify products to build a search engine
#30[flagged]
Google is a data scraper. OpenAI is a data scraper. Pretty much every major company uses data obtained from a data scraper.