Basically it’s Amazon
Show HN: I scraped 25M Shopify products to build a search engine
11–20 of 286 posts
Re: Show HN: I scraped 25M Shopify products to build a search engine
#12Earlier quoted context omitted.
Scraper is built in Javascript and a Mongo database. Probably not the most scalable way to do it, but I found that all Shopify stores have a public JSON file available at [Base URL]/products.json. So found a list of stores, built a crawler to go store-by-store, and standardized the data on my end. Here's an example: https://www.wildfox.com/products.json
ooo that is a hot tip!
[deleted]
Re: Show HN: I scraped 25M Shopify products to build a search engine
#13What technology did you use to build the scraper and how did you get around the usual challenges (anti bot, ip banning, etc) with scraping large amounts of data?
Scraper is built in Javascript and a Mongo database. Probably not the most scalable way to do it, but I found that all Shopify stores have a public JSON file available at [Base URL]/products.json. So found a list of stores, built a crawler to go store-by-store, and standardized the data on my end. Here's an example: https://www.wildfox.com/products.json
Excellent work
Re: Show HN: I scraped 25M Shopify products to build a search engine
#14What technology did you use to build the scraper and how did you get around the usual challenges (anti bot, ip banning, etc) with scraping large amounts of data?
Scraper is built in Javascript and a Mongo database. Probably not the most scalable way to do it, but I found that all Shopify stores have a public JSON file available at [Base URL]/products.json. So found a list of stores, built a crawler to go store-by-store, and standardized the data on my end. Here's an example: https://www.wildfox.com/products.json
Oh nice, you deserve great things in life for this comment!
Re: Show HN: I scraped 25M Shopify products to build a search engine
#15Hey, I have a Shopify store that sells e-paper calendars / smart screens. I tried to search for it but I could not find it. What should I do so your crawler can find me? https://shop.invisible-computers.com
Super cool product! I'm currently using a list of Shopify stores, so it's still limited (i.e. wanted to start with a relatively small list to focus on the search experience). I'll submit your URL to the crawler now. If you want to reach out to support @ searchagora.com , I'd love to get your feedback as a Shopify store owner.
Re: Show HN: I scraped 25M Shopify products to build a search engine
#16What technology did you use to build the scraper and how did you get around the usual challenges (anti bot, ip banning, etc) with scraping large amounts of data?
Scraper is built in Javascript and a Mongo database. Probably not the most scalable way to do it, but I found that all Shopify stores have a public JSON file available at [Base URL]/products.json. So found a list of stores, built a crawler to go store-by-store, and standardized the data on my end. Here's an example: https://www.wildfox.com/products.json
How did you detect that it was a Shopify store?
Re: Show HN: I scraped 25M Shopify products to build a search engine
#17Cool project but Shopify already has this. https://shop.app
Re: Show HN: I scraped 25M Shopify products to build a search engine
#18This is amazing for finding cute collectibles from my favorite TV show that I would otherwise not noticed among random t-shirt and other "slap the picture and call it co-branded" products! I'm not super sure how long it is going to be around, but I think I'm gonna keep playing with it for a while.
Re: Show HN: I scraped 25M Shopify products to build a search engine
#19Earlier quoted context omitted.
Scraper is built in Javascript and a Mongo database. Probably not the most scalable way to do it, but I found that all Shopify stores have a public JSON file available at [Base URL]/products.json. So found a list of stores, built a crawler to go store-by-store, and standardized the data on my end. Here's an example: https://www.wildfox.com/products.json
How did you detect that it was a Shopify store?
Not OP but:
"Has the site a /products.json file?" is a good first check :) And if it does, "Does that format match with the format a Shopify store?" is another good followup question.
Re: Show HN: I scraped 25M Shopify products to build a search engine
#20Cool project but Shopify already has this. https://shop.app https://shop.app/search/results?query=red%20shoes
Was about to share the same link, it seems like competing against Shopify would prove quite the challenge.
The real way to differentiate IMO is with a targeted UX for different niches rather than the one search engine to satisfy all queries.