Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I scraped 25M Shopify products to build a search engine

searchagora.com

Post reply on HN
Print view View original

1–10 of 286 posts

Show HN: I scraped 25M Shopify products to build a search engine

#1

Post by pencildiver »

Hi HN! I built Agora as a side-project leading up to the holiday season. I wanted to find an easier way to find Christmas gifts, without needing to go store-by-store.

My wife asked me for a a pair of red shoes for Christmas. I quickly typed it into Google and found a combination of ads from large retailers and links to a 1948 movie called 'Red Shoes'. I decided to build Agora to solve my own problem (and stay happily married). The product is a search engine that automatically crawls thousands of Shopify stores and makes them easily accessible with a search interface. There's a few additional features to enhance the buying experience including saving products, filters, reviews, and popular products.

I've started with exclusively Shopify stores and plan to expand the crawler to other e-commerce platforms like BigCommerce, WooCommerce, Wix, etc. The technical challenge I've found is keeping the search speed and performance strong as the data set becomes larger. There's about 25 million products on Agora right now. I'll ramp this up carefully to make sure we don't compromise the search speed and user experience.

I'd love any feedback!

Show HN: I scraped 25M Shopify products to build a search engine
searchagora.com

Re: Show HN: I scraped 25M Shopify products to build a search engine

#6

Post by cldellow »

misterbwong wrote: post #2

What technology did you use to build the scraper and how did you get around the usual challenges (anti bot, ip banning, etc) with scraping large amounts of data?

(not the OP, but I have some experience with Shopify)

Shopify stores publish their product catalog at /products.json. From personal experience, you can hammer it pretty hard without being rate limited.

A challenge is that the pricing info in that endpoint is based on the stock Shopify catalog fields, and can be misleading depending on the specific theme customizations that the merchant uses.

Re: Show HN: I scraped 25M Shopify products to build a search engine

#7

Post by pencildiver »

misterbwong wrote: post #2

What technology did you use to build the scraper and how did you get around the usual challenges (anti bot, ip banning, etc) with scraping large amounts of data?

Scraper is built in Javascript and a Mongo database. Probably not the most scalable way to do it, but I found that all Shopify stores have a public JSON file available at [Base URL]/products.json. So found a list of stores, built a crawler to go store-by-store, and standardized the data on my end.

Here's an example: https://www.wildfox.com/products.json

Re: Show HN: I scraped 25M Shopify products to build a search engine

#9

Post by thomasfromcdnjs »

pencildiver wrote: post #7
misterbwong wrote: post #2

What technology did you use to build the scraper and how did you get around the usual challenges (anti bot, ip banning, etc) with scraping large amounts of data?

Scraper is built in Javascript and a Mongo database. Probably not the most scalable way to do it, but I found that all Shopify stores have a public JSON file available at [Base URL]/products.json. So found a list of stores, built a crawler to go store-by-store, and standardized the data on my end. Here's an example: https://www.wildfox.com/products.json

ooo that is a hot tip!

Re: Show HN: I scraped 25M Shopify products to build a search engine

#10

Post by muratsu »

Agora also doesn't return red shoes for the search query "red shoes". Seems like you haven't fully solved the problem yet :)

From a technical perspective, crawling 25M products is impressive but the search itself doesn't provide much value to me. I already use large e-commerce sites (amazon, wallmart, ...) and targeted ones (Nordstrom, SSENSE, ...). Sure I may not be searching through all the shopify, wix stores but I need to know why that's valuable to me to begin with. Perhaps understanding the value prop of SMBs and educating me about it would be a better positioning for Agora than simply being a search engine.

Post reply on HN