Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I scraped 25M Shopify products to build a search engine

searchagora.com

Post reply on HN
Print view View original

281–286 of 286 posts

Re: Show HN: I scraped 25M Shopify products to build a search engine

#281

Post by russum »

pencildiver wrote: post #26
Asparagirl wrote: post #21

Cool! But how did you get the initial dataset of 643,000+ Shopify stores (data as per your “About” page) in the first place, to then scrape the products from their /products.json feeds? Or did you just try a huge list of domain names at random?

Bought an initial list of 2m stores for a few hundred dollars from a website called "Built With". Think they are used for building sales outreach lists. Then narrowed down the focus to stores to US only and between $100k - $1m in revenue to keep the initial data set manageable (and the CPU / Storage costs reasonable).

> and between $100k - $1m in revenue

Does "Built With" provide that data? How accurate do you think it might be?

Re: Show HN: I scraped 25M Shopify products to build a search engine

#283

Post by 4runner »

pencildiver wrote: post #80
screye wrote: post #57

What was the process for scraping 25M products ? I have always used standard python tools like selenium, bs4 and the like. But I'm guessing none of these work at scale. Could you talk about your process and key bottlenecks at that scale a little bit ? Also, how much did it cost ? ______________ A recommendation for how to improve search. Your base captions will be pretty bad. You can use spot instances on a smaller G…

Great suggestions, looking into this right now. First time building something like this so definitely new to some of these tools. For scraping: Found that every Shopify store has a public JSON file that is available in the same route. The JSON file appears on the [Base URL]/products.json. For example, the store for Wild Fox has their JSON file available here: https://www.wildfox.com/products.json . Built a crawler in…

Sounds like you used an incorrect instance type/size on Atlas

Re: Show HN: I scraped 25M Shopify products to build a search engine

#284

Post by pencildiver »

callmeed wrote: post #167

I built this a couple years ago (now defunct) for the same reason :) The public JSON endpoints on shopify stores make it pretty easy to get the data. You mentioned using Mongo but it sounds expensive. I honestly think you could do this with just elastic or even postgres full text search and save money. Here's a pro tip + feature you should implement: Shopify has a semi-hidden hack where you can link directly to check…

Thanks for the heads up! I spent some time trying to get the cart route to work. Doesn't seem to be supported anymore (link you sent leads to a 404 page). Tried it with every combination of Product ID, Variant ID, etc. Let me know if you have any ideas on how to get this to work. It would be a great feature to add to Agora.

And I agree on quality over quantity. Writing a script to remove all stores that are shutdown, products that are sold out, and a few other characteristics. Heavily focusing on the search algorithm and data quality now.

Re: Show HN: I scraped 25M Shopify products to build a search engine

#285

Post by charliejuggler »

pencildiver wrote: post #23
muratsu wrote: post #10

Agora also doesn't return red shoes for the search query "red shoes". Seems like you haven't fully solved the problem yet :) From a technical perspective, crawling 25M products is impressive but the search itself doesn't provide much value to me. I already use large e-commerce sites (amazon, wallmart, ...) and targeted ones (Nordstrom, SSENSE, ...). Sure I may not be searching through all the shopify, wix stores but…

Definitely have not solved the problem yet! The search algorithm prioritizes the brand called "Red Wing Shoe" so still figuring out ways to show real 'red shoes'. Have been thinking about passing the images through a detection tool and tag them to enhance the search experience. Re: Value Proposition. Absolutely, I think focusing on the SMB-angle and 'local shopping' will help direct users better. I'll definitely take…

You could try the method we used for our vector search demo for e-commerce (all open source, natch) - use CLIP to get vector embeddings for product pictures and then use these for boosting or matching. https://opensourceconnections.com/blog/2023/03/22/building-v... Our demo works pretty well for searches like 'blue network cable' when the colour isn't always explicitly mentioned in the product data.

Re: Show HN: I scraped 25M Shopify products to build a search engine

#286

Post by SantiagoRuberto »

senecaso wrote: post #252
DeathArrow wrote: post #168

Earlier quoted context omitted.

>but in the end, it still wouldn't have mattered if I couldn't figure out how to acquire users In EU there are many price comparison engines with millions or billions of products. I don't know how popular they are. Some monetize trough ads, some have partnership with stores and you can buy directly from the search results. I generally search first on the local Amazon equivalent, if I don't like what I see, I search o…

We were intentionally limiting the number of products and shops we were indexing due to opex. We needed to keep it low enough to provide ourselves with enough runway to keep things floating for longer. pricerunner is another site which operates in a similar space. We had plans to build out the price tracking and a number of other features, so that we would appeal more to users who had your use cases. Sadly, we weren'…

>

What were your plans to solve this problem?

Post reply on HN