Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I scraped 25M Shopify products to build a search engine

searchagora.com

Post reply on HN
Print view View original

271–280 of 286 posts

Re: Show HN: I scraped 25M Shopify products to build a search engine

#271

Post by grumpyviscacha »

senecaso wrote: post #130

I hope you have better luck than I did! A few years ago, my partner and I built vendazzo.com (now defunct). It was an e-commerce search engine on products listed on Shopify shops (sound familiar? :)). At the time, we had > 100m products listed, and I don't remember how many shops we were indexing.. over 100k I think, but we had access to over a million. Overall, I think your approach is very similar to ours, but we m…

Wow, it's cool to see this idea trending on HN! Full disclosure, I'm one of the co-founders at https://www.marmalade.co. Speaking from personal experience, it’s been a long road getting from the universe of all Shopify products to a curated inventory that’s easy for people to shop on. While ChatGPT isn't going to replace human curation anytime soon, the AI tailwind has made it much easier to build search and recommendation systems. On our end, we've definitely caught the semantic search bug. Watch out for it - you’ll wake up one day with a cross-modal hybrid search index on pinecone and any number of models on huggingface :). However, as you rightly point out, user growth is still the key. We're working toward launching a community aspect of the platform in the coming months as a solution.

Re: Show HN: I scraped 25M Shopify products to build a search engine

#272

Post by senecaso »

grumpyviscacha wrote: post #271
senecaso wrote: post #130

I hope you have better luck than I did! A few years ago, my partner and I built vendazzo.com (now defunct). It was an e-commerce search engine on products listed on Shopify shops (sound familiar? :)). At the time, we had > 100m products listed, and I don't remember how many shops we were indexing.. over 100k I think, but we had access to over a million. Overall, I think your approach is very similar to ours, but we m…

Wow, it's cool to see this idea trending on HN! Full disclosure, I'm one of the co-founders at https://www.marmalade.co . Speaking from personal experience, it’s been a long road getting from the universe of all Shopify products to a curated inventory that’s easy for people to shop on. While ChatGPT isn't going to replace human curation anytime soon, the AI tailwind has made it much easier to build search and recomme…

You site looks good, and your results are fantastic! Job well done. I did hit a server error though, so obviously still some issues to work out, but overall, really well done. Moving to semantic search was one of my top priorities before we went under, but I struggled to justify the costs of it as we were operating on a shoestring budget.

Best of luck to you and your team on user acquisition!

Re: Show HN: I scraped 25M Shopify products to build a search engine

#273

Post by katella »

tehlike wrote: post #124
pencildiver wrote: post #80

Earlier quoted context omitted.

Great suggestions, looking into this right now. First time building something like this so definitely new to some of these tools. For scraping: Found that every Shopify store has a public JSON file that is available in the same route. The JSON file appears on the [Base URL]/products.json. For example, the store for Wild Fox has their JSON file available here: https://www.wildfox.com/products.json . Built a crawler in…

Disclaimer: I am building https://pricetracker.wtf You may want to look at Hetzner, and cut your costs by about 90%. Feel free to reach me, email in profile.

In your footer you have a lot of links like "kitchenaid price tracker" and "best buy price tracker". Have these helped links helped?

Re: Show HN: I scraped 25M Shopify products to build a search engine

#276

Post by pencildiver »

difradev wrote: post #275

Amazing job! I've one question: how did you find the price of every products? I mean, every product page has a different id or class that identify a price. Do you use a regex?

Thanks! Actually a lot easier than you'd expect. Not touching anything on the front-end of the Shopify store.

Every Shopify store has a public JSON file at [Base URL]/products.json with 'price' as a field. Example here: https://wildfox.com/products.json

One thing I messed up on originally was not pulling the 'currency' field which is actually in a different public JSON file called 'meta.json'. Example here: https://wildfox.com/meta.json

Separately, this was primary reason to only start with US stores: to make sure the currency shows up correctly and to purposely limit the initial audience to keep loading times reasonable. Working on adding all Shopify stores in the world now (a list of about 5 million active stores from what I have found).

Re: Show HN: I scraped 25M Shopify products to build a search engine

#277

Post by difradev »

pencildiver wrote: post #276
difradev wrote: post #275

Amazing job! I've one question: how did you find the price of every products? I mean, every product page has a different id or class that identify a price. Do you use a regex?

Thanks! Actually a lot easier than you'd expect. Not touching anything on the front-end of the Shopify store. Every Shopify store has a public JSON file at [Base URL]/products.json with 'price' as a field. Example here: https://wildfox.com/products.json One thing I messed up on originally was not pulling the 'currency' field which is actually in a different public JSON file called 'meta.json'. Example here: https://w…

Clear! Thanks!! :)

Re: Show HN: I scraped 25M Shopify products to build a search engine

#278

Post by pencildiver »

HN— Not sure if anyone will see this but I wanted to thank you all for the support. Although I haven't slept much since going live, it has been amazing getting early feedback from the community.

Agora is still in MVP stage but getting better by the day. Just pushed a big update: fixing an image shifting bug, a blur effect on loading, Redis for caching, brand pages, architecture fixes, and several other things. Currently working on improving the relevancy algorithm, adding all ~5 million Shopify stores, and then adding WooCommerce stores over the next few days.

If you have any suggestions or ideas, reach out to me at support @ searchagora .com :)

Post reply on HN