Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I scraped 25M Shopify products to build a search engine

searchagora.com

Post reply on HN
Print view View original

131–140 of 286 posts

Re: Show HN: I scraped 25M Shopify products to build a search engine

#132

Post by xp84 »

satvikpendem wrote: post #16
pencildiver wrote: post #7

Earlier quoted context omitted.

Scraper is built in Javascript and a Mongo database. Probably not the most scalable way to do it, but I found that all Shopify stores have a public JSON file available at [Base URL]/products.json. So found a list of stores, built a crawler to go store-by-store, and standardized the data on my end. Here's an example: https://www.wildfox.com/products.json

How did you detect that it was a Shopify store?

There are lots of telltale endpoints that you could just HEAD for a 200 vs 404. Or even just the products.json itself is a pretty good giveaway.

Or an even better way I’ve done in the past (to check which competitor’s platform a list of prospects is using in bulk) is just to use the DNS — a Shopify shop will be CNAMEd to a certain Shopify hostname.

Re: Show HN: I scraped 25M Shopify products to build a search engine

#133

Post by senecaso »

yoru-sulfur wrote: post #131

For those unaware, Shopify already has platform wide search. You can use https://shop.app/ (or the app), and it also has some chatbot thing that can offer suggestions

Yes, this has been available for a few years now. Initially, they only indexed a very small number of shops, so it was less useful. Based on a few queries, it seems like the are still using some form of text-based search with rank boosting. Seems like they still aren't searching their entire base of shops, but they have increased the number of shops for sure, and they seem to be continuing to invest in the product, which is nice. It seems more useful now than it did the last time I checked!

Re: Show HN: I scraped 25M Shopify products to build a search engine

#134

Post by bytearray »

senecaso wrote: post #130

I hope you have better luck than I did! A few years ago, my partner and I built vendazzo.com (now defunct). It was an e-commerce search engine on products listed on Shopify shops (sound familiar? :)). At the time, we had > 100m products listed, and I don't remember how many shops we were indexing.. over 100k I think, but we had access to over a million. Overall, I think your approach is very similar to ours, but we m…

What strategies did you consider or implement to attract more users, and what would you do differently now to ensure better user acquisition?

Re: Show HN: I scraped 25M Shopify products to build a search engine

#136

Post by senecaso »

bytearray wrote: post #134
senecaso wrote: post #130

I hope you have better luck than I did! A few years ago, my partner and I built vendazzo.com (now defunct). It was an e-commerce search engine on products listed on Shopify shops (sound familiar? :)). At the time, we had > 100m products listed, and I don't remember how many shops we were indexing.. over 100k I think, but we had access to over a million. Overall, I think your approach is very similar to ours, but we m…

What strategies did you consider or implement to attract more users, and what would you do differently now to ensure better user acquisition?

We had no capital, so advertising or solutions that basically involved "throwing money at the problem" were off the table for us.

We spent time posting in forums helping people find items they were looking for, and we had a few posts here on HN that generated short-lived, explosive traffic bursts. I remember those days we had posts get picked up on HN, it was always an exciting night!

We were looking at influencers and getting our name getting bloggers to talk about us, but, again, without capital, our options were very limited here. I'm sure someone with more of a marketing background would have found a bunch of ways we could have generated organic user growth, but neither me or my business partner had that skill set.

If I were to do it again, I think I would try to get someone with a marketing background involved to help gain traction. Without that, even the best product in the world will die of starvation if no one finds it.

Re: Show HN: I scraped 25M Shopify products to build a search engine

#140

Post by ashvardanian »

Cool project!

As you scale, you may benefit from these two projects I maintain, and the Big Tech uses :)

https://github.com/unum-cloud/usearch - for faster search

https://github.com/unum-cloud/uform - for cheaper multi-lingual multi-modal embeddings

Feel free to reach out with feedback and feature requests!

Post reply on HN