Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I scraped 25M Shopify products to build a search engine

searchagora.com

Post reply on HN
Print view View original

191–200 of 286 posts

Re: Show HN: I scraped 25M Shopify products to build a search engine

#191

Post by pbmonster »

olivermuty wrote: post #185
DeathArrow wrote: post #168

Earlier quoted context omitted.

>but in the end, it still wouldn't have mattered if I couldn't figure out how to acquire users In EU there are many price comparison engines with millions or billions of products. I don't know how popular they are. Some monetize trough ads, some have partnership with stores and you can buy directly from the search results. I generally search first on the local Amazon equivalent, if I don't like what I see, I search o…

What do you consider the local Amazon variant? And which country?

Amazon has no direct presence in Switzerland, but you can order a fraction of its products from neighboring countries. Many products are not available, mainly because nobody wants to deal with customs once the product crosses the EU boarder.

Amazon itself never moved into Switzerland in the first place for many reasons (small market, unusual customs situation, relatively high salary for warehouse workers), and in the meantime the largest Swiss supermarket chain created an Amazon clone which became hugely popular pretty much immediately: Galaxus.ch

Re: Show HN: I scraped 25M Shopify products to build a search engine

#192

Post by julesvr »

olivermuty wrote: post #185
DeathArrow wrote: post #168

Earlier quoted context omitted.

>but in the end, it still wouldn't have mattered if I couldn't figure out how to acquire users In EU there are many price comparison engines with millions or billions of products. I don't know how popular they are. Some monetize trough ads, some have partnership with stores and you can buy directly from the search results. I generally search first on the local Amazon equivalent, if I don't like what I see, I search o…

What do you consider the local Amazon variant? And which country?

hagglezon.com to compare Amazon variant prices

Re: Show HN: I scraped 25M Shopify products to build a search engine

#195

Post by berkes »

DeathArrow wrote: post #168
senecaso wrote: post #130

I hope you have better luck than I did! A few years ago, my partner and I built vendazzo.com (now defunct). It was an e-commerce search engine on products listed on Shopify shops (sound familiar? :)). At the time, we had > 100m products listed, and I don't remember how many shops we were indexing.. over 100k I think, but we had access to over a million. Overall, I think your approach is very similar to ours, but we m…

>but in the end, it still wouldn't have mattered if I couldn't figure out how to acquire users In EU there are many price comparison engines with millions or billions of products. I don't know how popular they are. Some monetize trough ads, some have partnership with stores and you can buy directly from the search results. I generally search first on the local Amazon equivalent, if I don't like what I see, I search o…

Unfortunately many of these "comparison" websites have a businesses model built on affiliate fees.

It doesn't take much imagination to predict which products show up as "best" or "cheapest".

And the fairer ones have to keep playing cat and mouse with shops lowering pricing when they detect a scraper coming by. Or employ tricks to make their shipping seem free, lowering their overall price on the comparison platform.

Re: Show HN: I scraped 25M Shopify products to build a search engine

#196

Post by berkes »

slim wrote: post #150
senecaso wrote: post #136

Earlier quoted context omitted.

We had no capital, so advertising or solutions that basically involved "throwing money at the problem" were off the table for us. We spent time posting in forums helping people find items they were looking for, and we had a few posts here on HN that generated short-lived, explosive traffic bursts. I remember those days we had posts get picked up on HN, it was always an exciting night! We were looking at influencers a…

looks like simptoms of no market. maybe you were solving a problem already solved by amazon ? most shops on shopify also use amazon

Not in all countries though. Amazon isn't present or popular, or as omnipresent in many countries.

That's an opportunity, I guess.

Re: Show HN: I scraped 25M Shopify products to build a search engine

#197

Post by berkes »

ganesha727 wrote: post #194

Idea! Shopify has a ton of resellers that sell junk from China. If you figure out how to avoid them, your life would be 10x easier.

I was wondering on a tech solution for this lately.

What I now do when I shop, is that I compare the images (and descriptions) of listings on Amazon, bol.com or Shopify with listings on Ali and Temu.

If they're exactly the same, run. If they differ slightly, look closer (and most likely run). I guess automating this could make for a solution to at least detect cheap resellers.

Re: Show HN: I scraped 25M Shopify products to build a search engine

#199

Post by berkes »

pencildiver wrote: post #80
screye wrote: post #57

What was the process for scraping 25M products ? I have always used standard python tools like selenium, bs4 and the like. But I'm guessing none of these work at scale. Could you talk about your process and key bottlenecks at that scale a little bit ? Also, how much did it cost ? ______________ A recommendation for how to improve search. Your base captions will be pretty bad. You can use spot instances on a smaller G…

Great suggestions, looking into this right now. First time building something like this so definitely new to some of these tools. For scraping: Found that every Shopify store has a public JSON file that is available in the same route. The JSON file appears on the [Base URL]/products.json. For example, the store for Wild Fox has their JSON file available here: https://www.wildfox.com/products.json . Built a crawler in…

> site called "Built With",

Do you have Alink. And are they any good?

Re: Show HN: I scraped 25M Shopify products to build a search engine

#200

Post by Semaphor »

berkes wrote: post #195
DeathArrow wrote: post #168

Earlier quoted context omitted.

>but in the end, it still wouldn't have mattered if I couldn't figure out how to acquire users In EU there are many price comparison engines with millions or billions of products. I don't know how popular they are. Some monetize trough ads, some have partnership with stores and you can buy directly from the search results. I generally search first on the local Amazon equivalent, if I don't like what I see, I search o…

Unfortunately many of these "comparison" websites have a businesses model built on affiliate fees. It doesn't take much imagination to predict which products show up as "best" or "cheapest". And the fairer ones have to keep playing cat and mouse with shops lowering pricing when they detect a scraper coming by. Or employ tricks to make their shipping seem free, lowering their overall price on the comparison platform.

> It doesn't take much imagination to predict which products show up as "best" or "cheapest".

Never seen a "best" outside of amazon, which does weird shit even without any affiliate fees. And "cheapest" is not really up to the site, unless they want to go under quite quickly.

Post reply on HN