Live data from Hacker News

Show HN: I scraped 25M Shopify products to build a search engine

searchagora.com

181–190 of 286 posts

Re: Show HN: I scraped 25M Shopify products to build a search engine

#181
post #179

The Terms page goes to "Jaggi Enterprises", "A Modern Investment Fund. We buy, build, and invest in software companies with recurring revenue.". So maybe this is not really something a guy built for his wife, but some anonymous startup that googled "Which terms rank best on Hacker News" and then wrote the "I did ... my wife .." story?

Jaggi is a fake it until you make it fake portfolio. Most of the companies it runs are just lorem ipsum fake sites. I think it is likely true that this is a solo dev.

Re: Show HN: I scraped 25M Shopify products to build a search engine

#182
post #179

The Terms page goes to "Jaggi Enterprises", "A Modern Investment Fund. We buy, build, and invest in software companies with recurring revenue.". So maybe this is not really something a guy built for his wife, but some anonymous startup that googled "Which terms rank best on Hacker News" and then wrote the "I did ... my wife .." story?

[deleted]

Re: Show HN: I scraped 25M Shopify products to build a search engine

#183
post #179

The Terms page goes to "Jaggi Enterprises", "A Modern Investment Fund. We buy, build, and invest in software companies with recurring revenue.". So maybe this is not really something a guy built for his wife, but some anonymous startup that googled "Which terms rank best on Hacker News" and then wrote the "I did ... my wife .." story?

Jaggi is a fake it until you make it fake portfolio. Most of the companies it runs are just lorem ipsum fake sites. I think it is likely true that this is a solo dev.

Yeah you're right: https://www.linkedin.com/company/jaggi/

Re: Show HN: I scraped 25M Shopify products to build a search engine

#184
post #115

Earlier quoted context omitted.

Great suggestions, looking into this right now. First time building something like this so definitely new to some of these tools. For scraping: Found that every Shopify store has a public JSON file that is available in the same route. The JSON file appears on the [Base URL]/products.json. For example, the store for Wild Fox has their JSON file available here: https://www.wildfox.com/products.json . Built a crawler in…

If you're already on AWS, I recommend switching to postgres for now. For context, I have 3 RDS instances, each multi zone, with the biggest instance storing several billion records. My total bill for all 3 last month was $661. Postgres has full text search, vector search, and jsonb. With jsonb you can store and index json documents like you would in Mongo. - https://www.postgresql.org/docs/current/textsearch.html - h…

I have troubles seeing how this is possible.

$220 dollars per instance gets you 8Gb of RAM which is way, way, below the index size if you are indexing billions of vectors.

Re: Show HN: I scraped 25M Shopify products to build a search engine

#185

I hope you have better luck than I did! A few years ago, my partner and I built vendazzo.com (now defunct). It was an e-commerce search engine on products listed on Shopify shops (sound familiar? :)). At the time, we had > 100m products listed, and I don't remember how many shops we were indexing.. over 100k I think, but we had access to over a million. Overall, I think your approach is very similar to ours, but we m…

>but in the end, it still wouldn't have mattered if I couldn't figure out how to acquire users In EU there are many price comparison engines with millions or billions of products. I don't know how popular they are. Some monetize trough ads, some have partnership with stores and you can buy directly from the search results. I generally search first on the local Amazon equivalent, if I don't like what I see, I search o…

What do you consider the local Amazon variant? And which country?

Re: Show HN: I scraped 25M Shopify products to build a search engine

#186
post #115

Earlier quoted context omitted.

If you're already on AWS, I recommend switching to postgres for now. For context, I have 3 RDS instances, each multi zone, with the biggest instance storing several billion records. My total bill for all 3 last month was $661. Postgres has full text search, vector search, and jsonb. With jsonb you can store and index json documents like you would in Mongo. - https://www.postgresql.org/docs/current/textsearch.html - h…

I have troubles seeing how this is possible. $220 dollars per instance gets you 8Gb of RAM which is way, way, below the index size if you are indexing billions of vectors.

[deleted]

Re: Show HN: I scraped 25M Shopify products to build a search engine

#187

Shopify has tried a few times to build a tool like this but hasn’t ever managed to get any traction. I think that missing any curation at all could be what eventually kills it. Their current attempt is https://shop.app and a query for red shoes is mostly red shoes.

> a query for red shoes is mostly red shoes

well I get mostly black shoes lol

Edit: ah no, they just use half a page for shoe shops first with black shoes as logo??

Re: Show HN: I scraped 25M Shopify products to build a search engine

#188

Earlier quoted context omitted.

>but in the end, it still wouldn't have mattered if I couldn't figure out how to acquire users In EU there are many price comparison engines with millions or billions of products. I don't know how popular they are. Some monetize trough ads, some have partnership with stores and you can buy directly from the search results. I generally search first on the local Amazon equivalent, if I don't like what I see, I search o…

What do you consider the local Amazon variant? And which country?

There are alternatives throughout Europe. The Balkans have Emag, Benelux has bol.com. I think in both regions Amazon is less popular. I'm sure there are other examples.

Re: Show HN: I scraped 25M Shopify products to build a search engine

#189

Earlier quoted context omitted.

>but in the end, it still wouldn't have mattered if I couldn't figure out how to acquire users In EU there are many price comparison engines with millions or billions of products. I don't know how popular they are. Some monetize trough ads, some have partnership with stores and you can buy directly from the search results. I generally search first on the local Amazon equivalent, if I don't like what I see, I search o…

What do you consider the local Amazon variant? And which country?

The Netherlands has plenty of them. Tweakers.net is a price tracker for electronics and such (eg: computer parts, phones, laptops etc) and usually it's easier to find a shop cheaper than Amazon. I have some go to stores for my needs because their content is organised way better than Amazon. I also find some alternatives better than Amazon because they have free next day shipping, something that's not free on Amazon.

Re: Show HN: I scraped 25M Shopify products to build a search engine

#190
There are a few conferences dedicated to ecommerce search. Mices is pretty good. I did not go there this year but I know some of the people behind it. Good community and lots of stuff happening.

Two points here.

- 25 million is really not a lot for most search engines. Something like Elasticsearch can easily deal with that if you deal with it properly. And there are plenty of equally capable solutions. I have worked with logging clusters that processes log entries by those numbers on a daily basis. A modestly sized cluster goes a long way for that. Bare metal is cheaper than cloud for this. But a couple of simple servers with decent CPUs and memory and SSDs should go a long way here. Start worrying once you hit a few hundred GB of storage used. Anything below that is easy to deal with.

- The key challenge with this volume is not performance but search quality. Building a competitive search engine is hard. You might have thousands of potential matches out of millions for any given query and your job is to pick the best 3, 5, 10 (whatever fits on your screen) ones. This is hard.

So, what makes for a good answer is the key question to answer. All the naive solutions for this problem put you at the bottom of the market in terms of competitiveness. If you can't do better, you are just another low quality search engine not quite solving the problem. The bar is high these days for a good search engine and most of the better ecommerce companies have highly skilled search teams working on this.

Post reply on HN