Maybe I'm clearly ignorant, but how does this differ from Klarna (https://www.klarna.com)?
Show HN: I scraped 25M Shopify products to build a search engine
161–170 of 286 posts
Re: Show HN: I scraped 25M Shopify products to build a search engine
#162Maybe I'm clearly ignorant, but how does this differ from Klarna ( https://www.klarna.com )?
Do you mean pricerunner?
There's similar price comparison sites, but they don't index every store available.
You basically have to submit your listing.
Re: Show HN: I scraped 25M Shopify products to build a search engine
#163Maybe I'm clearly ignorant, but how does this differ from Klarna ( https://www.klarna.com )?
Is there a similarity at all? One is a search engine, the other a leeching "buy now pay later" scheme as a service.
Re: Show HN: I scraped 25M Shopify products to build a search engine
#164Earlier quoted context omitted.
Great suggestions, looking into this right now. First time building something like this so definitely new to some of these tools. For scraping: Found that every Shopify store has a public JSON file that is available in the same route. The JSON file appears on the [Base URL]/products.json. For example, the store for Wild Fox has their JSON file available here: https://www.wildfox.com/products.json . Built a crawler in…
If you're already on AWS, I recommend switching to postgres for now. For context, I have 3 RDS instances, each multi zone, with the biggest instance storing several billion records. My total bill for all 3 last month was $661. Postgres has full text search, vector search, and jsonb. With jsonb you can store and index json documents like you would in Mongo. - https://www.postgresql.org/docs/current/textsearch.html - h…
how big is the disk for the biggest instance?
Re: Show HN: I scraped 25M Shopify products to build a search engine
#165Worked well for me, great job. I searched for something I've been looking for and found some interesting options I haven't seen before.
Re: Show HN: I scraped 25M Shopify products to build a search engine
#166Clicking an item could show you similar items before it takes you to the item (or have capability for similar)
Re: Show HN: I scraped 25M Shopify products to build a search engine
#167I built this a couple years ago (now defunct) for the same reason :) The public JSON endpoints on shopify stores make it pretty easy to get the data. You mentioned using Mongo but it sounds expensive. I honestly think you could do this with just elastic or even postgres full text search and save money.
Here's a pro tip + feature you should implement: Shopify has a semi-hidden hack where you can link directly to checkout of a product if you know the variant ID. You could add a BUY NOW button to your site without forcing the user to navigate the original site or checkout flow. Example: https://hapaboardshop.com/cart/42165521907955 (it also supports quantities and coupon codes)
A word of caution: more products isn't necessarily better. I definitely found there to be a long tail of really bad shopify stores and products. IMO it's better to curate or audit the stores you index–otherwise you risk your site being littered with kitchy t-shirts or drop-shipping garbage.
Re: Show HN: I scraped 25M Shopify products to build a search engine
#168I hope you have better luck than I did! A few years ago, my partner and I built vendazzo.com (now defunct). It was an e-commerce search engine on products listed on Shopify shops (sound familiar? :)). At the time, we had > 100m products listed, and I don't remember how many shops we were indexing.. over 100k I think, but we had access to over a million. Overall, I think your approach is very similar to ours, but we m…
>but in the end, it still wouldn't have mattered if I couldn't figure out how to acquire users
In EU there are many price comparison engines with millions or billions of products. I don't know how popular they are. Some monetize trough ads, some have partnership with stores and you can buy directly from the search results.
I generally search first on the local Amazon equivalent, if I don't like what I see, I search on a smaller store. If I still can't find or dislike the products or prices I search Google. If I am still not contended with the results, I will go search on comparison engines.
And I also have a browser extension called Pricy who polls the comparison engines, so once I land in a product page I know which store has the better price and what was the price history through last year.
Probably many people have similar patterns. I expect people in US to search Amazon first, if it's not a very niche product they are after.
I think you can have a better monetization proposal, if instead of just search you build a sales platform, so people can directly buy after searching, without hoping to various websites.
Re: Show HN: I scraped 25M Shopify products to build a search engine
#169I built this a couple years ago (now defunct) for the same reason :) The public JSON endpoints on shopify stores make it pretty easy to get the data. You mentioned using Mongo but it sounds expensive. I honestly think you could do this with just elastic or even postgres full text search and save money. Here's a pro tip + feature you should implement: Shopify has a semi-hidden hack where you can link directly to check…
>otherwise you risk your site being littered with kitchy t-shirts or drop-shipping garbage.
You mean like Amazon?
Re: Show HN: I scraped 25M Shopify products to build a search engine
#170Earlier quoted context omitted.
Im curious why you consider lack of users to be the problem. I would have described it as lack of revenue. What plans did you have for generating revenue from the site? (Serious question - given your low costs it would seem like a tiny amount of revenue would gave been enough.)
Our business model revolved around referrals, so lack of users directly translated to lack of revenue. While its true that even if we had millions of users but none of them were buying sponsored items we would have had a revenue problem, that wasn't the problem we were facing, as the few users we did have were in fact purchasing sponsored items.
Then the problem seem to be the lack of users.
Have you tried having an YouTube channel, TikTok, Facebook, Twitter, blog and explain daily how you built the website, how your platform is going to help users?