Live data from Hacker News

Show HN: I scraped 200M Shopify products to build a search engine

searchagora.com

11–20 of 44 posts

Re: Show HN: I scraped 200M Shopify products to build a search engine

#11
post #8

Don't take this as a harsh criticism but I want to know what problem are you solving? Is this just for fun? This is a lose-lose game. You will never be able to catch up to the providers (shopify and woocommerce and others). What you are doing is not a search problem. It is a traffic problem of which you have little to none. There is a reason why Instagram and FB works as a driver for ecommerce products. My suggestion…

Finding products that aren't on Amazon or other top sites is a problem for consumers. Google Shopping is pretty bad. Connecting buyers direct to merchants is a win and something they all want. Every single entity that sells online is trying to boost their revenue share for direct to consumer as a mandate. Because they get more margin and they get to know their customer for ongoing marketing. The pandemic exposed many…

You refute my point and then agree with me that it can be expensive.

I worked for largest eCommerce employer twice and have built scraping and search systems at scale. I know a thing or two in eCommerce shopping sites. Do you know how much infra and people it takes to keep pricing, product, variant data up-to-date? Hint: It's significant capex investment.

Problem is entry point funnel in the shopping journey. If all he becomes an aggregator, it is no different than thousands out there. There is Shop.app which has all access to seller data and analytics. As I said, this is not a search problem. It is product discovery problem which you also seem to agree with. Unfortunately, not a lot of people/teams have managed to solve dicovery problem when site itself is fairly unknown.

Michael from YC also talks about product discovery being a tarpit idea.

I am not saying his search won't work. Search in itself is a solved problem here. You enter criteria/product name and you get results. But, unless you truly solve discovery, which in my opinion he can't do due to lack of first-party data, it won't do much besides a hopper site.

Re: Show HN: I scraped 200M Shopify products to build a search engine

#14
post #8

Earlier quoted context omitted.

Finding products that aren't on Amazon or other top sites is a problem for consumers. Google Shopping is pretty bad. Connecting buyers direct to merchants is a win and something they all want. Every single entity that sells online is trying to boost their revenue share for direct to consumer as a mandate. Because they get more margin and they get to know their customer for ongoing marketing. The pandemic exposed many…

You refute my point and then agree with me that it can be expensive. I worked for largest eCommerce employer twice and have built scraping and search systems at scale. I know a thing or two in eCommerce shopping sites. Do you know how much infra and people it takes to keep pricing, product, variant data up-to-date? Hint: It's significant capex investment. Problem is entry point funnel in the shopping journey. If all…

> You enter criteria/product name and you get results

Not on Amazon. At least not since 2020 or so.

Re: Show HN: I scraped 200M Shopify products to build a search engine

#17

Earlier quoted context omitted.

You refute my point and then agree with me that it can be expensive. I worked for largest eCommerce employer twice and have built scraping and search systems at scale. I know a thing or two in eCommerce shopping sites. Do you know how much infra and people it takes to keep pricing, product, variant data up-to-date? Hint: It's significant capex investment. Problem is entry point funnel in the shopping journey. If all…

> You enter criteria/product name and you get results Not on Amazon. At least not since 2020 or so.

GP is correct. There is not much value in search itself.

Are you serious about Amazon? I can absolutely find everything I need on Amazon. Who needs exact search when there are better alternatives through recommendation, people also buy etc?

I am curious to know — if you think search on amazon as broken as you think, why do you think people keep buying from Amazon? They have strong logistic network. But, to even start that logistics part, you need to search and buy.

Re: Show HN: I scraped 200M Shopify products to build a search engine

#18
Search is not working. Also, I seem to get shoes for anything I search. Did you hardcode it by any chance?

Are you open to collaborate with others? I might have an automated method of curating products. Please drop a line to comp [dot] turkey [at] gmail.com.

Re: Show HN: I scraped 200M Shopify products to build a search engine

#19
post #7

Oh is the site currently down? I tried a few queries, including the ones on the landing page. It gave me empty results.

Yes, we're currently down. Working as fast as possible to get our search API back live. Really sorry about that, server crashed a few hours ago.

Re: Show HN: I scraped 200M Shopify products to build a search engine

#20
post #8

Don't take this as a harsh criticism but I want to know what problem are you solving? Is this just for fun? This is a lose-lose game. You will never be able to catch up to the providers (shopify and woocommerce and others). What you are doing is not a search problem. It is a traffic problem of which you have little to none. There is a reason why Instagram and FB works as a driver for ecommerce products. My suggestion…

Finding products that aren't on Amazon or other top sites is a problem for consumers. Google Shopping is pretty bad. Connecting buyers direct to merchants is a win and something they all want. Every single entity that sells online is trying to boost their revenue share for direct to consumer as a mandate. Because they get more margin and they get to know their customer for ongoing marketing. The pandemic exposed many…

[dead]
Post reply on HN